Deploying postgre sql on amazon ec2
0 likes9,638 views
Denish Patel deployed PostgreSQL on Amazon EC2 for a startup that had its entire IT architecture on the Amazon cloud. The initial deployment involved setting up master-slave configurations across two EC2 environments with issues like weekly instance failures and lack of monitoring. Patel then consolidated the environments, configured high availability using replication across availability zones and regions, implemented automation using Puppet, and added monitoring and backups to improve the stability and management of the PostgreSQL deployment.
![Tools
✤ tracking schema in git : pg_extractor
✤ pg_bloat_report
✤ Pg log errors reporting : tail_n_mail
✤ Log analysis: pgbadger
✤ log_min_duration_statement = 50 ms
✤ log_line_prefix= '%t [%r] [%p]: [%l-1] user=%u,db=%d,e=%e '
27
Tuesday, May 7, 13](https://image.slidesharecdn.com/deployingpostgresqlonamazonec2-130507094812-phpapp02/85/Deploying-postgre-sql-on-amazon-ec2-27-320.jpg)
Deploying postgre sql on amazon ec2
- 1. Denish Patel Database Architect, OmniTI Deploying PostgreSQL on Amazon EC2 A case study 1 Tuesday, May 7, 13
- 2. Story teller? ✤ Denish Patel ✤ [email protected] ✤ Database Architect at OmniTI ✤ Heterogeneous Environment ✤ PostgreSQL, Oracle , MySQL, NoSQL ✤ Scalability, Performance & Simplicity ✤ OmniTI is hiring Database Engineer! 2 Tuesday, May 7, 13
- 3. Situation ✤ Startup ✤ Ruby on Rails Shop ✤ Entire IT architecture on Amazon Cloud ✤ 2 Identical Database Environments on EC2 ✤ 4 Production Instances ✤ 50GB Each PostgreSQL 9.0 Master - Slave ( 2 Pairs) ✤ 2 QA and 2 Dev environments 3 Tuesday, May 7, 13
- 4. Situation ✤ Weekly flake out issues on EC2 instances ✤ Weekly backups using pg_dump ✤ No file system level backup ✤ Master-slave wasn’t working for months! ✤ Schema mismatch between 2 environments ✤ Almost NO monitoring in-place for DBs 4 Tuesday, May 7, 13
- 5. Situation ✤ Prod , QA and Dev ... ✤ M1 Extra large instances (m1.xlarge) ✤ 15 GB RAM, 4 Virtual Cores with 2 EC2 compute units each ✤ Configuration Management tool - Puppet 5 Tuesday, May 7, 13
- 6. The Amazon Cloud ✤ Business Benefits ✤ Offloading heavy lifting ✤ Lowers cost ✤ Reduces time to market 6 Tuesday, May 7, 13
- 7. Amazon Data Centers 7 Tuesday, May 7, 13
- 8. Amazon Instances ✤ High CPU ✤ High IO ✤ High Memory 8 Tuesday, May 7, 13
- 9. Amazon StorageTypes 9 Tuesday, May 7, 13
- 10. Network and Security ✤ Amazon Security Groups ✤ Keep your Amazon EC2 firewall permissions as restrictive as possible. ✤ Elastic IP Address (EIP) 10 Tuesday, May 7, 13
- 11. Deployment Strategies ✤ Sizing Instances and Storage Types ✤ Configuration Management ✤ High Availability ✤ Replication ✤ Failover ✤ Operations ✤ Upgrade, Backups, Maintenance tools ✤ Monitoring 11 Tuesday, May 7, 13
- 12. Sizing Instances/Storage ✤ Understand your database , schema and data ✤ Env 1: 50GB , Env 2: 25GB ✤ 2 large archival tables ✤ Working Set about 20-30GB in total ✤ Both environments are inter-related. ✤ Decided to consolidate both environment into single cluster 12 Tuesday, May 7, 13
- 13. Sizing Instances/Storage ✤ m2.2xlarge: 34.2GB RAM, 4 Virtual Core ✤ Storage ✤ EBS ✤ RAID 10 ✤ Not EBS optimized ( Cost Savings!) ✤ 850 GB 13 Tuesday, May 7, 13
- 14. Deployment Phase #1 us-east-1 AV1 AV2 us-west-1 App Servers db-master db-slave- backup db-slave Streaming Replication WAL shipping (OmniPITR) 14 Tuesday, May 7, 13
- 15. Configuration Management ✤ Automation ✤ Repeatability ✤ Maintainability ✤ Security 15 Tuesday, May 7, 13
- 16. Configuration Management 16 Tuesday, May 7, 13
- 17. Puppetizing Postgres ✤ DB server role ✤ PostgreSQL installation ✤ Installing common tools ✤ 3 Roles : prod-master role , prod-slave role, prod-slave-backup role ✤ PostgreSQL Configuration Files: postgresql.conf, pg_hba.conf ✤ Scripts, crontabs 17 Tuesday, May 7, 13
- 18. High Availability ✤ Replication ✤ Failover 18 Tuesday, May 7, 13
- 19. Replication ✤ Streaming replication ✤ WAL shipping , OmniPITR ✤ Different AV zones ✤ At least 1 slave in different Region 19 Tuesday, May 7, 13
- 20. Failover ✤ EIP - about 3 minutes to propagate changes after failover ✤ Failover tools ✤ Semi-automated script ✤ pgbouncer ✤ puppet 20 Tuesday, May 7, 13
- 21. Deployment Phase #2 us-east-1 AV1 AV2 us-west-1 App Servers db-master db-slave- backup db-slave Streaming Replication WAL shipping (OmniPITR) pgbouncer 21 Tuesday, May 7, 13
- 22. pgbouncer ✤ Session Pooling ✤ Supports all Postgres features ✤ Transactions Pooling ✤ Does NOT support PREPARE transactions 22 Tuesday, May 7, 13
- 24. Database Upgrade ✤ Upgraded from PostgreSQL 9.0 to PostgreSQL 9.2 ✤ Minimal outage upgrade using Mimeo ✤ http://www.pateldenish.com/2012/10/an-easy-way-to-reduce- outage-window-for-postgresql-upgrade.html 24 Tuesday, May 7, 13
- 25. Beware!! ✤ Sept 24th, 2012 ✤ PostgreSQL 9.1.0 to 9.1.5 & 9.2.0 ✤ VACUUM all tables in your database & Rebuild Indices ✤ http://wiki.postgresql.org/wiki/20120924updaterelease ✤ April 4th, 2013 ✤ PostgreSQL 9.0.* to 9.1.* & 9.2.* ✤ Security release ✤ http://www.postgresql.org/support/security/faq/2013-04-04/ 25 Tuesday, May 7, 13
- 26. Backups ✤ Online Backups ✤ Daily, db-slave , OmniPITR ✤ Test the backups!!!! ✤ Schema Backups ✤ pg_extractor ✤ Offsite Backup ✤ sending backups to Amazon S3 26 Tuesday, May 7, 13
- 27. Tools ✤ tracking schema in git : pg_extractor ✤ pg_bloat_report ✤ Pg log errors reporting : tail_n_mail ✤ Log analysis: pgbadger ✤ log_min_duration_statement = 50 ms ✤ log_line_prefix= '%t [%r] [%p]: [%l-1] user=%u,db=%d,e=%e ' 27 Tuesday, May 7, 13
- 29. Database Checks 29 Tuesday, May 7, 13
- 30. Database Checks 30 Tuesday, May 7, 13
- 31. System Checks 31 Tuesday, May 7, 13
- 32. Business Metrics 32 Tuesday, May 7, 13
- 34. References ✤ http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ Storage.html ✤ OmniTI labs tools ✤ https://github.com/omniti-labs/pgtreats ✤ OmniPITR ✤ https://github.com/omniti-labs/omnipitr 34 Tuesday, May 7, 13