Deploying Maximum HA Architecture With PostgreSQL
7 likes3,890 views
This document proposes a "Maximum HA architecture" for PostgreSQL that aims to provide 99.99% application uptime and reduce mean time to recovery (MTTR) for planned and unplanned outages. It discusses using PostgreSQL features like streaming replication, failover, hot backups, WAL archiving and point-in-time recovery (PITR) to achieve high availability, prevent data loss from failures, and allow fast recovery from outages through redundancy and automation. The architecture builds upon traditional high availability techniques and aims to handle different types of failures including system failures, site failures, data failures, and planned changes through comprehensive disaster recovery planning.
More Related Content
What's hot (20)
Similar to Deploying Maximum HA Architecture With PostgreSQL (20)
More from Denish Patel (17)
Deploying Maximum HA Architecture With PostgreSQL
- 1. Deploying Maximum HA architecture with PostgreSQL / Denish Patel Database Architect
- 2. Who am I ? • Denish Patel • Database Architect with OmniTI for more than 5 years • Expertise in PostgreSQL , Oracle, MySQL, MS SQL Server • Contact : [email protected] • Blog: http://denishjpatel.blogspot.com/ • Providing Solutions for business problems to deliver • Scalability • Reliability • High Availability We are hiring!! Apply @ l42.org/lg • Consistency • Security 1
- 3. Agendum • Why do you need HA architecture ? • Why PostgreSQL ? • Traditional HA Architecture • Goals for Maximum HA • Maximum HA Solution 2
- 4. Assumptions • Consistency and Availability Matters (CAP theorem) • Good to reduce MTTF but you have “real” control on MTTR. 3
- 5. Why do you need HA architecture? Application Unavailability of Downtime Data Loss of productivity Loss of Revenue Dissatisfied Customers 4
- 6. Why do you need HA architecture? System Unplanned Failures Outages Data Failures Prevent Tolerate System Recover Fast Planned Changes Outages Data Changes 5
- 7. Why PostreSQL ? • Best protection at Lowest Cost • No additional software costs for providing maximum Availability compared to closed source databases • Provide free feature sets to prevent outages, tolerate them and recover fast. 6
- 8. Traditional HA Architecture Master Standby Database Database Copy WAL files WAL WAL PostgreSQL 8 7
- 9. Traditional HA Architecture Master Hot Standby Database Database Steaming Replication Copy WAL files WAL WAL PostgreSQL 9 8
- 10. Goals for Maximum HA Architecture • 99.99% Uptime of application • Reduce MTTR • Planned outages • Unplanned outages 9
- 11. Plan to reduce MTTR • How do you manage failover ? • Is it transparent to your application? • Hot Backups/ Dumps • Are you running on production server? • Schema backups • How often? Are they under revision control ? • WAL files copy scripts • Do all of your prod servers using same copy of the script ? • Where is your reporting queries pointing to ? • Production DB? 10
- 12. System Failures Server Node Fails Storage Fails System Failures Site Fails Unplanned Outages 11
- 13. Handle System Failures inet Floating IP/ VIP App Server Master Failover 12
- 14. Site Failures Server Node Fails Storage Fails System Failures Site Fails Unplanned Outages 13
- 15. Handle Site Failures Offsite Bkp inet Floating IP/ WAL VIP apply App Server Ship WAL Files Master Failover SRHS 14
- 16. Data Failures Unplanned Outages Human Error Data Failures Data Corruption 15
- 17. Handle Data Failures • PITR slave lag using OMNIpitr • 1 hour lag on wal apply • Periodic pg_dump tables from slave • Run pg_extractor • https://github.com/omniti-labs/pg_extractor • Track schema changes into subversion/git 16
- 18. Data Corruption Unplanned Outages Human Error Data Failures Data Corruption 17
- 19. Handle Data Corruption • File System level backups • Backups on Slave database using OMNIpitr • Regular recovery testing • Snapshot backups for faster recovery • Solaris ZFS is recommended! • Monthly pg_dump backups • Backups on slave 18
- 20. System Changes OS Upgrade Database Upgrade System Changes Network Changes Planned Outages 19
- 21. Handle OS Upgrades Floating IP Master SRHS Master SRHS Failover Read WAL Slave 1 Copy NAS 20
- 22. Handle OS Upgrades Floating IP Master Upgrade OS SRHS Master SRHS New Read WAL Master Slave 1 Copy NAS 21
- 23. Handle OS Upgrades Floating IP Master SRHS New SRHS Failover New Read WAL Master Slave 1 Copy NAS 22
- 24. System Changes OS Upgrade Database Upgrade System Changes Network Changes Planned Outages 23
- 25. Handle Database Upgrade Yes No PG 8.3+ ? Outage acceptable ? Outage No acceptable? Yes pg_upgrad No Yes e –check pass? Third Yes party Rep pg_dump No i.e Slony pg_restore Drop incompatible tables before pg_upgrade upgrade and * Only showing recommended options restore after 24
- 26. Handle Data Changes Planned Outages Alter Schemas Data Changes Data growth 25
- 27. Handle Alter schemas • Transactional DDL • CREATE or REPLACE views • NOT VALID • Checks • FKs • Add column without scanning entire table • NULLABLE • No Default 26
- 28. Handle Data Changes Planned Outages Alter Schemas Data Changes Data growth 27
- 29. Handle Data Growth PostgreSQL Bloat removal • Offline • VACUUM FULL • CLUSTER • Online • Rebuild index CONCURRENTLY • Rebuild table online using pg_reorg http://denishjpatel.blogspot.com/2011/03/extreme-training-session-at-pgeast-p90x.html 28
- 30. Now we have …. 9 PITR Floating IP pg_extractor pg_reorg 29
- 31. Maximum HA Architecture App Floating IP Master LB SRHS Bkp Failover Master SRHS Read WAL Slave 2 apply NAS Read Bkp Salve 1 30
- 32. References • PostgreSQL Documentations • http://www.postgresql.org/docs/ • OmniTI Labs • https://labs.omniti.com/ • OMNIpitr • pg_extractor 30
- 33. Thanks • PG Day NYC Conference Committee • OmniTI • You!! 31
- 34. Questions? 32