re:Invent 2022 DAT316 Build resilient applications using Amazon RDS and Aurora PostgreSQL
- 1. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 2. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Build resilient applications using Amazon RDS and Aurora PostgreSQL D A T 3 1 6 Grant McAlister Senior Principal Engineer Amazon Web Services Jim Mlodgenski Principal Database Engineer Amazon Web Services
- 3. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Primary The case for resiliency • Everything fails at some point Availability Zone 1 Availability Zone 2 AWS Region Applications
- 4. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Primary The case for resiliency • Everything fails at some point Availability Zone 1 Availability Zone 2 AWS Region Applications
- 5. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Primary The case for resiliency • Everything fails at some point Availability Zone 1 Availability Zone 2 AWS Region Applications
- 6. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Primary The case for resiliency • Everything fails at some point Availability Zone 1 Availability Zone 2 AWS Region Applications
- 7. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Primary The case for resiliency • Everything fails at some point Availability Zone 1 Availability Zone 2 AWS Region Applications
- 8. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Primary The case for resiliency • Everything fails at some point Availability Zone 1 Availability Zone 2 AWS Region Applications
- 9. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Primary The case for resiliency • Everything fails at some point • Planned and unplanned Availability Zone 1 Availability Zone 2 AWS Region Applications
- 10. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Primary The case for resiliency • Everything fails at some point • Planned and unplanned Availability Zone 1 Availability Zone 2 AWS Region Applications
- 11. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Standby Primary The case for resiliency • Everything fails at some point • Planned and unplanned Availability Zone 1 Availability Zone 2 AWS Region Applications Applications
- 12. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Standby Primary The case for resiliency • Everything fails at some point • Planned and unplanned • Does the system operate as designed Availability Zone 1 Availability Zone 2 AWS Region Applications Applications
- 13. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. The case for resiliency • Everything fails at some point • Planned and unplanned • Does the system operate as designed Availability Zone 1 Availability Zone 2 AWS Region Applications Applications Standby Primary
- 14. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. The case for resiliency • Everything fails at some point • Planned and unplanned • Does the system operate as designed Availability Zone 1 Availability Zone 2 AWS Region Applications Applications Standby Primary
- 15. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. The case for resiliency • Everything fails at some point • Planned and unplanned • Does the system operate as designed Availability Zone 1 Availability Zone 2 AWS Region Applications Applications Standby Primary
- 16. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. The case for resiliency • Everything fails at some point • Planned and unplanned • Does the system operate as designed • Grey failures Availability Zone 1 Availability Zone 2 AWS Region Applications Applications Standby Primary
- 17. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora with PostgreSQL compatibility Availability Zone 1 AWS Region Applications Availability Zone 2 Availability Zone 3 Aurora Storage Applications Applications
- 18. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora with PostgreSQL compatibility Availability Zone 1 AWS Region Applications Availability Zone 2 Availability Zone 3 Aurora Storage Applications Applications • 4 of 6 write quorum across 3 AZ’s 1 2 3 4 5 6
- 19. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora with PostgreSQL compatibility Availability Zone 1 AWS Region Applications Availability Zone 2 Availability Zone 3 Aurora Storage Applications Applications • 4 of 6 write quorum across 3 AZ’s • Self repairing storage system 1 2 3 4 5 6
- 20. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora with PostgreSQL compatibility Availability Zone 1 AWS Region Applications Availability Zone 2 Availability Zone 3 Aurora Storage Applications Applications • 4 of 6 write quorum across 3 AZ’s • Self repairing storage system • One or more db instances per AZ 1 2 3 4 5 6 writer reader reader
- 21. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora with PostgreSQL compatibility Availability Zone 1 AWS Region Applications Availability Zone 2 Availability Zone 3 Aurora Storage Applications Applications • 4 of 6 write quorum across 3 AZ’s • Self repairing storage system • One or more db instances per AZ 1 2 3 4 5 6 writer reader reader
- 22. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora with PostgreSQL compatibility Availability Zone 1 AWS Region Applications Availability Zone 2 Availability Zone 3 Aurora Storage Applications Applications • 4 of 6 write quorum across 3 AZ’s • Self repairing storage system • One or more db instances per AZ 1 2 3 4 5 6 reader
- 23. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora with PostgreSQL compatibility Availability Zone 1 AWS Region Applications Availability Zone 2 Availability Zone 3 Aurora Storage Applications Applications • 4 of 6 write quorum across 3 AZ’s • Self repairing storage system • One or more db instances per AZ 1 2 3 4 5 6 reader writer
- 24. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora with PostgreSQL compatibility Availability Zone 1 AWS Region Applications Availability Zone 2 Availability Zone 3 Aurora Storage Applications Applications • 4 of 6 write quorum across 3 AZ’s • Self repairing storage system • One or more db instances per AZ 1 2 3 4 5 6 reader writer
- 25. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora with PostgreSQL compatibility Availability Zone 1 AWS Region Applications Availability Zone 2 Availability Zone 3 Aurora Storage Applications Applications • 4 of 6 write quorum across 3 AZ’s • Self repairing storage system • One or more db instances per AZ 1 2 3 4 5 6 reader writer
- 26. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora with PostgreSQL compatibility Availability Zone 1 AWS Region Applications Availability Zone 2 Availability Zone 3 Aurora Storage Applications Applications • 4 of 6 write quorum across 3 AZ’s • Self repairing storage system • One or more db instances per AZ 1 2 3 4 5 6 reader writer
- 27. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora with PostgreSQL compatibility Availability Zone 1 AWS Region Applications Availability Zone 2 Availability Zone 3 Aurora Storage Applications Applications • 4 of 6 write quorum across 3 AZ’s • Self repairing storage system • One or more db instances per AZ • AWS JDBC driver for faster failover 1 2 3 4 5 6 reader AWS JDBC writer
- 28. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Aurora with PostgreSQL compatibility Availability Zone 1 AWS Region Applications Availability Zone 2 Availability Zone 3 Aurora Storage Applications Applications • 4 of 6 write quorum across 3 AZ’s • Self repairing storage system • One or more db instances per AZ • AWS JDBC driver for faster failover 1 2 3 4 5 6 reader AWS JDBC writer
- 29. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Standby Amazon RDS Multi-AZ with two readable standbys • PostgreSQL replication across 3 AZ’s with a 2 of 3 quorum • Write Ahead Log written to local storage • Failover time a factor of transaction apply lag Availability Zone 3 AWS Region Applications Availability Zone 1 Primary Availability Zone 2 Standby WAL WAL WAL
- 30. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Standby Primary Amazon RDS Multi-AZ with one standby • PostgreSQL community version with easy configuration and management • High availability across two availability zones • All writes synchronously replicated to the standby • Failover a factor of checkpoints Availability Zone 1 Availability Zone 2 AWS Region Applications
- 31. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Recovery Time 3 GiB Redo Recovered in 19 seconds 10 GiB Redo Recovered in 50 seconds 30 GiB Redo Recovered in 123 seconds 0 20 40 60 80 100 120 140 160 0 20,000 40,000 60,000 80,000 100,000 120,000 140,000 Recovery Time in Seconds (less is better) Writes / Second (more is better) RECOVERY TIME FROM CRASH UNDER LOAD Bubble size represents redo log, which must be recovered As PostgreSQL throughput goes up, so does log size and crash recovery time
- 32. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Recovery Time 3 GiB Redo Recovered in 19 seconds 10 GiB Redo Recovered in 50 seconds 30 GiB Redo Recovered in 123 seconds 0 20 40 60 80 100 120 140 160 0 20,000 40,000 60,000 80,000 100,000 120,000 140,000 Recovery Time in Seconds (less is better) Writes / Second (more is better) RECOVERY TIME FROM CRASH UNDER LOAD Bubble size represents redo log, which must be recovered As PostgreSQL throughput goes up, so does log size and crash recovery time Amazon Aurora has no redo. Recovered in 3 seconds while maintaining greater throughput.
- 33. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Failover time 0 50,000 100,000 150,000 200,000 250,000 300,000 350,000 400,000 0 60 120 180 240 300 360 420 480 540 600 660 720 780 840 900 960 1020 1080 1140 1200 Transactions per Second (TPS) Seconds PGBench 20X RO / 1X RW 160GB Cached - Failover at 600 Seconds Baseline 32 seconds
- 34. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Failover time 0 50,000 100,000 150,000 200,000 250,000 300,000 350,000 400,000 0 60 120 180 240 300 360 420 480 540 600 660 720 780 840 900 960 1020 1080 1140 1200 Transactions per Second (TPS) Seconds PGBench 20X RO / 1X RW 160GB Cached - Failover at 600 Seconds Baseline 340 seconds
- 35. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. RO Application Cluster Cache Management (CCM) Feature RW Application RO Application Async Invalidation & Update Availability zone 1 Availability zone 3 Availability zone 2 Aurora Storage RO RO RO RO
- 36. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. RO Application Cluster Cache Management (CCM) Feature RW Application RO Application Async Invalidation & Update Availability zone 1 Availability zone 3 Availability zone 2 Aurora Storage RO RO RO RO apg_ccm_enabled=on
- 37. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. RO Application Cluster Cache Management (CCM) Feature RW Application RO Application Async Invalidation & Update Availability zone 1 Availability zone 3 Availability zone 2 Aurora Storage RO RO RO RO apg_ccm_enabled=on
- 38. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. RO Application Cluster Cache Management (CCM) Feature RW Application RO Application Async Invalidation & Update Availability zone 1 Availability zone 3 Availability zone 2 Aurora Storage RO RO RO RO apg_ccm_enabled=on
- 39. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. RO Application Cluster Cache Management (CCM) Feature RW Application RO Application Async Invalidation & Update Availability zone 1 Availability zone 3 Availability zone 2 Aurora Storage RO RO RO RO apg_ccm_enabled=on
- 40. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Failover time - Cluster Cache Management 0 50,000 100,000 150,000 200,000 250,000 300,000 350,000 400,000 0 60 120 180 240 300 360 420 480 540 600 660 720 780 840 900 960 1020 1080 1140 1200 Transactions per Second (TPS) Seconds PGBench 20X RO / 1X RW 160GB Cached - Failover at 600 Seconds Baseline CCM Enabled 340 seconds
- 41. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Failover time - Cluster Cache Management 0 50,000 100,000 150,000 200,000 250,000 300,000 350,000 400,000 0 60 120 180 240 300 360 420 480 540 600 660 720 780 840 900 960 1020 1080 1140 1200 Transactions per Second (TPS) Seconds PGBench 20X RO / 1X RW 160GB Cached - Failover at 600 Seconds Baseline CCM Enabled 32 seconds 340 seconds
- 42. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Pg_prewarm • Extension available in all supported versions of PostgreSQL • Can manually load tables and indexes into cache • PostgreSQL 11 introduced auto prewarm to restore the cache after a restart or failover Standby Primary Availability Zone 1 Availability Zone 2 AWS Region
- 43. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Pg_prewarm • Extension available in all supported versions of PostgreSQL • Can manually load tables and indexes into cache • PostgreSQL 11 introduced auto prewarm to restore the cache after a restart or failover Standby Primary Availability Zone 1 Availability Zone 2 AWS Region
- 44. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Pg_prewarm • Extension available in all supported versions of PostgreSQL • Can manually load tables and indexes into cache • PostgreSQL 11 introduced auto prewarm to restore the cache after a restart or failover Standby Primary Availability Zone 1 Availability Zone 2 AWS Region
- 45. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Pg_prewarm • Extension available in all supported versions of PostgreSQL • Can manually load tables and indexes into cache • PostgreSQL 11 introduced auto prewarm to restore the cache after a restart or failover Standby Primary Availability Zone 1 Availability Zone 2 AWS Region
- 46. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Pg_prewarm • Extension available in all supported versions of PostgreSQL • Can manually load tables and indexes into cache • PostgreSQL 11 introduced auto prewarm to restore the cache after a restart or failover Standby Primary Availability Zone 1 Availability Zone 2 AWS Region
- 47. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Pg_prewarm • Extension available in all supported versions of PostgreSQL • Can manually load tables and indexes into cache • PostgreSQL 11 introduced auto prewarm to restore the cache after a restart or failover Primary Availability Zone 1 Availability Zone 2 AWS Region
- 48. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Pg_prewarm • Extension available in all supported versions of PostgreSQL • Can manually load tables and indexes into cache • PostgreSQL 11 introduced auto prewarm to restore the cache after a restart or failover Primary Availability Zone 1 Availability Zone 2 AWS Region
- 49. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Standby Primary Fault Injection Availability Zone 1 Availability Zone 2 AWS Region Applications Applications
- 50. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Standby Primary Fault Injection • AWS Fault Injection Simulator • Service API’s/cli Availability Zone 1 Availability Zone 2 AWS Region Applications Applications
- 51. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Standby Primary Fault Injection • AWS Fault Injection Simulator • Service API’s/cli EC2 - stop-instances Availability Zone 1 Availability Zone 2 AWS Region Applications Applications
- 52. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Standby Primary Fault Injection • AWS Fault Injection Simulator • Service API’s/cli EC2 - stop-instances Availability Zone 1 Availability Zone 2 AWS Region Applications Applications
- 53. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Standby Primary Fault Injection • AWS Fault Injection Simulator • Service API’s/cli EC2 - stop-instances Availability Zone 1 Availability Zone 2 AWS Region Applications Applications
- 54. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Standby Primary Fault Injection • AWS Fault Injection Simulator • Service API’s/cli EC2 - stop-instances RDS - reboot-instance --force-failover Aurora – failover-db-cluster Availability Zone 1 Availability Zone 2 AWS Region Applications Applications
- 55. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Fault Injection • AWS Fault Injection Simulator • Service API’s/cli EC2 - stop-instances RDS - reboot-instance --force-failover Aurora – failover-db-cluster Availability Zone 1 Availability Zone 2 AWS Region Applications Applications Standby Primary
- 56. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Timeouts • Timeouts prevent runaway queries Availability Zone 1 Availability Zone 2 AWS Region Applications
- 57. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Timeouts • Timeouts prevent runaway queries Availability Zone 1 Availability Zone 2 AWS Region Applications
- 58. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Timeouts • Timeouts prevent runaway queries • Timeouts can be implemented at any or all layers in the application stack Availability Zone 1 Availability Zone 2 AWS Region Applications 4 sec 10 sec 30 sec
- 59. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Timeouts • Timeouts prevent runaway queries • Timeouts can be implemented at any or all layers in the application stack • Improperly coordinated timeouts can cause connection storms Availability Zone 1 Availability Zone 2 AWS Region Applications 4 sec 10 sec 30 sec
- 60. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Timeouts • Timeouts prevent runaway queries • Timeouts can be implemented at any or all layers in the application stack • Improperly coordinated timeouts can cause connection storms • The database statement timeout needs to be the shortest duration Availability Zone 1 Availability Zone 2 AWS Region Applications 30 sec 10 sec 4 sec
- 61. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Thank you! © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Grant McAlister Jim Mlodgenski