Deep Dive into RDS PostgreSQL Universe
1 like731 views
The document presents an in-depth overview of Amazon RDS for PostgreSQL, detailing its robust features such as multi-version concurrency control, automated backups, and high availability options. It emphasizes security and compliance measures, including IAM governance and encryption practices, while discussing deployment flexibility across on-premises and cloud environments. Additionally, the document covers instance types, performance monitoring tools like CloudWatch, and the support for PostgreSQL extensions and logical replication.
Deep Dive into RDS PostgreSQL Universe
- 1. Jignesh Shah, Amazon RDS PGConf India 2019 Deep Dive on Amazon RDS PostgreSQL
- 2. PostgreSQL Robust feature sets and extensions Multi-Version Concurrency Control (MVCC), point in time recovery, granular access controls, tablespaces, asynchronous replication, nested transactions, online/hot backups, a refined query planner/optimizer, and write ahead logging Supports international character sets, multi-byte character encodings, Unicode, and it is locale-aware for sorting, case- sensitivity, and formatting Reliable High fault tolerance, ACID compliance, and full support for foreign keys, joins, views, triggers, and stored procedures Standards-compliant Includes most SQL:2008 data types, including INTEGER, NUMERIC, BOOLEAN, CHAR, VARCHAR, DATE, INTERVAL, and TIMESTAMP. Supports storage of binary large objects, including pictures, sounds, or video
- 3. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. PostgreSQL Deployment Options On-Premises Hosted EC2 DB Services Managed
- 4. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Full control over parameters of server, OS, and database Remote access to host via ssh Customer can install 3rd party applications and extensions Self Managed PostgreSQL • Customer has full responsibility for upgrades and backup • Customer has major responsibility for security • High Availability and replication are expensive, complex, and require a lot of engineering Running PostgreSQL on EC2
- 5. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Relational Database Service Multi-engine support: Amazon Aurora, MySQL, MariaDB, PostgreSQL, Oracle, SQL Server Automated provisioning, patching, scaling, replicas, backup/restore, failover High availability with RDS Multi-AZ and Amazon Aurora
- 6. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Security and Compliance Cloud security at AWS is the highest priority Compliance is important to meet industry and local regulations
- 7. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Security Groups Database IP firewall protection Protocol Port Range Source TCP 3306 172.31.0.0/16 TCP 3306 “Application security group” Corporate address admins Application tier
- 8. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Virtual Private Cloud (Amazon VPC) Private Network Security Group for fine grained control Availability Zone AWS Region 10.1.0.0/16 10.1.1.0/24
- 9. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. At Rest Encryption Default key available for encryption • Cannot share outside account Recommended to security use separate keys for each instances Benefits: • Ability to share encrypted snapshots across accounts • Limited risks of a compromised key Leveraging AWS Key Management Service (KMS) Data key 1 Data key 2 Data key 3 Data key 4 Customer master key(s) Amazon RDS instance 3 Amazon RDS instance 2 Amazon RDS instance 1
- 10. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Forcing SSL on all connections DB Instance Snapshot Application Host SSL Log Backups Security Group VPC Encryption at Rest ssl_mode=disable rds.force_ssl=1 (default 0)
- 11. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. IAM Governed Access Use AWS Identity and Access Management (IAM) to control who can perform actions on RDS using web console or “aws” command line interface Examples: 1. Deploy, Modify, delete instances 2. Create, delete, promote read replicas 3. Reboot, Start/Stop instance DBA and Ops RDS Controlled with IAM
- 12. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Database Users Access Connects to the database instance using PostgreSQL Clients Examples: CREATE USER foobar ALTER DATABASE testdb RENAME to proddb Your database Users and DBAApplications Controlled with database grants
- 13. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Identity and Access Management (IAM) authentication PostgreSQL authentication is managed externally using IAM • Available for Amazon RDS PostgreSQL and Aurora PostgreSQL Authentication tokens are used to validate the user • Tokens have a lifetime of 15 minutes • Generated using AWS Signature Version 4 New role rds_iam available when IAM integration is enabled
- 14. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Restrict Password changes on PostgreSQL instances Simplifies integration of home grown or 3rd party password management tools New database parameter to restrict password changes • rds.restrict_password_commands = on/off Flexibility to assign a role to allow certain users to • GRANT rds_password TO tom;
- 15. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Compliance Singapore MTCS 27001/9001 27017/27018 Aurora SOC 1, 2, 3 ISO 20001/9001 ISO 27107/27018 PCI HIPAA BAA RDS PostgreSQL SOC 1, 2, 3 ISO 20001/9001 ISO 27107/27018 PCI FedRAMP HIPAA BAA UK Gov. Programs Singapore MTCS EC2 SOC 1, 2, 3 ISO 20001/9001 ISO 27107/27018 PCI FedRAMP HIPAA BAA UK Gov. Programs Singapore MTCS Details: https://aws.amazon.com/compliance/services-in-scope/
- 16. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Database Parameter Groups Create a standard group based on your needs (IAM protected) rds.force_ssl=true shared_preload_libraries=pgaudit, auto_explain,pg_stat_statements,pg_re pack pgaudit.role = rds_pgaudit huge_pages = on Use your standard group or copy to instance specific parameter group Avoid typos while editing parameters
- 17. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Database server instance types General purpose (T2/T3) • 1 vCPU / 1 GB RAM to 8 vCPU 32 GB RAM • Moderate networking performance • Good for smaller or variable workloads • T2.micro is eligible for free tier General purpose (M4/M5) • 2 vCPU / 8 GiB RAM to 96 vCPU 384 GiB RAM • High performance networking • Good for running CPU intensive workloads (e.g. WordPress) Memory optimized (R4/R5) • 2 vCPU / 16 GiB RAM to 96 vCPU 768 GiB RAM • High performance networking • Good for query intensive workloads or high connection counts
- 18. High performance database storage General purpose (GP2) • SSD storage • Maximum of 32 TB • Latency in milliseconds • IOPS determined by volume size • Bursts to 3,000 IOPS (applicable below 1.3 TB) • Affordable performance Provisioned IOPS (IO1) • SSD storage • Maximum of 32 TB • Single digit millisecond latencies • Maximum of 40 K IOPS • Delivers within 10% of the IOPS performance 99.9% of the time • High performance and consistency
- 19. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Automated Backup Snapshots RDS PostgreSQL Snapshots • Scheduled daily volume backup of entire instance • Archive database change logs (WAL) • 35-day retention • Multiple copies in each AZ when running multi-AZ • Taken from standby when running multi-AZ Aurora PostgreSQL Snapshots • Automatic, continuous, incremental backups • No impact on database performance • 35-day retention Every day during your backup window, RDS creates a storage volume snapshot of your database Every five minutes, RDS backs up the transaction logs of your database
- 20. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Availability – Read and Write – Multi-AZ Physical Synchronous Replication AZ1 AZ2 DNS cname update Primary Update
- 21. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Read Replicas • Relieve pressure on your master node with additional read capacity • Bring data close to your applications in different regions • Promote a Read Replica to a master for faster recovery in the event of disaster
- 22. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Read Replicas = Availability Sync Replication Multi-AZ Async Replication
- 23. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cross Region Replicas – Reduce Latency AZ1 AZ2 AZ1 Async Replication US-EAST-1 EU-WEST-1
- 24. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cross Region Replicas – DR & Moves AZ1 AZ2 AZ1 Async Replication US-EAST-1 EU-WEST-1 EU-WEST-1 AZ2
- 25. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Minor version upgrade Prod 10.1 Prod 10.3 reboot Minor Versions upgrade on managed services - Shutdown instance - Replace version binaries - Start instance
- 26. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Major version upgrade Prod 9.6 Prod 10 pg_upgrade Backup Backup No PITR Test 9.6 Test 10 pg_upgrade Restore to a test instance Application Testing
- 27. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Upgrade with minimum downtime using DMS Create new target instance Create Schema on Target ( SCT) Start a replication instance Connect to source and target databases Select tables, schemas, or databases RDS PostgreSQL 9.5 Application Users RDS PostgreSQL 10 Let the AWS Database Migration Service truncate tables and load data Uses change data capture to keep them in sync Switch applications over to the target at your convenience AWS Database Migration Service
- 28. Amazon RDS for PostgreSQL Support for latest minor releases • 10.6, 9.6.11, 9.5.15, 9.4.20 60+ extensions supported • Pglogical, pg_similarity, orafce, pageinspect, amcheck PostgreSQL Version 11 available in preview https://aws.amazon.com/rds/databasepreview/
- 29. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. PostgreSQL Extensions/Modules Supported Email: [email protected] 9.3 Original - 32 9.3 Current - 35 9.4 Current - 39 9.5 Current - 46 Future - ??? 9.6 Current - 58 10 Current - 60
- 30. New PostgreSQL Extensions Supported Extensions Description pglogical Support logical replication –for PostgreSQL 9.6 and PostgreSQL 10 pg_similarity Extension for supporting similar text queries pageinspect Allows to inspect the contents of database pages at a low level protobuf Enable Map Box Vector Tiles support in PostGIS amcheck Allows verify the logical consistency of the structure of indexes orafce Implements commonly used functions to ease migration from Oracle prefix Makes it easy to match prefix using @> operator
- 31. Replication in Amazon RDS PostgreSQL • Statement based • Trigger BasedLogical - SQL • Standard PostgreSQL • Extension “pglogical” • AWS DMS • Third-party Logical - Engine • Read replicas • Multi-AZPhysical - Engine
- 32. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Logical Replication Support • Set rds.logical_replication parameter to 1 • As user who has rds_replication & rds_superuser role SELECT * FROM pg_create_logical_replication_slot('test_slot', 'test_decoding'); pg_recvlogical -d postgres --slot test_slot -U master --host $rds_hostname -f - -- start • Support for Event Triggers • Now support • Native Logical replication • Pglogical • wal2json • decoder_raw RDS Postgres RDS Postgres Logical Replica Redshift EC2 Postgres On Premise Postgres DMS RDS/Aurora PostgreSQL Custom Logical Handler NoSQL DB
- 33. Monitoring Amazon RDS Performance Insights • Measures database load to help you identify bottlenecks • Top SQL/most intensive queries • Adjustable timeframe: hour, day, week, longer • Compliments other key tools • query execution plans • pg_stat_statements
- 34. Monitoring Amazon RDS Enhanced Monitoring Enhanced Monitoring for Amazon RDS Access to over 50 CPU, memory, file system, and disk I/O metrics Access to top processes As low as 1 second intervals
- 35. Monitoring Amazon RDS Amazon CloudWatch Amazon CloudWatch metrics Displayed in the Amazon RDS Console or personalized CloudWatch dashboards As low as one minute intervals Amazon CloudWatch alarms Trigger actions based on a metric value relative to a threshold you set
- 36. Upload PostgreSQL Logs to Amazon Cloudwatch Amazon CloudWatch Amazon CloudWatch Upload PostgreSQL logs to CloudWatch Export logs to S3 from CloudWatch
- 37. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Forums EC2 • https://forums.aws.amazon.com/forum.jspa?forumID=30 Amazon RDS • https://forums.aws.amazon.com/forum.jspa?forumID=60 Aurora with PostgreSQL compatibility • https://forums.aws.amazon.com/forum.jspa?forumID=227
- 38. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon RDS PostgreSQL Customers