Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
0 likes233 views
The document discusses a PostgreSQL meetup focused on connection pooling, highlighting the benefits of using connection poolers like PgBouncer, PgPool-II, and PgCat to manage database connections efficiently. Key topics include how connection pooling reduces resource consumption, improves scalability, and the operational considerations for implementing these tools. Additionally, it emphasizes testing load balancing and ensuring optimal system settings for production environments.
![CLI
pgbouncer=# show help;
• NOTICE: Console usage
• DETAIL:
• SHOW HELP|CONFIG|DATABASES|POOLS|CLIENTS|SERVERS|USERS|VERSION
• SHOW FDS|SOCKETS|ACTIVE_SOCKETS|LISTS|MEM
• SHOW DNS_HOSTS|DNS_ZONES
• SHOW STATS|STATS_TOTALS|STATS_AVERAGES|TOTALS
• SET key = arg
• RELOAD
• PAUSE [<db>]
• RESUME [<db>]
• DISABLE <db>
• ENABLE <db>
• RECONNECT [<db>]
• KILL <db>
• SUSPEND
• SHUTDOWN](https://image.slidesharecdn.com/bglppt-fin-240705055339-b085ba0c/85/Scaling-Connections-in-PostgreSQL-Postgres-Bangalore-PGBLR-Meetup-2-Mydbops-59-320.jpg)
![CLI
pgbouncer=# show pools;
-[ RECORD 1 ]-----------
• database | test
• user | mydbops
• cl_active | 50
• cl_waiting | 50
• sv_active | 50
• sv_idle | 0
• sv_used | 0
• sv_tested | 0
• sv_login | 0
• maxwait | 0
• maxwait_us | 19535
• pool_mode | transaction](https://image.slidesharecdn.com/bglppt-fin-240705055339-b085ba0c/85/Scaling-Connections-in-PostgreSQL-Postgres-Bangalore-PGBLR-Meetup-2-Mydbops-60-320.jpg)
Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2 - Mydbops
- 1. Scaling Connections in PostgreSQL Postgres Bangalore(PGBLR) Meetup-2, 29th June 2024 - Aakash M
- 2. About Me • PostgreSQL Tech Lead at Mydbops • Expertise in databases like MySQL, SQL Server • Performance tuning / Cost Optimization • Tech Speaker on Open Source Events • Active Blogger • 6+ years of Experience
- 3. About Us
- 4. Our Services Focus on MySQL, MongoDB, PostgreSQL, TiDB, Cassandra Consulting Services Consulting Services Managed Support Services 24*7 DBA Team Targeted Engagement
- 5. Agenda • Connection Pooling • Why pooling is required ? • Available Connection Poolers • pgbouncer • Benchmark Comparison
- 7. Application What is Connection Pooling • A pool of database connections • Reusing unused connections from the pool • Once purpose completes, return back to the pool Connections to the database
- 8. Application Connection Pooler What is Connection Pooling • A pool of database connections • Reusing unused connections from the pool • Once purpose completes, return back to the pool Connections to the Pool Connections to the database
- 9. Advantages of Connection Pooling • Limiting connections to the database • Overhead of opening/closing connection is reduced
- 10. Why pooling is required ?
- 11. Why pooling is required ? • Process based connection
- 12. Statistics Collector WAL Writer Background Writer Checkpointer Autovacuum Archiver Shared Memory Postgres(Server Process) Log Writer Why pooling is required ? Application Postgres Architecture PostgreSQL Server
- 13. Statistics Collector WAL Writer Background Writer Checkpointer Autovacuum Archiver Shared Memory Postgres(Server Process) Log Writer Why pooling is required ? Application Postgres(Backend Process) Postgres(Backend Process) PostgreSQL Server
- 14. Disadvantages • More Resources (CPU & Memory) • Slower Context Switching
- 15. Impact of Idle Connections Reference: https://aws.amazon.com/blogs/database/resources-consumed-by-idle-postgresql-connections/
- 16. MySQL is handling More Connections ??
- 17. Receiver Thread Connection Creation in MySQL Application MySQL Server mysqld process Connection Requests Thread Cache User Threads THDS
- 18. Connection Creation in MySQL • Resource Efficiency (CPU & Memory) • Faster Context Switching • Better Scalability
- 20. Connection Pooler Available Since Setup Complexity Github Repository pgbouncer 2007 Simple https://github.com/pgbo uncer/pgbouncer Pgpool-II 2003 Medium to High https://github.com/pgpo ol/pgpool2 PgCat 2020 Medium https://github.com/postg resml/pgcat Odyssey 2017 Medium to High https://github.com/yand ex/odyssey pgagroal 2020 Low to Medium https://github.com/agroa l/pgagroal Available Connection Poolers
- 21. pgbouncer
- 22. Pgbouncer • Lightweight Process • Easy to setup • Single Threaded • Same authentication as database
- 23. Pgbouncer
- 24. Configuration
- 25. Pooling Mode • Session • Transaction • Statement
- 26. Session Pooling Mode • Once the client disconnections, Connection will be released back to the pool • Client and Server Connection is paired
- 27. Application Connection Pooler Session Pooling Mode • Once the client disconnections, Connection will be released back to the pool • Client and Server Connection is paired Connection 1 Connection 1 Connection 2 Connection 2
- 28. Tranaction Pooling Mode • Server Connection is assigned to Client whenever a new transaction is started • Released back to the pool when transaction closes • Server and client connection is not paired • No support for session variables and prepared statements
- 29. Tranaction Pooling Mode Application Connection Pooler User1 Trx1: Begin; statement ; commit or rollback User2 Trx2: Begin; statement ;
- 30. Statement Pooling Mode • Server connection returns to pool immediately after statement completes • Multi statement is not supported • Autocommit is enabled
- 31. Statement Pooling Mode Application Connection Pooler User1 drop table x; drop table y; User2 select 1; select 2; select 3; select 1; select 2;
- 32. Authentication Modes • Authentication with Auth_file • Authentication with Auth_user
- 33. Auth File • Text file contains username and password • Location of Auth_file requires
- 34. Auth File • auth_type = md5 or password • auth_file = /etc/pgbouncer/userlist.txt cat /etc/pgbouncer/userlist.txt "appuser" "md5e10adc3949ba59abbe56e057f20f883e" # Password: 123456 "reporting_user" "md5e10adc543ba59abbe56e057f20f8899" "admin" "md5e10000543ba59abbe56e057f20f8877"
- 35. Auth User • Load user/password directly from the server • No maintenance overhead
- 36. Auth User • auth_user=”low_priv_user” • auth_query=”select usename, passwd FROM pg_shadow WHERE usename=$1” • auth_file = /etc/pgbouncer/userlist.txt cat /etc/pgbouncer/userlist.txt "low_priv_user" "md5e10adc3949ba59abbe56e057f20f883e"
- 37. Parameter Description Default Max_client_conn Maximum number of client connections allowed 100 Default_Pool_Size How many server connections to allow 20 min_pool_size Minimum number of server connections to maintain 0(Disabled) Configuration
- 38. Parameter Description Default Reserve_pool_size Additional connections allowed to the pool 0(Disabled) Reserve_pool_timeout If timeout, use additional connections in the pool 0(Disabled) Listen_port Port to listen 6432 Configuration
- 40. Working of pgbouncer • Max_client_conn=10 • Default_pool_size=3 • Reserve_pool_size=1 • Reserve_pool_timeout=10 • Listen_port=6542 • Pooling_mode=Session
- 45. Application pgbouncer Working of pgbouncer Pool Area Reserve Pool Area Waiting Area 2 connections
- 46. Application pgbouncer Working of pgbouncer Pool Area Reserve Pool Area Waiting Area 2 connections 2 connections
- 47. Application pgbouncer Working of pgbouncer Pool Area Reserve Pool Area Waiting Area 2 connections 2 connections
- 48. Application pgbouncer Working of pgbouncer Pool Area Reserve Pool Area Waiting Area 2 connections 2 connections
- 50. Application pgbouncer Working of pgbouncer Pool Area Reserve Pool Area Waiting Area 4 connections
- 51. Application pgbouncer Working of pgbouncer Pool Area Reserve Pool Area Waiting Area 4 connections 3 connections
- 52. Application pgbouncer Working of pgbouncer Pool Area Reserve Pool Area Waiting Area 4 connections 3 connections
- 53. Application pgbouncer Working of pgbouncer Pool Area Reserve Pool Area Waiting Area 4 connections 2 connections
- 54. Application pgbouncer Working of pgbouncer Pool Area Reserve Pool Area Waiting Area 4 connections 2 connections
- 55. Application pgbouncer Working of pgbouncer Pool Area Reserve Pool Area Waiting Area 4 connections 3 connections
- 56. Application pgbouncer Working of pgbouncer Pool Area Reserve Pool Area Waiting Area 4 connections 4 connections
- 58. CLI • Admin Console • psql -h127.0.0.1 -Upostgres pgbouncer root@ip-172-31-3-53:~/# psql -h172.31.3.53 -Umydbops pgbouncer -p6432 psql (16.3 (Ubuntu 16.3-1.pgdg20.04+1), server 1.12.0/bouncer) pgbouncer=#
- 59. CLI pgbouncer=# show help; • NOTICE: Console usage • DETAIL: • SHOW HELP|CONFIG|DATABASES|POOLS|CLIENTS|SERVERS|USERS|VERSION • SHOW FDS|SOCKETS|ACTIVE_SOCKETS|LISTS|MEM • SHOW DNS_HOSTS|DNS_ZONES • SHOW STATS|STATS_TOTALS|STATS_AVERAGES|TOTALS • SET key = arg • RELOAD • PAUSE [<db>] • RESUME [<db>] • DISABLE <db> • ENABLE <db> • RECONNECT [<db>] • KILL <db> • SUSPEND • SHUTDOWN
- 60. CLI pgbouncer=# show pools; -[ RECORD 1 ]----------- • database | test • user | mydbops • cl_active | 50 • cl_waiting | 50 • sv_active | 50 • sv_idle | 0 • sv_used | 0 • sv_tested | 0 • sv_login | 0 • maxwait | 0 • maxwait_us | 19535 • pool_mode | transaction
- 61. Grafana • Open Source Exporters and Dashboards https://github.com/spreaker/prometheus-pgbouncer-exporter https://blog.searce.com/grafana-dashboard-for-pgbouncer-and-monitor-with-percona-pmm- 3170d3eb4d14
- 62. Grafana
- 64. Benchmark • 50 pool size • Hosted on Application Server • Small Instance Size • Pgbench for benchmarking • 1, 50, 75, 100, 200, 400, 500 connections
- 65. Benchmark
- 66. Benchmark
- 67. Benchmark • Direct connection performs good when number of connection is less • Only for OLTP workload • Maintain resource usage
- 68. Considerations
- 69. Few Considerations • Single Point of Failure • Single Threaded • Multiple Instance for More Connections • OS should be optimised
- 71. Production Use Case • High CPU usage for last 1 week • AWS RDS db.r5.2xlarge (8 CPU, 64G RAM) • No new deployment
- 73. Production Use Case • No CPU causing queries • Increased connections • Killing Idle Connections
- 79. Pgpool - II
- 80. pgpool-II • multi threaded Architecture • Connection Pooling • Read Write Routing (Load Balancing) • Query Caching • In-Built HA Architecture
- 81. General Considerations • Complexity • Higher Resource Usage • Latency
- 82. Pgcat
- 83. pgcat • multi threaded Architecture • Simple to setup • Sharding • Read Write Routing (Load Balancing) • Query Caching • In-Built HA Architecture
- 84. pgcat vs pgbouncer Ref: https://tembo.io/blog/postgres-connection- poolers
- 85. pgcat vs pgbouncer Ref: https://tembo.io/blog/postgres-connection- poolers Ref: https://tembo.io/blog/postgres-connection- poolers
- 86. Summary • pgbouncer works well for simpler setups • pgpool/pgcat has features and complex settings • Ensure to test load balancing • OS tuning is mandatory • Do not forget HA • Good for OLTP workload
- 87. Reference: • https://tembo.io/blog/postgres-connection-poolers • https://www.mydbops.com/blog/getting-started-with-pgbouncer-in-postgresql/ • https://blog.searce.com/grafana-dashboard-for-pgbouncer-and-monitor-with- percona-pmm-3170d3eb4d14 • https://postgresml.org/docs/product/pgcat/features • https://www.pgpool.net/mediawiki/index.php/Main_Page
- 88. Thank You