Scaling PostgreSQL with Skytools
18 likes5,090 views
This document discusses scaling PostgreSQL databases using open-source tools from Skype including pgBouncer, PGQ, Londiste, and PL/Proxy. It describes how these tools can be used to improve concurrency, add read replicas, replicate data changes, and shard queries across multiple databases. Configuration files and SQL functions are shown as examples of setting up and using these tools to horizontally scale a PostgreSQL deployment.
![Easy to run
Usage: pgbouncer [OPTION]... config.ini
-d, --daemon Run in background (as a daemon)
-R, --restart Do a online restart
-q, --quiet Run quietly
-v, --verbose Increase verbosity
-u, --user=<username> Assume identity of <username>
-V, --version Show version
-h, --help Show this help screen and exit](https://image.slidesharecdn.com/65304692-scaling-with-skytools-120313170916-phpapp01/85/Scaling-PostgreSQL-with-Skytools-32-320.jpg)
![Specifying Connections
[databases]
; foodb over unix socket
foodb =
; redirect bardb to bazdb on localhost
bardb = host=localhost dbname=bazdb
; access to dest database will go with single user
forcedb = host=127.0.0.1 port=300 user=baz password=foo
client_encoding=UNICODE datestyle=ISO connect_query='SELECT
1'](https://image.slidesharecdn.com/65304692-scaling-with-skytools-120313170916-phpapp01/85/Scaling-PostgreSQL-with-Skytools-35-320.jpg)
![Base Daemon Config
[pgbouncer]
logfile = pgbouncer.log
pidfile = pgbouncer.pid
; ip address or * which means all ip-s
listen_addr = 127.0.0.1
listen_port = 6432
; unix socket is also used for -R.
;unix_socket_dir = /tmp](https://image.slidesharecdn.com/65304692-scaling-with-skytools-120313170916-phpapp01/85/Scaling-PostgreSQL-with-Skytools-36-320.jpg)
![ticker.ini
[pgqadm]
job_name = pgopen_ticker
db = dbname=pgopen
# how often to run maintenance [seconds]
maint_delay = 600
# how often to check for activity [seconds]
loop_delay = 0.1
logfile = ~/Source/pgopen_skytools/%(job_name)s.log
pidfile = ~/Source/pgopen_skytools/%(job_name)s.pid](https://image.slidesharecdn.com/65304692-scaling-with-skytools-120313170916-phpapp01/85/Scaling-PostgreSQL-with-Skytools-45-320.jpg)
![replication.ini
[londiste]
job_name = pgopen_to_destination
provider_db = dbname=pgopen
subscriber_db = dbname=destination
# it will be used as sql ident so no dots/spaces
pgq_queue_name = pgopen
logfile = ~/Source/pgopen_skytools/%(job_name)s.log
pidfile = ~/Source/pgopen_skytools/%(job_name)s.pid](https://image.slidesharecdn.com/65304692-scaling-with-skytools-120313170916-phpapp01/85/Scaling-PostgreSQL-with-Skytools-48-320.jpg)
Scaling PostgreSQL with Skytools
- 1. Scaling with SkyTools & More Scaling-Out Postgres with Skype’s Open-Source Toolset Gavin M. Roy September 14th, 2011
- 2. About Me • PostgreSQL ~ 6.5 • CTO @myYearbook.com • Scaled initial infrastructure • Not as involved day-to-day database operational and development • Twitter: @Crad
- 3. Scaling?
- 4. Concurrency Requests per Second 6am 8am 10am 12pm 2pm 4pm 6pm 8pm 10pm 12am 2am 4am 6am Hourly breakdown
- 5. Increasing Size-On-Disk Size in GB Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
- 7. Size on Disk
- 8. Tuples, Indexes, Overhead
- 9. Table Size + Size of all combined Indexes Relations Indexes
- 10. Constraints • Available Memory • Disk Speed • IO Bus Speed
- 11. Keep it in memory.
- 12. Get Fast Disks & I/O.
- 13. Process Forking + Locks
- 15. One Connection per Concurrent Request
- 16. Apache+PHP One connection per backend for each pg_connect
- 17. Python One connection per connection*
- 18. ODBC One connection to Postgres per ODBC connection
- 19. Master Process Lock Stats Collector Contention? Autovacuum Each backend for a connected Wall Writer client has to check for locks Wall Writer Connection Client Connection Backend
- 20. Master Process New Client Stats Collector Connection? Autovacuum Wall Writer Access Share Access Exclusive Wall Writer Exclusive Share Connection Client Connection Backend Share Row Exclusive Share Update Connection Client Connection Row Share Backend Row Exclusive
- 21. Master Process Stats Collector Too many Autovacuum connections? Wall Writer Wall Writer Slow performance Connection Client Connection Backend Connection Client Connection Backend ... Connection Client Connection Backend
- 22. 250 Apache Backends x 1 Connection per Backend x 250 Servers = 62,500 Connections
- 24. The Trailblazers
- 26. pgBouncer
- 27. Session Pooling
- 30. Connection Pooling Clients Clients Clients Hundreds Hundreds Hundreds pgBouncer Tens Tens Tens Postgres Postgres Postgres Server #1 Server #2 Server #3
- 31. Add Local Pooling Clients Clients Clients Hundreds Hundreds Hundreds Local pgBouncer Local pgBouncer Local pgBouncer Tens Tens Tens pgBouncer Tens Tens Tens Postgres Postgres Postgres Server #1 Server #2 Server #3
- 32. Easy to run Usage: pgbouncer [OPTION]... config.ini -d, --daemon Run in background (as a daemon) -R, --restart Do a online restart -q, --quiet Run quietly -v, --verbose Increase verbosity -u, --user=<username> Assume identity of <username> -V, --version Show version -h, --help Show this help screen and exit
- 34. pgbouncer.ini
- 35. Specifying Connections [databases] ; foodb over unix socket foodb = ; redirect bardb to bazdb on localhost bardb = host=localhost dbname=bazdb ; access to dest database will go with single user forcedb = host=127.0.0.1 port=300 user=baz password=foo client_encoding=UNICODE datestyle=ISO connect_query='SELECT 1'
- 36. Base Daemon Config [pgbouncer] logfile = pgbouncer.log pidfile = pgbouncer.pid ; ip address or * which means all ip-s listen_addr = 127.0.0.1 listen_port = 6432 ; unix socket is also used for -R. ;unix_socket_dir = /tmp
- 37. Authentication ; any, trust, plain, crypt, md5 auth_type = trust #auth_file = 8.0/main/global/pg_auth auth_file = etc/userlist.txt admin_users = user2, someadmin, otheradmin stats_users = stats, root
- 38. Stats Users? SHOW HELP|CONFIG|DATABASES|POOLS|CLIENTS|SERVERS|VERSION SHOW FDS|SOCKETS|ACTIVE_SOCKETS|LISTS|MEM pgbouncer=# SHOW CLIENTS; type | user | database | state | addr | port | local_addr | local_port | connect_time ------+-------+-----------+--------+-----------+-------+------------+------------+--------------------- C | stats | pgbouncer | active | 127.0.0.1 | 47229 | 127.0.0.1 | 6000 | 2011-09-13 17:55:46 * Truncated columns for display purposes
- 39. psql 9.0+ Problem? psql -U stats -p 6432 pgbouncer psql: ERROR: Unknown startup parameter Add to pgbouncer.ini: ignore_startup_parameters = application_name
- 40. Pooling Behavior pool_mode = statement server_check_query = select 1 server_check_delay = 10 max_client_conn = 1000 default_pool_size = 20 server_connect_timeout = 15 server_lifetime = 1200 server_idle_timeout = 60
- 41. Skytools
- 42. Scale-Out Reads Clients Clients Clients Clients pgBouncer Load Balancer Read Only Copy Read Only Copy Read Only Copy Read Only Copy Canonical Database
- 43. PGQ
- 44. The Ticker
- 45. ticker.ini [pgqadm] job_name = pgopen_ticker db = dbname=pgopen # how often to run maintenance [seconds] maint_delay = 600 # how often to check for activity [seconds] loop_delay = 0.1 logfile = ~/Source/pgopen_skytools/%(job_name)s.log pidfile = ~/Source/pgopen_skytools/%(job_name)s.pid
- 46. Getting PGQ Running Setup our ticker: pgqadm.py ticker.ini install Run the ticker daemon: pgqadm.py ticker.ini ticker -d
- 47. Londiste
- 48. replication.ini [londiste] job_name = pgopen_to_destination provider_db = dbname=pgopen subscriber_db = dbname=destination # it will be used as sql ident so no dots/spaces pgq_queue_name = pgopen logfile = ~/Source/pgopen_skytools/%(job_name)s.log pidfile = ~/Source/pgopen_skytools/%(job_name)s.pid
- 49. Install Londiste londiste.py replication.ini provider install londiste.py replication.ini subscriber install
- 50. Start Replication Daemon londiste.py replication.ini replay -d
- 51. DDL?
- 52. Add the Provider Tables and Sequences londiste.py replication.ini provider add public.auth_user
- 53. Add the Subscriber Tables and Sequences londiste.py replication.ini subscriber add public.auth_user
- 54. Great Success!
- 55. PL/Proxy
- 56. Scale-Out Reads & Writes plProxy Server A-F Server G-L Server M-R Server S-Z Server
- 57. How does it work?
- 58. Simple Remote Connection CREATE FUNCTION get_user_email(username text) RETURNS SETOF text AS $$ CONNECT 'dbname=remotedb'; SELECT email FROM users WHERE username = $1; $$ LANGUAGE plproxy;
- 59. Sharded Request CREATE FUNCTION get_user_email(username text) RETURNS SETOF text AS $$ CLUSTER “usercluster”; RUN ON hashtext(username); $$ LANGUAGE plproxy;
- 60. Sharding Setup • Need 3 Functions: • plproxy.get_cluster_partitions(cluster_name text) • plproxy.get_cluster_version(cluster_name text) • plproxy.get_cluster_config(in cluster_name text, out key text, out val text)
- 61. get_cluster_partitions CREATE OR REPLACE FUNCTION plproxy.get_cluster_partitions(cluster_name text) RETURNS SETOF text AS $$ BEGIN IF cluster_name = 'usercluster' THEN RETURN NEXT 'dbname=part00 host=127.0.0.1'; RETURN NEXT 'dbname=part01 host=127.0.0.1'; RETURN; END IF; RAISE EXCEPTION 'Unknown cluster'; END; $$ LANGUAGE plpgsql;
- 62. get_cluster_version CREATE OR REPLACE FUNCTION plproxy.get_cluster_version(cluster_name text) RETURNS int4 AS $$ BEGIN IF cluster_name = 'usercluster' THEN RETURN 1; END IF; RAISE EXCEPTION 'Unknown cluster'; END; $$ LANGUAGE plpgsql;
- 63. get_cluster_config CREATE OR REPLACE FUNCTION plproxy.get_cluster_config( in cluster_name text, out key text, out val text) RETURNS SETOF record AS $$ BEGIN -- lets use same config for all clusters key := 'connection_lifetime'; val := 30*60; -- 30m RETURN NEXT; RETURN; END; $$ LANGUAGE plpgsql;
- 64. get_cluster_config values • connection_lifetime • query_timeout • disable_binary • keepalive_idle • keepalive_interval • keepalive_count
- 65. SQL/MED
- 66. SQL/Med Cluster Definition CREATE SERVER a_cluster FOREIGN DATA WRAPPER plproxy OPTIONS ( connection_lifetime '1800', disable_binary '1', p0 'dbname=part00 hostname=127.0.0.1', p1 'dbname=part01 hostname=127.0.0.1', p2 'dbname=part02 hostname=127.0.0.1', p3 'dbname=part03 hostname=127.0.0.1' );
- 67. PLProxy + SQL/Med Behavior • PL/Proxy will prefer SQL/Med cluster definitions over the plproxy.get_* functions • PL/Proxy will fallback to plproxy.get_* functions if there are no SQL/Med clusters
- 68. SQL/MED User Mapping CREATE USER MAPPING FOR bob SERVER a_cluster OPTIONS (user 'bob', password 'secret'); CREATE USER MAPPING FOR public SERVER a_cluster OPTIONS (user 'plproxy', password 'foo');
- 69. plproxyrc • plpgsql based api for table based management of PL/Proxy • Used to manage complicated PL/Proxy infrastructure @myYearbook • BSD Licensed https://github.com/myYearbook/plproxyrc
- 70. “Server-to-Server” Postgres Postgres Postgres Server #1 Server #2 Server #3 pgBouncer
- 72. Complex PL/Proxy and pgBouncer Environment Clients Local pgBouncer Load Balancer Clients Local pgBouncer Clients Local pgBouncer pgBouncer pgBouncer Postgres plProxy Server plProxy Server Server #1 pgBouncer Postgres Load Balancer Server #3 pgBouncer Postgres Server #3
- 73. Other Tools and Methods?
- 74. Questions?