pg_proctab: Accessing System Stats in PostgreSQL
3 likes3,756 views
The document discusses the use of pg_proctab, a collection of C stored functions for accessing PostgreSQL system and operating system statistics, including process activity and resource usage. It provides examples of how to query various statistics and create snapshots to monitor system performance. Additionally, it highlights the utility of pg_proctab for generating custom reports and monitoring remote databases.
![Calculate Processor Utilization
$ ./ps-processor-utilization.sh [pid] [before] [after]
$ ./ps-processor-utilization.sh 4590 1 2
Processor Utilization = 1.00 %
What the script does (partially) should be the same as top:
SELECT stime, utime, stime + utime AS total,
extract(epoch FROM time)
FROM ps_snaps a, ps_procstat b
WHERE pid = ${PID}
AND a.snap = b.snap
AND a.snap = ${SNAP1}](https://image.slidesharecdn.com/pgeast2010-pgproctab-100327162327-phpapp01/85/pg_proctab-Accessing-System-Stats-in-PostgreSQL-30-320.jpg)
pg_proctab: Accessing System Stats in PostgreSQL
- 1. pg proctab Accessing System Stats in PostgreSQL Mark Wong [email protected] PostgreSQL Conference East 2010 March 25-28, 2010
- 2. Slides available on slideshare http://www.slideshare.net/markwkm
- 3. Agenda Brief review of what PostgreSQL stats are available How pg proctab may help
- 4. Review Some of the PostgreSQL system catalog tables: pg stat activity pg stat database pg stat all tables pg stat all indexes pg statio all tables pg statio all indexes
- 5. Example: pg stat activity SELECT datname, procpid, usename, current_query FROM pg_stat_activity WHERE current_query <> ’<IDLE>’; datname dbt5 procpid 3260 usename postgres current_query SELECT * FROM TradeLookupFrame3( ’2006-2-27 9:15:0’,43000050000,20, ’2005-11-23 12:6:8’,’ENGAPRB’) ...
- 6. Example: pg stat database SELECT datname, numbackends, xact_commit, xact_rollback FROM pg_stat_database ORDER BY datname; datname dbt5 numbackends 11 xact_commit 228458 xact_rollback 320
- 7. Example: pg stat all tables SELECT seq_scan, idx_scan, n_tup_ins, n_tup_upd, n_tup_del FROM pg_stat_all_tables WHERE relname = ’customer’; seq_scan 10 idx_scan 152795 n_tup_ins 5000 n_tup_upd 0 n_tup_del 0
- 8. Example: pg statio all tables SELECT heap_blks_read, heap_blks_hit, idx_blks_read, idx_blks_hit FROM pg_statio_all_tables WHERE relname = ’customer’; heap_blks_read 26897 heap_blks_hit 141581 idx_blks_read 5577 idx_blks_hit 328770
- 9. __ __ / ~~~/ . o O ( Want more! ) ,----( oo ) / __ __/ /| ( |( ^ /___ / | |__| |__|-"
- 10. What about operating system statistics? I/O? — iostat Processor Utilization? — mpstat Per Process Statistics? — pidstat Other system activity? — sar and so on...
- 11. Introducing pg proctab pg proctab is a collection of four C stored functions: pg cputime pg loadavg pg memusage pg proctab
- 12. __ __ / ~~~/ . o O ( Examples? ) ,----( oo ) / __ __/ /| ( |( ^ /___ / | |__| |__|-"
- 13. Some of the things pg proctab should help with Query operating system process table Query operating system statistics Processor time Load averages Memory usage Without escaping out to a shell! ...plus generate reports about timeslices Note: Following examples are from Linux based systems.
- 14. pg cputime() Example SELECT * FROM pg_cputime(); user 31529387 nice 76 system 6865679 idle 574707718 iowait 1985455
- 15. pg cputime() Column Description From Linux kernel source code at Documentation/filesystems/proc.txt: user: normal processes executing in user mode nice: niced processes executing in user mode system: processes executing in kernel mode idle: processes twiddling thumbs iowait: waiting for I/O to complete
- 16. pg loadavg() Example SELECT * FROM pg_loadavg(); load1 7.71 load5 7.73 load15 7.62 last_pid 4623
- 17. pg loadavg() Column Description load1: load average of last minute load5: load average of last 5 minutes load15: load average of last 15 minutes last pid: last pid running
- 18. pg memusage() Example SELECT * FROM pg_memusage(); memused 32793836 memfree 157704 memshared 0 membuffers 94216 memcached 31749292 swapused 13960 swapfree 3986216 swapcached 1264
- 19. pg memusage() Column Description Paraphrased from Linux kernel source code at Documentation/filesystems/proc.txt: memused: Total physical RAM used memfree: Total physical RAM not used memshared: Not used, always 0. (For Solaris.) membuffers: Temporary storage for raw disk blocks memcached: In-memory cache for files read from disk swapused: Total swap space used swapfree: Memory evicted from RAM that is now temporary on disk swapcached: Memory that was swapped out, now swapped in but still in swap
- 20. pg proctab() Partial Column Description Everything from the operating system such as /proc/<pid>/stat, /proc/<pid>/io and /proc/<pid>/cmdline as well as data from PostgreSQL system catalog such as pg stat activity table are available but we’ll only cover some of the fields here: Informative: pid comm - filename of the executable fullcomm (/proc/<pid>/cmdline) uid username Processor: utime - user mode jiffies stime - kernel mode jiffies ...
- 21. pg proctab() Partial Column Description (cont.) Memory: vsize - virtual memory size rss - resident set memory size I/O: syscr - number of read I/O operations syscw - number of write I/O operations reads - number of bytes which this process really did cause to be fetched from the storage layer writes - number of bytes which this process really did cause to be sent from the storage layer cwrites - number of bytes which this process caused to not happen, by truncating pagecache
- 22. __ __ / ~~~/ . o O ( More useful examples? ) ,----( oo ) / __ __/ /| ( |( ^ /___ / | |__| |__|-"
- 23. pg proctab() Example SELECT datname, procpid, processor, state, fullcomm FROM pg_stat_activity, pg_proctab() WHERE procpid = pid; datname dbt5 procpid 3260 processor 6 state R fullcomm postgres: postgres dbt5 207.173.203.228(48950) SELECT datname dbt5 procpid 3261 processor 1 state R fullcomm postgres: postgres dbt5 207.173.203.228(48953) SELECT
- 24. __ __ / / ~~~/ . o O | Measuring performance | ,----( oo ) | of a query. | / __ __/ / /| ( |( ^ /___ / | |__| |__|-" You can find the following helper scripts in the pg proctab contrib directory.
- 25. Create snapshot tables Create a set of tables to hold all of the information returned by these 4 stored functions. Also creates a table to timestamp when a snapshot of data is taken. psql -f create-ps_procstat-tables.sql
- 26. Identify yourself. dbt3=# SELECT * FROM pg_backend_pid(); pg_backend_pid ---------------- 4590 (1 row)
- 27. Take a snapshot before running the query dbt3=# i ps_procstat-snap.sql BEGIN ps_snap_stats --------------- 1 (1 row) COMMIT
- 28. Execute an SQL statement Don’t focus too much on the actual query, the idea is that is you want to collect statistics for a single query: SELECT nation, o_year, Sum(amount) AS sum_profit FROM (SELECT n_name AS nation, Extract(YEAR FROM o_orderdate) AS o_year, l_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity AS amount FROM part, supplier, lineitem, partsupp, orders, nation WHERE s_suppkey = l_suppkey AND ps_suppkey = l_suppkey AND ps_partkey = l_partkey AND p_partkey = l_partkey AND o_orderkey = l_orderkey AND s_nationkey = n_nationkey AND p_name LIKE ’%white%’) AS profit GROUP BY nation, o_year ORDER BY nation, o_year DESC;
- 29. Take a snapshot after running the query dbt3=# i ps_procstat-snap.sql BEGIN ps_snap_stats --------------- 2 (1 row) COMMIT
- 30. Calculate Processor Utilization $ ./ps-processor-utilization.sh [pid] [before] [after] $ ./ps-processor-utilization.sh 4590 1 2 Processor Utilization = 1.00 % What the script does (partially) should be the same as top: SELECT stime, utime, stime + utime AS total, extract(epoch FROM time) FROM ps_snaps a, ps_procstat b WHERE pid = ${PID} AND a.snap = b.snap AND a.snap = ${SNAP1}
- 31. Calculate Disk Utilization $ ./ps-io-utilization.sh 4590 1 2 Reads = 276981 Writes = 63803 Reads (Bytes) = 2164604928 Writes (Bytes) = 508166144 Cancelled (Bytes) = 36880384 SELECT syscr, syscw, reads, writes, cwrites FROM ps_snaps a, ps_procstat b WHERE pid = ${PID} AND a.snap = b.snap AND a.snap = ${SNAP1}
- 32. __ __ / / ~~~/ . o O | Creating Custom | ,----( oo ) | Reports! | / __ __/ / /| ( |( ^ /___ / | |__| |__|-" ps-report-pl
- 33. __ __ / / ~~~/ . o O | Warning! Too much data | ,----( oo ) | to fit on screen! | / __ __/ / /| ( |( ^ /___ / | |__| |__|-"
- 34. Creating Reports: Section 1 Database : dbt5 Snapshot Start : 2010-03-26 15:24:51.516226-07 Snapshot End : 2010-03-26 15:25:51.57661-07 ------------------- Database Statistics ------------------- Commits : 421 Rollbacks : 2 Blocks Read : 13919368 Blocks Hit : 7876506
- 35. Creating Reports: Section 2 ================ Table Statistics ================ ------------------------------------------ -------- ------------ -------- ------------- --------- -------- Schema.Relation Seq Scan Seq Tup Read Idx Scan Idx Tup Fetch N Tup Ins N Tup Up ------------------------------------------ -------- ------------ -------- ------------- --------- -------- ... public.account_permission 0 0 3 3 0 public.address 0 0 488 732 0 public.broker 169 8067 259 259 0 3 public.cash_transaction 0 0 952 928 37 7 public.charge 39 585 0 0 0 public.commission_rate 60 9820 18 44 0 public.company 2 5000 1496 1496 0 public.company_competitor 0 0 61 183 0 public.customer 0 0 375 375 0 public.customer_account 0 0 690 968 0 3 public.customer_taxrate 20 200000 40 40 0 public.daily_market 0 0 4322 4962 0 ...
- 36. Creating Reports: Section 2 - Falling off the right side... N Tup Upd N Tup Del Last Vacuum Last Autovacuum Last Analyze Last Autoanalyze
- 37. Creating Reports: Section 3 ================ Index Statistics ================ --------------------------------------------------------------------- -------- ------------ ------------- Schema.Relation.Index Idx Scan Idx Tup Read Idx Tup Fetch --------------------------------------------------------------------- -------- ------------ ------------- ... public.account_permission.pk_account_permission 3 3 3 public.address.pk_address 488 732 732 public.broker.pk_broker 259 259 259 public.cash_transaction.pk_cash_transaction 952 998 928 public.charge.pk_charge 0 0 0 public.commission_rate.pk_commission_rate 18 44 0 public.company.i_co_name 14 14 14 public.company.pk_company 1482 1482 1482 public.company_competitor.pk_company_competitor 61 183 183 public.customer.i_c_tax_id 27 27 27 public.customer.pk_customer 348 348 348 public.customer_account.i_ca_c_id 198 477 476 ...
- 38. What else can we do with pg proctab? Enable pg top to monitor remote databases by providing access to the database system’s operating system process table.
- 39. pg top
- 40. __ __ / ~~~/ . o O ( Thank you! ) ,----( oo ) / __ __/ /| ( |( ^ /___ / | |__| |__|-"
- 41. . . . the fine print . . . Links: http://git.postgresql.org/gitweb?p=pg_proctab.git git clone git://git.postgresql.org/git/pg proctab.git
- 42. Acknowledgements Haley Jane Wakenshaw __ __ / ~~~/ ,----( oo ) / __ __/ /| ( |( ^ /___ / | |__| |__|-"
- 43. License This work is licensed under a Creative Commons Attribution 3.0 Unported License. To view a copy of this license, (a) visit http://creativecommons.org/licenses/by/3.0/us/; or, (b) send a letter to Creative Commons, 171 2nd Street, Suite 300, San Francisco, California, 94105, USA.