SlideShare a Scribd company logo

OSMC 2021 | pg_stat_monitor: A cool extension for better database (PostgreSQL) monitoring

NETWAYS, profile picture
NETWAYS

The document discusses pg_stat_monitor, an extension for PostgreSQL that provides enhanced monitoring of database performance by capturing detailed statistics on queries. It explains the importance of various data points for identifying poor performance due to 'bad queries' and outlines how to install and configure the extension effectively. Additionally, it includes information about the monitoring capabilities and data collection methods used by pg_stat_monitor.

Software
1 of 40
Download to read offline
1
2
3
4
5
6
7
8
9
Most read
10
Most read
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
pg_stat_monitor A cool extension for better database (PostgreSQL) monitoring
© 2021 Percona Ibrar Ahmed - The Author ● Software industries since 1998 ● Open Source Contribution other than PostgreSQL ○ Cross platform implementation of FTP for Google Chrome ○ Testing Framework for Google Chromium Project ○ Cross Platform Telnet application ● Working on PostgreSQL Since 2006 ● EnterpriseDB ○ Associate Software Architect core Database Engine 2006-2009 ○ Software Architect core Database Engine 2011 - 2016 ○ Senior Software Architect core Database Engine 2016 – 2018 ● Percona ○ Senior Software Architect core Database Engine 2018 – Present 2 Twitter: @ibrar_ahmad LinkedIn: ibrarahmed74 pgelephant.com
© 2021 Percona Peter Zaitsev - The Mentor ● CEO, Percona ● Database Performance Geek ● Know enough PostgreSQL to be dangerous 3 Twitter: @PeterZaitsev LinkedIn: peterzaitsev
© 2021 Percona Denys Kondratenko ● Engineering Manager in Percona ○ Percona Monitoring and Management ● Past experience around distributed storage as eng mgr ○ SUSE - Ceph ○ Seagate - Lustre ● Monitoring experience in Massive Solutions ● Embedded engineer in Mindspeed before that ● love linux, run full Wayland, openSUSEway (sway) 4 github: denisok LinkedIn: kondratenko https://per.co.na/discord
© 2021 Percona | Confidential Database Performance Basics ● Percona - 15+ Years Open Source Database Performance Passion ● Most Common Reason of Poor Performance “Bad Queries” ● Users often do not even know they have such queries ● When they may not know source of such queries ● Or why they are bad and how to fix them 5
© 2021 Percona | Confidential Sourcing the Data ● System State Sampling ○ Limited Overhead ○ Limited Accuracy ○ Can have measurement Artifacts ● Event Counting ○ Risk of high overhead ○ Very Good accuracy ● pg_stat_monitor is Event Counting Approach 6
© 2021 Percona | Confidential Why Some Data Points are Important ? ● Actual parameters in the statements ○ So you can run explain or modify the query and see if it improves ● Time Bucketing ○ High Fidelity stats, Better Network Quality Tolerance ● Multi-Dimensional grouping, like Client IP ○ Security Analyses. Spot where workload which should not be running runs ● Capture Failed Queries ○ Some poorly Performing Queries never complete (ERROR, WARNING and LOG error levels). Catch errors in your application ● Capture Touched Relation (tables accessed by the statement) ○ Easily spot all queries touching given relation 7
© 2021 Percona | Confidential Why Some Data Points are Important ? ● Response Time Distribution (histograms) ○ “Average” query execution time does not tell the whole story ● Bucketing by Different Query Plans ○ Because Different Query Plans can have drastically difference performance behavior ● Plan Information ○ Having Plan for actual query executions is helpful with unstable plans ● Query Hierarchy ○ When Query is called from Stored Routine both query and caller are very meaningful 8
© 2021 Percona Download pg_stat_monitor Source code ○ git clone https://github.com/percona/pg_stat_monitor.git ○ cd pg_stat_monitor ○ make USE_PGXS=1 install Packages ○ Percona ■ https://www.percona.com/downloads/postgresql-distribution-13 ■ https://repo.percona.com/ppg-13/yum/release/7/RPMS/x86_64/ ○ PostgreSQL yum repositories ■ https://yum.postgresql.org/packages/ ○ PostgreSQL Extensions Network (PGXN) ■ pgxn install pg_stat_monitor 9
© 2021 Percona Install pg_stat_monitor ● pg_stat_monitor library can be loaded by setting shared_preload_libraries in postgresql.conf file. shared_preload_libraries = 'pg_stat_monitor' # (change requires restart) ● The same parameter shared_preload_libraries can be changed using ALTER SYSTEM command. postgres=# ALTER SYSTEM SET shared_preload_libraries=pg_stat_monitor; ALTER SYSTEM ● Restart the server # sudo systemctl restart postgresql* ● Create the extension using the create extension command. postgres=# CREATE EXTENSION pg_stat_monitor; CREATE EXTENSION 10
© 2021 Percona SQL version for pg_stat_monitor # pg_stat_monitor extension comment = 'The pg_stat_monitor is a PostgreSQL Query Performance Monitoring tool, based on PostgreSQL contrib module pg_stat_ statements. pg_stat_monitor provides aggregated statistics, client information, plan details including plan, and histogram in formation.' default_version = '1.0' module_pathname = '$libdir/pg_stat_monitor' relocatable = true $ ls -lrt *.sql -rw-r--r-- 1 vagrant vagrant 6756 Oct 28 19:18 pg_stat_monitor--1.0.sql 11
© 2021 Percona Build version for pg_stat_monitor postgres=# SELECT pg_stat_monitor_version(); pg_stat_monitor_version ------------------------- 1.0.0-beta-3 (1 row) 12
© 2021 Percona pg_stat_monitor additional columns 13 Column Type Description bucket integer Data collection unit. The number shows what bucket in a chain a record belongs to bucket_start_time timestamp with time zone The start time of the bucket client_ip inet The IP address of a client that run the query planid text An internally generated ID of a query plan query_plan text The sequence of steps used to execute a query. This parameter is available only when the pgsm_enable_query_plan is enabled. top_query text Shows the top query used in a statement top_queryid text Shows the top query id used in a statement application_name text Shows the name of the application connected to the database relations text[] The list of tables involved in the query
© 2021 Percona pg_stat_monitor additional columns 14 Column Type Description cmd_type integer Type of the query executed cmd_type_text text elevel integer Shows the error level of a query (WARNING, ERROR, LOG) sqlcode integer The IP address of a client that run the query message text The error message resp_calls text[] Call histogram cpu_user_time double precision The time (in ms) the CPU spent on running the query cpu_sys_time double precision The time (in ms) the CPU spent on executing the kernel code state_code bigint Shows the state code of a query state text The state message
© 2021 Percona postgres=# SELECT * FROM pg_stat_monitor_settings; name | value | default_value | description | minimum | maximum | restart ------------------------------------------+--------+---------------+----------------------------------------------------+---------+------------+--------- pg_stat_monitor.pgsm_max | 100 | 100 | Sets the maximum size of shared memory in (MB).... | 1 | 1000 | 1 pg_stat_monitor.pgsm_query_max_len | 1024 | 1024 | Sets the maximum length of query. | 1024 | 2147483647 | 1 pg_stat_monitor.pgsm_enable | 1 | 1 | Enable/Disable statistics collector. | 0 | 0 | 0 pg_stat_monitor.pgsm_track_utility | 1 | 1 | Selects whether utility commands are tracked. | 0 | 0 | 0 pg_stat_monitor.pgsm_normalized_query | 0 | 1 | Selects whether save query in normalized format. | 0 | 0 | 0 pg_stat_monitor.pgsm_max_buckets | 10 | 10 | Sets the maximum number of buckets. | 1 | 10 | 1 pg_stat_monitor.pgsm_bucket_time | 60 | 60 | Sets the time in seconds per bucket. | 1 | 2147483647 | 1 pg_stat_monitor.pgsm_histogram_min | 0 | 0 | Sets the time in millisecond. | 0 | 2147483647 | 1 pg_stat_monitor.pgsm_histogram_max | 100000 | 100000 | Sets the time in millisecond. | 10 | 2147483647 | 1 pg_stat_monitor.pgsm_histogram_buckets | 10 | 10 | Sets the maximum number of histogram buckets | 2 | 2147483647 | 1 pg_stat_monitor.pgsm_query_shared_buffer | 20 | 20 | Sets the maximum size of shared memory in (MB) | 1 | 10000 | 1 pg_stat_monitor.pgsm_overflow_target | 0 | 1 | Sets the overflow target for pg_stat_monitor | 0 | 1 | 1 pg_stat_monitor.pgsm_enable_query_plan | 0 | 0 | Enable/Disable query plan monitoring | 0 | 0 | 0 pg_stat_monitor.pgsm_track_planning | 1 | 1 | Selects whether planning statistics are tracked. | 0 | 0 | 0 (14 rows) 15
© 2021 Percona name | value | default_value | description | minimum | maximum | restart ------------------------------------------+--------+---------------+----------------------------------------------------+---------+------------+--------- pg_stat_monitor.pgsm_max | 100 | 100 | Sets the maximum size of shared memory in (MB).... | 1 | 1000 | 1 ● Name ○ pg_stat_monitor.pgsm_max ● Value ○ 100 ● Default Value ○ 100 ● Description ○ Sets the maximum size of shared memory in (MB) used for statement's metadata tracked by pg_stat_monitor. ● Minimum ○ 1 ● Maximum ○ 1000 ● Restart ○ 1 16
© 2021 Percona Buckets SELECT bucket, bucket_start_time, substr(query,0,50)|| '...' AS query, calls FROM pg_stat_monitor; bucket | bucket_start_time | query | calls -------+---------------------+--------------------------- +----- 0 | 2020-10-14 09:30:00 | INSERT INTO pgbench_tell...| 2 1 | 2020-10-14 09:30:30 | INSERT INTO pgbench_bran...| 1 1 | 2020-10-14 09:30:30 | SELECT relname, relkind ...| 1 1 | 2020-10-14 09:30:30 | SELECT queryid, bucket, ...| 1 2 | 2020-10-14 09:31:00 | CREATE table pgbench_his...| 1 2 | 2020-10-14 09:31:00 | SELECT queryid, bucket, ...| 1 2 | 2020-10-14 09:31:00 | SELECT queryid, bucket ...| 1 (7 rows) 17 b u c k e t 4 Bucket 3 Bucket 2 Bucket 0 B u c k e t 1 Bucket Start Time B u c k e t T i m e Aggregates
© 2021 Percona Query information postgres=# SELECT userid, datname, queryid, substr(query,0, 50) AS query, calls FROM pg_stat_monitor; userid | datname | queryid | query | calls ---------+----------+------------------+---------------------------------------------------+------- vagrant | postgres | 939C2F56E1F6A174 | END | 561 vagrant | postgres | 2A4437C4905E0E23 | SELECT abalance FROM pgbench_accounts WHERE aid = | 561 vagrant | postgres | 4EE9ED0CDF143477 | SELECT userid, datname, queryid, substr(query,$1 | 1 vagrant | postgres | 8867FEEB8A5388AC | vacuum pgbench_branches | 1 vagrant | postgres | 41D1168FB0733CAB | select count(*) from pgbench_branches | 1 vagrant | postgres | E5A889A8FF37C2B1 | UPDATE pgbench_accounts SET abalance = abalance + | 561 vagrant | postgres | 4876BBA9A8FCFCF9 | truncate pgbench_history | 1 vagrant | postgres | 22B76AE84689E4DC | INSERT INTO pgbench_history (tid, bid, aid, delta | 561 vagrant | postgres | F6DA9838660825CA | vacuum pgbench_tellers | 1 vagrant | postgres | 214646CE6F9B1A85 | BEGIN | 561 vagrant | postgres | 27462943E814C5B5 | UPDATE pgbench_tellers SET tbalance = tbalance + | 561 vagrant | postgres | 4F66D46F3D4151E | SELECT userid, dbid, queryid, substr(query,0, 50 | 1 vagrant | postgres | 6A02C123488B95DB | UPDATE pgbench_branches SET bbalance = bbalance + | 561 (13 rows) 18
© 2021 Percona Query information postgres=# SELECT application_name, client_ip, substr(query,0,100) AS query FROM pg_stat_monitor; application_name | client_ip | query ------------------+-----------+----------------------------------------------------------------------------- pgbench | 127.0.0.1 | truncate pgbench_history pgbench | 127.0.0.1 | SELECT abalance FROM pgbench_accounts WHERE aid = $1 pgbench | 127.0.0.1 | UPDATE pgbench_accounts SET abalance = abalance + $1 WHERE aid = $2 pgbench | 127.0.0.1 | BEGIN; pgbench | 127.0.0.1 | INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES ($1, $2, $3 pgbench | 127.0.0.1 | END; pgbench | 127.0.0.1 | vacuum pgbench_branches pgbench | 127.0.0.1 | UPDATE pgbench_tellers SET tbalance = tbalance + $1 WHERE tid = $2 pgbench | 127.0.0.1 | vacuum pgbench_tellers pgbench | 127.0.0.1 | UPDATE pgbench_branches SET bbalance = bbalance + $1 WHERE bid = $2 pgbench | 127.0.0.1 | select o.n, p.partstrat, pg_catalog.count(i.inhparent) from pg_catalog.pg_ psql | 127.0.0.1 | SELECT application_name, client_ip, substr(query,$1,$2) as query FROM pg_s pgbench | 127.0.0.1 | select count(*) from pgbench_branches (13 rows) 19
© 2021 Percona Query Monitoring postgres=# SET pg_stat_monitor.pgsm_normalized_query = true; postgres=# SELECT a FROM foo where a = 10; a --- (0 rows) postgres=# SELECT queryid, substr(query,0,50)|| '...' AS query,calls FROM pg_stat_monitor; queryid | query | calls ------------------+------------------------------------------------------+------- 55F88A754A1BC5FF | SELECT a from foo where a = $1... | 1 postgres=# SET pg_stat_monitor.pgsm_normalized_query = false; postgres=# SELECT a,2 FROM foo where a = 10; a | ?column? ---+---------- (0 rows) postgres=# SELECT queryid, substr(query,0,50)|| '...' AS query,calls FROM pg_stat_monitor; queryid | query | calls ------------------+------------------------------------------------------+------- EF380BA0410F35EC | SELECT a,2 from foo where a = 10;... | 1 20
© 2021 Percona Error Logging SELECT decode_error_level(elevel) AS elevel, sqlcode, query, message FROM pg_stat_monitor WHERE elevel != 0; elevel | sqlcode | query | message --------+----------+-----------------------+---------------------------------- ERROR | 16908420 | SELECT * FROM pg_foo; | relation "pg_foo" does not exist ERROR | 33816706 | SELECT 1/0; | division by zero 21
© 2021 Percona Query Timing Histogram postgres=# SELECT substr(query,0,40) || '... ' AS query, calls, resp_calls FROM pg_stat_monitor LIMIT 5; query | calls | resp_calls ---------------------------------------------+-------+------------------------ SELECT * FROM pg_stat_database;... | 24 | {23,0,1,0,0,0,0,0,0,0} select extract(epoch from current_times... | 24 | {24,0,0,0,0,0,0,0,0,0} SELECT /* pmm-agent:pgstatmonitor */ pg... | 1 | {1,0,0,0,0,0,0,0,0,0} SELECT pg_database.datname,tmp.mode,COA... | 24 | {24,0,0,0,0,0,0,0,0,0} SELECT /* pmm-agent:pgstatmonitor */ "p... | 1 | {1,0,0,0,0,0,0,0,0,0} (5 rows) 22
© 2021 Percona Query Timing Histogram SELECT resp_calls, query FROM pg_stat_monitor; resp_calls | query --------------------------------------------------+---------------------------------------------- {1," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 0"} | select client_ip, query from pg_stat_monitor {3," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 1"} | select * from pg_stat_monitor_reset() {3," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 1"} | SELECT * FROM foo SELECT * FROM histogram(0, 'F44CD1B4B33A47AF') AS a(range TEXT, freq INT, bar TEXT); range | freq | bar --------------------+------+-------------------------------- (0 - 3)} | 2 | ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ (3 - 10)} | 0 | (10 - 31)} | 1 | ■■■■■■■■■■■■■■■ (31 - 100)} | 0 | (100 - 316)} | 0 | (316 - 1000)} | 0 | (1000 - 3162)} | 0 | (3162 - 10000)} | 0 | (10000 - 31622)} | 0 | (31622 - 100000)} | 0 | (10 rows) 23
© 2021 Percona Planning Statistics SELECT queryid, plans_calls, rows_retrieved, substr(query,0,40) || '... ' AS query, calls FROM pg_stat_monitor; 24
© 2021 Percona Planning Statistics queryid | plans_calls | rows_retrieved | query | calls ------------------+-------------+----------------+-----------------------------+------- BA6EC88C00347CF7 | 0 | 0 | truncate table pgbench_... | 2 1B12EDE3C70B8F88 | 100 | 100 | insert into pgbench_tel... | 100 3FD5C490B83F760D | 0 | 0 | create table pgbench_br... | 2 B1B991EE89D61CB6 | 0 | 0 | create table pgbench_ac... | 2 29A99577F1695D28 | 0 | 11000000 | copy pgbench_accounts ... | 2 772222CE7E8765 | 0 | 0 | alter table pgbench_bra... | 2 D59D5F4391AA6B3 | 10 | 10 | insert into pgbench_bra... | 10 25
© 2021 Percona Query Plan: SELECT substr(query,0,50) AS query FROM pg_stat_monitor LIMIT 10; query | -------------------------------------------------- + SELECT abalance FROM pgbench_accounts WHERE aid = | | SELECT substr(query,$1,$2),calls, planid,query_pl | 26
© 2021 Percona Query Plan: SELECT substr(query,0,50) AS query_plan FROM pg_stat_monitor LIMIT 10; query_plan ---------------------------------------------------------------------------+ Index Scan using pgbench_accounts_pkey on pgbench_accounts | Index Cond: (aid = 102232) | Limit | -> Subquery Scan on pg_stat_monitor | -> Result | -> Sort | Sort Key: p.bucket_start_time | -> Hash Join | Hash Cond: (p.dbid = d.oid) | -> Function Scan on pg_stat_monitor_internal p | ... 27
© 2021 Percona Top Query CREATE OR REPLACE function add2(int, int) RETURNS INTEGER AS $$ BEGIN return (select $1 + $2); END; $$ language plpgsql; SELECT add2(1,2); add2 ------ 3 (1 row) 28
© 2021 Percona Top Query SELECT queryid, top_queryid, query, top_query FROM pg_stat_monitor; queryid | top_queryid | query | top_query ------------------+------------------+--------------------------+-------------- 762B99349F6C7F31 | 7209043A6BB0BCD5 | SELECT (select $1 + $2) | SELECT add2($1,$2) 7209043A6BB0BCD5 | | SELECT add2($1,$2) | (3 rows) 29
© 2021 Percona Query / Command Type SELECT bucket, substr(query,0, 50) AS query, cmd_type_text FROM pg_stat_monitor WHERE elevel = 0; bucket | query | cmd_type_text --------+---------------------------------------------------+---------- 4 | END | 4 | SELECT abalance FROM pgbench_accounts WHERE aid = | SELECT 4 | vacuum pgbench_branches | 4 | select count(*) from pgbench_branches | SELECT 4 | UPDATE pgbench_accounts SET abalance = abalance + | UPDATE 4 | truncate pgbench_history | 4 | INSERT INTO pgbench_history (tid, bid, aid, delta | INSERT 5 | SELECT relations query FROM pg_stat_monitor | SELECT 9 | SELECT bucket, substr(query,$1, $2) AS query, cmd | 30
© 2021 Percona pg_stat_monitor: Others postgres=# d pg_stat_monitor View "public.pg_stat_monitor" Column | Type | ---------------------+------------------+ bucket | bigint | bucket_start_time | text | userid | regrole | datname | name | client_ip | inet | queryid | text | top_queryid | text | query | text | comments | text | 31 planid | text | query_plan | text | top_query | text | application_name | text | relations | text[] | cmd_type | integer | cmd_type_text | text | elevel | integer | sqlcode | text | message | text | calls | bigint | total_time | double precision | min_time | double precision |
© 2021 Percona pg_stat_monitor: Others max_time | double precision | mean_time | double precision | stddev_time | double precision | rows_retrieved | bigint | plans_calls | bigint | plan_total_time | double precision | plan_min_time | double precision | plan_max_time | double precision | plan_mean_time | double precision | shared_blks_hit | bigint | shared_blks_read | bigint | shared_blks_dirtied | bigint | shared_blks_written | bigint | local_blks_hit | bigint | 32 local_blks_read | bigint | local_blks_dirtied | bigint | local_blks_written | bigint | temp_blks_read | bigint | temp_blks_written | bigint | blk_read_time | double precision | blk_write_time | double precision | resp_calls | text[] | cpu_user_time | double precision | cpu_sys_time | double precision | wal_records | bigint | wal_fpi | bigint | wal_bytes | numeric | state_code | bigint | state | text |
© 2021 Percona 33 Vanila PostgreSQL pg_stat_statment pg_stat_monitor Database Size TPS 1.6GB 3256 16GB 840 160GB 608 Database Size TPS 1.6GB 3268 16GB 843 160GB 610 Database Size TPS 1.6GB 3262 16GB 844 160GB 611 Performance Metrics
© 2021 Percona | Confidential pg_stat_monitor in PMM 34
© 2021 Percona | Confidential pg_stat_monitor in PMM 35
© 2021 Percona | Confidential pg_stat_monitor in PMM 36
© 2021 Percona | Confidential pg_stat_monitor in PMM 37
© 2021 Percona pg_stat_monitor add service to PMM # pmm-admin add postgresql --query-source=pgstatmonitor 38
© 2021 Percona Feedback ● If you found a bug or have a feature request in pg_stat_monitor, please submit the report to the https://jira.percona.com/projects/PG/issues ● If there is no existing report, submit your report following these steps: ○ Sign in to Jira issue tracker. You will need to create an account if you do not have one. ○ In the Summary, Description, Steps To Reproduce, Affects Version fields describe the problem you have detected. As a general rule of thumb, try to create bug reports that are: ● Reproducible: describe the steps to reproduce the problem. ● Unique: check if there already exists a JIRA ticket to describe the problem. ● Scoped to a Single Bug: only report one bug in one JIRA ticket. 39
Question

More Related Content

PDF
Docker swarm introduction
Evan Lin
 
PDF
Git - An Introduction
Behzad Altaf
 
PPTX
Introduction to Ansible
CoreStack
 
PPTX
Power shell training
David Brabant
 
PPTX
Github basics
Radoslav Georgiev
 
PPTX
GitLab for CI/CD process
HYS Enterprise
 
PDF
Terraform 0.12 + Terragrunt
Anton Babenko
 
PDF
How to Integrate Kubernetes in OpenStack
Meng-Ze Lee
 
Docker swarm introduction
Evan Lin
 
Git - An Introduction
Behzad Altaf
 
Introduction to Ansible
CoreStack
 
Power shell training
David Brabant
 
Github basics
Radoslav Georgiev
 
GitLab for CI/CD process
HYS Enterprise
 
Terraform 0.12 + Terragrunt
Anton Babenko
 
How to Integrate Kubernetes in OpenStack
Meng-Ze Lee
 

What's hot (20)

PDF
Prometheus course
Jorn Jambers
 
PPTX
Docker introduction
dotCloud
 
PDF
Linux Hardening - nullhyd
n|u - The Open Security Community
 
PDF
Ansible
Rahul Bajaj
 
PPTX
Docker Swarm for Beginner
Shahzad Masud
 
PPTX
Getting started with Jenkins
Edureka!
 
PDF
Introduction to container based virtualization with docker
Bangladesh Network Operators Group
 
PDF
Deep Dive into Docker Swarm Mode
Ajeet Singh Raina
 
PPTX
Intro to git and git hub
Venkat Malladi
 
PPT
Introduction to PowerShell
Salaudeen Rajack
 
PDF
How Zalando runs Kubernetes clusters at scale on AWS - AWS re:Invent
Henning Jacobs
 
PDF
[PHP 也有 Day #64] PHP 升級指南
Shengyou Fan
 
PDF
Cloud adoption fails - 5 ways deployments go wrong and 5 solutions
Yevgeniy Brikman
 
PDF
Rancher 2.0 Technical Deep Dive
LINE Corporation
 
DOCX
3PAR: HOW TO CHANGE THE IP ADDRESS OF HP 3PAR SAN
Saroj Sahu
 
PPTX
Terraform modules restructured
Ami Mahloof
 
PPT
Introduction to Git Commands and Concepts
Carl Brown
 
PDF
Introduction to docker
Instruqt
 
PDF
CI/CD with Jenkins and Docker - DevOps Meetup Day Thailand
Troublemaker Khunpech
 
PDF
Prometheus on EKS
Jo Hoon
 
Prometheus course
Jorn Jambers
 
Docker introduction
dotCloud
 
Linux Hardening - nullhyd
n|u - The Open Security Community
 
Ansible
Rahul Bajaj
 
Docker Swarm for Beginner
Shahzad Masud
 
Getting started with Jenkins
Edureka!
 
Introduction to container based virtualization with docker
Bangladesh Network Operators Group
 
Deep Dive into Docker Swarm Mode
Ajeet Singh Raina
 
Intro to git and git hub
Venkat Malladi
 
Introduction to PowerShell
Salaudeen Rajack
 
How Zalando runs Kubernetes clusters at scale on AWS - AWS re:Invent
Henning Jacobs
 
[PHP 也有 Day #64] PHP 升級指南
Shengyou Fan
 
Cloud adoption fails - 5 ways deployments go wrong and 5 solutions
Yevgeniy Brikman
 
Rancher 2.0 Technical Deep Dive
LINE Corporation
 
3PAR: HOW TO CHANGE THE IP ADDRESS OF HP 3PAR SAN
Saroj Sahu
 
Terraform modules restructured
Ami Mahloof
 
Introduction to Git Commands and Concepts
Carl Brown
 
Introduction to docker
Instruqt
 
CI/CD with Jenkins and Docker - DevOps Meetup Day Thailand
Troublemaker Khunpech
 
Prometheus on EKS
Jo Hoon
 
Ad

Similar to OSMC 2021 | pg_stat_monitor: A cool extension for better database (PostgreSQL) monitoring (20)

PDF
Time series denver an introduction to prometheus
Bob Cotton
 
PDF
What's New in MariaDB Server 10.2 and MariaDB MaxScale 2.1
MariaDB plc
 
PDF
What's New in MariaDB Server 10.2 and MariaDB MaxScale 2.1
MariaDB plc
 
PDF
PostgreSQL 9.5 - Major Features
InMobi Technology
 
PPTX
Zendcon scaling magento
Mathew Beane
 
ODP
Common schema my sql uc 2012
Roland Bouman
 
ODP
Common schema my sql uc 2012
Roland Bouman
 
PPTX
Performance Tuning Cheat Sheet for MongoDB
Severalnines
 
PDF
Sprint 123
ManageIQ
 
PDF
PGConf APAC 2018 - Monitoring PostgreSQL at Scale
PGConf APAC
 
PDF
Apache Cassandra at Macys
DataStax Academy
 
PDF
query_tuning.pdf
ssuserf99076
 
PDF
The Accidental DBA
PostgreSQL Experts, Inc.
 
PDF
OSMC 2023 | What’s new with Grafana Labs’s Open Source Observability stack by...
NETWAYS
 
PDF
PoC Oracle Exadata - Retour d'expérience
Swiss Data Forum Swiss Data Forum
 
PDF
Operating PostgreSQL at Scale with Kubernetes
Jonathan Katz
 
PDF
Towards a ZooKeeper-less Pulsar, etcd, etcd, etcd. - Pulsar Summit SF 2022
StreamNative
 
PPTX
PostgreSQL-as-a-Service with Crunchy PostgreSQL for PKS
VMware Tanzu
 
PPTX
PostgreSQL-as-a-Service with Crunchy PostgreSQL for PKS
Carlos Andrés García
 
PDF
Sprint 44 review
ManageIQ
 
Time series denver an introduction to prometheus
Bob Cotton
 
What's New in MariaDB Server 10.2 and MariaDB MaxScale 2.1
MariaDB plc
 
What's New in MariaDB Server 10.2 and MariaDB MaxScale 2.1
MariaDB plc
 
PostgreSQL 9.5 - Major Features
InMobi Technology
 
Zendcon scaling magento
Mathew Beane
 
Common schema my sql uc 2012
Roland Bouman
 
Common schema my sql uc 2012
Roland Bouman
 
Performance Tuning Cheat Sheet for MongoDB
Severalnines
 
Sprint 123
ManageIQ
 
PGConf APAC 2018 - Monitoring PostgreSQL at Scale
PGConf APAC
 
Apache Cassandra at Macys
DataStax Academy
 
query_tuning.pdf
ssuserf99076
 
The Accidental DBA
PostgreSQL Experts, Inc.
 
OSMC 2023 | What’s new with Grafana Labs’s Open Source Observability stack by...
NETWAYS
 
PoC Oracle Exadata - Retour d'expérience
Swiss Data Forum Swiss Data Forum
 
Operating PostgreSQL at Scale with Kubernetes
Jonathan Katz
 
Towards a ZooKeeper-less Pulsar, etcd, etcd, etcd. - Pulsar Summit SF 2022
StreamNative
 
PostgreSQL-as-a-Service with Crunchy PostgreSQL for PKS
VMware Tanzu
 
PostgreSQL-as-a-Service with Crunchy PostgreSQL for PKS
Carlos Andrés García
 
Sprint 44 review
ManageIQ
 
Ad

Recently uploaded (20)

PDF
Design an Analysis of Algorithms II-SECS-1021-03
DilanEscobar1
 
PDF
Softaken Excel to vCard Converter Software.pdf
markwillsonmw004
 
PDF
Internet Downloader Manager (IDM) Crack 6.42 Build 41
ghrom2211g
 
PPTX
Operating system designcfffgfgggggggvggggggggg
rmskumara09
 
PDF
Wondershare Filmora 15 Crack With Activation Key [2025
ghrom2211g
 
PDF
T3DD25 TYPO3 Content Blocks - Deep Dive by André Kraus
Andre Kraus
 
PDF
EN-Survey-Report-SAP-LeanIX-EA-Insights-2025.pdf
deepakkhaitan3
 
PPTX
Essential Infomation Tech presentation.pptx
pradeepjava2016
 
PDF
How Creative Agencies Leverage Project Management Software.pdf
Orangescrum
 
PPTX
Lecture 3: Operating Systems Introduction to Computer Hardware Systems
hci7se
 
PDF
How to Choose the Right IT Partner for Your Business in Malaysia
Asian Business Services & Technologies
 
PDF
Addressing The Cult of Project Management Tools-Why Disconnected Work is Hold...
OnePlan Solutions
 
PDF
Claude Code: Everyone is a 10x Developer - A Comprehensive AI-Powered CLI Tool
William Chong
 
PDF
Nekopoi APK 2025 free lastest update
umarali35kp
 
PPTX
Oracle E-Business Suite: A Comprehensive Guide for Modern Enterprises
Conacent Consulting
 
PPTX
ai tools demonstartion for schools and inter college
harshavardhanreddyka11
 
PDF
Raksha Bandhan Grocery Pricing Trends in India 2025.pdf
Actowiz Solustions
 
PDF
Which alternative to Crystal Reports is best for small or large businesses.pdf
Varsha Nayak
 
PDF
medical staffing services at VALiNTRY
Tiyan Grayson
 
PPTX
Transform Your Business with a Software ERP System
Canada Software Company
 
Design an Analysis of Algorithms II-SECS-1021-03
DilanEscobar1
 
Softaken Excel to vCard Converter Software.pdf
markwillsonmw004
 
Internet Downloader Manager (IDM) Crack 6.42 Build 41
ghrom2211g
 
Operating system designcfffgfgggggggvggggggggg
rmskumara09
 
Wondershare Filmora 15 Crack With Activation Key [2025
ghrom2211g
 
T3DD25 TYPO3 Content Blocks - Deep Dive by André Kraus
Andre Kraus
 
EN-Survey-Report-SAP-LeanIX-EA-Insights-2025.pdf
deepakkhaitan3
 
Essential Infomation Tech presentation.pptx
pradeepjava2016
 
How Creative Agencies Leverage Project Management Software.pdf
Orangescrum
 
Lecture 3: Operating Systems Introduction to Computer Hardware Systems
hci7se
 
How to Choose the Right IT Partner for Your Business in Malaysia
Asian Business Services & Technologies
 
Addressing The Cult of Project Management Tools-Why Disconnected Work is Hold...
OnePlan Solutions
 
Claude Code: Everyone is a 10x Developer - A Comprehensive AI-Powered CLI Tool
William Chong
 
Nekopoi APK 2025 free lastest update
umarali35kp
 
Oracle E-Business Suite: A Comprehensive Guide for Modern Enterprises
Conacent Consulting
 
ai tools demonstartion for schools and inter college
harshavardhanreddyka11
 
Raksha Bandhan Grocery Pricing Trends in India 2025.pdf
Actowiz Solustions
 
Which alternative to Crystal Reports is best for small or large businesses.pdf
Varsha Nayak
 
medical staffing services at VALiNTRY
Tiyan Grayson
 
Transform Your Business with a Software ERP System
Canada Software Company
 

OSMC 2021 | pg_stat_monitor: A cool extension for better database (PostgreSQL) monitoring

  • 1. pg_stat_monitor A cool extension for better database (PostgreSQL) monitoring
  • 2. © 2021 Percona Ibrar Ahmed - The Author ● Software industries since 1998 ● Open Source Contribution other than PostgreSQL ○ Cross platform implementation of FTP for Google Chrome ○ Testing Framework for Google Chromium Project ○ Cross Platform Telnet application ● Working on PostgreSQL Since 2006 ● EnterpriseDB ○ Associate Software Architect core Database Engine 2006-2009 ○ Software Architect core Database Engine 2011 - 2016 ○ Senior Software Architect core Database Engine 2016 – 2018 ● Percona ○ Senior Software Architect core Database Engine 2018 – Present 2 Twitter: @ibrar_ahmad LinkedIn: ibrarahmed74 pgelephant.com
  • 3. © 2021 Percona Peter Zaitsev - The Mentor ● CEO, Percona ● Database Performance Geek ● Know enough PostgreSQL to be dangerous 3 Twitter: @PeterZaitsev LinkedIn: peterzaitsev
  • 4. © 2021 Percona Denys Kondratenko ● Engineering Manager in Percona ○ Percona Monitoring and Management ● Past experience around distributed storage as eng mgr ○ SUSE - Ceph ○ Seagate - Lustre ● Monitoring experience in Massive Solutions ● Embedded engineer in Mindspeed before that ● love linux, run full Wayland, openSUSEway (sway) 4 github: denisok LinkedIn: kondratenko https://per.co.na/discord
  • 5. © 2021 Percona | Confidential Database Performance Basics ● Percona - 15+ Years Open Source Database Performance Passion ● Most Common Reason of Poor Performance “Bad Queries” ● Users often do not even know they have such queries ● When they may not know source of such queries ● Or why they are bad and how to fix them 5
  • 6. © 2021 Percona | Confidential Sourcing the Data ● System State Sampling ○ Limited Overhead ○ Limited Accuracy ○ Can have measurement Artifacts ● Event Counting ○ Risk of high overhead ○ Very Good accuracy ● pg_stat_monitor is Event Counting Approach 6
  • 7. © 2021 Percona | Confidential Why Some Data Points are Important ? ● Actual parameters in the statements ○ So you can run explain or modify the query and see if it improves ● Time Bucketing ○ High Fidelity stats, Better Network Quality Tolerance ● Multi-Dimensional grouping, like Client IP ○ Security Analyses. Spot where workload which should not be running runs ● Capture Failed Queries ○ Some poorly Performing Queries never complete (ERROR, WARNING and LOG error levels). Catch errors in your application ● Capture Touched Relation (tables accessed by the statement) ○ Easily spot all queries touching given relation 7
  • 8. © 2021 Percona | Confidential Why Some Data Points are Important ? ● Response Time Distribution (histograms) ○ “Average” query execution time does not tell the whole story ● Bucketing by Different Query Plans ○ Because Different Query Plans can have drastically difference performance behavior ● Plan Information ○ Having Plan for actual query executions is helpful with unstable plans ● Query Hierarchy ○ When Query is called from Stored Routine both query and caller are very meaningful 8
  • 9. © 2021 Percona Download pg_stat_monitor Source code ○ git clone https://github.com/percona/pg_stat_monitor.git ○ cd pg_stat_monitor ○ make USE_PGXS=1 install Packages ○ Percona ■ https://www.percona.com/downloads/postgresql-distribution-13 ■ https://repo.percona.com/ppg-13/yum/release/7/RPMS/x86_64/ ○ PostgreSQL yum repositories ■ https://yum.postgresql.org/packages/ ○ PostgreSQL Extensions Network (PGXN) ■ pgxn install pg_stat_monitor 9
  • 10. © 2021 Percona Install pg_stat_monitor ● pg_stat_monitor library can be loaded by setting shared_preload_libraries in postgresql.conf file. shared_preload_libraries = 'pg_stat_monitor' # (change requires restart) ● The same parameter shared_preload_libraries can be changed using ALTER SYSTEM command. postgres=# ALTER SYSTEM SET shared_preload_libraries=pg_stat_monitor; ALTER SYSTEM ● Restart the server # sudo systemctl restart postgresql* ● Create the extension using the create extension command. postgres=# CREATE EXTENSION pg_stat_monitor; CREATE EXTENSION 10
  • 11. © 2021 Percona SQL version for pg_stat_monitor # pg_stat_monitor extension comment = 'The pg_stat_monitor is a PostgreSQL Query Performance Monitoring tool, based on PostgreSQL contrib module pg_stat_ statements. pg_stat_monitor provides aggregated statistics, client information, plan details including plan, and histogram in formation.' default_version = '1.0' module_pathname = '$libdir/pg_stat_monitor' relocatable = true $ ls -lrt *.sql -rw-r--r-- 1 vagrant vagrant 6756 Oct 28 19:18 pg_stat_monitor--1.0.sql 11
  • 12. © 2021 Percona Build version for pg_stat_monitor postgres=# SELECT pg_stat_monitor_version(); pg_stat_monitor_version ------------------------- 1.0.0-beta-3 (1 row) 12
  • 13. © 2021 Percona pg_stat_monitor additional columns 13 Column Type Description bucket integer Data collection unit. The number shows what bucket in a chain a record belongs to bucket_start_time timestamp with time zone The start time of the bucket client_ip inet The IP address of a client that run the query planid text An internally generated ID of a query plan query_plan text The sequence of steps used to execute a query. This parameter is available only when the pgsm_enable_query_plan is enabled. top_query text Shows the top query used in a statement top_queryid text Shows the top query id used in a statement application_name text Shows the name of the application connected to the database relations text[] The list of tables involved in the query
  • 14. © 2021 Percona pg_stat_monitor additional columns 14 Column Type Description cmd_type integer Type of the query executed cmd_type_text text elevel integer Shows the error level of a query (WARNING, ERROR, LOG) sqlcode integer The IP address of a client that run the query message text The error message resp_calls text[] Call histogram cpu_user_time double precision The time (in ms) the CPU spent on running the query cpu_sys_time double precision The time (in ms) the CPU spent on executing the kernel code state_code bigint Shows the state code of a query state text The state message
  • 15. © 2021 Percona postgres=# SELECT * FROM pg_stat_monitor_settings; name | value | default_value | description | minimum | maximum | restart ------------------------------------------+--------+---------------+----------------------------------------------------+---------+------------+--------- pg_stat_monitor.pgsm_max | 100 | 100 | Sets the maximum size of shared memory in (MB).... | 1 | 1000 | 1 pg_stat_monitor.pgsm_query_max_len | 1024 | 1024 | Sets the maximum length of query. | 1024 | 2147483647 | 1 pg_stat_monitor.pgsm_enable | 1 | 1 | Enable/Disable statistics collector. | 0 | 0 | 0 pg_stat_monitor.pgsm_track_utility | 1 | 1 | Selects whether utility commands are tracked. | 0 | 0 | 0 pg_stat_monitor.pgsm_normalized_query | 0 | 1 | Selects whether save query in normalized format. | 0 | 0 | 0 pg_stat_monitor.pgsm_max_buckets | 10 | 10 | Sets the maximum number of buckets. | 1 | 10 | 1 pg_stat_monitor.pgsm_bucket_time | 60 | 60 | Sets the time in seconds per bucket. | 1 | 2147483647 | 1 pg_stat_monitor.pgsm_histogram_min | 0 | 0 | Sets the time in millisecond. | 0 | 2147483647 | 1 pg_stat_monitor.pgsm_histogram_max | 100000 | 100000 | Sets the time in millisecond. | 10 | 2147483647 | 1 pg_stat_monitor.pgsm_histogram_buckets | 10 | 10 | Sets the maximum number of histogram buckets | 2 | 2147483647 | 1 pg_stat_monitor.pgsm_query_shared_buffer | 20 | 20 | Sets the maximum size of shared memory in (MB) | 1 | 10000 | 1 pg_stat_monitor.pgsm_overflow_target | 0 | 1 | Sets the overflow target for pg_stat_monitor | 0 | 1 | 1 pg_stat_monitor.pgsm_enable_query_plan | 0 | 0 | Enable/Disable query plan monitoring | 0 | 0 | 0 pg_stat_monitor.pgsm_track_planning | 1 | 1 | Selects whether planning statistics are tracked. | 0 | 0 | 0 (14 rows) 15
  • 16. © 2021 Percona name | value | default_value | description | minimum | maximum | restart ------------------------------------------+--------+---------------+----------------------------------------------------+---------+------------+--------- pg_stat_monitor.pgsm_max | 100 | 100 | Sets the maximum size of shared memory in (MB).... | 1 | 1000 | 1 ● Name ○ pg_stat_monitor.pgsm_max ● Value ○ 100 ● Default Value ○ 100 ● Description ○ Sets the maximum size of shared memory in (MB) used for statement's metadata tracked by pg_stat_monitor. ● Minimum ○ 1 ● Maximum ○ 1000 ● Restart ○ 1 16
  • 17. © 2021 Percona Buckets SELECT bucket, bucket_start_time, substr(query,0,50)|| '...' AS query, calls FROM pg_stat_monitor; bucket | bucket_start_time | query | calls -------+---------------------+--------------------------- +----- 0 | 2020-10-14 09:30:00 | INSERT INTO pgbench_tell...| 2 1 | 2020-10-14 09:30:30 | INSERT INTO pgbench_bran...| 1 1 | 2020-10-14 09:30:30 | SELECT relname, relkind ...| 1 1 | 2020-10-14 09:30:30 | SELECT queryid, bucket, ...| 1 2 | 2020-10-14 09:31:00 | CREATE table pgbench_his...| 1 2 | 2020-10-14 09:31:00 | SELECT queryid, bucket, ...| 1 2 | 2020-10-14 09:31:00 | SELECT queryid, bucket ...| 1 (7 rows) 17 b u c k e t 4 Bucket 3 Bucket 2 Bucket 0 B u c k e t 1 Bucket Start Time B u c k e t T i m e Aggregates
  • 18. © 2021 Percona Query information postgres=# SELECT userid, datname, queryid, substr(query,0, 50) AS query, calls FROM pg_stat_monitor; userid | datname | queryid | query | calls ---------+----------+------------------+---------------------------------------------------+------- vagrant | postgres | 939C2F56E1F6A174 | END | 561 vagrant | postgres | 2A4437C4905E0E23 | SELECT abalance FROM pgbench_accounts WHERE aid = | 561 vagrant | postgres | 4EE9ED0CDF143477 | SELECT userid, datname, queryid, substr(query,$1 | 1 vagrant | postgres | 8867FEEB8A5388AC | vacuum pgbench_branches | 1 vagrant | postgres | 41D1168FB0733CAB | select count(*) from pgbench_branches | 1 vagrant | postgres | E5A889A8FF37C2B1 | UPDATE pgbench_accounts SET abalance = abalance + | 561 vagrant | postgres | 4876BBA9A8FCFCF9 | truncate pgbench_history | 1 vagrant | postgres | 22B76AE84689E4DC | INSERT INTO pgbench_history (tid, bid, aid, delta | 561 vagrant | postgres | F6DA9838660825CA | vacuum pgbench_tellers | 1 vagrant | postgres | 214646CE6F9B1A85 | BEGIN | 561 vagrant | postgres | 27462943E814C5B5 | UPDATE pgbench_tellers SET tbalance = tbalance + | 561 vagrant | postgres | 4F66D46F3D4151E | SELECT userid, dbid, queryid, substr(query,0, 50 | 1 vagrant | postgres | 6A02C123488B95DB | UPDATE pgbench_branches SET bbalance = bbalance + | 561 (13 rows) 18
  • 19. © 2021 Percona Query information postgres=# SELECT application_name, client_ip, substr(query,0,100) AS query FROM pg_stat_monitor; application_name | client_ip | query ------------------+-----------+----------------------------------------------------------------------------- pgbench | 127.0.0.1 | truncate pgbench_history pgbench | 127.0.0.1 | SELECT abalance FROM pgbench_accounts WHERE aid = $1 pgbench | 127.0.0.1 | UPDATE pgbench_accounts SET abalance = abalance + $1 WHERE aid = $2 pgbench | 127.0.0.1 | BEGIN; pgbench | 127.0.0.1 | INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES ($1, $2, $3 pgbench | 127.0.0.1 | END; pgbench | 127.0.0.1 | vacuum pgbench_branches pgbench | 127.0.0.1 | UPDATE pgbench_tellers SET tbalance = tbalance + $1 WHERE tid = $2 pgbench | 127.0.0.1 | vacuum pgbench_tellers pgbench | 127.0.0.1 | UPDATE pgbench_branches SET bbalance = bbalance + $1 WHERE bid = $2 pgbench | 127.0.0.1 | select o.n, p.partstrat, pg_catalog.count(i.inhparent) from pg_catalog.pg_ psql | 127.0.0.1 | SELECT application_name, client_ip, substr(query,$1,$2) as query FROM pg_s pgbench | 127.0.0.1 | select count(*) from pgbench_branches (13 rows) 19
  • 20. © 2021 Percona Query Monitoring postgres=# SET pg_stat_monitor.pgsm_normalized_query = true; postgres=# SELECT a FROM foo where a = 10; a --- (0 rows) postgres=# SELECT queryid, substr(query,0,50)|| '...' AS query,calls FROM pg_stat_monitor; queryid | query | calls ------------------+------------------------------------------------------+------- 55F88A754A1BC5FF | SELECT a from foo where a = $1... | 1 postgres=# SET pg_stat_monitor.pgsm_normalized_query = false; postgres=# SELECT a,2 FROM foo where a = 10; a | ?column? ---+---------- (0 rows) postgres=# SELECT queryid, substr(query,0,50)|| '...' AS query,calls FROM pg_stat_monitor; queryid | query | calls ------------------+------------------------------------------------------+------- EF380BA0410F35EC | SELECT a,2 from foo where a = 10;... | 1 20
  • 21. © 2021 Percona Error Logging SELECT decode_error_level(elevel) AS elevel, sqlcode, query, message FROM pg_stat_monitor WHERE elevel != 0; elevel | sqlcode | query | message --------+----------+-----------------------+---------------------------------- ERROR | 16908420 | SELECT * FROM pg_foo; | relation "pg_foo" does not exist ERROR | 33816706 | SELECT 1/0; | division by zero 21
  • 22. © 2021 Percona Query Timing Histogram postgres=# SELECT substr(query,0,40) || '... ' AS query, calls, resp_calls FROM pg_stat_monitor LIMIT 5; query | calls | resp_calls ---------------------------------------------+-------+------------------------ SELECT * FROM pg_stat_database;... | 24 | {23,0,1,0,0,0,0,0,0,0} select extract(epoch from current_times... | 24 | {24,0,0,0,0,0,0,0,0,0} SELECT /* pmm-agent:pgstatmonitor */ pg... | 1 | {1,0,0,0,0,0,0,0,0,0} SELECT pg_database.datname,tmp.mode,COA... | 24 | {24,0,0,0,0,0,0,0,0,0} SELECT /* pmm-agent:pgstatmonitor */ "p... | 1 | {1,0,0,0,0,0,0,0,0,0} (5 rows) 22
  • 23. © 2021 Percona Query Timing Histogram SELECT resp_calls, query FROM pg_stat_monitor; resp_calls | query --------------------------------------------------+---------------------------------------------- {1," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 0"} | select client_ip, query from pg_stat_monitor {3," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 1"} | select * from pg_stat_monitor_reset() {3," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 0"," 1"} | SELECT * FROM foo SELECT * FROM histogram(0, 'F44CD1B4B33A47AF') AS a(range TEXT, freq INT, bar TEXT); range | freq | bar --------------------+------+-------------------------------- (0 - 3)} | 2 | ■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■ (3 - 10)} | 0 | (10 - 31)} | 1 | ■■■■■■■■■■■■■■■ (31 - 100)} | 0 | (100 - 316)} | 0 | (316 - 1000)} | 0 | (1000 - 3162)} | 0 | (3162 - 10000)} | 0 | (10000 - 31622)} | 0 | (31622 - 100000)} | 0 | (10 rows) 23
  • 24. © 2021 Percona Planning Statistics SELECT queryid, plans_calls, rows_retrieved, substr(query,0,40) || '... ' AS query, calls FROM pg_stat_monitor; 24
  • 25. © 2021 Percona Planning Statistics queryid | plans_calls | rows_retrieved | query | calls ------------------+-------------+----------------+-----------------------------+------- BA6EC88C00347CF7 | 0 | 0 | truncate table pgbench_... | 2 1B12EDE3C70B8F88 | 100 | 100 | insert into pgbench_tel... | 100 3FD5C490B83F760D | 0 | 0 | create table pgbench_br... | 2 B1B991EE89D61CB6 | 0 | 0 | create table pgbench_ac... | 2 29A99577F1695D28 | 0 | 11000000 | copy pgbench_accounts ... | 2 772222CE7E8765 | 0 | 0 | alter table pgbench_bra... | 2 D59D5F4391AA6B3 | 10 | 10 | insert into pgbench_bra... | 10 25
  • 26. © 2021 Percona Query Plan: SELECT substr(query,0,50) AS query FROM pg_stat_monitor LIMIT 10; query | -------------------------------------------------- + SELECT abalance FROM pgbench_accounts WHERE aid = | | SELECT substr(query,$1,$2),calls, planid,query_pl | 26
  • 27. © 2021 Percona Query Plan: SELECT substr(query,0,50) AS query_plan FROM pg_stat_monitor LIMIT 10; query_plan ---------------------------------------------------------------------------+ Index Scan using pgbench_accounts_pkey on pgbench_accounts | Index Cond: (aid = 102232) | Limit | -> Subquery Scan on pg_stat_monitor | -> Result | -> Sort | Sort Key: p.bucket_start_time | -> Hash Join | Hash Cond: (p.dbid = d.oid) | -> Function Scan on pg_stat_monitor_internal p | ... 27
  • 28. © 2021 Percona Top Query CREATE OR REPLACE function add2(int, int) RETURNS INTEGER AS $$ BEGIN return (select $1 + $2); END; $$ language plpgsql; SELECT add2(1,2); add2 ------ 3 (1 row) 28
  • 29. © 2021 Percona Top Query SELECT queryid, top_queryid, query, top_query FROM pg_stat_monitor; queryid | top_queryid | query | top_query ------------------+------------------+--------------------------+-------------- 762B99349F6C7F31 | 7209043A6BB0BCD5 | SELECT (select $1 + $2) | SELECT add2($1,$2) 7209043A6BB0BCD5 | | SELECT add2($1,$2) | (3 rows) 29
  • 30. © 2021 Percona Query / Command Type SELECT bucket, substr(query,0, 50) AS query, cmd_type_text FROM pg_stat_monitor WHERE elevel = 0; bucket | query | cmd_type_text --------+---------------------------------------------------+---------- 4 | END | 4 | SELECT abalance FROM pgbench_accounts WHERE aid = | SELECT 4 | vacuum pgbench_branches | 4 | select count(*) from pgbench_branches | SELECT 4 | UPDATE pgbench_accounts SET abalance = abalance + | UPDATE 4 | truncate pgbench_history | 4 | INSERT INTO pgbench_history (tid, bid, aid, delta | INSERT 5 | SELECT relations query FROM pg_stat_monitor | SELECT 9 | SELECT bucket, substr(query,$1, $2) AS query, cmd | 30
  • 31. © 2021 Percona pg_stat_monitor: Others postgres=# d pg_stat_monitor View "public.pg_stat_monitor" Column | Type | ---------------------+------------------+ bucket | bigint | bucket_start_time | text | userid | regrole | datname | name | client_ip | inet | queryid | text | top_queryid | text | query | text | comments | text | 31 planid | text | query_plan | text | top_query | text | application_name | text | relations | text[] | cmd_type | integer | cmd_type_text | text | elevel | integer | sqlcode | text | message | text | calls | bigint | total_time | double precision | min_time | double precision |
  • 32. © 2021 Percona pg_stat_monitor: Others max_time | double precision | mean_time | double precision | stddev_time | double precision | rows_retrieved | bigint | plans_calls | bigint | plan_total_time | double precision | plan_min_time | double precision | plan_max_time | double precision | plan_mean_time | double precision | shared_blks_hit | bigint | shared_blks_read | bigint | shared_blks_dirtied | bigint | shared_blks_written | bigint | local_blks_hit | bigint | 32 local_blks_read | bigint | local_blks_dirtied | bigint | local_blks_written | bigint | temp_blks_read | bigint | temp_blks_written | bigint | blk_read_time | double precision | blk_write_time | double precision | resp_calls | text[] | cpu_user_time | double precision | cpu_sys_time | double precision | wal_records | bigint | wal_fpi | bigint | wal_bytes | numeric | state_code | bigint | state | text |
  • 33. © 2021 Percona 33 Vanila PostgreSQL pg_stat_statment pg_stat_monitor Database Size TPS 1.6GB 3256 16GB 840 160GB 608 Database Size TPS 1.6GB 3268 16GB 843 160GB 610 Database Size TPS 1.6GB 3262 16GB 844 160GB 611 Performance Metrics
  • 34. © 2021 Percona | Confidential pg_stat_monitor in PMM 34
  • 35. © 2021 Percona | Confidential pg_stat_monitor in PMM 35
  • 36. © 2021 Percona | Confidential pg_stat_monitor in PMM 36
  • 37. © 2021 Percona | Confidential pg_stat_monitor in PMM 37
  • 38. © 2021 Percona pg_stat_monitor add service to PMM # pmm-admin add postgresql --query-source=pgstatmonitor 38
  • 39. © 2021 Percona Feedback ● If you found a bug or have a feature request in pg_stat_monitor, please submit the report to the https://jira.percona.com/projects/PG/issues ● If there is no existing report, submit your report following these steps: ○ Sign in to Jira issue tracker. You will need to create an account if you do not have one. ○ In the Summary, Description, Steps To Reproduce, Affects Version fields describe the problem you have detected. As a general rule of thumb, try to create bug reports that are: ● Reproducible: describe the steps to reproduce the problem. ● Unique: check if there already exists a JIRA ticket to describe the problem. ● Scoped to a Single Bug: only report one bug in one JIRA ticket. 39