SlideShare a Scribd company logo

Introduction into MySQL Query Tuning for Dev[Op]s

Sveta Smirnova
Sveta Smirnova

The document provides a comprehensive guide to MySQL query tuning targeted at DevOps. It covers topics like index usage, optimizer histograms, and techniques for diagnosing and tuning slow queries. Additionally, it emphasizes the importance of understanding data structures and query plans for effective performance improvement.

Software
1 of 121
Download to read offline
1
2
Most read
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
MySQL Query Tuning for Dev[Op]s. Introduction
•Basics •Indexes How Do They Work? When MySQL Uses Indexes Optimizer Histograms •Diagnostics EXPLAIN: How Optimizer Works Real Numbers: Inside Storage Engine Real Numbers: Inside the Server •How to Affect Query Plans Table of Contents 2 ©2021 Percona
• Sveta Smirnova • MySQL Support engineer • Author of • MySQL Troubleshooting • JSON UDF functions • FILTER clause for MySQL • Speaker • Percona Live, OOW, Fosdem, DevConf, HighLoad... 3 ©2021 Percona
Basics
Concurrency Environment Query Tuning Troubleshooting Workflow 5 ©2021 Percona
Concurrency Environment Query Tuning Troubleshooting Workflow: This Session 6 ©2021 Percona
$system = System::factory() ->setName($this->form->get(Field::NAME)) ->setDescription( $this->form->get(Field::DESCRIPTION) ); DAO::system()->take($system); The Query 7 ©2021 Percona
$system = System::factory() ->setName($this->form->get(Field::NAME)) ->setDescription( $this->form->get(Field::DESCRIPTION) ); DAO::system()->take($system); The Query 7 ©2021 Percona
cursor = conn.cursor() q = ’’’UPDATE ‘foo‘ SET my_date=NOW(), subject = %s, msg = %s, address = %s, updated_at = NOW() WHERE id=%s ’’’ cursor.execute(q, [ remote_resp.get(’subject’), remote_resp.get(’msg’), remote_resp.get(’address’), my_id ]) The Query 7 ©2021 Percona
cursor = conn.cursor() q = ’’’UPDATE ‘foo‘ SET my_date=NOW(), subject = %s, msg = %s, address = %s, updated_at = NOW() WHERE id=%s ’’’ cursor.execute(q, [ remote_resp.get(’subject’), remote_resp.get(’msg’), remote_resp.get(’address’), my_id ]) The Query 7 ©2021 Percona
SELECT dept_name, title, gender, min(salary) AS mins, max(salary) AS maxs FROM employees JOIN salaries USING(emp_no) JOIN titles USING(emp_no) JOIN dept_emp USING(emp_no) JOIN departments USING(dept_no) JOIN dept_manager USING(dept_no) WHERE dept_manager.to_date = ’9999-01-01’ GROUP BY dept_name, title, gender ORDER BY gender, maxs DESC; The Query 7 ©2021 Percona
SELECT dept_name, title, gender, min(salary) AS mins, max(salary) AS maxs FROM employees JOIN salaries USING(emp_no) JOIN titles USING(emp_no) JOIN dept_emp USING(emp_no) JOIN departments USING(dept_no) JOIN dept_manager USING(dept_no) WHERE dept_manager.to_date = ’9999-01-01’ GROUP BY dept_name, title, gender ORDER BY gender, maxs DESC; The Query 7 ©2021 Percona
• PMM QAN Always Tune Raw Query 8 ©2021 Percona
• PMM QAN • Slow Query Log Always Tune Raw Query 8 ©2021 Percona
• PMM QAN • Slow Query Log • Application log Always Tune Raw Query 8 ©2021 Percona
• PMM QAN • Slow Query Log • Application log • ... Always Tune Raw Query 8 ©2021 Percona
• Mind your data! • 75,000,000 rows • (INT, INT) 75,000,000 * (4 + 4) = 600,000,000 B = 572 MB • (INT, INT, DATETIME, VARCHAR(255), VARCHAR(255)) 75,000,000 * (4 + 4 + 8 + 256 + 256) = 39,600,000,000 bytes = 37 G • 39,600,000,000 / 600,000,000 = 66 Slow is Relative 9 ©2021 Percona
• Mind your data! • Mind use case • Popular website • Admin interface • Weekly cron job Slow is Relative 9 ©2021 Percona
• Mind your data! • Mind use case • Mind location • Server, used by multiple connections • Dedicated for OLAP queries Slow is Relative 9 ©2021 Percona
• MySQL performs a job to execute it Why is the Query Slow 10 ©2021 Percona
• MySQL performs a job to execute it • In the worst case scenario it will do a full table scan • CREATE INDEX • ANALYZE TABLE ... UPDATE HISTOGRAM ON ... Why is the Query Slow 10 ©2021 Percona
• MySQL performs a job to execute it • In the worst case scenario it will do a full table scan • CREATE INDEX • ANALYZE TABLE ... UPDATE HISTOGRAM ON ... • Incorrect index can be used Why is the Query Slow 10 ©2021 Percona
Query sent Connection Pool: Authentication, Caches; SQL interface; Parser Optimizer Storage engines Hardware Query Execution Workflow 11 ©2021 Percona
• Selecting a lot of data • SELECT * FROM many columns table You May not Be Able to Change a Slow Query 12 ©2021 Percona
• Selecting a lot of data • SELECT * FROM many columns table • Badly written • LEFT JOIN instead of INNER JOIN • Many values in IN() • Retrieving large result set, then discarding • Not effective SQL For particular MySQL version You May not Be Able to Change a Slow Query 12 ©2021 Percona
• Selecting a lot of data • SELECT * FROM many columns table • Badly written • LEFT JOIN instead of INNER JOIN • Many values in IN() • Retrieving large result set, then discarding • Not effective SQL For particular MySQL version • We can still improve performance You May not Be Able to Change a Slow Query 12 ©2021 Percona
Indexes
Indexes How Do They Work?
SELECT name FROM users WHERE id=12 1 2 5 6 7 9 12 16 18 21 Full Table Scan 15 ©2021 Percona
SELECT name FROM users WHERE id=12 1 2 5 6 7 9 12 16 18 21 Index Access 16 ©2021 Percona
d001 d003 d008 d009 d003****** d009****** d008****** d009****** d001****** d003****** d009****** d008****** d009****** d001****** d008****** d008****** MySQL Indexes - B-Tree Mostly - LSM Tree - Fractal Tree - R-Tree Spatial - Hash Memory SE - Engine’s 17 ©2021 Percona
• Single column CREATE INDEX index name ON the table(the column) • Multiple columns CREATE INDEX index name ON the table(column1, column2) How to Create an Index 18 ©2021 Percona
• Single column ALTER TABLE table name ADD INDEX [index name] (the column) • Multiple columns ALTER TABLE table name ADD INDEX [index name] (column1, column2) How to Create an Index 18 ©2021 Percona
Indexes When MySQL Uses Indexes
• WHERE the column = a value • WHERE the column IN(value1, value2, value3) • WHERE the column LIKE ’value%’ • WHERE the column LIKE ’%value’ Conditions 20 ©2021 Percona
• WHERE left part = value1 AND right part = value2 • WHERE left part = value1 OR right part = value2 • WHERE right part = value1 AND left part = value2 • WHERE right part = value1 OR left part = value2 Conditions 20 ©2021 Percona
• table1 JOIN table2 ON table1.column1 = table2.column2 Joins 21 ©2021 Percona
• table1 JOIN table2 ON table1.column1 = table2.column2 • Same as FROM table1, table2 WHERE table1.column1 = table2.column2 Joins 21 ©2021 Percona
• GROUP BY the column • GROUP BY left part, right part • GROUP BY right part, left part • GROUP BY the index, another index GROUP BY 22 ©2021 Percona
• ORDER BY the column • ORDER BY left part, right part • ORDER BY right part, left part • ORDER BY the index, another index ORDER BY 23 ©2021 Percona
5.7 ORDER BY left part DESC, right part ASC 8.0 ORDER BY left part DESC, right part ASC • left part must be descending • right part must be ascending • the index(left part DESC, right part ASC) ORDER BY 23 ©2021 Percona
• Deterministic, built-in • Return same value for the same argument • WHERE the column = FLOOR(123.45) Expressions 24 ©2021 Percona
• Deterministic, built-in • Return same value for the same argument • WHERE the column = FLOOR(123.45) • Non-deterministic • Return different values for different invocations • WHERE the column = RAND() ∗ 100 Expressions 24 ©2021 Percona
• Deterministic, built-in • Return same value for the same argument • WHERE the column = FLOOR(123.45) • Non-deterministic • Return different values for different invocations • WHERE the column = RAND() ∗ 100 • Stored functions and UDFs • Indexes are not used Use generated column indexes Expressions 24 ©2021 Percona
• Identify queries not using indexes • Status variables mysql select * from performance_schema.session_status - where variable_name in (’Created_tmp_tables’, - ’Created_tmp_disk_tables’, ’Select_full_join’, - ’Select_full_range_join’, ’Select_range’, - ’Select_range_check’, ’Select_scan’, - ’Sort_merge_passes’, ’Sort_range’, ’Sort_rows’, - ’Sort_scan’) and variable_value 0; +------------------------+----------------+ | VARIABLE_NAME | VARIABLE_VALUE | +------------------------+----------------+ | Select_scan | 2 | +------------------------+----------------+ 1 row in set (0,00 sec) When to Add Indexes? 25 ©2021 Percona
• Identify queries not using indexes mysql show global status like ’Handler_read%’; +-----------------------+-------+ | Variable_name | Value | +-----------------------+-------+ | Handler_read_first | 31 | | Handler_read_key | 1909 | | Handler_read_last | 0 | | Handler_read_next | 4393 | | Handler_read_prev | 0 | | Handler_read_rnd | 0 | | Handler_read_rnd_next | 1135 | +-----------------------+-------+ When to Add Indexes? 25 ©2021 Percona
• Identify queries not using indexes • PERFORMANCE SCHEMA.EVENTS STATEMENTS * mysql select * from events_statements_history_long - where sql_text like ’select count(*) from employees join %’G *************************** 1. row **************************** ... ROWS_SENT: 1 SELECT_RANGE_CHECK: 0 ROWS_EXAMINED: 541058 SELECT_SCAN: 1 CREATED_TMP_DISK_TABLES: 0 SORT_MERGE_PASSES: 0 CREATED_TMP_TABLES: 0 SORT_RANGE: 0 SELECT_FULL_JOIN: 0 SORT_ROWS: 0 SELECT_FULL_RANGE_JOIN: 0 SORT_SCAN: 0 SELECT_RANGE: 0 NO_INDEX_USED: 0 When to Add Indexes? 25 ©2021 Percona
• Identify queries not using indexes • Slow query log # Time: 2020-10-11T23:34:03.701871Z # User@Host: root[root] @ localhost [127.0.0.1] Id: 506 # Query_time: 0.024106 Lock_time: 0.000091 Rows_sent: 1 Rows_examined: 1000 SET timestamp=1602459243; SELECT c FROM sbtest1 WHERE id=996; When to Add Indexes? 25 ©2021 Percona
• Identify queries not using indexes • QAN in PMM When to Add Indexes? 25 ©2021 Percona
• Identify queries not using indexes • Analyze if adding an index will help When to Add Indexes? 25 ©2021 Percona
• Identify queries not using indexes • Analyze if adding an index will help • Add the index When to Add Indexes? 25 ©2021 Percona
• Identify queries not using indexes • Analyze if adding an index will help • Add the index • Test first! • ADD INDEX is an expensive operation Use pt-online-schema-change When to Add Indexes? 25 ©2021 Percona
Indexes Optimizer Histograms
• Since version 8.0 Histogram Statistics 27 ©2021 Percona
• Since version 8.0 • Collected and used by the Optimizer Histogram Statistics 27 ©2021 Percona
• Since version 8.0 • Collected and used by the Optimizer • Visible in Information Schema mysql select HISTOGRAM from information_schema.column_statistics - where table_name=’example’G *************************** 1. row *************************** HISTOGRAM: {buckets: [[1, 0.6], [2, 0.8], [3, 1.0]], data-type: int, null-values: 0.0, collation-id: 8, last-updated: 2018-11-07 09:07:19.791470, sampling-rate: 1.0, histogram-type: singleton, number-of-buckets-specified: 3} 1 row in set (0.00 sec) Histogram Statistics 27 ©2021 Percona
• Since version 8.0 • Collected and used by the Optimizer • Visible in Information Schema • Affects query execution plan, but not access • SELECT ... FROM a JOIN b vs SELECT ... FROM b JOIN a Histogram Statistics 27 ©2021 Percona
• Since version 8.0 • Collected and used by the Optimizer • Visible in Information Schema • Affects query execution plan, but not access More details Histogram Statistics 27 ©2021 Percona
1 2 3 4 5 6 7 8 9 10 0 200 400 600 800 Indexes: Number of Items with Same Value 28 ©2021 Percona
1 2 3 4 5 6 7 8 9 10 0 200 400 600 800 Indexes: Cardinality 29 ©2021 Percona
1 2 3 4 5 6 7 8 9 10 0 200 400 600 800 Histograms: Number of Values in Each Bucket 30 ©2021 Percona
1 2 3 4 5 6 7 8 9 10 0 0.2 0.4 0.6 0.8 1 Histograms: Data in the Histogram 31 ©2021 Percona
• Data distribution is far from uniform • Query accesses two or more tables • Indexes cannot be used • Maintaining them is too expensive • They are already used for different conditions Create Histograms 32 ©2021 Percona
Diagnostics
Diagnostics EXPLAIN: How Optimizer Works
• EXPLAIN • Estimates the query execution plan 5.6- EXTENDED 5.6- PARTITIONS 5.6+ FORMAT=JSON 8.0+ FORMAT=TREE How to Find how MySQL Uses Indexes 35 ©2021 Percona
• EXPLAIN • Estimates the query execution plan 5.6- EXTENDED 5.6- PARTITIONS 5.6+ FORMAT=JSON 8.0+ FORMAT=TREE • INFORMATION SCHEMA.OPTIMIZER TRACE • Real data, collected when query was executing • Advanced topic How to Find how MySQL Uses Indexes 35 ©2021 Percona
mysql EXPLAIN SELECT first_name, last_name, title, salary FROM employees - JOIN salaries USING(emp_no) JOIN titles USING(emp_no) - WHERE salary = (SELECT MAX(salary) FROM salaries - JOIN titles WHERE titles.to_date CURDATE()); +----+-------------+-----------+------------+------+---------------+---------+---------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | +----+-------------+-----------+------------+------+---------------+---------+---------+ | 1 | PRIMARY | employees | NULL | ALL | PRIMARY | NULL | NULL ... | 1 | PRIMARY | salaries | NULL | ref | PRIMARY | PRIMARY | 4 ... | 1 | PRIMARY | titles | NULL | ref | PRIMARY | PRIMARY | 4 ... | 2 | SUBQUERY | titles | NULL | ALL | NULL | NULL | NULL ... | 2 | SUBQUERY | salaries | NULL | ALL | NULL | NULL | NULL ... +----+-------------+-----------+------------+------+---------------+---------+---------+ +----+-...-+----------------------------+---------+----------+-------------------------------+ | id | ... | ref | rows | filtered | Extra | +----+-...-+----------------------------+---------+----------+-------------------------------+ | 1 | ... | NULL | 299113 | 100.00 | NULL | | 1 | ... | employees.employees.emp_no | 9 | 10.00 | Using where | | 1 | ... | employees.employees.emp_no | 1 | 100.00 | Using index | | 2 | ... | NULL | 442189 | 33.33 | Using where | | 2 | ... | NULL | 2838426 | 100.00 | Using join buffer (hash join) | +----+-...-+----------------------------+---------+----------+-------------------------------+ 5 rows in set, 1 warning (0,01 sec) 299113 * 9 * 1 * 442189 * 2838426 = 3,378,806,408,204,514,738 EXPLAIN: rows 36 ©2021 Percona
mysql EXPLAIN SELECT title, MAX(salary) AS maxs FROM employees - JOIN salaries USING(emp_no) JOIN titles USING(emp_no) - GROUP BY title ORDER BY maxs DESC; +----+-------------+-----------+------------+-------+---------------+---------+---------+ | id | select_type | table | partitions | type | possible_keys | key | key_len +----+-------------+-----------+------------+-------+---------------+---------+---------+ | 1 | SIMPLE | employees | NULL | index | PRIMARY | PRIMARY | 4 ... | 1 | SIMPLE | titles | NULL | ref | PRIMARY,title | PRIMARY | 4 ... | 1 | SIMPLE | salaries | NULL | ref | PRIMARY | PRIMARY | 4 ... +----+-------------+-----------+------------+-------+---------------+---------+---------+ EXPLAIN: type 37 ©2021 Percona
ALL Read all rows Query Execution Type 38 ©2021 Percona
ALL Read all rows index Read all rows from the index Query Execution Type 38 ©2021 Percona
ALL Read all rows index Read all rows from the index range Range: N ... M Query Execution Type 38 ©2021 Percona
ALL Read all rows index Read all rows from the index range Range: N ... M subquery Subqueries index subquery unique subquery Query Execution Type 38 ©2021 Percona
ALL Read all rows index Read all rows from the index range Range: N ... M subquery Subqueries merge Combination of two indexes index merge Query Execution Type 38 ©2021 Percona
ALL Read all rows index Read all rows from the index range Range: N ... M subquery Subqueries merge Combination of two indexes ref Comparision with single value ref or null ref eq ref Query Execution Type 38 ©2021 Percona
ALL Read all rows index Read all rows from the index range Range: N ... M subquery Subqueries merge Combination of two indexes ref Comparision with single value fulltext Fulltext index Query Execution Type 38 ©2021 Percona
ALL Read all rows index Read all rows from the index range Range: N ... M subquery Subqueries merge Combination of two indexes ref Comparision with single value fulltext Fulltext index const Single row Query Execution Type 38 ©2021 Percona
ALL Read all rows index Read all rows from the index range Range: N ... M subquery Subqueries merge Combination of two indexes ref Comparision with single value fulltext Fulltext index const Single row system Table has single row Query Execution Type 38 ©2021 Percona
mysql EXPLAIN SELECT title, MAX(salary) AS maxs FROM employees - JOIN salaries USING(emp_no) JOIN titles USING(emp_no) - GROUP BY title ORDER BY maxs DESC; +----+-------------+-----------+------------+-------+---------------+---------+---------+ | id | select_type | table | partitions | type | possible_keys | key | key_len +----+-------------+-----------+------------+-------+---------------+---------+---------+ | 1 | SIMPLE | employees | NULL | index | PRIMARY | PRIMARY | 4 ... | 1 | SIMPLE | titles | NULL | ref | PRIMARY,title | PRIMARY | 4 ... | 1 | SIMPLE | salaries | NULL | ref | PRIMARY | PRIMARY | 4 ... +----+-------------+-----------+------------+-------+---------------+---------+---------+ EXPLAIN: key and possible keys 39 ©2021 Percona
mysql EXPLAIN SELECT title, MAX(salary) AS maxs FROM employees - JOIN salaries USING(emp_no) - JOIN titles USING(emp_no) - GROUP BY title ORDER BY maxs DESC; +----+-------------+-----------+------------+-------+---------------+---------+---------+ | id | select_type | table | partitions | type | possible_keys | key | key_len +----+-------------+-----------+------------+-------+---------------+---------+---------+ | 1 | SIMPLE | employees | NULL | index | PRIMARY | PRIMARY | 4 ... | 1 | SIMPLE | titles | NULL | ref | PRIMARY,title | PRIMARY | 4 ... | 1 | SIMPLE | salaries | NULL | ref | PRIMARY | PRIMARY | 4 ... +----+-------------+-----------+------------+-------+---------------+---------+---------+ EXPLAIN: JOIN order 40 ©2021 Percona
mysql EXPLAIN SELECT first_name, last_name, title, salary FROM employees - JOIN salaries USING(emp_no) JOIN titles USING(emp_no) - WHERE salary IN (SELECT MAX(salary) FROM salaries - JOIN titles WHERE titles.to_date CURDATE()); ... 5 rows in set, 1 warning (0,01 sec) mysql SHOW WARNINGSG *************************** 1. row *************************** Level: Note Code: 1003 Message: /* select#1 */ select ‘employees‘.‘employees‘.‘first_name‘ AS ‘first_name‘, ‘employees‘.‘employees‘.‘last_name‘ AS ‘last_name‘,‘employees‘.‘salaries‘.‘salary‘ AS ‘salary‘ from ‘employees‘.‘employees‘ join ‘employees‘.‘salaries‘ where ((‘employees‘.‘salaries‘.‘emp_no‘ = ‘employees‘.‘employees‘.‘emp_no‘) and in_optimizer(‘employees‘.‘salaries‘.‘salary‘,‘employees‘.‘salaries‘.‘salary‘ in ( materialize (/* select#2 */ select max(‘employees‘.‘salaries‘.‘salary‘) from ‘employees‘.‘salaries‘ join ‘employees‘.‘titles‘ where (‘employees‘.‘titles‘.‘to_date‘ cache(curdate())) having true ), primary_index_lookup(‘employees‘.‘salaries‘.‘salary‘ in temporary table on auto_distinct_key where ((‘employees‘.‘salaries‘.‘salary‘ = ‘materialized_subquery‘.‘MAX(salary)‘)))))) 1 row in set (0,00 sec) Actual Query 41 ©2021 Percona
mysql explain select * from t1G *************************** 1. row *************************** ... rows: 12 Extra: NULL mysql explain select * from t1 where f2=12G *************************** 1. row *************************** ... key: NULL ... rows: 12 Extra: Using where Effect of Indexes: Before 42 ©2021 Percona
mysql alter table t1 add index(f2); Query OK, 12 rows affected (0.07 sec) Records: 12 Duplicates: 0 Warnings: 0 mysql explain select * from t1 where f2=12G *************************** 1. row *************************** ... key: f2 key_len: 5 ref: const rows: 1 Extra: NULL 1 row in set (0.00 sec) Effect of Indexes: After 43 ©2021 Percona
Diagnostics Real Numbers: Inside Storage Engine
• EXPLAIN is optimistic mysql explain select * from ol - where thread_id=10432 and site_id != 9939 - order by id limit 3G *************************** 1. row *************************** id: 1 | ref: NULL select_type: SIMPLE | rows: 33 table: ol | filtered: 8.07 partitions: NULL | Extra: Using where type: index possible_keys: thread_id key: PRIMARY key_len: 4 1 row in set, 1 warning (0,00 sec) Handler * Status Variables 45 ©2021 Percona
• ’Handler *’ show truth mysql FLUSH STATUS; mysql select * from ol - where thread_id=10432 and site_id != 9939 - order by id limit 3; mysql show status like ’Handler%’; +----------------------------+--------+ | Variable_name | Value | +----------------------------+--------+ ... | Handler_read_first | 1 | | Handler_read_key | 1 | | Handler_read_last | 0 | | Handler_read_next | 100000 | ... Handler * Status Variables 45 ©2021 Percona
Diagnostics Real Numbers: Inside the Server
• SHOW [FULL] PROCESSLIST • INFORMATION SCHEMA.PROCESSLIST • performance schema.THREADS PROCESSLIST 47 ©2021 Percona
• SHOW [FULL] PROCESSLIST • INFORMATION SCHEMA.PROCESSLIST • performance schema.THREADS • Your first alert of performance issue PROCESSLIST 47 ©2021 Percona
• SHOW [FULL] PROCESSLIST • INFORMATION SCHEMA.PROCESSLIST • performance schema.THREADS • Your first alert of performance issue • Shows all running queries PROCESSLIST 47 ©2021 Percona
• Can be seen in the PROCESSLIST mysql show processlistG ************************ 1. row ************************ Id: 7 User: root Host: localhost:48799 db: employees Command: Query Time: 2 State: Sending data Info: select count(*) from employees join titles using(emp_no) where title=’Senior Engineer’ ... Execution Stages 48 ©2021 Percona
• Can be seen in the PROCESSLIST • Very useful when you need to answer on the question: ”What is my server doing now?” Execution Stages 48 ©2021 Percona
• PERFORMANCE SCHEMA.EVENTS STAGES * mysql select eshl.event_name, substr(sql_text, 1, 15) ‘sql‘, - eshl.timer_wait/1000000000000 w_s - from events_stages_history_long - eshl join events_statements_history_long esthl on - (eshl.nesting_event_id = esthl.event_id) where - esthl.current_schema=’employees’ and sql_text like - ’select count(*) from employees%’ - order by eshl.timer_start asc; +--------------------------------+-----------------+--------+ | event_name | sql | w_s | +--------------------------------+-----------------+--------+ | stage/sql/starting | select count(*) | 0.0002 | | stage/sql/checking permissions | select count(*) | 0.0000 | ... Execution Stages 48 ©2021 Percona
• PERFORMANCE SCHEMA.EVENTS STAGES * ... | stage/sql/checking permissions | select count(*) | 0.0000 | | stage/sql/Opening tables | select count(*) | 0.0000 | | stage/sql/init | select count(*) | 0.0001 | | stage/sql/System lock | select count(*) | 0.0000 | | stage/sql/optimizing | select count(*) | 0.0000 | | stage/sql/statistics | select count(*) | 0.0001 | | stage/sql/preparing | select count(*) | 0.0000 | | stage/sql/executing | select count(*) | 0.0000 | | stage/sql/Sending data | select count(*) | 5.4915 | | stage/sql/end | select count(*) | 0.0000 | ... Execution Stages 48 ©2021 Percona
• Status variables mysql flush status; Query OK, 0 rows affected (0,01 sec) mysql select count(*) from employees join titles using(emp_no) - where title=’Senior Engineer’; +----------+ | count(*) | +----------+ | 97750 | +----------+ 1 row in set (5,44 sec) Temporary Tables and Other Job 49 ©2021 Percona
• Status variables mysql select * from performance_schema.session_status - where variable_name in (’Created_tmp_tables’, - ’Created_tmp_disk_tables’, ’Select_full_join’, - ’Select_full_range_join’, ’Select_range’, - ’Select_range_check’, ’Select_scan’, ’Sort_merge_passes’, - ’Sort_range’, ’Sort_rows’, ’Sort_scan’) - and variable_value 0; +------------------------+----------------+ | VARIABLE_NAME | VARIABLE_VALUE | +------------------------+----------------+ | Select_scan | 2 | +------------------------+----------------+ 1 row in set (0,00 sec) Temporary Tables and Other Job 49 ©2021 Percona
• PERFORMANCE SCHEMA.EVENTS STATEMENTS * mysql select * from events_statements_history_long - where sql_text like - ’select count(*) from employees join %’G *************************** 1. row **************************** ... ROWS_SENT: 1 SELECT_RANGE_CHECK: 0 ROWS_EXAMINED: 541058 SELECT_SCAN: 1 CREATED_TMP_DISK_TABLES: 0 SORT_MERGE_PASSES: 0 CREATED_TMP_TABLES: 0 SORT_RANGE: 0 SELECT_FULL_JOIN: 0 SORT_ROWS: 0 SELECT_FULL_RANGE_JOIN: 0 SORT_SCAN: 0 SELECT_RANGE: 0 NO_INDEX_USED: 0 Temporary Tables and Other Job 49 ©2021 Percona
• sys.statement analysis mysql select * from statement_analysis where query like - ’SELECT COUNT ( * ) FROM ‘emplo%’ and db=’employees’G *************************** 1. row *************************** query: SELECT COUNT ( * ) FROM ‘emplo ... ‘emp_no‘ ) WHE... db: employees max_latency: 5.59 s full_scan: avg_latency: 5.41 s exec_count: 7 lock_latency: 2.24 ms err_count: 0 rows_sent: 7 warn_count: 0 rows_sent_avg: 1 total_latency: 37.89 s rows_examined: 3787406 Temporary Tables and Other Job 49 ©2021 Percona
• sys.statement analysis rows_examined_avg: 541058 rows_affected: 0 rows_affected_avg: 0 tmp_tables: 0 tmp_disk_tables: 0 rows_sorted: 0 sort_merge_passes: 0 digest: 4086bc3dc6510a1d9c8f2fe1f59f0943 first_seen: 2016-04-14 15:19:19 last_seen: 2016-04-14 16:13:14 Temporary Tables and Other Job 49 ©2021 Percona
How to Affect Query Plans
• Index statistics • Histogram statistics • Optimizer switches • Bugs in optimizer What has an Effect on Query Optimizer Plans? 51 ©2021 Percona
• Collected by storage engine Index Statistics 52 ©2021 Percona
• Collected by storage engine • Used by Optimizer Index Statistics 52 ©2021 Percona
• Can be examined by SHOW INDEX mysql show index from sbtest1; +---------+----------+-------------+-------------+ | Table | Key_name | Column_name | Cardinality | +---------+----------+-------------+-------------+ | sbtest1 | k_1 | k | 49142 | +---------+----------+-------------+-------------+ mysql select count(distinct id), - count(distinct k) from sbtest1; +--------------------+-------------------+ | count(distinct id) | count(distinct k) | +--------------------+-------------------+ | 100000 | 17598 | +--------------------+-------------------+ Index Statistics 52 ©2021 Percona
• Can be updated • ANALYZE TABLE • If does not help: rebuild the table OPTIMIZE TABLE ALTER TABLE ENGINE=INNODB; ANALYZE TABLE Index Statistics 52 ©2021 Percona
• Since version 8.0 Histogram Statistics 53 ©2021 Percona
• Since version 8.0 • Collected and used by the Optimizer Histogram Statistics 53 ©2021 Percona
• Since version 8.0 • Collected and used by the Optimizer • Visible in Information Schema mysql select HISTOGRAM from information_schema.column_statistics - where table_name=’example’G *************************** 1. row *************************** HISTOGRAM: {buckets: [[1, 0.6], [2, 0.8], [3, 1.0]], data-type: int, null-values: 0.0, collation-id: 8, last-updated: 2018-11-07 09:07:19.791470, sampling-rate: 1.0, histogram-type: singleton, number-of-buckets-specified: 3} 1 row in set (0.00 sec) Histogram Statistics 53 ©2021 Percona
• Since version 8.0 • Collected and used by the Optimizer • Visible in Information Schema More details Histogram Statistics 53 ©2021 Percona
mysql select @@optimizer_switchG *************************** 1. row *************************** @@optimizer_switch: index_merge=on,index_merge_union=on, index_merge_sort_union=on,index_merge_intersection=on, engine_condition_pushdown=on,index_condition_pushdown=on, mrr=on,mrr_cost_based=on, block_nested_loop=on,batched_key_access=off, materialization=on,semijoin=on,loosescan=on,firstmatch=on, duplicateweedout=on,subquery_materialization_cost_based=on, use_index_extensions=on,condition_fanout_filter=on, derived_merge=on 1 row in set (0,00 sec) Optimizer Switches 54 ©2021 Percona
• Turn ON and OFF particular optimization Optimizer Switches 54 ©2021 Percona
• Turn ON and OFF particular optimization • Can be unhelpful • Especially for queries, tuned for previous versions Optimizer Switches 54 ©2021 Percona
• Turn ON and OFF particular optimization • Can be unhelpful • Work with them as with any other option • Turn OFF and try SET optimizer_switch = ’use_index_extensions=off’; SELECT ... EXPLAIN SELECT ... Optimizer Switches 54 ©2021 Percona
• Turn ON and OFF particular optimization • Can be unhelpful • Work with them as with any other option • If helps implement in queries SELECT /*+ SEMIJOIN(FIRSTMATCH, LOOSESCAN) */ * FROM t1 ...; SELECT /*+ BKA(t1) NO_BKA(t2) */ * FROM t1 INNER JOIN t2 WHERE ...; Optimizer Switches 54 ©2021 Percona
• Optimizer choses wrong plan for no reason Bugs in Optimizer 55 ©2021 Percona
• Optimizer choses wrong plan for no reason • Statistics are up to date • Histograms are not usable Bugs in Optimizer 55 ©2021 Percona
• Optimizer choses wrong plan for no reason • Statistics are up to date • Histograms are not usable • Solution • Use index hints FORCE INDEX IGNORE INDEX Bugs in Optimizer 55 ©2021 Percona
• Optimizer choses wrong plan for no reason • Statistics are up to date • Histograms are not usable • Solution • On every upgrade • Remove index hints • Test if the query improved • You must do it even for minor version upgrades! Bugs in Optimizer 55 ©2021 Percona
• EXPLAIN is essential for query tuning • Real job is done by storage engine • Index statistics affect query execution plan • All index hints, optimizer hints and other workarounds must be validated on each upgrade Summary 56 ©2021 Percona
EXPLAIN Syntax EXPLAIN FORMAT=JSON is Cool! Troubleshooting add-ons Optimizer Statistics aka Histograms Optimizer Hints Tracing the Optimizer More Information 57 ©2021 Percona
www.slideshare.net/SvetaSmirnova twitter.com/svetsmirnova github.com/svetasmirnova Thank you! 58 ©2021 Percona
Ad

Recommended

PPTX
Query logging with proxysql
YoungHeon (Roy) Kim
 
PDF
MySQL/MariaDB Proxy Software Test
I Goo Lee
 
PPTX
My sql failover test using orchestrator
YoungHeon (Roy) Kim
 
PDF
New features in ProxySQL 2.0 (updated to 2.0.9) by Rene Cannao (ProxySQL)
Altinity Ltd
 
PDF
Intro ProxySQL
I Goo Lee
 
PPTX
Mysql data replication
Tuấn Ngô
 
PDF
Wars of MySQL Cluster ( InnoDB Cluster VS Galera )
Mydbops
 
PDF
Automated master failover
Yoshinori Matsunobu
 
PPSX
Oracle Performance Tuning Fundamentals
Carlos Sierra
 
PPTX
ProxySQL for MySQL
Mydbops
 
PDF
Secret Management with Hashicorp’s Vault
AWS Germany
 
PDF
Galera cluster for high availability
Mydbops
 
PDF
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)
Jean-François Gagné
 
PDF
MySQL 5.7 トラブルシューティング 性能解析入門編
Mikiya Okuno
 
PDF
Mongodb - Scaling write performance
Daum DNA
 
PDF
Hardening Kubernetes by Securing Pods
Suraj Deshmukh
 
PDF
Kubernetes security
Thomas Fricke
 
PDF
MariaDB 10.5 binary install (바이너리 설치)
NeoClova
 
PDF
PostgreSQL 15の新機能を徹底解説
Masahiko Sawada
 
PDF
MySQL Performance for DevOps
Sveta Smirnova
 
PDF
CentOS 8で標準搭載！ 「389-ds」で構築する 認証サーバーについて
Nobuyuki Sasaki
 
PPTX
MySQL_MariaDB로의_전환_기술요소-202212.pptx
NeoClova
 
PDF
MySQL Performance Tuning: Top 10 Tips
OSSCube
 
PDF
[Pgday.Seoul 2019] Citus를 이용한 분산 데이터베이스
PgDay.Seoul
 
DOCX
Keepalived+MaxScale+MariaDB_운영매뉴얼_1.0.docx
NeoClova
 
PDF
Advanced Namespaces and cgroups
Kernel TLV
 
PDF
MySQL Advanced Administrator 2021 - 네오클로바
NeoClova
 
PDF
CMake best practices
Henry Schreiner
 
PPTX
Работа с индексами - лучшие практики для MySQL 5.6, Петр Зайцев (Percona)
Ontico
 
PDF
Troubleshooting MySQL Performance
Sveta Smirnova
 

More Related Content

What's hot (20)

PPSX
Oracle Performance Tuning Fundamentals
Carlos Sierra
 
PPTX
ProxySQL for MySQL
Mydbops
 
PDF
Secret Management with Hashicorp’s Vault
AWS Germany
 
PDF
Galera cluster for high availability
Mydbops
 
PDF
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)
Jean-François Gagné
 
PDF
MySQL 5.7 トラブルシューティング 性能解析入門編
Mikiya Okuno
 
PDF
Mongodb - Scaling write performance
Daum DNA
 
PDF
Hardening Kubernetes by Securing Pods
Suraj Deshmukh
 
PDF
Kubernetes security
Thomas Fricke
 
PDF
MariaDB 10.5 binary install (바이너리 설치)
NeoClova
 
PDF
PostgreSQL 15の新機能を徹底解説
Masahiko Sawada
 
PDF
MySQL Performance for DevOps
Sveta Smirnova
 
PDF
CentOS 8で標準搭載！ 「389-ds」で構築する 認証サーバーについて
Nobuyuki Sasaki
 
PPTX
MySQL_MariaDB로의_전환_기술요소-202212.pptx
NeoClova
 
PDF
MySQL Performance Tuning: Top 10 Tips
OSSCube
 
PDF
[Pgday.Seoul 2019] Citus를 이용한 분산 데이터베이스
PgDay.Seoul
 
DOCX
Keepalived+MaxScale+MariaDB_운영매뉴얼_1.0.docx
NeoClova
 
PDF
Advanced Namespaces and cgroups
Kernel TLV
 
PDF
MySQL Advanced Administrator 2021 - 네오클로바
NeoClova
 
PDF
CMake best practices
Henry Schreiner
 
Oracle Performance Tuning Fundamentals
Carlos Sierra
 
ProxySQL for MySQL
Mydbops
 
Secret Management with Hashicorp’s Vault
AWS Germany
 
Galera cluster for high availability
Mydbops
 
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)
Jean-François Gagné
 
MySQL 5.7 トラブルシューティング 性能解析入門編
Mikiya Okuno
 
Mongodb - Scaling write performance
Daum DNA
 
Hardening Kubernetes by Securing Pods
Suraj Deshmukh
 
Kubernetes security
Thomas Fricke
 
MariaDB 10.5 binary install (바이너리 설치)
NeoClova
 
PostgreSQL 15の新機能を徹底解説
Masahiko Sawada
 
MySQL Performance for DevOps
Sveta Smirnova
 
CentOS 8で標準搭載！ 「389-ds」で構築する 認証サーバーについて
Nobuyuki Sasaki
 
MySQL_MariaDB로의_전환_기술요소-202212.pptx
NeoClova
 
MySQL Performance Tuning: Top 10 Tips
OSSCube
 
[Pgday.Seoul 2019] Citus를 이용한 분산 데이터베이스
PgDay.Seoul
 
Keepalived+MaxScale+MariaDB_운영매뉴얼_1.0.docx
NeoClova
 
Advanced Namespaces and cgroups
Kernel TLV
 
MySQL Advanced Administrator 2021 - 네오클로바
NeoClova
 
CMake best practices
Henry Schreiner
 

Similar to Introduction into MySQL Query Tuning for Dev[Op]s (20)

PPTX
Работа с индексами - лучшие практики для MySQL 5.6, Петр Зайцев (Percona)
Ontico
 
PDF
Troubleshooting MySQL Performance
Sveta Smirnova
 
PDF
Introduction to MySQL Query Tuning for Dev[Op]s
Sveta Smirnova
 
PDF
Introduction into MySQL Query Tuning
Sveta Smirnova
 
PDF
Query Optimization with MySQL 5.6: Old and New Tricks
MYXPLAIN
 
PPTX
MySQL Indexing - Best practices for MySQL 5.6
MYXPLAIN
 
PPTX
MySQL performance tuning
Anurag Srivastava
 
PPTX
Alkin Tezuysal "MySQL Query Optimization Best Practices and Indexing"
Fwdays
 
PDF
Percona Live 2012PPT: MySQL Query optimization
mysqlops
 
PPTX
Mysql query optimization best practices and indexing
Alkin Tezuysal
 
PDF
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013
Jaime Crespo
 
PDF
MySQL Query Optimisation 101
Federico Razzoli
 
PDF
Scaling MySQL Strategies for Developers
Jonathan Levin
 
PPTX
Optimizing MySQL queries
GMO-Z.com Vietnam Lab Center
 
PDF
Quick Wins
HighLoad2009
 
PDF
Using MySQL in a web-scale environment
David Landgren
 
PDF
MySQL Database Monitoring: Must, Good and Nice to Have
Sveta Smirnova
 
PDF
Webinar 2013 advanced_query_tuning
晓 周
 
PDF
Practical my sql performance optimization
Marian Marinov
 
PDF
MySQL Query And Index Tuning
Manikanda kumar
 
Работа с индексами - лучшие практики для MySQL 5.6, Петр Зайцев (Percona)
Ontico
 
Troubleshooting MySQL Performance
Sveta Smirnova
 
Introduction to MySQL Query Tuning for Dev[Op]s
Sveta Smirnova
 
Introduction into MySQL Query Tuning
Sveta Smirnova
 
Query Optimization with MySQL 5.6: Old and New Tricks
MYXPLAIN
 
MySQL Indexing - Best practices for MySQL 5.6
MYXPLAIN
 
MySQL performance tuning
Anurag Srivastava
 
Alkin Tezuysal "MySQL Query Optimization Best Practices and Indexing"
Fwdays
 
Percona Live 2012PPT: MySQL Query optimization
mysqlops
 
Mysql query optimization best practices and indexing
Alkin Tezuysal
 
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013
Jaime Crespo
 
MySQL Query Optimisation 101
Federico Razzoli
 
Scaling MySQL Strategies for Developers
Jonathan Levin
 
Optimizing MySQL queries
GMO-Z.com Vietnam Lab Center
 
Quick Wins
HighLoad2009
 
Using MySQL in a web-scale environment
David Landgren
 
MySQL Database Monitoring: Must, Good and Nice to Have
Sveta Smirnova
 
Webinar 2013 advanced_query_tuning
晓 周
 
Practical my sql performance optimization
Marian Marinov
 
MySQL Query And Index Tuning
Manikanda kumar
 
Ad

More from Sveta Smirnova (20)

PDF
War Story: Removing Offensive Language from Percona Toolkit
Sveta Smirnova
 
PDF
MySQL 2024: Зачем переходить на MySQL 8, если в 5.х всё устраивает?
Sveta Smirnova
 
PDF
Database in Kubernetes: Diagnostics and Monitoring
Sveta Smirnova
 
PDF
MySQL Cookbook: Recipes for Developers
Sveta Smirnova
 
PDF
MySQL Test Framework для поддержки клиентов и верификации багов
Sveta Smirnova
 
PDF
MySQL Cookbook: Recipes for Your Business
Sveta Smirnova
 
PDF
Производительность MySQL для DevOps
Sveta Smirnova
 
PDF
MySQL Performance for DevOps
Sveta Smirnova
 
PDF
How to Avoid Pitfalls in Schema Upgrade with Percona XtraDB Cluster
Sveta Smirnova
 
PDF
How to migrate from MySQL to MariaDB without tears
Sveta Smirnova
 
PDF
Modern solutions for modern database load: improvements in the latest MariaDB...
Sveta Smirnova
 
PDF
How Safe is Asynchronous Master-Master Setup?
Sveta Smirnova
 
PDF
Современному хайлоду - современные решения: MySQL 8.0 и улучшения Percona
Sveta Smirnova
 
PDF
How to Avoid Pitfalls in Schema Upgrade with Galera
Sveta Smirnova
 
PDF
How Safe is Asynchronous Master-Master Setup?
Sveta Smirnova
 
PDF
Billion Goods in Few Categories: How Histograms Save a Life?
Sveta Smirnova
 
PDF
A Billion Goods in a Few Categories: When Optimizer Histograms Help and When ...
Sveta Smirnova
 
PDF
Что нужно знать о трёх топовых фичах MySQL
Sveta Smirnova
 
PDF
Billion Goods in Few Categories: How Histograms Save a Life?
Sveta Smirnova
 
PDF
MySQL Performance Schema in Action
Sveta Smirnova
 
War Story: Removing Offensive Language from Percona Toolkit
Sveta Smirnova
 
MySQL 2024: Зачем переходить на MySQL 8, если в 5.х всё устраивает?
Sveta Smirnova
 
Database in Kubernetes: Diagnostics and Monitoring
Sveta Smirnova
 
MySQL Cookbook: Recipes for Developers
Sveta Smirnova
 
MySQL Test Framework для поддержки клиентов и верификации багов
Sveta Smirnova
 
MySQL Cookbook: Recipes for Your Business
Sveta Smirnova
 
Производительность MySQL для DevOps
Sveta Smirnova
 
MySQL Performance for DevOps
Sveta Smirnova
 
How to Avoid Pitfalls in Schema Upgrade with Percona XtraDB Cluster
Sveta Smirnova
 
How to migrate from MySQL to MariaDB without tears
Sveta Smirnova
 
Modern solutions for modern database load: improvements in the latest MariaDB...
Sveta Smirnova
 
How Safe is Asynchronous Master-Master Setup?
Sveta Smirnova
 
Современному хайлоду - современные решения: MySQL 8.0 и улучшения Percona
Sveta Smirnova
 
How to Avoid Pitfalls in Schema Upgrade with Galera
Sveta Smirnova
 
How Safe is Asynchronous Master-Master Setup?
Sveta Smirnova
 
Billion Goods in Few Categories: How Histograms Save a Life?
Sveta Smirnova
 
A Billion Goods in a Few Categories: When Optimizer Histograms Help and When ...
Sveta Smirnova
 
Что нужно знать о трёх топовых фичах MySQL
Sveta Smirnova
 
Billion Goods in Few Categories: How Histograms Save a Life?
Sveta Smirnova
 
MySQL Performance Schema in Action
Sveta Smirnova
 
Ad

Recently uploaded (20)

PPTX
declaration of Variables and constants.pptx
meemee7378
 
PPTX
Test Case Design Techniques – Practical Examples & Best Practices in Software...
Muhammad Fahad Bashir
 
PPTX
Threat Modeling a Batch Job Framework - Teri Radichel - AWS re:Inforce 2025
2nd Sight Lab
 
PPTX
IDM Crack with Internet Download Manager 6.42 Build 41 [Latest 2025]
pcprocore
 
PDF
Humans vs AI Call Agents - Qcall.ai's Special Report
Udit Goenka
 
PPTX
Top Time Tracking Solutions for Accountants
oliviareed320
 
PDF
Best Practice for LLM Serving in the Cloud
Alluxio, Inc.
 
PDF
Automated Testing and Safety Analysis of Deep Neural Networks
Lionel Briand
 
DOCX
Zoho Creator Solution for EI by Elsner Technologies.docx
Elsner Technologies Pvt. Ltd.
 
DOCX
Enable Your Cloud Journey With Microsoft Trusted Partner | IFI Tech
IFI Techsolutions
 
DOCX
Best AI-Powered Wearable Tech for Remote Health Monitoring in 2025
SEOLIFT - SEO Company London
 
PPTX
Sap basis role in public cloud in s/4hana.pptx
htmlprogrammer987
 
PDF
Why Edge Computing Matters in Mobile Application Tech.pdf
IMG Global Infotech
 
PPTX
Simplify Insurance Regulations with Compliance Management Software
Insurance Tech Services
 
PDF
Building Geospatial Data Warehouse for GIS by GIS with FME
Safe Software
 
PPTX
ERP Systems in the UAE: Driving Business Transformation with Smart Solutions
dheeodoo
 
PDF
A Guide to Telemedicine Software Development.pdf
Olivero Bozzelli
 
PDF
Streamlining CI/CD with FME Flow: A Practical Guide
Safe Software
 
PDF
University Campus Navigation for All - Peak of Data & AI
Safe Software
 
PDF
On-Device AI: Is It Time to Go All-In, or Do We Still Need the Cloud?
Hassan Abid
 
declaration of Variables and constants.pptx
meemee7378
 
Test Case Design Techniques – Practical Examples & Best Practices in Software...
Muhammad Fahad Bashir
 
Threat Modeling a Batch Job Framework - Teri Radichel - AWS re:Inforce 2025
2nd Sight Lab
 
IDM Crack with Internet Download Manager 6.42 Build 41 [Latest 2025]
pcprocore
 
Humans vs AI Call Agents - Qcall.ai's Special Report
Udit Goenka
 
Top Time Tracking Solutions for Accountants
oliviareed320
 
Best Practice for LLM Serving in the Cloud
Alluxio, Inc.
 
Automated Testing and Safety Analysis of Deep Neural Networks
Lionel Briand
 
Zoho Creator Solution for EI by Elsner Technologies.docx
Elsner Technologies Pvt. Ltd.
 
Enable Your Cloud Journey With Microsoft Trusted Partner | IFI Tech
IFI Techsolutions
 
Best AI-Powered Wearable Tech for Remote Health Monitoring in 2025
SEOLIFT - SEO Company London
 
Sap basis role in public cloud in s/4hana.pptx
htmlprogrammer987
 
Why Edge Computing Matters in Mobile Application Tech.pdf
IMG Global Infotech
 
Simplify Insurance Regulations with Compliance Management Software
Insurance Tech Services
 
Building Geospatial Data Warehouse for GIS by GIS with FME
Safe Software
 
ERP Systems in the UAE: Driving Business Transformation with Smart Solutions
dheeodoo
 
A Guide to Telemedicine Software Development.pdf
Olivero Bozzelli
 
Streamlining CI/CD with FME Flow: A Practical Guide
Safe Software
 
University Campus Navigation for All - Peak of Data & AI
Safe Software
 
On-Device AI: Is It Time to Go All-In, or Do We Still Need the Cloud?
Hassan Abid
 

Introduction into MySQL Query Tuning for Dev[Op]s

  • 1. MySQL Query Tuning for Dev[Op]s. Introduction
  • 2. •Basics •Indexes How Do They Work? When MySQL Uses Indexes Optimizer Histograms •Diagnostics EXPLAIN: How Optimizer Works Real Numbers: Inside Storage Engine Real Numbers: Inside the Server •How to Affect Query Plans Table of Contents 2 ©2021 Percona
  • 3. • Sveta Smirnova • MySQL Support engineer • Author of • MySQL Troubleshooting • JSON UDF functions • FILTER clause for MySQL • Speaker • Percona Live, OOW, Fosdem, DevConf, HighLoad... 3 ©2021 Percona
  • 9. cursor = conn.cursor() q = ’’’UPDATE ‘foo‘ SET my_date=NOW(), subject = %s, msg = %s, address = %s, updated_at = NOW() WHERE id=%s ’’’ cursor.execute(q, [ remote_resp.get(’subject’), remote_resp.get(’msg’), remote_resp.get(’address’), my_id ]) The Query 7 ©2021 Percona
  • 10. cursor = conn.cursor() q = ’’’UPDATE ‘foo‘ SET my_date=NOW(), subject = %s, msg = %s, address = %s, updated_at = NOW() WHERE id=%s ’’’ cursor.execute(q, [ remote_resp.get(’subject’), remote_resp.get(’msg’), remote_resp.get(’address’), my_id ]) The Query 7 ©2021 Percona
  • 11. SELECT dept_name, title, gender, min(salary) AS mins, max(salary) AS maxs FROM employees JOIN salaries USING(emp_no) JOIN titles USING(emp_no) JOIN dept_emp USING(emp_no) JOIN departments USING(dept_no) JOIN dept_manager USING(dept_no) WHERE dept_manager.to_date = ’9999-01-01’ GROUP BY dept_name, title, gender ORDER BY gender, maxs DESC; The Query 7 ©2021 Percona
  • 12. SELECT dept_name, title, gender, min(salary) AS mins, max(salary) AS maxs FROM employees JOIN salaries USING(emp_no) JOIN titles USING(emp_no) JOIN dept_emp USING(emp_no) JOIN departments USING(dept_no) JOIN dept_manager USING(dept_no) WHERE dept_manager.to_date = ’9999-01-01’ GROUP BY dept_name, title, gender ORDER BY gender, maxs DESC; The Query 7 ©2021 Percona
  • 13. • PMM QAN Always Tune Raw Query 8 ©2021 Percona
  • 14. • PMM QAN • Slow Query Log Always Tune Raw Query 8 ©2021 Percona
  • 15. • PMM QAN • Slow Query Log • Application log Always Tune Raw Query 8 ©2021 Percona
  • 16. • PMM QAN • Slow Query Log • Application log • ... Always Tune Raw Query 8 ©2021 Percona
  • 17. • Mind your data! • 75,000,000 rows • (INT, INT) 75,000,000 * (4 + 4) = 600,000,000 B = 572 MB • (INT, INT, DATETIME, VARCHAR(255), VARCHAR(255)) 75,000,000 * (4 + 4 + 8 + 256 + 256) = 39,600,000,000 bytes = 37 G • 39,600,000,000 / 600,000,000 = 66 Slow is Relative 9 ©2021 Percona
  • 18. • Mind your data! • Mind use case • Popular website • Admin interface • Weekly cron job Slow is Relative 9 ©2021 Percona
  • 19. • Mind your data! • Mind use case • Mind location • Server, used by multiple connections • Dedicated for OLAP queries Slow is Relative 9 ©2021 Percona
  • 20. • MySQL performs a job to execute it Why is the Query Slow 10 ©2021 Percona
  • 21. • MySQL performs a job to execute it • In the worst case scenario it will do a full table scan • CREATE INDEX • ANALYZE TABLE ... UPDATE HISTOGRAM ON ... Why is the Query Slow 10 ©2021 Percona
  • 22. • MySQL performs a job to execute it • In the worst case scenario it will do a full table scan • CREATE INDEX • ANALYZE TABLE ... UPDATE HISTOGRAM ON ... • Incorrect index can be used Why is the Query Slow 10 ©2021 Percona
  • 23. Query sent Connection Pool: Authentication, Caches; SQL interface; Parser Optimizer Storage engines Hardware Query Execution Workflow 11 ©2021 Percona
  • 24. • Selecting a lot of data • SELECT * FROM many columns table You May not Be Able to Change a Slow Query 12 ©2021 Percona
  • 25. • Selecting a lot of data • SELECT * FROM many columns table • Badly written • LEFT JOIN instead of INNER JOIN • Many values in IN() • Retrieving large result set, then discarding • Not effective SQL For particular MySQL version You May not Be Able to Change a Slow Query 12 ©2021 Percona
  • 26. • Selecting a lot of data • SELECT * FROM many columns table • Badly written • LEFT JOIN instead of INNER JOIN • Many values in IN() • Retrieving large result set, then discarding • Not effective SQL For particular MySQL version • We can still improve performance You May not Be Able to Change a Slow Query 12 ©2021 Percona
  • 29. SELECT name FROM users WHERE id=12 1 2 5 6 7 9 12 16 18 21 Full Table Scan 15 ©2021 Percona
  • 30. SELECT name FROM users WHERE id=12 1 2 5 6 7 9 12 16 18 21 Index Access 16 ©2021 Percona
  • 32. • Single column CREATE INDEX index name ON the table(the column) • Multiple columns CREATE INDEX index name ON the table(column1, column2) How to Create an Index 18 ©2021 Percona
  • 33. • Single column ALTER TABLE table name ADD INDEX [index name] (the column) • Multiple columns ALTER TABLE table name ADD INDEX [index name] (column1, column2) How to Create an Index 18 ©2021 Percona
  • 35. • WHERE the column = a value • WHERE the column IN(value1, value2, value3) • WHERE the column LIKE ’value%’ • WHERE the column LIKE ’%value’ Conditions 20 ©2021 Percona
  • 36. • WHERE left part = value1 AND right part = value2 • WHERE left part = value1 OR right part = value2 • WHERE right part = value1 AND left part = value2 • WHERE right part = value1 OR left part = value2 Conditions 20 ©2021 Percona
  • 37. • table1 JOIN table2 ON table1.column1 = table2.column2 Joins 21 ©2021 Percona
  • 38. • table1 JOIN table2 ON table1.column1 = table2.column2 • Same as FROM table1, table2 WHERE table1.column1 = table2.column2 Joins 21 ©2021 Percona
  • 39. • GROUP BY the column • GROUP BY left part, right part • GROUP BY right part, left part • GROUP BY the index, another index GROUP BY 22 ©2021 Percona
  • 40. • ORDER BY the column • ORDER BY left part, right part • ORDER BY right part, left part • ORDER BY the index, another index ORDER BY 23 ©2021 Percona
  • 41. 5.7 ORDER BY left part DESC, right part ASC 8.0 ORDER BY left part DESC, right part ASC • left part must be descending • right part must be ascending • the index(left part DESC, right part ASC) ORDER BY 23 ©2021 Percona
  • 42. • Deterministic, built-in • Return same value for the same argument • WHERE the column = FLOOR(123.45) Expressions 24 ©2021 Percona
  • 43. • Deterministic, built-in • Return same value for the same argument • WHERE the column = FLOOR(123.45) • Non-deterministic • Return different values for different invocations • WHERE the column = RAND() ∗ 100 Expressions 24 ©2021 Percona
  • 44. • Deterministic, built-in • Return same value for the same argument • WHERE the column = FLOOR(123.45) • Non-deterministic • Return different values for different invocations • WHERE the column = RAND() ∗ 100 • Stored functions and UDFs • Indexes are not used Use generated column indexes Expressions 24 ©2021 Percona
  • 45. • Identify queries not using indexes • Status variables mysql select * from performance_schema.session_status - where variable_name in (’Created_tmp_tables’, - ’Created_tmp_disk_tables’, ’Select_full_join’, - ’Select_full_range_join’, ’Select_range’, - ’Select_range_check’, ’Select_scan’, - ’Sort_merge_passes’, ’Sort_range’, ’Sort_rows’, - ’Sort_scan’) and variable_value 0; +------------------------+----------------+ | VARIABLE_NAME | VARIABLE_VALUE | +------------------------+----------------+ | Select_scan | 2 | +------------------------+----------------+ 1 row in set (0,00 sec) When to Add Indexes? 25 ©2021 Percona
  • 46. • Identify queries not using indexes mysql show global status like ’Handler_read%’; +-----------------------+-------+ | Variable_name | Value | +-----------------------+-------+ | Handler_read_first | 31 | | Handler_read_key | 1909 | | Handler_read_last | 0 | | Handler_read_next | 4393 | | Handler_read_prev | 0 | | Handler_read_rnd | 0 | | Handler_read_rnd_next | 1135 | +-----------------------+-------+ When to Add Indexes? 25 ©2021 Percona
  • 47. • Identify queries not using indexes • PERFORMANCE SCHEMA.EVENTS STATEMENTS * mysql select * from events_statements_history_long - where sql_text like ’select count(*) from employees join %’G *************************** 1. row **************************** ... ROWS_SENT: 1 SELECT_RANGE_CHECK: 0 ROWS_EXAMINED: 541058 SELECT_SCAN: 1 CREATED_TMP_DISK_TABLES: 0 SORT_MERGE_PASSES: 0 CREATED_TMP_TABLES: 0 SORT_RANGE: 0 SELECT_FULL_JOIN: 0 SORT_ROWS: 0 SELECT_FULL_RANGE_JOIN: 0 SORT_SCAN: 0 SELECT_RANGE: 0 NO_INDEX_USED: 0 When to Add Indexes? 25 ©2021 Percona
  • 48. • Identify queries not using indexes • Slow query log # Time: 2020-10-11T23:34:03.701871Z # User@Host: root[root] @ localhost [127.0.0.1] Id: 506 # Query_time: 0.024106 Lock_time: 0.000091 Rows_sent: 1 Rows_examined: 1000 SET timestamp=1602459243; SELECT c FROM sbtest1 WHERE id=996; When to Add Indexes? 25 ©2021 Percona
  • 49. • Identify queries not using indexes • QAN in PMM When to Add Indexes? 25 ©2021 Percona
  • 50. • Identify queries not using indexes • Analyze if adding an index will help When to Add Indexes? 25 ©2021 Percona
  • 51. • Identify queries not using indexes • Analyze if adding an index will help • Add the index When to Add Indexes? 25 ©2021 Percona
  • 52. • Identify queries not using indexes • Analyze if adding an index will help • Add the index • Test first! • ADD INDEX is an expensive operation Use pt-online-schema-change When to Add Indexes? 25 ©2021 Percona
  • 54. • Since version 8.0 Histogram Statistics 27 ©2021 Percona
  • 55. • Since version 8.0 • Collected and used by the Optimizer Histogram Statistics 27 ©2021 Percona
  • 56. • Since version 8.0 • Collected and used by the Optimizer • Visible in Information Schema mysql select HISTOGRAM from information_schema.column_statistics - where table_name=’example’G *************************** 1. row *************************** HISTOGRAM: {buckets: [[1, 0.6], [2, 0.8], [3, 1.0]], data-type: int, null-values: 0.0, collation-id: 8, last-updated: 2018-11-07 09:07:19.791470, sampling-rate: 1.0, histogram-type: singleton, number-of-buckets-specified: 3} 1 row in set (0.00 sec) Histogram Statistics 27 ©2021 Percona
  • 57. • Since version 8.0 • Collected and used by the Optimizer • Visible in Information Schema • Affects query execution plan, but not access • SELECT ... FROM a JOIN b vs SELECT ... FROM b JOIN a Histogram Statistics 27 ©2021 Percona
  • 58. • Since version 8.0 • Collected and used by the Optimizer • Visible in Information Schema • Affects query execution plan, but not access More details Histogram Statistics 27 ©2021 Percona
  • 59. 1 2 3 4 5 6 7 8 9 10 0 200 400 600 800 Indexes: Number of Items with Same Value 28 ©2021 Percona
  • 60. 1 2 3 4 5 6 7 8 9 10 0 200 400 600 800 Indexes: Cardinality 29 ©2021 Percona
  • 61. 1 2 3 4 5 6 7 8 9 10 0 200 400 600 800 Histograms: Number of Values in Each Bucket 30 ©2021 Percona
  • 62. 1 2 3 4 5 6 7 8 9 10 0 0.2 0.4 0.6 0.8 1 Histograms: Data in the Histogram 31 ©2021 Percona
  • 63. • Data distribution is far from uniform • Query accesses two or more tables • Indexes cannot be used • Maintaining them is too expensive • They are already used for different conditions Create Histograms 32 ©2021 Percona
  • 66. • EXPLAIN • Estimates the query execution plan 5.6- EXTENDED 5.6- PARTITIONS 5.6+ FORMAT=JSON 8.0+ FORMAT=TREE How to Find how MySQL Uses Indexes 35 ©2021 Percona
  • 67. • EXPLAIN • Estimates the query execution plan 5.6- EXTENDED 5.6- PARTITIONS 5.6+ FORMAT=JSON 8.0+ FORMAT=TREE • INFORMATION SCHEMA.OPTIMIZER TRACE • Real data, collected when query was executing • Advanced topic How to Find how MySQL Uses Indexes 35 ©2021 Percona
  • 68. mysql EXPLAIN SELECT first_name, last_name, title, salary FROM employees - JOIN salaries USING(emp_no) JOIN titles USING(emp_no) - WHERE salary = (SELECT MAX(salary) FROM salaries - JOIN titles WHERE titles.to_date CURDATE()); +----+-------------+-----------+------------+------+---------------+---------+---------+ | id | select_type | table | partitions | type | possible_keys | key | key_len | +----+-------------+-----------+------------+------+---------------+---------+---------+ | 1 | PRIMARY | employees | NULL | ALL | PRIMARY | NULL | NULL ... | 1 | PRIMARY | salaries | NULL | ref | PRIMARY | PRIMARY | 4 ... | 1 | PRIMARY | titles | NULL | ref | PRIMARY | PRIMARY | 4 ... | 2 | SUBQUERY | titles | NULL | ALL | NULL | NULL | NULL ... | 2 | SUBQUERY | salaries | NULL | ALL | NULL | NULL | NULL ... +----+-------------+-----------+------------+------+---------------+---------+---------+ +----+-...-+----------------------------+---------+----------+-------------------------------+ | id | ... | ref | rows | filtered | Extra | +----+-...-+----------------------------+---------+----------+-------------------------------+ | 1 | ... | NULL | 299113 | 100.00 | NULL | | 1 | ... | employees.employees.emp_no | 9 | 10.00 | Using where | | 1 | ... | employees.employees.emp_no | 1 | 100.00 | Using index | | 2 | ... | NULL | 442189 | 33.33 | Using where | | 2 | ... | NULL | 2838426 | 100.00 | Using join buffer (hash join) | +----+-...-+----------------------------+---------+----------+-------------------------------+ 5 rows in set, 1 warning (0,01 sec) 299113 * 9 * 1 * 442189 * 2838426 = 3,378,806,408,204,514,738 EXPLAIN: rows 36 ©2021 Percona
  • 69. mysql EXPLAIN SELECT title, MAX(salary) AS maxs FROM employees - JOIN salaries USING(emp_no) JOIN titles USING(emp_no) - GROUP BY title ORDER BY maxs DESC; +----+-------------+-----------+------------+-------+---------------+---------+---------+ | id | select_type | table | partitions | type | possible_keys | key | key_len +----+-------------+-----------+------------+-------+---------------+---------+---------+ | 1 | SIMPLE | employees | NULL | index | PRIMARY | PRIMARY | 4 ... | 1 | SIMPLE | titles | NULL | ref | PRIMARY,title | PRIMARY | 4 ... | 1 | SIMPLE | salaries | NULL | ref | PRIMARY | PRIMARY | 4 ... +----+-------------+-----------+------------+-------+---------------+---------+---------+ EXPLAIN: type 37 ©2021 Percona
  • 70. ALL Read all rows Query Execution Type 38 ©2021 Percona
  • 71. ALL Read all rows index Read all rows from the index Query Execution Type 38 ©2021 Percona
  • 72. ALL Read all rows index Read all rows from the index range Range: N ... M Query Execution Type 38 ©2021 Percona
  • 73. ALL Read all rows index Read all rows from the index range Range: N ... M subquery Subqueries index subquery unique subquery Query Execution Type 38 ©2021 Percona
  • 74. ALL Read all rows index Read all rows from the index range Range: N ... M subquery Subqueries merge Combination of two indexes index merge Query Execution Type 38 ©2021 Percona
  • 75. ALL Read all rows index Read all rows from the index range Range: N ... M subquery Subqueries merge Combination of two indexes ref Comparision with single value ref or null ref eq ref Query Execution Type 38 ©2021 Percona
  • 76. ALL Read all rows index Read all rows from the index range Range: N ... M subquery Subqueries merge Combination of two indexes ref Comparision with single value fulltext Fulltext index Query Execution Type 38 ©2021 Percona
  • 77. ALL Read all rows index Read all rows from the index range Range: N ... M subquery Subqueries merge Combination of two indexes ref Comparision with single value fulltext Fulltext index const Single row Query Execution Type 38 ©2021 Percona
  • 78. ALL Read all rows index Read all rows from the index range Range: N ... M subquery Subqueries merge Combination of two indexes ref Comparision with single value fulltext Fulltext index const Single row system Table has single row Query Execution Type 38 ©2021 Percona
  • 79. mysql EXPLAIN SELECT title, MAX(salary) AS maxs FROM employees - JOIN salaries USING(emp_no) JOIN titles USING(emp_no) - GROUP BY title ORDER BY maxs DESC; +----+-------------+-----------+------------+-------+---------------+---------+---------+ | id | select_type | table | partitions | type | possible_keys | key | key_len +----+-------------+-----------+------------+-------+---------------+---------+---------+ | 1 | SIMPLE | employees | NULL | index | PRIMARY | PRIMARY | 4 ... | 1 | SIMPLE | titles | NULL | ref | PRIMARY,title | PRIMARY | 4 ... | 1 | SIMPLE | salaries | NULL | ref | PRIMARY | PRIMARY | 4 ... +----+-------------+-----------+------------+-------+---------------+---------+---------+ EXPLAIN: key and possible keys 39 ©2021 Percona
  • 80. mysql EXPLAIN SELECT title, MAX(salary) AS maxs FROM employees - JOIN salaries USING(emp_no) - JOIN titles USING(emp_no) - GROUP BY title ORDER BY maxs DESC; +----+-------------+-----------+------------+-------+---------------+---------+---------+ | id | select_type | table | partitions | type | possible_keys | key | key_len +----+-------------+-----------+------------+-------+---------------+---------+---------+ | 1 | SIMPLE | employees | NULL | index | PRIMARY | PRIMARY | 4 ... | 1 | SIMPLE | titles | NULL | ref | PRIMARY,title | PRIMARY | 4 ... | 1 | SIMPLE | salaries | NULL | ref | PRIMARY | PRIMARY | 4 ... +----+-------------+-----------+------------+-------+---------------+---------+---------+ EXPLAIN: JOIN order 40 ©2021 Percona
  • 81. mysql EXPLAIN SELECT first_name, last_name, title, salary FROM employees - JOIN salaries USING(emp_no) JOIN titles USING(emp_no) - WHERE salary IN (SELECT MAX(salary) FROM salaries - JOIN titles WHERE titles.to_date CURDATE()); ... 5 rows in set, 1 warning (0,01 sec) mysql SHOW WARNINGSG *************************** 1. row *************************** Level: Note Code: 1003 Message: /* select#1 */ select ‘employees‘.‘employees‘.‘first_name‘ AS ‘first_name‘, ‘employees‘.‘employees‘.‘last_name‘ AS ‘last_name‘,‘employees‘.‘salaries‘.‘salary‘ AS ‘salary‘ from ‘employees‘.‘employees‘ join ‘employees‘.‘salaries‘ where ((‘employees‘.‘salaries‘.‘emp_no‘ = ‘employees‘.‘employees‘.‘emp_no‘) and in_optimizer(‘employees‘.‘salaries‘.‘salary‘,‘employees‘.‘salaries‘.‘salary‘ in ( materialize (/* select#2 */ select max(‘employees‘.‘salaries‘.‘salary‘) from ‘employees‘.‘salaries‘ join ‘employees‘.‘titles‘ where (‘employees‘.‘titles‘.‘to_date‘ cache(curdate())) having true ), primary_index_lookup(‘employees‘.‘salaries‘.‘salary‘ in temporary table on auto_distinct_key where ((‘employees‘.‘salaries‘.‘salary‘ = ‘materialized_subquery‘.‘MAX(salary)‘)))))) 1 row in set (0,00 sec) Actual Query 41 ©2021 Percona
  • 82. mysql explain select * from t1G *************************** 1. row *************************** ... rows: 12 Extra: NULL mysql explain select * from t1 where f2=12G *************************** 1. row *************************** ... key: NULL ... rows: 12 Extra: Using where Effect of Indexes: Before 42 ©2021 Percona
  • 83. mysql alter table t1 add index(f2); Query OK, 12 rows affected (0.07 sec) Records: 12 Duplicates: 0 Warnings: 0 mysql explain select * from t1 where f2=12G *************************** 1. row *************************** ... key: f2 key_len: 5 ref: const rows: 1 Extra: NULL 1 row in set (0.00 sec) Effect of Indexes: After 43 ©2021 Percona
  • 85. • EXPLAIN is optimistic mysql explain select * from ol - where thread_id=10432 and site_id != 9939 - order by id limit 3G *************************** 1. row *************************** id: 1 | ref: NULL select_type: SIMPLE | rows: 33 table: ol | filtered: 8.07 partitions: NULL | Extra: Using where type: index possible_keys: thread_id key: PRIMARY key_len: 4 1 row in set, 1 warning (0,00 sec) Handler * Status Variables 45 ©2021 Percona
  • 86. • ’Handler *’ show truth mysql FLUSH STATUS; mysql select * from ol - where thread_id=10432 and site_id != 9939 - order by id limit 3; mysql show status like ’Handler%’; +----------------------------+--------+ | Variable_name | Value | +----------------------------+--------+ ... | Handler_read_first | 1 | | Handler_read_key | 1 | | Handler_read_last | 0 | | Handler_read_next | 100000 | ... Handler * Status Variables 45 ©2021 Percona
  • 88. • SHOW [FULL] PROCESSLIST • INFORMATION SCHEMA.PROCESSLIST • performance schema.THREADS PROCESSLIST 47 ©2021 Percona
  • 89. • SHOW [FULL] PROCESSLIST • INFORMATION SCHEMA.PROCESSLIST • performance schema.THREADS • Your first alert of performance issue PROCESSLIST 47 ©2021 Percona
  • 90. • SHOW [FULL] PROCESSLIST • INFORMATION SCHEMA.PROCESSLIST • performance schema.THREADS • Your first alert of performance issue • Shows all running queries PROCESSLIST 47 ©2021 Percona
  • 91. • Can be seen in the PROCESSLIST mysql show processlistG ************************ 1. row ************************ Id: 7 User: root Host: localhost:48799 db: employees Command: Query Time: 2 State: Sending data Info: select count(*) from employees join titles using(emp_no) where title=’Senior Engineer’ ... Execution Stages 48 ©2021 Percona
  • 92. • Can be seen in the PROCESSLIST • Very useful when you need to answer on the question: ”What is my server doing now?” Execution Stages 48 ©2021 Percona
  • 93. • PERFORMANCE SCHEMA.EVENTS STAGES * mysql select eshl.event_name, substr(sql_text, 1, 15) ‘sql‘, - eshl.timer_wait/1000000000000 w_s - from events_stages_history_long - eshl join events_statements_history_long esthl on - (eshl.nesting_event_id = esthl.event_id) where - esthl.current_schema=’employees’ and sql_text like - ’select count(*) from employees%’ - order by eshl.timer_start asc; +--------------------------------+-----------------+--------+ | event_name | sql | w_s | +--------------------------------+-----------------+--------+ | stage/sql/starting | select count(*) | 0.0002 | | stage/sql/checking permissions | select count(*) | 0.0000 | ... Execution Stages 48 ©2021 Percona
  • 94. • PERFORMANCE SCHEMA.EVENTS STAGES * ... | stage/sql/checking permissions | select count(*) | 0.0000 | | stage/sql/Opening tables | select count(*) | 0.0000 | | stage/sql/init | select count(*) | 0.0001 | | stage/sql/System lock | select count(*) | 0.0000 | | stage/sql/optimizing | select count(*) | 0.0000 | | stage/sql/statistics | select count(*) | 0.0001 | | stage/sql/preparing | select count(*) | 0.0000 | | stage/sql/executing | select count(*) | 0.0000 | | stage/sql/Sending data | select count(*) | 5.4915 | | stage/sql/end | select count(*) | 0.0000 | ... Execution Stages 48 ©2021 Percona
  • 95. • Status variables mysql flush status; Query OK, 0 rows affected (0,01 sec) mysql select count(*) from employees join titles using(emp_no) - where title=’Senior Engineer’; +----------+ | count(*) | +----------+ | 97750 | +----------+ 1 row in set (5,44 sec) Temporary Tables and Other Job 49 ©2021 Percona
  • 96. • Status variables mysql select * from performance_schema.session_status - where variable_name in (’Created_tmp_tables’, - ’Created_tmp_disk_tables’, ’Select_full_join’, - ’Select_full_range_join’, ’Select_range’, - ’Select_range_check’, ’Select_scan’, ’Sort_merge_passes’, - ’Sort_range’, ’Sort_rows’, ’Sort_scan’) - and variable_value 0; +------------------------+----------------+ | VARIABLE_NAME | VARIABLE_VALUE | +------------------------+----------------+ | Select_scan | 2 | +------------------------+----------------+ 1 row in set (0,00 sec) Temporary Tables and Other Job 49 ©2021 Percona
  • 97. • PERFORMANCE SCHEMA.EVENTS STATEMENTS * mysql select * from events_statements_history_long - where sql_text like - ’select count(*) from employees join %’G *************************** 1. row **************************** ... ROWS_SENT: 1 SELECT_RANGE_CHECK: 0 ROWS_EXAMINED: 541058 SELECT_SCAN: 1 CREATED_TMP_DISK_TABLES: 0 SORT_MERGE_PASSES: 0 CREATED_TMP_TABLES: 0 SORT_RANGE: 0 SELECT_FULL_JOIN: 0 SORT_ROWS: 0 SELECT_FULL_RANGE_JOIN: 0 SORT_SCAN: 0 SELECT_RANGE: 0 NO_INDEX_USED: 0 Temporary Tables and Other Job 49 ©2021 Percona
  • 98. • sys.statement analysis mysql select * from statement_analysis where query like - ’SELECT COUNT ( * ) FROM ‘emplo%’ and db=’employees’G *************************** 1. row *************************** query: SELECT COUNT ( * ) FROM ‘emplo ... ‘emp_no‘ ) WHE... db: employees max_latency: 5.59 s full_scan: avg_latency: 5.41 s exec_count: 7 lock_latency: 2.24 ms err_count: 0 rows_sent: 7 warn_count: 0 rows_sent_avg: 1 total_latency: 37.89 s rows_examined: 3787406 Temporary Tables and Other Job 49 ©2021 Percona
  • 99. • sys.statement analysis rows_examined_avg: 541058 rows_affected: 0 rows_affected_avg: 0 tmp_tables: 0 tmp_disk_tables: 0 rows_sorted: 0 sort_merge_passes: 0 digest: 4086bc3dc6510a1d9c8f2fe1f59f0943 first_seen: 2016-04-14 15:19:19 last_seen: 2016-04-14 16:13:14 Temporary Tables and Other Job 49 ©2021 Percona
  • 100. How to Affect Query Plans
  • 101. • Index statistics • Histogram statistics • Optimizer switches • Bugs in optimizer What has an Effect on Query Optimizer Plans? 51 ©2021 Percona
  • 102. • Collected by storage engine Index Statistics 52 ©2021 Percona
  • 103. • Collected by storage engine • Used by Optimizer Index Statistics 52 ©2021 Percona
  • 104. • Can be examined by SHOW INDEX mysql show index from sbtest1; +---------+----------+-------------+-------------+ | Table | Key_name | Column_name | Cardinality | +---------+----------+-------------+-------------+ | sbtest1 | k_1 | k | 49142 | +---------+----------+-------------+-------------+ mysql select count(distinct id), - count(distinct k) from sbtest1; +--------------------+-------------------+ | count(distinct id) | count(distinct k) | +--------------------+-------------------+ | 100000 | 17598 | +--------------------+-------------------+ Index Statistics 52 ©2021 Percona
  • 105. • Can be updated • ANALYZE TABLE • If does not help: rebuild the table OPTIMIZE TABLE ALTER TABLE ENGINE=INNODB; ANALYZE TABLE Index Statistics 52 ©2021 Percona
  • 106. • Since version 8.0 Histogram Statistics 53 ©2021 Percona
  • 107. • Since version 8.0 • Collected and used by the Optimizer Histogram Statistics 53 ©2021 Percona
  • 108. • Since version 8.0 • Collected and used by the Optimizer • Visible in Information Schema mysql select HISTOGRAM from information_schema.column_statistics - where table_name=’example’G *************************** 1. row *************************** HISTOGRAM: {buckets: [[1, 0.6], [2, 0.8], [3, 1.0]], data-type: int, null-values: 0.0, collation-id: 8, last-updated: 2018-11-07 09:07:19.791470, sampling-rate: 1.0, histogram-type: singleton, number-of-buckets-specified: 3} 1 row in set (0.00 sec) Histogram Statistics 53 ©2021 Percona
  • 109. • Since version 8.0 • Collected and used by the Optimizer • Visible in Information Schema More details Histogram Statistics 53 ©2021 Percona
  • 110. mysql select @@optimizer_switchG *************************** 1. row *************************** @@optimizer_switch: index_merge=on,index_merge_union=on, index_merge_sort_union=on,index_merge_intersection=on, engine_condition_pushdown=on,index_condition_pushdown=on, mrr=on,mrr_cost_based=on, block_nested_loop=on,batched_key_access=off, materialization=on,semijoin=on,loosescan=on,firstmatch=on, duplicateweedout=on,subquery_materialization_cost_based=on, use_index_extensions=on,condition_fanout_filter=on, derived_merge=on 1 row in set (0,00 sec) Optimizer Switches 54 ©2021 Percona
  • 111. • Turn ON and OFF particular optimization Optimizer Switches 54 ©2021 Percona
  • 112. • Turn ON and OFF particular optimization • Can be unhelpful • Especially for queries, tuned for previous versions Optimizer Switches 54 ©2021 Percona
  • 113. • Turn ON and OFF particular optimization • Can be unhelpful • Work with them as with any other option • Turn OFF and try SET optimizer_switch = ’use_index_extensions=off’; SELECT ... EXPLAIN SELECT ... Optimizer Switches 54 ©2021 Percona
  • 114. • Turn ON and OFF particular optimization • Can be unhelpful • Work with them as with any other option • If helps implement in queries SELECT /*+ SEMIJOIN(FIRSTMATCH, LOOSESCAN) */ * FROM t1 ...; SELECT /*+ BKA(t1) NO_BKA(t2) */ * FROM t1 INNER JOIN t2 WHERE ...; Optimizer Switches 54 ©2021 Percona
  • 115. • Optimizer choses wrong plan for no reason Bugs in Optimizer 55 ©2021 Percona
  • 116. • Optimizer choses wrong plan for no reason • Statistics are up to date • Histograms are not usable Bugs in Optimizer 55 ©2021 Percona
  • 117. • Optimizer choses wrong plan for no reason • Statistics are up to date • Histograms are not usable • Solution • Use index hints FORCE INDEX IGNORE INDEX Bugs in Optimizer 55 ©2021 Percona
  • 118. • Optimizer choses wrong plan for no reason • Statistics are up to date • Histograms are not usable • Solution • On every upgrade • Remove index hints • Test if the query improved • You must do it even for minor version upgrades! Bugs in Optimizer 55 ©2021 Percona
  • 119. • EXPLAIN is essential for query tuning • Real job is done by storage engine • Index statistics affect query execution plan • All index hints, optimizer hints and other workarounds must be validated on each upgrade Summary 56 ©2021 Percona
  • 120. EXPLAIN Syntax EXPLAIN FORMAT=JSON is Cool! Troubleshooting add-ons Optimizer Statistics aka Histograms Optimizer Hints Tracing the Optimizer More Information 57 ©2021 Percona