SlideShare a Scribd company logo

Advanced query optimization

MYXPLAIN, profile picture
MYXPLAIN

The document summarizes several case studies on optimizing large SQL queries. Case study 1 examines a query that runs slowly due to a column used as a bitmap being compared with a bitwise operator rather than being indexed directly. Case study 2 shows a query that could use indexes but data type mismatches prevent it. Case study 3 involves a dependent subquery that is rewritten as a left join to better use indexes.

Technology
1 of 43
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
Advanced query optimization techniques on large queries Peter Boros Percona Webinar
Agenda • Showing the title slide • Going through this agenda • Prerequisites • Case study #1 • Case study #2 • Case study #3 • Case study #4 www.percona.com
This talk is not about ● SQL basics ● Reading explain ● Indexing basics www.percona.com
Case Study #1
#1: Query ● Query runs in the SELECT t.id, ts.meta1, lt.meta2 FROM things t 10 seconds range INNER JOIN things_stuff ts ON t.fk_things_stuff_id = ts.id INNER JOIN yet_an_other_table yat ● According to ON t.fk_yat_id = yat.id WHERE t.active = 1 explain output it AND t.type != 'guest' AND t.title like '%the%' AND LOWER(t.username) like '%e%' can use some AND t.grants & 1 != 0 AND t.attribute1=3; indexes ● Mostly single- column indexes were defined www.percona.com
#1: Query ● Rows examined in SELECT t.id, ts.meta1, lt.meta2 FROM things t the 100k range INNER JOIN things_stuff ts ON t.fk_things_stuff_id = ts.id INNER JOIN yet_an_other_table yat ● Rows returned is ON t.fk_yat_id = yat.id WHERE t.active = 1 single digit AND t.type != 'guest' AND t.title like '%the%' AND LOWER(t.username) like '%e%' ● This means AND t.grants & 1 != 0 AND t.attribute1=3; something is selective, but most likely not indexed www.percona.com
#1: issue identification ********************** 1. row ********************** id: 1 select_type: SIMPLE table: t This is the issue, the rest type: const of the tables are just some key: idx_active metadata fetching (type is eq_ref) key_len: 4 rows: 332873 Extra: Using where ********************** 2. row ********************** id: 1 select_type: SIMPLE table: ts type: eq_ref key: PRIMARY key_len: 4 rows: 1 Extra: ********************** 3. row ***************** type: eq_ref rows: 1 www.percona.com
#1: finding the filtering clause ● In the original query, filtering in the WHERE clause are all for table t (things) ● Find out if any of them are selective alone (it's possible that they are not, but some combination of them will be selective) SELECT COUNT(t.id) FROM things t WHERE t.active=1; +----------+ | COUNT(1) | +----------+ | 168520 | +----------+ 1 row in set (10.15 sec) www.percona.com
#1: finding the filtering clause SELECT COUNT(t.id) FROM things t WHERE t.status!='guest'; +----------+ | COUNT(1) | +----------+ | 284582 | +----------+ 1 row in set (10.13 sec) SELECT COUNT(t.id) FROM things t WHERE t.title LIKE '%the%'; +----------+ | COUNT(1) | +----------+ | 173895 | +----------+ 1 row in set (10.12 sec) SELECT COUNT(t.id) FROM things t WHERE lower(t.username) like '%e%'; +----------+ | COUNT(1) | +----------+ | 190345 | +----------+ 1 row in set (10.15 sec) www.percona.com
#1: finding the filtering clause SELECT COUNT(t.id) FROM things t WHERE t.grants & 1 != 0; +----------+ | COUNT(1) | +----------+ | 4 | +----------+ 1 row in set (10.13 sec) ● This is selective ● In the original table definition the grants column has an index on it, but it's not used (the index on the not too selective active column is used instead) ● Data type of grants in TINYINT www.percona.com
#1: reason for slowness SELECT COUNT(t.id) FROM things t WHERE t.grants & 1 != 0; +----------+ | COUNT(1) | +----------+ | 4 | +----------+ 1 row in set (10.13 sec) ● The & is bitwise AND operator ● On the left hand side of comparison it's not the column itself, which is indexed, but some kind of function of the column is compared, which is not indexed. www.percona.com
#1: TINYINT column used as bitmap ● The grants in this case used as 1 raw byte (8 bits), 1 is a mask we compare to ● A common case for this is storing privileges Does some document / user / page has privilege “H”? 1 0 1 0 0 0 0 1 & Privilege “A” Privilege “H” 0 0 0 0 0 0 0 1 The result is non-0 when the appropriate privilege bit is 1. 0 0 0 0 0 0 0 1 www.percona.com
#1: solution ● The solution itself is very space efficient, but the data is not accessible. ● If only privilege “H” is checked frequently, just create a privilege “H” column, which is indexed and store data redundantly ● If other checks are frequent, use separate columns for each privilege ● Other antipatterns in the query, which are not making it slower www.percona.com
Case Study #2
#2: Query ● Query runs in the SELECT ta.*, tb.attr1, 10 seconds range tb.attr2, tb.attr3, tb.attr4, ● Rows examined is tc.attr1 FROM table_a ta, much higher than table_b tb, table_c tc WHERE tb.ID = ta.tb_id rows returned AND tc.attr1 = ta.attr1 AND tc.attr2 = 1 AND ta.attr3 = 'non_selective' AND ta.attr4 = 715988 AND ta.attr5 BETWEEN 1 AND 10 ORDER BY ta.attr6 ASC www.percona.com
#2: Explain *************************** 1. row *************************** id: 1 select_type: SIMPLE table: ta partitions: NULL type: ref possible_keys: idx_attr4,...,idx_attr3,... key: idx_attr3 key_len: 202 ref: const rows: 100280 Extra: Using where; Using filesort *************************** 2. row *************************** id: 1 table: tb type: eq_ref key: PRIMARY rows: 1 *************************** 3. row *************************** id: 1 table: tc type: ref key: idx_attr1 rows: 2 www.percona.com
#2: Show indexes and data types mysql> SHOW INDEX FROM table_a WHERE Key_name = "attr4" OR Key_name = "attr3"G *************************** 1. row *************************** Table: table_a Key_name: attr4 Seq_in_index: 1 Column_name: attr4 Cardinality: 523608 *************************** 2. row *************************** Table: table_a Key_name: attr3 Seq_in_index: 1 Column_name: attr4 Cardinality: 21 2 rows in set (0.01 sec) `attr3` varchar(200) DEFAULT NULL, `attr4` varchar(200) DEFAULT NULL www.percona.com
#2: solution SELECT ta.*, SELECT ta.*, tb.attr1, tb.attr1, tb.attr2, tb.attr2, tn.attr3, tn.attr3, tb.attr4, tb.attr4, tc.attr1 tc.attr1 FROM table_a ta, FROM table_a ta, table_b tb, table_b tb, table_c tc table_c tc WHERE tb.ID = ta.tb_id WHERE tb.ID = ta.tb_id AND tc.attr1 = ta.attr1 AND tc.attr1 = ta.attr1 AND tc.attr2 = 1 AND tc.attr2 = 1 AND ta.attr3 = 'non_selective' AND ta.attr3 = 'non_selective' AND ta.attr4 = 715988 AND ta.attr4 = '715988' AND ta.attr5 BETWEEN 1 AND 10 AND ta.attr5 BETWEEN 1 AND 10 ORDER BY a.date_recorded ASC ORDER BY a.date_recorded ASC After this change, the right index is used (changing column type could be better) www.percona.com
#2: conclusion ● Always quote strings, never quote numbers ● Similar thing can happen when there is a data type mismatch between joined tables ● For example an id is in a INT column is one table, VARCHAR(x) in the other, join won't use indexes on them www.percona.com
Case Study #3
#3: Query SELECT * FROM things WHERE ((thing_id=87459234 AND thing_flag1 = 1) OR (thing_other_thing_id=87459234 AND thing_flag2 = 1)) AND (thing_id=87459234 OR other_thing_id=87459234) AND (thing_id <> 3846571 AND other_thing_id <> 3846571) AND (thing_prop1 IS NOT NULL) AND (thing_rangedate >= '2012-12-01') OR (thing_rangedate_min > '2012-12-03' AND thing_rangedate_max < '2012-12-04') AND (NOT EXISTS (SELECT 1 FROM thing_stuffs WHERE fk_things_id = things.id AND important_num BETWEEN 1 and 4 AND other_range_value between 1 and 3)); If a query is big enough, always start with formatting the query. www.percona.com
#3: Query SELECT * FROM things First issue is clear from WHERE explain output: ((thing_id=87459234 AND thing_flag1 = 1) DEPENDENT SUBQUERY OR (thing_other_thing_id=87459234 AND thing_flag2 = 1)) AND (thing_id=87459234 OR other_thing_id=87459234) AND (thing_id <> 3846571 AND other_thing_id <> 3846571) AND (thing_prop1 IS NOT NULL) AND (thing_rangedate >= '2012-12-01') OR (thing_rangedate_min > '2012-12-03' AND thing_rangedate_max < '2012-12-04') AND (NOT EXISTS (SELECT 1 FROM thing_stuffs WHERE fk_things_id = things.id AND important_num BETWEEN 1 and 4 AND other_range_value BETWEEN 1 and 3)); www.percona.com
#3: Explain *************************** 1. row *************************** id: 1 select_type: PRIMARY table: things type: ref possible_keys: idx_thing_id,idx_other_thing_id,idx_prop1 key: idx_thing_id key_len: 4 rows: 23917933 Extra: Using where *************************** 2. row *************************** id: 2 select_type: DEPENDENT SUBQUERY table: thing_stuffs type: subquery possible_keys: PRIMARY,fk_things_id key: PRIMARY key_len: 4 rows: 1 Extra: Using where 2 rows in set (0.01 sec) www.percona.com
#3: Rewrite DEPENDENT SUBQUERY SELECT * FROM things WHERE AND (NOT EXISTS (SELECT 1 FROM thing_stuffs WHERE fk_things_id = things.id AND important_num BETWEEN 1 and 4 AND other_range_value BETWEEN 1 and 3)); SELECT t.* FROM things t LEFT JOIN thing_stuffs ts ON ts.fk_things_id = things.id WHERE ts.fk_things_id IS NULL AND ts.important_num BETWEEN 1 AND 4 AND ts.other_range_value BETWEEN 1 AND 3 www.percona.com
#3: Explain after rewrite *************************** 1. row *************************** id: 1 select_type: SIMPLE Composite index on important_num table: ts and other_range_value type: ref possible_keys: idx_important_num_other_range key: idx_important_num_other_range key_len: 4 ref: const rows: 93854 Extra: Using where *************************** 2. row *************************** id: 1 select_type: SIMPLE table: t type: ref possible_keys: PRIMARY key: PRIMARY key_len: 4 ref: ts.fk_things_id rows: 5 Extra: Using where 2 rows in set (0.01 sec) www.percona.com
#3: Examining further Still more rows examined than returned. possible_keys: idx_important_num_other_range key: idx_important_num_other_range key_len: 4 rows: 93854 The index is a composite index, however only important_num is used mysql> SELECT COUNT(1) FROM thing_stuffs WHERE important_num BETWEEN 1 AND 4; +----------+ | count(1) | +----------+ | 87520 | +----------+ 1 row in set (0.15 sec) mysql> SELECT COUNT(1) FROM thing_stuffs WHERE other_range_value BETWEEN 1 AND 3; +----------+ | count(1) | +----------+ | 134526 | +----------+ 1 row in set (0.17 sec) www.percona.com
#3: Examining further mysql> SELECT COUNT(1) FROM thing_stuff WHERE important_num BETWEEN 1 AND 4 AND other_range_value BETWEEN 1 AND 3; +----------+ | count(1) | +----------+ | 125 | +----------+ 1 row in set (0.15 sec) ● Together they are selective ● Both of them are range conditions, so they can't be used together in a composite index ● Unless we rewrite www.percona.com
#3: Second rewrite SELECT COUNT(1) from thing_stuff WHERE important_num IN (1,2,3,4) AND other_range_value IN (1,2,3); *************************** 1. row *************************** id: 1 select_type: SIMPLE table: ts type: ref key: idx_important_num_other_range key_len: 8 ref: const rows: 125 Extra: Using where *************************** 2. row *************************** id: 1 select_type: SIMPLE table: t type: ref key: PRIMARY key_len: 4 ref: ts.fk_things_id rows: 5 Extra: Using where 2 rows in set (0.01 sec) www.percona.com
#3: Using it in the original query The query is still slow. SELECT t.* FROM things t LEFT JOIN thing_stuffs ts ON ts.fk_things_id = t.id A lot more rows examined WHERE ((t.thing_id=87459234 AND t.thing_flag1 = 1) Than returned. OR (t.other_thing_id=87459234 AND t.thing_flag2 = 1)) AND (t.thing_id=87459234 OR t.other_thing_id=87459234) AND (t.thing_id <> 3846571 Checking explain the AND t.other_thing_id <> 3846571) relevant part is that the AND (t.thing_prop1 IS NOT NULL) query uses an index merge. AND (t.thing_rangedate >= '2012-12-01') OR (t.thing_rangedate_min > '2012-12-03' AND t.thing_rangedate_max < '2012-12-04') AND ts.important_num IN (1,2,3,4) AND ts.other_range_value IN (1,2,3) AND ts.fk_things_id IS NULL; Extra: Using union(idx_thing_id,idx_other_thing_id); Using where www.percona.com
#3: Relevant part SELECT * FROM things WHERE (thing_id=87459234 OR other_thing_id=87459234) AND thing_id <> 3846571 AND other_thing_id <> 3846571; *************************** 1. row *************************** id: 1 select_type: SIMPLE table: things type: index_merge possible_keys: idx_thing_id,idx_other_thing_id key: idx_thing_id,idx_other_thing_id key_len: 4,4 ref: NULL rows: 59834 Extra: Using union(idx_thing_id,idx_other_thing_id); Using where 1 row in set (0.00 sec) www.percona.com
#3: Rewritten query ● Index merge is an expensive SELECT * FROM things operation WHERE thing_id=87459234 AND other_thing_id<>3846571 ● Especially if indexes are large UNION SELECT * FROM things ● UNION always create an on-disk WHERE other_thing_id=87459234 temp table AND thing_id<>3846571; ● Just turning off index merge could be enough for a significant performance advantage mysql> select @@optimizer_switchG *************************** 1. row *************************** @@optimizer_switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_interse ction=on,engine_condition_pushdown=on 1 row in set (0.00 sec) www.percona.com
#3: Putting it together SELECT t.* FROM things t LEFT JOIN things_stuff ts ON ts.fk_things_id = t.id WHERE t.thing_id=87459234 AND t.other_thing_id<>3846571 AND t.thing_flag1=1 AND t.thing_prop IS NOT NULL SELECT t.* FROM things t AND (t.thing_rangedate_min >= '2012-12-01') LEFT JOIN t.things_stuff ts AND ((t.thing_rangedate_min <= '2012-12-03' ON ts.fk_things_id = t.id AND t.thing_rangedate_max > '2012-12-03') WHERE t.other_thing_id=87459234 OR (t.thing_rangedate_min < '2012-12-04' AND t.thing_id<>3846571 AND t.thing_rangedate_max >= '2012-12-04') AND t.thing_flag2=1 OR (t.thing_rangedate_min > '2012-12-03' AND t.thing_prop1 IS NOT NULL AND t.thing_rangerate_max < '2012-12-04')) AND (t.thing_rangedate_min >= '2012-12-01') AND ts.fk_things_id IS NULL AND ((t.thing_rangedate_min <= '2012-12-03' AND ts.important_num IN (1,2,3,4) AND t.thing_rangedate_max > '2012-12-03') AND ts.other_range_value IN (1,2,3) OR (t.thing_rangedate_min < '2012-12-04' UNION AND t.thing_rangedate_max >= '2012-12-04') OR (t.thing_rangedate_min > '2012-12-03' AND t.thing_rangedate_max < '2012-12-04')) AND ts.fk_things_id IS NULL AND ts.important_num IN (1,2,3,4) AND ts.other_range_value IN (1,2,3); www.percona.com
Case Study #4
#4: Summary ● Very large, 20 table join ● Explain looks ok ● Query runs for almost a hour ● Handler status variables are way off www.percona.com
#4: Query SELECT tmain.id, tmain.date, CONCAT_WS("_", COALESCE(m1.col1, 'n/a'), COALESCE(m2.col2, 'n/a') /* a lot more columns selected */) FROM main_table as tmain ah LEFT JOIN (metadata_table1 m1 JOIN metadata_table2 m2 ON m1.attr1 = m2.attr1 AND m1.const = 'some text') ON tmain.id = m1.tmain_id AND tmain.important_flag=1 AND ((m1.attr11='a') OR (m2.attr12 IS NOT NULL) AND 3>=m1.attr13 AND 5<m2.attr14) OR m1.attr15 IS NOT NULL AND m2.attr16 BETWEEN 1 AND 10 LEFT JOIN (metadata_table3 m3 JOIN metadata_table4 m4 ON m3.attr1 = m4.attr1 AND m1.const = 'some text') ON tmain.id = m4.tmain_id AND tmain.important_flag=1 AND ((m3.attr11='a') OR (m4.attr12 IS NOT NULL) AND 3>=m3.attr13 AND 5<m4.attr14) OR m3.attr15 IS NOT NULL AND m4.attr16 BETWEEN 11 AND 20 LEFT JOIN (metadata_table5 m5 JOIN metadata_table2 m6 ON m5.attr1 = m6.attr1 AND m1.const = 'some text') ON tmain.id = m5.tmain_id AND tmain.important_flag=1 AND ((m5.attr11='a') OR (m6.attr12 IS NOT NULL) AND 3>=m5.attr13 AND 5<m6.attr14) OR m1.attr15 IS NOT NULL AND m2.attr16 BETWEEN 21 AND 30 LEFT JOIN (metadata_table1 m7 JOIN metadata_table2 m8 ON m7.attr1 = m8.attr1 AND m7.const = 'some text') ON tmain.id = m7.tmain_id AND tmain.important_flag=1 AND ((m7.attr11='a') OR (m8.attr12 IS NOT NULL) AND 3>=m7.attr13 AND 5<m8.attr14) OR m7.attr15 IS NOT NULL AND m8.attr16 BETWEEN 31 AND 40 /* 6 more joins exactly like the previous ones... */ WHERE tmain.important_flag = 1; www.percona.com
#4: Explain (only relevant part) +----+-------------+----------+--------+------+-------------+ | id | select_type | table | type | rows | Extra | +----+-------------+----------+--------+------+-------------+ | 1 | SIMPLE | tmain | ref | 1786 | | | 1 | SIMPLE | m1 | ref | 4 | | | 1 | SIMPLE | m2 | eq_ref | 1 | Using index | | 1 | SIMPLE | m3 | ref | 4 | | | 1 | SIMPLE | m4 | eq_ref | 1 | Using index | |....|.............|..........|........|......|.............| | 1 | SIMPLE | m19 | ref | 4 | | | 1 | SIMPLE | m20 | eq_ref | 1 | Using index | +----+-------------+----------+--------+------+-------------+ www.percona.com
#4: Handler operations mysql> show status like 'Handler%'; +----------------------------+-----------+ Run FLUSH STATUS before the query | Variable_name | Value | +----------------------------+-----------+ to reset the counters. | Handler_commit | 1 | | Handler_delete | 0 | | Handler_discover | 0 | FLUSH STATUS; | Handler_prepare | 0 | QUERY; | Handler_read_first | 0 | | Handler_read_key | 61341 | SHOW STATUS LIKE 'Handler%'; | Handler_read_last | 0 | | Handler_read_next | 198008370 | | Handler_read_prev | 0 | | Handler_read_rnd | 0 | | Handler_read_rnd_next | 0 | | Handler_rollback | 0 | | Handler_savepoint | 0 | | Handler_savepoint_rollback | 0 | | Handler_update | 0 | | Handler_write | 0 | +----------------------------+-----------+ www.percona.com
#4: Problematic part LEFT JOIN (metadata_table1 m7 JOIN metadata_table2 m8 ON m7.attr1 = m8.attr1 AND m7.const = 'some text') ON tmain.id = m7.tmain_id AND tmain.important_flag=1; SELECT AVG(LENGTH(m1.somefield)) FROM metadata_table1 m1 JOIN metadata_table2 m2 ON m1.attr1 = m2.attr1 AND m1.const='some text' +----+-------------+----------+--------+-------+-------------+ | id | select_type | table | type | rows | Extra | +----+-------------+----------+--------+-------+-------------+ | 1 | SIMPLE | m1 | ref | 19764 | | | 1 | SIMPLE | m2 | eq_ref | 102 | Using index | +----+-------------+----------+--------+-------+-------------+ MySQL executes the join like this. One “problematic part takes roughly 3 minutes, so It adds up. www.percona.com
#4: Rewrite SELECT tmain.id FROM main_table tmain LEFT JOIN metadata_table1 m1 ON tmain.id=m1.tmain_id LEFT JOIN metadata_table2 m2 ON m1.attr1 = m2.attr1 AND m1.const = 'some text' WHERE tmain.important_flag = 1; ● In order to leave out () LEFT JOIN has to be used to find non-matching elements. ● Runtime for query is 0.2 sec instead of 3 minutes. +----+-------------+----------+--------+------+-------------+ | id | select_type | table | type | rows | Extra | +----+-------------+----------+--------+------+-------------+ | 1 | SIMPLE | tmain | ref | 1786 | | | 1 | SIMPLE | m1 | ref | 4 | | | 1 | SIMPLE | m2 | eq_ref | 1 | Using index | +----+-------------+----------+--------+------+-------------+ www.percona.com
#4: Similarly large queries ● After applying the tuning to the original query, I expected it t execute in a few seconds. ● It executed in 4 minutes instead. ● Used Handler operations looked good. ● Took a look at explain, and explain took 4 minutes also. ● This means the optimizer spends time in constructing the query plan. ● Solution: STRAIGHT_JOIN and/or optimizer_search_depth www.percona.com
Annual Percona Live MySQL Conference and Expo Visit: http://www.percona.com/live/mysql-conference-2013/ www.percona.com
Q&A
Thank you for attention.

More Related Content

PPT
Recovery of lost or corrupted inno db tables(mysql uc 2010)
guest808c167
 
PDF
pyton Notes6
Amba Research
 
PPT
C1320prespost
FALLEE31188
 
DOCX
Technical
ved prakash
 
PDF
Slicing in Python - What is It?
💾 Radek Fabisiak
 
PDF
Cis 336 final exam 2
lifesgood12
 
PPT
Linked list
Trupti Agrawal
 
PDF
CIS 336 Final Exam 2 (Devry)p
critterc07
 
Recovery of lost or corrupted inno db tables(mysql uc 2010)
guest808c167
 
pyton Notes6
Amba Research
 
C1320prespost
FALLEE31188
 
Technical
ved prakash
 
Slicing in Python - What is It?
💾 Radek Fabisiak
 
Cis 336 final exam 2
lifesgood12
 
Linked list
Trupti Agrawal
 
CIS 336 Final Exam 2 (Devry)p
critterc07
 

Viewers also liked (15)

PDF
TOPPS Cairo May 17th 2016 v1.2
TAQAMISR
 
PPTX
Techos verdes
Danni Parra Calderon
 
PDF
Platinum Tools 12507C Data Sheet
JMAC Supply
 
PPT
давление в жидкостях и газах 2
fangizf68123
 
PDF
Ata 44 da ceext 1ª câmara
G. Gomes
 
PDF
TAQAMISR May 17th 2016
TAQAMISR
 
PDF
Ata 42 ceext transposição de servidores
G. Gomes
 
PDF
Tabela de Taxas Veiculares-2017
G. Gomes
 
PDF
The Power of MySQL Explain
MYXPLAIN
 
DOCX
Hasil keputusan rapat rt 03
Ikhsan Arvis
 
PDF
Apa referencing guide 6th ed 2014 update
Liza Ainurahmah II
 
PDF
Optimizing Queries with Explain
MYXPLAIN
 
PDF
QUALITY MANAGEMENT PRACTICES USED BY TELECOM ORGANIZATIONS,quality in telecom...
Micky Lyf
 
PDF
Binder1
John Johnson
 
PPTX
Rechtsupdate 2016 - Community Camp Berlin 2016 - #ccb16
Thomas Schwenke
 
TOPPS Cairo May 17th 2016 v1.2
TAQAMISR
 
Techos verdes
Danni Parra Calderon
 
Platinum Tools 12507C Data Sheet
JMAC Supply
 
давление в жидкостях и газах 2
fangizf68123
 
Ata 44 da ceext 1ª câmara
G. Gomes
 
TAQAMISR May 17th 2016
TAQAMISR
 
Ata 42 ceext transposição de servidores
G. Gomes
 
Tabela de Taxas Veiculares-2017
G. Gomes
 
The Power of MySQL Explain
MYXPLAIN
 
Hasil keputusan rapat rt 03
Ikhsan Arvis
 
Apa referencing guide 6th ed 2014 update
Liza Ainurahmah II
 
Optimizing Queries with Explain
MYXPLAIN
 
QUALITY MANAGEMENT PRACTICES USED BY TELECOM ORGANIZATIONS,quality in telecom...
Micky Lyf
 
Binder1
John Johnson
 
Rechtsupdate 2016 - Community Camp Berlin 2016 - #ccb16
Thomas Schwenke
 
Ad

Similar to Advanced query optimization (20)

PPTX
How to tune a query - ODTUG 2012
Connor McDonald
 
PDF
MySQL Query tuning 101
Sveta Smirnova
 
PPTX
REC-UNIT-2-DATABASEMANAGEMENTSYSTEMS.pptx
Uma Kakarlapudi
 
PPTX
Python
Sangita Panchal
 
PDF
MySQL Query Optimisation 101
Federico Razzoli
 
PPTX
Python Programming for basic beginners.pptx
mohitesoham12
 
PPTX
Data structures and algorithms
Julie Iskander
 
PPT
Python course in_mumbai
vibrantuser
 
PPT
Python course in_mumbai
vibrantuser
 
PPT
Data Structures 6
Dr.Umadevi V
 
PDF
Discover the power of Recursive SQL and query transformation with Informix da...
Ajay Gupte
 
PDF
MariaDB's join optimizer: how it works and current fixes
Sergey Petrunya
 
PPT
Ch02
Maqsood Hayat
 
PDF
Python Basics it will teach you about data types
NimitSinghal2
 
PPTX
Advance sql session - strings
Eyal Trabelsi
 
PDF
Python Programminng…………………………………………………..
somyaranjan27
 
PPTX
Improve Your Edge on Machine Learning - Day 1.pptx
CatherineVania1
 
PPT
Python study material
Satish Nagabhushan
 
PPTX
RELATIONALfsaaaaaaaaaaaakyagsgs MODEL.pptx
satish7588
 
PDF
SQL Joins and Query Optimization
Brian Gallagher
 
How to tune a query - ODTUG 2012
Connor McDonald
 
MySQL Query tuning 101
Sveta Smirnova
 
REC-UNIT-2-DATABASEMANAGEMENTSYSTEMS.pptx
Uma Kakarlapudi
 
Python
Sangita Panchal
 
MySQL Query Optimisation 101
Federico Razzoli
 
Python Programming for basic beginners.pptx
mohitesoham12
 
Data structures and algorithms
Julie Iskander
 
Python course in_mumbai
vibrantuser
 
Python course in_mumbai
vibrantuser
 
Data Structures 6
Dr.Umadevi V
 
Discover the power of Recursive SQL and query transformation with Informix da...
Ajay Gupte
 
MariaDB's join optimizer: how it works and current fixes
Sergey Petrunya
 
Ch02
Maqsood Hayat
 
Python Basics it will teach you about data types
NimitSinghal2
 
Advance sql session - strings
Eyal Trabelsi
 
Python Programminng…………………………………………………..
somyaranjan27
 
Improve Your Edge on Machine Learning - Day 1.pptx
CatherineVania1
 
Python study material
Satish Nagabhushan
 
RELATIONALfsaaaaaaaaaaaakyagsgs MODEL.pptx
satish7588
 
SQL Joins and Query Optimization
Brian Gallagher
 
Ad

More from MYXPLAIN (16)

PDF
Query Optimization with MySQL 5.6: Old and New Tricks
MYXPLAIN
 
PDF
Need for Speed: MySQL Indexing
MYXPLAIN
 
PDF
Advanced Query Optimizer Tuning and Analysis
MYXPLAIN
 
PDF
MySQL Index Cookbook
MYXPLAIN
 
PDF
Advanced MySQL Query and Schema Tuning
MYXPLAIN
 
PDF
Are You Getting the Best of your MySQL Indexes
MYXPLAIN
 
PDF
How to Design Indexes, Really
MYXPLAIN
 
PDF
MySQL 5.6 Performance
MYXPLAIN
 
PPTX
MySQL Indexing - Best practices for MySQL 5.6
MYXPLAIN
 
PDF
56 Query Optimization
MYXPLAIN
 
PDF
Tools and Techniques for Index Design
MYXPLAIN
 
PDF
Powerful Explain in MySQL 5.6
MYXPLAIN
 
PDF
Improving Performance with Better Indexes
MYXPLAIN
 
PDF
Explaining the MySQL Explain
MYXPLAIN
 
PDF
Covering indexes
MYXPLAIN
 
PDF
MySQL Optimizer Overview
MYXPLAIN
 
Query Optimization with MySQL 5.6: Old and New Tricks
MYXPLAIN
 
Need for Speed: MySQL Indexing
MYXPLAIN
 
Advanced Query Optimizer Tuning and Analysis
MYXPLAIN
 
MySQL Index Cookbook
MYXPLAIN
 
Advanced MySQL Query and Schema Tuning
MYXPLAIN
 
Are You Getting the Best of your MySQL Indexes
MYXPLAIN
 
How to Design Indexes, Really
MYXPLAIN
 
MySQL 5.6 Performance
MYXPLAIN
 
MySQL Indexing - Best practices for MySQL 5.6
MYXPLAIN
 
56 Query Optimization
MYXPLAIN
 
Tools and Techniques for Index Design
MYXPLAIN
 
Powerful Explain in MySQL 5.6
MYXPLAIN
 
Improving Performance with Better Indexes
MYXPLAIN
 
Explaining the MySQL Explain
MYXPLAIN
 
Covering indexes
MYXPLAIN
 
MySQL Optimizer Overview
MYXPLAIN
 

Recently uploaded (20)

PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
PPTX
ACSFv1EN-58255 AWS Academy Cloud Security Foundations.pptx
23bcla24
 
PDF
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
PDF
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
PDF
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
PDF
NewMind AI Weekly Chronicles - August'25-Week II
NewMind AI
 
PDF
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
PDF
Assigned Numbers - 2025 - Bluetooth® Document
Bloombase
 
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
PDF
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
PDF
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
PDF
Encapsulation theory and applications.pdf
gurumoop
 
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
PDF
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
PPTX
Spectroscopy.pptx food analysis technology
nawahassan45
 
PDF
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
PDF
Approach and Philosophy of On baking technology
30anthuong17
 
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
ACSFv1EN-58255 AWS Academy Cloud Security Foundations.pptx
23bcla24
 
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
NewMind AI Weekly Chronicles - August'25-Week II
NewMind AI
 
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
Assigned Numbers - 2025 - Bluetooth® Document
Bloombase
 
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
Encapsulation theory and applications.pdf
gurumoop
 
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
Spectroscopy.pptx food analysis technology
nawahassan45
 
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
Approach and Philosophy of On baking technology
30anthuong17
 

Advanced query optimization

  • 1. Advanced query optimization techniques on large queries Peter Boros Percona Webinar
  • 2. Agenda • Showing the title slide • Going through this agenda • Prerequisites • Case study #1 • Case study #2 • Case study #3 • Case study #4 www.percona.com
  • 3. This talk is not about ● SQL basics ● Reading explain ● Indexing basics www.percona.com
  • 5. #1: Query ● Query runs in the SELECT t.id, ts.meta1, lt.meta2 FROM things t 10 seconds range INNER JOIN things_stuff ts ON t.fk_things_stuff_id = ts.id INNER JOIN yet_an_other_table yat ● According to ON t.fk_yat_id = yat.id WHERE t.active = 1 explain output it AND t.type != 'guest' AND t.title like '%the%' AND LOWER(t.username) like '%e%' can use some AND t.grants & 1 != 0 AND t.attribute1=3; indexes ● Mostly single- column indexes were defined www.percona.com
  • 6. #1: Query ● Rows examined in SELECT t.id, ts.meta1, lt.meta2 FROM things t the 100k range INNER JOIN things_stuff ts ON t.fk_things_stuff_id = ts.id INNER JOIN yet_an_other_table yat ● Rows returned is ON t.fk_yat_id = yat.id WHERE t.active = 1 single digit AND t.type != 'guest' AND t.title like '%the%' AND LOWER(t.username) like '%e%' ● This means AND t.grants & 1 != 0 AND t.attribute1=3; something is selective, but most likely not indexed www.percona.com
  • 7. #1: issue identification ********************** 1. row ********************** id: 1 select_type: SIMPLE table: t This is the issue, the rest type: const of the tables are just some key: idx_active metadata fetching (type is eq_ref) key_len: 4 rows: 332873 Extra: Using where ********************** 2. row ********************** id: 1 select_type: SIMPLE table: ts type: eq_ref key: PRIMARY key_len: 4 rows: 1 Extra: ********************** 3. row ***************** type: eq_ref rows: 1 www.percona.com
  • 8. #1: finding the filtering clause ● In the original query, filtering in the WHERE clause are all for table t (things) ● Find out if any of them are selective alone (it's possible that they are not, but some combination of them will be selective) SELECT COUNT(t.id) FROM things t WHERE t.active=1; +----------+ | COUNT(1) | +----------+ | 168520 | +----------+ 1 row in set (10.15 sec) www.percona.com
  • 9. #1: finding the filtering clause SELECT COUNT(t.id) FROM things t WHERE t.status!='guest'; +----------+ | COUNT(1) | +----------+ | 284582 | +----------+ 1 row in set (10.13 sec) SELECT COUNT(t.id) FROM things t WHERE t.title LIKE '%the%'; +----------+ | COUNT(1) | +----------+ | 173895 | +----------+ 1 row in set (10.12 sec) SELECT COUNT(t.id) FROM things t WHERE lower(t.username) like '%e%'; +----------+ | COUNT(1) | +----------+ | 190345 | +----------+ 1 row in set (10.15 sec) www.percona.com
  • 10. #1: finding the filtering clause SELECT COUNT(t.id) FROM things t WHERE t.grants & 1 != 0; +----------+ | COUNT(1) | +----------+ | 4 | +----------+ 1 row in set (10.13 sec) ● This is selective ● In the original table definition the grants column has an index on it, but it's not used (the index on the not too selective active column is used instead) ● Data type of grants in TINYINT www.percona.com
  • 11. #1: reason for slowness SELECT COUNT(t.id) FROM things t WHERE t.grants & 1 != 0; +----------+ | COUNT(1) | +----------+ | 4 | +----------+ 1 row in set (10.13 sec) ● The & is bitwise AND operator ● On the left hand side of comparison it's not the column itself, which is indexed, but some kind of function of the column is compared, which is not indexed. www.percona.com
  • 12. #1: TINYINT column used as bitmap ● The grants in this case used as 1 raw byte (8 bits), 1 is a mask we compare to ● A common case for this is storing privileges Does some document / user / page has privilege “H”? 1 0 1 0 0 0 0 1 & Privilege “A” Privilege “H” 0 0 0 0 0 0 0 1 The result is non-0 when the appropriate privilege bit is 1. 0 0 0 0 0 0 0 1 www.percona.com
  • 13. #1: solution ● The solution itself is very space efficient, but the data is not accessible. ● If only privilege “H” is checked frequently, just create a privilege “H” column, which is indexed and store data redundantly ● If other checks are frequent, use separate columns for each privilege ● Other antipatterns in the query, which are not making it slower www.percona.com
  • 15. #2: Query ● Query runs in the SELECT ta.*, tb.attr1, 10 seconds range tb.attr2, tb.attr3, tb.attr4, ● Rows examined is tc.attr1 FROM table_a ta, much higher than table_b tb, table_c tc WHERE tb.ID = ta.tb_id rows returned AND tc.attr1 = ta.attr1 AND tc.attr2 = 1 AND ta.attr3 = 'non_selective' AND ta.attr4 = 715988 AND ta.attr5 BETWEEN 1 AND 10 ORDER BY ta.attr6 ASC www.percona.com
  • 16. #2: Explain *************************** 1. row *************************** id: 1 select_type: SIMPLE table: ta partitions: NULL type: ref possible_keys: idx_attr4,...,idx_attr3,... key: idx_attr3 key_len: 202 ref: const rows: 100280 Extra: Using where; Using filesort *************************** 2. row *************************** id: 1 table: tb type: eq_ref key: PRIMARY rows: 1 *************************** 3. row *************************** id: 1 table: tc type: ref key: idx_attr1 rows: 2 www.percona.com
  • 17. #2: Show indexes and data types mysql> SHOW INDEX FROM table_a WHERE Key_name = "attr4" OR Key_name = "attr3"G *************************** 1. row *************************** Table: table_a Key_name: attr4 Seq_in_index: 1 Column_name: attr4 Cardinality: 523608 *************************** 2. row *************************** Table: table_a Key_name: attr3 Seq_in_index: 1 Column_name: attr4 Cardinality: 21 2 rows in set (0.01 sec) `attr3` varchar(200) DEFAULT NULL, `attr4` varchar(200) DEFAULT NULL www.percona.com
  • 18. #2: solution SELECT ta.*, SELECT ta.*, tb.attr1, tb.attr1, tb.attr2, tb.attr2, tn.attr3, tn.attr3, tb.attr4, tb.attr4, tc.attr1 tc.attr1 FROM table_a ta, FROM table_a ta, table_b tb, table_b tb, table_c tc table_c tc WHERE tb.ID = ta.tb_id WHERE tb.ID = ta.tb_id AND tc.attr1 = ta.attr1 AND tc.attr1 = ta.attr1 AND tc.attr2 = 1 AND tc.attr2 = 1 AND ta.attr3 = 'non_selective' AND ta.attr3 = 'non_selective' AND ta.attr4 = 715988 AND ta.attr4 = '715988' AND ta.attr5 BETWEEN 1 AND 10 AND ta.attr5 BETWEEN 1 AND 10 ORDER BY a.date_recorded ASC ORDER BY a.date_recorded ASC After this change, the right index is used (changing column type could be better) www.percona.com
  • 19. #2: conclusion ● Always quote strings, never quote numbers ● Similar thing can happen when there is a data type mismatch between joined tables ● For example an id is in a INT column is one table, VARCHAR(x) in the other, join won't use indexes on them www.percona.com
  • 21. #3: Query SELECT * FROM things WHERE ((thing_id=87459234 AND thing_flag1 = 1) OR (thing_other_thing_id=87459234 AND thing_flag2 = 1)) AND (thing_id=87459234 OR other_thing_id=87459234) AND (thing_id <> 3846571 AND other_thing_id <> 3846571) AND (thing_prop1 IS NOT NULL) AND (thing_rangedate >= '2012-12-01') OR (thing_rangedate_min > '2012-12-03' AND thing_rangedate_max < '2012-12-04') AND (NOT EXISTS (SELECT 1 FROM thing_stuffs WHERE fk_things_id = things.id AND important_num BETWEEN 1 and 4 AND other_range_value between 1 and 3)); If a query is big enough, always start with formatting the query. www.percona.com
  • 22. #3: Query SELECT * FROM things First issue is clear from WHERE explain output: ((thing_id=87459234 AND thing_flag1 = 1) DEPENDENT SUBQUERY OR (thing_other_thing_id=87459234 AND thing_flag2 = 1)) AND (thing_id=87459234 OR other_thing_id=87459234) AND (thing_id <> 3846571 AND other_thing_id <> 3846571) AND (thing_prop1 IS NOT NULL) AND (thing_rangedate >= '2012-12-01') OR (thing_rangedate_min > '2012-12-03' AND thing_rangedate_max < '2012-12-04') AND (NOT EXISTS (SELECT 1 FROM thing_stuffs WHERE fk_things_id = things.id AND important_num BETWEEN 1 and 4 AND other_range_value BETWEEN 1 and 3)); www.percona.com
  • 23. #3: Explain *************************** 1. row *************************** id: 1 select_type: PRIMARY table: things type: ref possible_keys: idx_thing_id,idx_other_thing_id,idx_prop1 key: idx_thing_id key_len: 4 rows: 23917933 Extra: Using where *************************** 2. row *************************** id: 2 select_type: DEPENDENT SUBQUERY table: thing_stuffs type: subquery possible_keys: PRIMARY,fk_things_id key: PRIMARY key_len: 4 rows: 1 Extra: Using where 2 rows in set (0.01 sec) www.percona.com
  • 24. #3: Rewrite DEPENDENT SUBQUERY SELECT * FROM things WHERE AND (NOT EXISTS (SELECT 1 FROM thing_stuffs WHERE fk_things_id = things.id AND important_num BETWEEN 1 and 4 AND other_range_value BETWEEN 1 and 3)); SELECT t.* FROM things t LEFT JOIN thing_stuffs ts ON ts.fk_things_id = things.id WHERE ts.fk_things_id IS NULL AND ts.important_num BETWEEN 1 AND 4 AND ts.other_range_value BETWEEN 1 AND 3 www.percona.com
  • 25. #3: Explain after rewrite *************************** 1. row *************************** id: 1 select_type: SIMPLE Composite index on important_num table: ts and other_range_value type: ref possible_keys: idx_important_num_other_range key: idx_important_num_other_range key_len: 4 ref: const rows: 93854 Extra: Using where *************************** 2. row *************************** id: 1 select_type: SIMPLE table: t type: ref possible_keys: PRIMARY key: PRIMARY key_len: 4 ref: ts.fk_things_id rows: 5 Extra: Using where 2 rows in set (0.01 sec) www.percona.com
  • 26. #3: Examining further Still more rows examined than returned. possible_keys: idx_important_num_other_range key: idx_important_num_other_range key_len: 4 rows: 93854 The index is a composite index, however only important_num is used mysql> SELECT COUNT(1) FROM thing_stuffs WHERE important_num BETWEEN 1 AND 4; +----------+ | count(1) | +----------+ | 87520 | +----------+ 1 row in set (0.15 sec) mysql> SELECT COUNT(1) FROM thing_stuffs WHERE other_range_value BETWEEN 1 AND 3; +----------+ | count(1) | +----------+ | 134526 | +----------+ 1 row in set (0.17 sec) www.percona.com
  • 27. #3: Examining further mysql> SELECT COUNT(1) FROM thing_stuff WHERE important_num BETWEEN 1 AND 4 AND other_range_value BETWEEN 1 AND 3; +----------+ | count(1) | +----------+ | 125 | +----------+ 1 row in set (0.15 sec) ● Together they are selective ● Both of them are range conditions, so they can't be used together in a composite index ● Unless we rewrite www.percona.com
  • 28. #3: Second rewrite SELECT COUNT(1) from thing_stuff WHERE important_num IN (1,2,3,4) AND other_range_value IN (1,2,3); *************************** 1. row *************************** id: 1 select_type: SIMPLE table: ts type: ref key: idx_important_num_other_range key_len: 8 ref: const rows: 125 Extra: Using where *************************** 2. row *************************** id: 1 select_type: SIMPLE table: t type: ref key: PRIMARY key_len: 4 ref: ts.fk_things_id rows: 5 Extra: Using where 2 rows in set (0.01 sec) www.percona.com
  • 29. #3: Using it in the original query The query is still slow. SELECT t.* FROM things t LEFT JOIN thing_stuffs ts ON ts.fk_things_id = t.id A lot more rows examined WHERE ((t.thing_id=87459234 AND t.thing_flag1 = 1) Than returned. OR (t.other_thing_id=87459234 AND t.thing_flag2 = 1)) AND (t.thing_id=87459234 OR t.other_thing_id=87459234) AND (t.thing_id <> 3846571 Checking explain the AND t.other_thing_id <> 3846571) relevant part is that the AND (t.thing_prop1 IS NOT NULL) query uses an index merge. AND (t.thing_rangedate >= '2012-12-01') OR (t.thing_rangedate_min > '2012-12-03' AND t.thing_rangedate_max < '2012-12-04') AND ts.important_num IN (1,2,3,4) AND ts.other_range_value IN (1,2,3) AND ts.fk_things_id IS NULL; Extra: Using union(idx_thing_id,idx_other_thing_id); Using where www.percona.com
  • 30. #3: Relevant part SELECT * FROM things WHERE (thing_id=87459234 OR other_thing_id=87459234) AND thing_id <> 3846571 AND other_thing_id <> 3846571; *************************** 1. row *************************** id: 1 select_type: SIMPLE table: things type: index_merge possible_keys: idx_thing_id,idx_other_thing_id key: idx_thing_id,idx_other_thing_id key_len: 4,4 ref: NULL rows: 59834 Extra: Using union(idx_thing_id,idx_other_thing_id); Using where 1 row in set (0.00 sec) www.percona.com
  • 31. #3: Rewritten query ● Index merge is an expensive SELECT * FROM things operation WHERE thing_id=87459234 AND other_thing_id<>3846571 ● Especially if indexes are large UNION SELECT * FROM things ● UNION always create an on-disk WHERE other_thing_id=87459234 temp table AND thing_id<>3846571; ● Just turning off index merge could be enough for a significant performance advantage mysql> select @@optimizer_switchG *************************** 1. row *************************** @@optimizer_switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_interse ction=on,engine_condition_pushdown=on 1 row in set (0.00 sec) www.percona.com
  • 32. #3: Putting it together SELECT t.* FROM things t LEFT JOIN things_stuff ts ON ts.fk_things_id = t.id WHERE t.thing_id=87459234 AND t.other_thing_id<>3846571 AND t.thing_flag1=1 AND t.thing_prop IS NOT NULL SELECT t.* FROM things t AND (t.thing_rangedate_min >= '2012-12-01') LEFT JOIN t.things_stuff ts AND ((t.thing_rangedate_min <= '2012-12-03' ON ts.fk_things_id = t.id AND t.thing_rangedate_max > '2012-12-03') WHERE t.other_thing_id=87459234 OR (t.thing_rangedate_min < '2012-12-04' AND t.thing_id<>3846571 AND t.thing_rangedate_max >= '2012-12-04') AND t.thing_flag2=1 OR (t.thing_rangedate_min > '2012-12-03' AND t.thing_prop1 IS NOT NULL AND t.thing_rangerate_max < '2012-12-04')) AND (t.thing_rangedate_min >= '2012-12-01') AND ts.fk_things_id IS NULL AND ((t.thing_rangedate_min <= '2012-12-03' AND ts.important_num IN (1,2,3,4) AND t.thing_rangedate_max > '2012-12-03') AND ts.other_range_value IN (1,2,3) OR (t.thing_rangedate_min < '2012-12-04' UNION AND t.thing_rangedate_max >= '2012-12-04') OR (t.thing_rangedate_min > '2012-12-03' AND t.thing_rangedate_max < '2012-12-04')) AND ts.fk_things_id IS NULL AND ts.important_num IN (1,2,3,4) AND ts.other_range_value IN (1,2,3); www.percona.com
  • 34. #4: Summary ● Very large, 20 table join ● Explain looks ok ● Query runs for almost a hour ● Handler status variables are way off www.percona.com
  • 35. #4: Query SELECT tmain.id, tmain.date, CONCAT_WS("_", COALESCE(m1.col1, 'n/a'), COALESCE(m2.col2, 'n/a') /* a lot more columns selected */) FROM main_table as tmain ah LEFT JOIN (metadata_table1 m1 JOIN metadata_table2 m2 ON m1.attr1 = m2.attr1 AND m1.const = 'some text') ON tmain.id = m1.tmain_id AND tmain.important_flag=1 AND ((m1.attr11='a') OR (m2.attr12 IS NOT NULL) AND 3>=m1.attr13 AND 5<m2.attr14) OR m1.attr15 IS NOT NULL AND m2.attr16 BETWEEN 1 AND 10 LEFT JOIN (metadata_table3 m3 JOIN metadata_table4 m4 ON m3.attr1 = m4.attr1 AND m1.const = 'some text') ON tmain.id = m4.tmain_id AND tmain.important_flag=1 AND ((m3.attr11='a') OR (m4.attr12 IS NOT NULL) AND 3>=m3.attr13 AND 5<m4.attr14) OR m3.attr15 IS NOT NULL AND m4.attr16 BETWEEN 11 AND 20 LEFT JOIN (metadata_table5 m5 JOIN metadata_table2 m6 ON m5.attr1 = m6.attr1 AND m1.const = 'some text') ON tmain.id = m5.tmain_id AND tmain.important_flag=1 AND ((m5.attr11='a') OR (m6.attr12 IS NOT NULL) AND 3>=m5.attr13 AND 5<m6.attr14) OR m1.attr15 IS NOT NULL AND m2.attr16 BETWEEN 21 AND 30 LEFT JOIN (metadata_table1 m7 JOIN metadata_table2 m8 ON m7.attr1 = m8.attr1 AND m7.const = 'some text') ON tmain.id = m7.tmain_id AND tmain.important_flag=1 AND ((m7.attr11='a') OR (m8.attr12 IS NOT NULL) AND 3>=m7.attr13 AND 5<m8.attr14) OR m7.attr15 IS NOT NULL AND m8.attr16 BETWEEN 31 AND 40 /* 6 more joins exactly like the previous ones... */ WHERE tmain.important_flag = 1; www.percona.com
  • 36. #4: Explain (only relevant part) +----+-------------+----------+--------+------+-------------+ | id | select_type | table | type | rows | Extra | +----+-------------+----------+--------+------+-------------+ | 1 | SIMPLE | tmain | ref | 1786 | | | 1 | SIMPLE | m1 | ref | 4 | | | 1 | SIMPLE | m2 | eq_ref | 1 | Using index | | 1 | SIMPLE | m3 | ref | 4 | | | 1 | SIMPLE | m4 | eq_ref | 1 | Using index | |....|.............|..........|........|......|.............| | 1 | SIMPLE | m19 | ref | 4 | | | 1 | SIMPLE | m20 | eq_ref | 1 | Using index | +----+-------------+----------+--------+------+-------------+ www.percona.com
  • 37. #4: Handler operations mysql> show status like 'Handler%'; +----------------------------+-----------+ Run FLUSH STATUS before the query | Variable_name | Value | +----------------------------+-----------+ to reset the counters. | Handler_commit | 1 | | Handler_delete | 0 | | Handler_discover | 0 | FLUSH STATUS; | Handler_prepare | 0 | QUERY; | Handler_read_first | 0 | | Handler_read_key | 61341 | SHOW STATUS LIKE 'Handler%'; | Handler_read_last | 0 | | Handler_read_next | 198008370 | | Handler_read_prev | 0 | | Handler_read_rnd | 0 | | Handler_read_rnd_next | 0 | | Handler_rollback | 0 | | Handler_savepoint | 0 | | Handler_savepoint_rollback | 0 | | Handler_update | 0 | | Handler_write | 0 | +----------------------------+-----------+ www.percona.com
  • 38. #4: Problematic part LEFT JOIN (metadata_table1 m7 JOIN metadata_table2 m8 ON m7.attr1 = m8.attr1 AND m7.const = 'some text') ON tmain.id = m7.tmain_id AND tmain.important_flag=1; SELECT AVG(LENGTH(m1.somefield)) FROM metadata_table1 m1 JOIN metadata_table2 m2 ON m1.attr1 = m2.attr1 AND m1.const='some text' +----+-------------+----------+--------+-------+-------------+ | id | select_type | table | type | rows | Extra | +----+-------------+----------+--------+-------+-------------+ | 1 | SIMPLE | m1 | ref | 19764 | | | 1 | SIMPLE | m2 | eq_ref | 102 | Using index | +----+-------------+----------+--------+-------+-------------+ MySQL executes the join like this. One “problematic part takes roughly 3 minutes, so It adds up. www.percona.com
  • 39. #4: Rewrite SELECT tmain.id FROM main_table tmain LEFT JOIN metadata_table1 m1 ON tmain.id=m1.tmain_id LEFT JOIN metadata_table2 m2 ON m1.attr1 = m2.attr1 AND m1.const = 'some text' WHERE tmain.important_flag = 1; ● In order to leave out () LEFT JOIN has to be used to find non-matching elements. ● Runtime for query is 0.2 sec instead of 3 minutes. +----+-------------+----------+--------+------+-------------+ | id | select_type | table | type | rows | Extra | +----+-------------+----------+--------+------+-------------+ | 1 | SIMPLE | tmain | ref | 1786 | | | 1 | SIMPLE | m1 | ref | 4 | | | 1 | SIMPLE | m2 | eq_ref | 1 | Using index | +----+-------------+----------+--------+------+-------------+ www.percona.com
  • 40. #4: Similarly large queries ● After applying the tuning to the original query, I expected it t execute in a few seconds. ● It executed in 4 minutes instead. ● Used Handler operations looked good. ● Took a look at explain, and explain took 4 minutes also. ● This means the optimizer spends time in constructing the query plan. ● Solution: STRAIGHT_JOIN and/or optimizer_search_depth www.percona.com
  • 41. Annual Percona Live MySQL Conference and Expo Visit: http://www.percona.com/live/mysql-conference-2013/ www.percona.com
  • 42. Q&A
  • 43. Thank you for attention.