Need for Speed: MySQL Indexing
4 likes1,378 views
This document discusses MySQL indexing best practices. It explains that indexes can improve query performance but also waste space and time if unnecessary. The right balance must be found to achieve fast queries using an optimal index set. It provides examples of how to create indexes and how they can be used to optimize WHERE, JOIN, ORDER BY, and other clauses. It also discusses avoiding unnecessary or redundant indexes.
Need for Speed: MySQL Indexing
- 1. www.fromdual.com 1 / 29 Need for Speed: MySQL Indexing Percona Live 2013, November 11 – 12, London Oli Sennhauser Senior MySQL Consultant at FromDual GmbH [email protected]
- 2. www.fromdual.com 2 / 29 About FromDual GmbH ● FromDual provides neutral and independent: ● Consulting for MySQL, Percona Server, MariaDB ● Support for all MySQL and Galera Cluster ● Remote-DBA Services ● MySQL Training ● Oracle Silver Partner (OPN) www.fromdual.com
- 3. www.fromdual.com 3 / 29 Our customers
- 4. www.fromdual.com 4 / 29 MySQL and Indexing ● MySQL documentation says: The best way to improve the performance of SELECT operations is to create indexes on one or more of the columns that are tested in the query. ● Great! But: Unnecessary indexes waste space and waste time to determine which indexes to use. You must find the right balance to achieve fast queries using the optimal set of indexes. ● ... hmm so we have to think a bit... :-(
- 5. www.fromdual.com 5 / 29 What is an Index? ● Adams, Douglas: The Hitchhiker's Guide to the Galaxy? ● Sennhauser, Oli, Uster?
- 6. www.fromdual.com 6 / 29 What is an Index technically?
- 7. www.fromdual.com 7 / 29 MySQL uses indexes: ● To enforce uniqueness (PRIMARY KEY, UNIQUE KEY) ● To fast access and filter rows (WHERE) ● To perform joins fast (JOIN) ● To find MIN() and MAX() values ● For sorting and grouping (ORDER BY, GROUP BY) ● To avoid joins by using covering indexes ● To enforce FOREIGN KEY Constraints (FOREIGN KEY)
- 8. www.fromdual.com 8 / 29 WHERE clause 1 SELECT * FROM customers WHERE name = 'No ClSuHeO Wo fC RMEyASTQEL TLALBCL'E; customersG CREATE TABLE `customers` ( `customer_id` smallint(5) unsigned , `name` varchar(64) DEFAULT NULL , PRIMARY KEY (`customer_id`) EXPLAIN ) SELECT * FROM customers WHERE name = 'No Clue of MySQL LLC'; +-----------+------+---------------+------+-------+-------------+ | table | type | possible_keys | key | rows | Extra | +-----------+------+---------------+------+-------+-------------+ | customers | ALL | NULL | NULL | 31978 | Using where | +-----------+------+---------------+------+-------+-------------+
- 9. www.fromdual.com 9 / 29 How to create and Index? ALTER TABLE … ● ADD PRIMARY KEY (id); ● ADD UNIQUE KEY (uuid); ● ADD FOREIGN KEY (customer_id) REFERENCES customers (customer_id); ● ADD INDEX (last_name, first_name); ● ADD INDEX pre_ind (hash(8)); ● ADD FULLTEXT INDEX (last_name, first_name);
- 10. www.fromdual.com 10 / 29 WHERE clause 2 ALTER TABLE customers ADD INDEX (name); CREATE TABLE `customers` ( `customer_id` smallint(5) unsigned Gain: 20 ms → 5 ms , `name` varchar(64) DEFAULT NULL , PRIMARY KEY (`customer_id`) , KEY `name` (`name`) ) +-----------+------+---------------+------+---------+-------+------+ | table | type | possible_keys | key | key_len | ref | rows | +-----------+------+---------------+------+---------+-------+------+ | customers | ref | name | name | 67 | const | 1 | +-----------+------+---------------+------+---------+-------+------+
- 11. www.fromdual.com 11 / 29 JOIN clause EXPLAIN SELECT * FROM customers AS c JOIN orders AS o ON c.customer_id = o.customer_id WHERE c.name = 'No Clue of MySQL LLC'; +-------+------+---------------+------+---------+-------+---------+ | table | type | possible_keys | key | key_len | ref | rows | +-------+------+---------------+------+---------+-------+---------+ | c | ref | PRIMARY,Gain: name 450 | name ms | → 67 6 ms | const | 1 | | o | ALL | NULL | NULL | NULL | NULL | 1045105 | +-------+------+---------------+------+---------+-------+---------+ ALTER TABLE orders ADD INDEX (customer_id); +-------+------+---------------+-------------+---------+---------------+------+ | table | type | possible_keys | key | key_len | ref | rows | +-------+------+---------------+-------------+---------+---------------+------+ | c | ref | PRIMARY,name | name | 67 | const | 1 | | o | ref | customer_id | customer_id | 3 | c.customer_id | 8 | +-------+------+---------------+-------------+---------+---------------+------+
- 12. www.fromdual.com 12 / 29 For sorting/grouping tables ORDER BY, GROUP BY EXPLAIN SELECT * FROM contacts AS c WHERE last_name = 'Sennhauser' ORDER BY last_name, first_name; +-------+------+-----------+------+----------------------------------------------------+ | table | type | key | +-------+------+-----------+------+----------------------------------------------------+ Gain: rows | Extra 20 ms → 7 ms | | c | ref | last_name | 1561 | Using index condition; Using where; Using filesort | +-------+------+-----------+------+----------------------------------------------------+ ALTER TABLE contacts ADD INDEX (last_name, first_name); +----------+------+-------------+------+--------------------------+ | table | type | key | rows | Extra | +----------+------+-------------+------+--------------------------+ | contacts | ref | last_name_2 | 1561 | Using where; Using index | +----------+------+-------------+------+--------------------------+
- 13. www.fromdual.com 13 / 29 Covering Indexes EXPLAIN SELECT customer_id, amount FROM orders AS o WHERE customer_id = 59349; +-------+------+-------------+------+-------+ | table | type | key | rows | Extra | +-------+------+-------------+------+-------+ | o | ref | customer_id So | what? 15 | NULL | +-------+------+-------------+------+-------+ ALTER TABLE orders ADD INDEX (customer_id, amount); +-------+------+---------------+------+-------------+ | table | type | key | rows | Extra | +-------+------+---------------+------+-------------+ | o | ref | customer_id_2 | 15 | Using index | +-------+------+---------------+------+-------------+
- 14. www.fromdual.com 14 / 29 Benefit of Covering Indexes ● Why are Covering Indexes beneficial
- 15. www.fromdual.com 15 / 29 How-to find missing indexes? ● ER Diagram? :-( ● Most of them rely to you business logic... ● How-to FIND? ● Slow Query Log ● MySQL Variables: ● Since v5.1 on-line! +-------------------------------+----------+ | Variable_name | Value | +-------------------------------+----------+ | log_queries_not_using_indexes | ON | | long_query_time | 0.250000 | | min_examined_row_limit | 100 | | slow_query_log | ON | | slow_query_log_file | slow.log | +-------------------------------+----------+
- 16. www.fromdual.com 16 / 29 Indexes are not only good ● Indexes use space (Disk, hot data in RAM!) ● Indexes use time to maintain (CPU, RAM, I/O) ● Optimizer needs time to determine which indexes to use. ● Sometimes optimizer is completely confused and does wrong decisions if too many (similar) indexes are there. → You must find the right balance to achieve fast queries using the optimal set of indexes.
- 17. www.fromdual.com 17 / 29 Smaller indexes faster queries ● Better fit into memory (less I/O) ● Higher data density (rows/block) ● Less CPU cycles (to crawl through) ● Prefixed indexes: ADD INDEX pre_ind (hash(8));
- 18. www.fromdual.com 18 / 29 Avoid indexes ● Avoid redundant (and thus unnecessary ) indexes ● How does it happen? ● Developer 1: Creates a Foreign Key constraint → done ● Developer 2: Ouu! Query is slow → Oli told me to create an index! → done ● Developer 3: Ouu! Query is slow → Developer 2 is stupid! → Create and index → done ● Frameworks vs. Developer ● Upgrade process vs. Developer ● Avoid indexes which are not used / needed
- 19. www.fromdual.com 19 / 29 How to find such indexes? SHOW CREATE TABLE ...G mysqldump nodata > structure_dump.sql ● Since MySQL 5.6: PERFORMANCE_SCHEMA ● Percona Server / MariaDB: Userstats ● http://fromdual.com/mysql-performance-schema-hints SELECT object_schema, object_name, index_name FROM performance_schema.table_io_waits_summary_by_index_usage WHERE index_name IS NOT NULL AND count_star = 0 ORDER BY object_schema, object_name;
- 20. www.fromdual.com 20 / 29 Avoid partial redundant indexes ● INDEX (city, last_name, first_name) ● INDEX (city, last_name) ● INDEX (city) ● INDEX (last_name, city) ??? ● INDEX (first_name, last_name) !!!
- 21. www.fromdual.com SELECT status, COUNT(*) 21 / 29 Bad selectivity ● Remove indexes with bad selectivity (~= low cardinality) ● Candidates are: ● status ● gender ● active ● How to find if field has bad selectivity? Indexes (and Joins) are expensive!!! ● Break even between 15% and 66% FROM orders GROUP BY status; +--------+--------+ | status | cnt | +--------+--------+ | 0 | 393216 | | 1 | 262144 | | 2 | 12 | | 3 | 36 | | 4 | 24 | | 5 | 4 | | 6 | 8 | +--------+--------+ ● Lets see if the MySQL optimizer knows about it... :-)
- 22. www.fromdual.com SELECT status, COUNT(*) +--------+--------+ | status | cnt | +--------+--------+ | 0 | 393216 | | 1 | 262144 | | 2 | 12 | | 3 | 36 | | 4 | 24 | | 5 | 4 | | 6 | 8 | +--------+--------+ 22 / 29 Optimizer is wrong! SELECT * FROM orders WHERE status = 2; +--------+------+---------------+--------+------+ | table | type | possible_keys | key | rows | +--------+------+---------------+--------+------+ | orders | ref | status | status | 12 | +--------+------+---------------+--------+------+ SELECT * FROM orders WHERE status = 0; 1.43 s +--------+------+---------------+--------+--------+ | table | type | possible_keys | key | rows | +--------+------+---------------+--------+--------+ | orders | ref | status | status | 327469 | +--------+------+---------------+--------+--------+ FROM orders GROUP BY status; SELECT * FROM orders IGNORE INDEX (STATUS) WHERE status = 0; 0.44 s +--------+------+---------------+------+--------+ | table | type | possible_5.6.12 keys (after | key analyze | rows table) | +--------+------+---------------+------+--------+ | orders | ALL | NULL | NULL | 654939 | +--------+------+---------------+------+--------+
- 23. www.fromdual.com 23 / 29 InnoDB PK and SK ● InnoDB has ● Primary Keys and ● Secondary Keys
- 24. www.fromdual.com 24 / 29 Clustered Index ● InnoDB: Data = Leaf of Primary Key ● We call this an Index Clustered Table (IOT) → Data are sorted like PK (key is sorted)! → PK influences Locality of data (physical location) ● AUTO_INCREMENT ~= sorting by time! ● Good for many things ● where hot data = recent data ● Bad for time series ● Where hot data = per item data
- 25. www.fromdual.com 25 / 29 Example: InnoDB A_I ts v_id xpos ypos ... 1 17:30 #42 x, y, ... 2 17:30 #43 x, y, ... 3 17:30 #44 x, y, ... ... 2001 17:32 #42 x, y, ... 2002 17:32 #43 x, y, ... 2003 17:32 #44 x, y, ... #42 every 2' over the last 3 days ~ 2000 rows ~ 200 kbyte Q1: Δ in rows? A1: 1 row ~ 100 byte Q2: Δ in bytes? Q3: Default InnoDB block size? default: 16 kbyte Q4: Avg. # of rows of car #42 in 1 InnoDB block? A2: 3 d and 720 pt/d → ~2000 pt ~ 2000 rec ~ 2000 blk Q5: How long will this take and why (32 Mbyte)? ~ 2000 IOPS ~ 10s random read!!! S: All in RAM or strong I/O system or …? ~ 1 2000 trucks
- 26. www.fromdual.com 26 / 29 InnoDB PK safes the day! ts v_id xpos ypos ... 17:30 #42 x, y, ... 17:32 #42 x, y, ... 17:34 #42 x, y, ... ... #42 every 2' 17:30 #43 x, y, ... 17:32 #43 x, y, ... 17:34 #43 x, y, ... over the last 3 days Q1: Avg. # of rows of car #42 in 1 InnoDB block? A1: 3 d and 720 pt/d → ~2000 pt ~ 2000 rec ~ 20 blk Q2: How long will this take and why (320 kbyte)? ~ 1-2 IOPS ~ 10-20 ms sequential read! S: Wow f=50 faster! Any drawbacks? ~ 120 2000 trucks 17:30 #44 x, y, ... ...
- 27. www.fromdual.com 27 / 29 Index hints ● MySQL optimizer is sometimes wrong! ● We have to help (= hint) him... ● Index hints are: ● USE INDEX (ind1, ind2) ● Only consider these indexes ● FORCE INDEX (ind3) ● Use this index without considering anything else ● IGNORE INDEX (ind1, ind3) ● Do NOT consider these indexes but everything else ● Hints should be used only as a last resort
- 28. www.fromdual.com 28 / 29 MySQL Variables ● MySQL variables influencing index use ● MyISAM: key_buffer_size ● InnoDB: innodb_buffer_pool_size / innodb_buffer_pool_instances ● InnoDB Change Buffer ● innodb_change_buffer_max_size ● innodb_change_buffering ● Adaptive Hash Index (AHI) ● MySQL 5.6.3 / 5.5.14 index length 767 → 3072 bytes ● innodb_large_prefix
- 29. www.fromdual.com 29 / 29 Q & A Questions ? Discussion? We have time for some face-to-face talks... ● FromDual provides neutral and independent: ● Consulting ● Remote-DBA ● Support for MySQL, Galera, Percona Server and MariaDB ● Training www.fromdual.com/presentations