OUGLS 2016: How profiling works in MySQL

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |
How Profiling Works in MySQL
Georgi Kodinov
MySQL team lead
Oracle Confidential – Internal/Restricted/Highly Restricted

Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
Oracle Confidential – Internal/Restricted/Highly Restricted 2

Agenda
 Status variables
 PERFORMANCE_SCHEMA
 SYS schema
 Analyzing MySQL performance

Status variables
The traditional tool

Status variables: how do they work ?
• Global and/or session
• One mutex protected global copy
• Each session has a copy of it
– taken at session creation time
• Session variable updates are done on the THD copy: no synchronization
• Session results are aggregated to the global copy at thread termination
– Or FLUSH STATUS
• You can observe P_S.SESISON_STATUS (global) and P_S.GLOBAL_STATUS
(current session) or P_S.STATUS_BY_THREAD (all sessions)

Interesting status variables
Variable Scope What ?
Threads_connected Global Number of open connections
Created_tmp_disk_tables Global/Session Number of temp tables created by statements (sort etc)
Handler_read_first Global/Session Indication of index scans
Innodb_buffer_pool_wait_free Global Waits for pages to be flushed
Max_used_connections Global High watermark of the active connections
Slow_queries Global/Session Number of queries taking more than long_query_time
Select_full_join Global/Session Number of joins performing full table scans (indexes ?)

PERFORMANCE_SCHEMA

PERFORMANCE_SCHEMA
Overview of performance_schema Instrumentation
Queries





_history
Σ
_summary_
Objects
Objects
Objects
Events
Events
Events

How Does Instrumentation Work in MySQL ?
• Tracks the duration of various events
• Duration can be exposed with up to a picosecond precision
• The instrumentation of each event is configurable
• Data are stored into fixed size ring buffers in memory
• The collected data are directed into consumers
• Various aggregations are automatically calculated
• Data collection is lockless !

What is PERFORMANCE_SCHEMA ?
• A generic system database for tables with volatile data
• Powered by an actual built-in MySQL storage engine
• Majority of tables expose instrumented event latency
• There are configuration tables that define the extent of instrumentation

Instrument Naming
• Hierarchical : component/subcomponent/sub-subcomponent/…
• Currently (5.7) 6 top level components:
– Wait
– Stage
– Statement
– Idle
– Memory
– Transaction

Wait Instruments
Event Name Mask
Table locks wait/lock/table/%
Network I/O wait/io/socket/%
Table I/O wait/io/table/%
File I/O wait/io/file/%
Mutexes wait/synch/mutex/%
Read/Write locks wait/synch/rwlock/%
Conditions wait/synch/cond/%

Stage Instruments
• Stage/<code area>/<stage name>
• Code area is e.g. sql, myisam, etc
• Stage name is a set of well known query execution stages
– Preparing
– Executing
– Opening tables
– etc

Statement Instruments
Event Name Mask
RPC command statement/com/%
SQL statement statement/sql/%
Abstract statement statement/abstract/%

Idle Instrument
• Tracks idle time on inactive sockets
• Socket is inactive when waiting for a request from the client
• It’s different from waiting on an active socket !

Memory Instruments
• memory/<code area>/<instrument name>
• Code area is e.g. sql, myisam, etc
• Instrument name is a named memory allocation counter

Transaction Instrument
• No further components

Types of Instrumentation Data Tables
• Instances: socket_instances, file_instances, mutex_instances,
cond_instances, etc
– Expose a current list and state of instrumented objects
• Events: events_waits_current, events_statements_history, etc
– Access to the a timeline of event instrumentations
– Typically 3 kinds per instrument : *_current, *_history, *_history_long
• Summary: events_waits_summary_by_instance,
events_statements_summary_by_thread_by_event_name, etc
– Aggregate events so you don’t have to

How to Configure Instrumentation
• Compile Time
– Completely or selectively remove
• Server Startup
– Disable completely
– Selectively enable instruments and consumers
• Mid Flight
– Update the setup tables
– Changes effective immediately for all but setup_actors !

Setup Tables Examples
Table Explanation
setup_actors What worker threads to monitor
setup_consumers What PERFORMANCE_SCHEMA data tables to fill
setup_instuments What instruments to enable
setup_objects What tables to monitor
setup_timers Precision to use when collecting instrument timing

Introduction to Analyzing MySQL
Performance
using PERFORMANCE_SCHEMA

Profiling Considerations
• It is easy to just enable everything, but there are overhead concerns
– On busy systems mutexes can be locked millions of times per second
• It is best to just pick higher latency event types
– Statements, Stages, Table IO, File IO, maybe Network IO
– It is good to enable all %_current consumers, and statement history
• For concurrency/profiling debugging toggle other instruments on an as
needed basis
22

Profiling Types
• Once you’ve narrowed down what you’re interested in, there are two ways
to start monitoring
• View raw data in the summary views
– Gives you an overall picture of usage on the instance
• Snapshot data, and compute deltas over time
– Gives you an idea of the rates of change for the events

Analyzing Global Waits
• Some waits can include other waits
– Table IO latency may also include some mutex and file IO latency
• Wait times include concurrency across all threads
• Do not assume you can sum all events in global tables and compare to wall
clock times

Top Waits By Latency
mysql> select event_name,
-> count_star as count,
-> sys.format_time(sum_timer_wait) as total_latency,
-> sys.format_time(avg_timer_wait) as avg_latency,
-> sys.format_time(max_timer_wait) as max_latency
-> from events_waits_summary_global_by_event_name
-> where event_name != 'idle'
-> order by sum_timer_wait desc limit 5;
+--------------------------------------+----------+---------------+-------------+-------------+
| event_name | count | total_latency | avg_latency | max_latency |
+--------------------------------------+----------+---------------+-------------+-------------+
| wait/io/table/sql/handler | 21888502 | 1.27h | 208.06 us | 2.21 s |
| wait/io/file/innodb/innodb_data_file | 4276800 | 00:48:12.49 | 676.32 us | 1.49 s |
| wait/io/file/innodb/innodb_log_file | 1948199 | 00:25:24.36 | 782.45 us | 1.30 s |
| wait/io/file/myisam/kfile | 4566406 | 00:13:45.92 | 180.87 us | 1.17 s |
| wait/io/file/myisam/dfile | 1277589 | 00:05:46.23 | 271.01 us | 1.18 s |
+--------------------------------------+----------+---------------+-------------+-------------+

Some Mutex events That Can Affect Global Concurrency
(if high in list)
• wait/synch/mutex/innodb/buf_pool_mutex
– Increase innodb_buffer_pool_instances
• wait/synch/mutex/sql/Query_cache::structure_guard_mutex
– Look in to disabling the Query Cache
• wait/synch/mutex/myisam/MYISAM_SHARE::intern_lock
– Use Innodb …

Some File IO Events To Watch For (if high in list):
• wait/io/file/sql/FRM
– Tune table_open_cache / table_definition_cache
• wait/io/file/sql/file_parser (view definition parsing)
– If high on 5.5, upgrade to 5.6, (which can cache these like tables)
• wait/io/file/sql/query_log and wait/io/file/sql/slow_log
– Disable the general log
– Disable or tune what is logged to the slow log (decent long_query_time)

Top Files By Total IO
mysql> select sys.format_path(file_name) as file,
-> count_read,
-> sys.format_bytes(sum_number_of_bytes_read) as total_read,
-> sys.format_bytes(IFNULL(sum_number_of_bytes_read / count_read, 0)) as avg_read,
-> count_write,
-> sys.format_bytes(sum_number_of_bytes_write) as total_written,
-> sys.format_bytes(IFNULL(sum_number_of_bytes_write / count_write, 0)) as avg_write,
-> sys.format_bytes(sum_number_of_bytes_read + sum_number_of_bytes_write) as total,
-> IFNULL(ROUND(100-((sum_number_of_bytes_read/(sum_number_of_bytes_read + sum_number_of_bytes_write))*100), 2), 0.00) as
write_pct
-> from file_summary_by_instance
-> order by (sum_number_of_bytes_read + sum_number_of_bytes_write) desc limit 5;
+----------------------------------+------------+------------+-----------+-------------+---------------+-----------+-------------+-----------+
| file | count_read | total_read | avg_read | count_write | total_written | avg_write | total | write_pct |
+----------------------------------+------------+------------+-----------+-------------+---------------+-----------+-------------+-----------+
| @@datadir/ibdata1 | 888 | 15.84 MiB | 18.27 KiB | 1089824 | 61.99 GiB | 59.64 KiB | 62.00 GiB | 99.98 |
| @@datadir/mem__events/events.ibd | 114 | 1.80 MiB | 16.14 KiB | 117370 | 2.14 GiB | 19.14 KiB | 2.14 GiB | 99.92 |
| @@datadir/cerberus-bin.000010 | 296103 | 1.01 GiB | 3.59 KiB | 362852 | 1.00 GiB | 2.89 KiB | 2.01 GiB | 49.63 |
| @@datadir/ib_logfile0 | 6 | 68.00 KiB | 11.33 KiB | 506837 | 1.46 GiB | 3.03 KiB | 1.46 GiB | 100.00 |
| @@datadir/ib_logfile1 | 0 | 0 bytes | 0 bytes | 476961 | 1.45 GiB | 3.19 KiB | 1.45 GiB | 100.00 |
+----------------------------------+------------+------------+-----------+-------------+---------------+-----------+-------------+-----------+
High IO on InnoDB per-tablespace tables
can show candidates for a separate
mountpoint/disk using “DATA
DIRECTORY” in CREATE TABLE

Analyzing User Activity
• All event_% summaries are exposed with a number of dimensions
• To analyze connection activity you can do this in 4 ways
– By User
– By Host
– By Account (User@Host)
– By Thread
• The follow examples are by user, but could be replaced with the host,
account or thread summary views

Top Users By Statement Latency
mysql> select user,
-> sum(count_star) as statements,
-> sys.format_time(sum(sum_timer_wait)) as total_latency,
-> sys.format_time(sum(sum_timer_wait) / sum(count_star)) as avg_latency
-> from events_statements_summary_by_user_by_event_name
-> where user is not null
-> group by user
-> order by sum(sum_timer_wait) desc;
+------+------------+---------------+-------------+
| user | statements | total_latency | avg_latency |
+------+------------+---------------+-------------+
| root | 7229032 | 15.17h | 7.55 ms |
| mark | 3072 | 1.77 s | 575.29 us |
+------+------------+---------------+-------------+

Top Users By IO Latency
mysql> select user, sum(count_star) as count,
-> sys.format_time(sum(sum_timer_wait)) as total_latency
-> from events_waits_summary_by_user_by_event_name
-> where event_name like 'wait/io/file/%'
-> and user is not null
-> group by user
-> order by sum(sum_timer_wait) desc;
+------+----------+---------------+
| user | count | total_latency |
+------+----------+---------------+
| root | 10892980 | 00:46:19.67 |
| mark | 20043 | 346.79 ms |
+------+----------+---------------+
Replace with some other
pattern here for
“Top User by wait class”

SYS SCHEMA

What is SYS Schema ?
• A set of objects that help interpreting data collected by
PERFORMANCE_SCHEMA
• Includes:
– Views that summarize Performance Schema data into more easily understandable
form.
– Stored procedures that perform operations such as Performance Schema
configuration and generating diagnostic reports.
– Stored functions that query Performance Schema configuration and provide
formatting services.
• Installed together with the server since 5.7
– CALL sys.ps_setup_reset_to_default(TRUE);

MySQL SYS Functions
• Make the raw data more readable to a human
– format_time() / format_bytes()
• Compress data for CLI output
– format_statement() / format_path()
• Extract some data to assist with JOIN in certain cases
– extract_[schema|table]_from_file_name()
• Other functions
– ps_is_account_enabled() / ps_thread_stack()

MySQL SYS Views
• Reference set of views solving various admin use cases
• Build upon both Performance Schema and INFORMATION_SCHEMA
• Both formatted and raw views are available
– All raw views are prefixed with x$
– Allows tooling to poll raw views, but humans to use normal ones

• High level user overview
• Breakdowns of IO usage
• Drill in to stages per user
• Drill in to statement details
per user
mysql> show tables like 'user%';
+-----------------------------------+
| Tables_in_sys (user%) |
+-----------------------------------+
| user_summary |
| user_summary_by_file_io |
| user_summary_by_file_io_type |
| user_summary_by_stages |
| user_summary_by_statement_latency |
| user_summary_by_statement_type |
+-----------------------------------+
User Summary Views

• IO breakdown by thread
• Global summaries by file and wait
class, by both bytes and latency
• Stream of last raw file IO stats
mysql> show tables like 'io_%';
+------------------------------+
| Tables_in_sys (io_%) |
+------------------------------+
| io_by_thread_by_latency |
| io_global_by_file_by_bytes |
| io_global_by_file_by_latency |
| io_global_by_wait_by_bytes |
| io_global_by_wait_by_latency |
| latest_file_io |
+------------------------------+
IO Summary Views

• Object overview
• Table usage stats
• Index usage stats
mysql> show tables like 'schema%';
+-------------------------------------+
| Tables_in_sys (schema%) |
+-------------------------------------+
| schema_index_statistics |
| schema_object_overview |
| schema_table_statistics |
| schema_table_statistics_with_buffer |
| schema_tables_with_full_table_scans |
| schema_unused_indexes |
+-------------------------------------+
Schema Analysis Views

• Buffer usage by schema
• Buffer usage by table
mysql> show tables like 'innodb%';
+-------------------------------+
| Tables_in_sys (innodb%) |
+-------------------------------+
| innodb_buffer_stats_by_schema |
| innodb_buffer_stats_by_table |
+-------------------------------+
InnoDB Buffer Usage Views

• Statement overview
• Find statements by errors, full
scans, sorting, temporary tables
• Find the statements with the
longest runtimes
mysql> show tables like 'statement%';
+---------------------------------------------+
| Tables_in_sys (statement%) |
+---------------------------------------------+
| statement_analysis |
| statements_with_errors_or_warnings |
| statements_with_full_table_scans |
| statements_with_runtimes_in_95th_percentile |
| statements_with_sorting |
| statements_with_temp_tables |
+---------------------------------------------+
Statement Analysis Views

• Wait summaries by class and per
instrument globally
• Wait details per user
mysql> show tables like 'wait%';
+------------------------------------+
| Tables_in_sys (wait%) |
+------------------------------------+
| wait_classes_global_by_avg_latency |
| wait_classes_global_by_latency |
| waits_by_user_by_latency |
| waits_global_by_latency |
+------------------------------------+
Wait Analysis Views

MySQL SYS Procedures - P_S Analysis Helpers
• Dump the data to create a graph dot file for a thread trace
– ps_trace_thread()
• Look in the statement history table trying to capture more info
– ps_trace_statement_digest()
• Reset all summary data
– ps_truncate_all_tables()

Tracing Statement Digests
• ps_trace_statement_digest() analyzes live traffic looking for certain
statement digest
• Captures statistics on each matching statement it finds
• Returns a report of the statistics
– An overall summary
– A break down for the longest running example
– An EXPLAIN (if the statement is not truncated)

Tracing Threads
• ps_trace_thread() monitors a specific thread for a period
• Captures as much information on the thread activity as possible
• Returns a “dot” formatted file, that can graph the event hierarchy as
previously seen
– http://en.wikipedia.org/wiki/DOT_(graph_description_language)

MySQL SYS Procedures - P_S Setup Helpers
• ps_setup_show_disabled() / ps_setup_show_enabled()
• ps_setup_disable_thread() / ps_setup_enable_thread()
• ps_setup_disable_background_threads() /
ps_setup_enable_background_threads()
• ps_setup_disable_instrument() / ps_setup_enable_instrument()
• ps_setup_disable_consumer() / ps_setup_enable_consumer()
• ps_setup_save() / ps_setup_reload_saved()
• ps_setup_reset_to_default()
• ps_truncate_all_tables()

OUGLS 2016: How profiling works in MySQL

OUGLS 2016: How profiling works in MySQL

More Related Content

What's hot (20)

Similar to OUGLS 2016: How profiling works in MySQL (20)

More from Georgi Kodinov (20)

Recently uploaded (20)

OUGLS 2016: How profiling works in MySQL

Editor's Notes