![www.percona.com
What MySQL DBA can do with gdb
โ Check stack traces (and variables), per thread:
thread apply all bt [full]
โ Print variables, up to complex one:
thread 1
print do_command::thd->query_string.string.str
โ Set new values for variables (global and per thread, even those formally
read-only in MySQL while itโs running):
set max_connections=5000
set opt_log_slave_updates=1
โ Call functions (that may do complex changes):
call rpl_filter->add_do_db(strdup("hehehe"))
โ Set breakpoints and watchpoints
โ Work interactively or use gdb as a command line utility (-batch)
โ Use macros/Python scripting, moreโฆ
โ All these may not work, fail, hang, crash, produce obscure errorsโฆ
โ You have to read and understand the source code
11](https://image.slidesharecdn.com/gdbbasicsformysqldbasopenfest2017final-171102202321/85/Gdb-basics-for-my-sql-db-as-openfest-2017-final-11-320.jpg)
![www.percona.com
pt-pmp (Poor Manโs Profiler)
โ http://www.percona.com/doc/percona-toolkit/2.2/pt-pmp.html
pt-pmp [-i 1] [-s 0] [-b mysqld] [-p pidofmysqld] [-l 0] [-k file] [--version]
โ It is based on original idea by Domas, http://poormansprofiler.org/
โ One of the recent examples how it is used: Bug #78277 - InnoDB deadlock,
thread stuck on kernel calls from transparent page compression, โOpenโ
โ When mysqld hangs or is slow, you can get some insight quickly: for
example, Bug #86902 (MySQL server may hang when we turn off binlog...)
โ When there are stalls, use pt-pmp to find out why (or what threads mostly
do at the moment): Bug #69810
โ Use in production as a last resort (may hang mysqld, --SIGCONT)
โ pt-pmp surely slows server down :) Hints:
โ https://bugs.launchpad.net/percona-toolkit/+bug/1320168 - partial
workaround
โ Use quickstack instead of gdb (check this discussion)
12](https://image.slidesharecdn.com/gdbbasicsformysqldbasopenfest2017final-171102202321/85/Gdb-basics-for-my-sql-db-as-openfest-2017-final-12-320.jpg)
![www.percona.com
Multi-threaded mysqld process and gdb
โ process/thread/frame concepts:
(gdb) thread 2
[Switching to thread 2 (Thread 0x7fe771550700 (LWP 2544))]
#0 0x0000000000605774 in Item_func_numhybrid::val_int (
this=<value optimized out>)
at /home/openxs/bzr2/percona-5.6/sql/item_func.cc:1013
1013 }
(gdb) bt
...
#12 0x00000000006f8a45 in dispatch_command (command=COM_QUERY,
thd=0x7fe760f94000, packet=0x7fe77154fac0 "", packet_length=0)
at /home/openxs/bzr2/percona-5.6/sql/sql_parse.cc:1434
...
(gdb) frame 12
#12 0x00000000006f8a45 in dispatch_command (command=COM_QUERY,
thd=0x7fe760f94000, packet=0x7fe77154fac0 "", packet_length=0)
at /home/openxs/bzr2/percona-5.6/sql/sql_parse.cc:1434
warning: Source file is more recent than executable.
1434 mysql_parse(thd, thd->query(), thd->query_length(), &parser_state);
(gdb) p thd->query_string.string.str
$2 = 0x7fe75301d010 "select benchmark(5", '0' <repeats 13 times>, ", 2*2)"
โ https://sourceware.org/gdb/onlinedocs/gdb/Variables.html
13](https://image.slidesharecdn.com/gdbbasicsformysqldbasopenfest2017final-171102202321/85/Gdb-basics-for-my-sql-db-as-openfest-2017-final-13-320.jpg)
![www.percona.com
Real life case: attaching to alive mysqld
This is how it goes:
[root@centos openxs]# mysql -uroot -e "show variables like
'innodb_autoinc_lock_mode'"
+--------------------------+-------+
| Variable_name | Value |
+--------------------------+-------+
| innodb_autoinc_lock_mode | 0 |
+--------------------------+-------+
[root@centos openxs]# mysql -uroot -e "set global
innodb_autoinc_lock_mode=1"
ERROR 1238 (HY000) at line 1: Variable 'innodb_autoinc_lock_mode' is a
read only variable
[root@centos openxs]# gdb -ex "set innobase_autoinc_lock_mode=1" -batch -p
`pidof mysqld`
โฆ
[Thread debugging using libthread_db enabled]
0x00007ff31d6830d3 in poll () from /lib64/libc.so.6
โฆ check the variable value again now
[root@centos openxs]# ps aux | grep mysqld
[root@centos openxs]# kill -SIGCONT `pidof mysqld`
17](https://image.slidesharecdn.com/gdbbasicsformysqldbasopenfest2017final-171102202321/85/Gdb-basics-for-my-sql-db-as-openfest-2017-final-17-320.jpg)
![www.percona.com
How to study InnoDB locks with gdb
โ Read the code (or blogs, or backtraces) to find out what
functions are called when InnoDB locks are requested:
โ lock_table - table level locks
โ lock_rec_lock - row level locks
โ Make sure there is debug info for mysqld binary you use
โ Attach gdb to running mysqld process in test env:
[root@centos ~]# gdb -p `pidof mysqld`
...
(gdb) b lock_table
...
(gdb) b lock_rec_lock
...
(gdb) c
โ Run SQL you want to study and check sequence of calls,
backtraces, variables...
18](https://image.slidesharecdn.com/gdbbasicsformysqldbasopenfest2017final-171102202321/85/Gdb-basics-for-my-sql-db-as-openfest-2017-final-18-320.jpg)
![www.percona.com
How to study metadata locks with gdb
โ Read the code (or blogs, or backtraces) to find out what
functions are called when metadata locks are requested:
โ MDL_request::init - metadata lock request
โ MDL_context::aquire_lock - attempt to acquire lock
โ Attach gdb to running mysqld process in test env:
[root@centos ~]# gdb -p `pidof mysqld`
...
(gdb) b MDL_request::init
...
(gdb) c
โ Run SQL you want to study and check sequence of calls,
backtraces, variables...
19](https://image.slidesharecdn.com/gdbbasicsformysqldbasopenfest2017final-171102202321/85/Gdb-basics-for-my-sql-db-as-openfest-2017-final-19-320.jpg)
![www.percona.com
Some gdb versions have Python, and this helps
โ If you like and know Python and have gdb linked with it (try py print(1+1))...
โ Use ~/.gdbinit file for complex Python macros
โ http://mysqlbugs.blogspot.com/2012/09/how-to-obtain-all-executing-queries.
html - Shane Bester on simplified navigation over threads, nice printing of
selected values etc,
โ https://mariadb.org/duel-gdb-vs-linked-lists-trees-hash-tables/ - โDuel: gdb
vs. linked lists, trees, and hash tablesโ. Sergei Golubchik on simplified
way to apply โsomethingโ (like print) to all/selected items of arrays, linked
lists etc. Check https://github.com/vuvova/gdb-tools
โ https://mariadb.org/making-life-prettier-gdb-prettyprinting-api/ - โMaking life
prettier with gdb PrettyPrinting APIโ. Sergei Golubchik on how to use
Python classes to pretty print almost anything inside MySQL code in gdb
โ Make sure to check if you have Python 2 or 3 in gdb! (pip vs pip3 etc):
[openxs@fc23 ~]$ ldd `which gdb` | grep pyt
libpython3.5m.so.1.0 => /lib64/libpython3.5m.so.1.0
(0x00007fee3957d000)
27](https://image.slidesharecdn.com/gdbbasicsformysqldbasopenfest2017final-171102202321/85/Gdb-basics-for-my-sql-db-as-openfest-2017-final-27-320.jpg)
Gdb basics for my sql db as (openfest 2017) final
- 1. gdb basics for MySQL DBAs or Using gdb to study MySQL internals and as a last resort Valerii Kravchuk, Principal Support Engineer, MariaDB [email protected] 1
- 2. www.percona.com Who am I? Valerii (aka Valeriy) Kravchuk: โ MySQL Support Engineer in MySQL AB, Sun and Oracle, 2005 - 2012 โ Principal Support Engineer in Percona, 2012 - 2016 โ Principal Support Engineer in MariaDB Corporation since March 2016 โ http://mysqlentomologist.blogspot.com - my blog about MySQL (a lot about MySQL bugs, but some HowTos as well) โ https://www.facebook.com/valerii.kravchuk - my Facebook page, a lot about MySQL (mostly bugsโฆ) โ http://bugs.mysql.com - my personal playground. 316 bugs reported in total, 8 in 2017 so far โ I like FOSDEM, see slides from my previous talks: โ http://www.slideshare.net/valeriikravchuk1/fosdem2015-gdb-tips-and-tricks-for-my-sql-db-as โ http://www.slideshare.net/ValeriyKravchuk/more-on-gdb-for-my-sql-db-as-fosdem-2016 โ https://www.slideshare.net/ValeriyKravchuk/applying-profilers-to-my-sql-fosdem-2017 2
- 3. www.percona.com What is this session about? โ Some historical remarks and URLs to known use cases/blog posts about gdb and MySQL troubleshooting โ Multi-threaded executables and gdb (threads, frames, variables) โ Basic gdb commands and โtricksโ โ Basic usage of pt-pmp tool, when to use โ Important MySQL data structures to explore: โ THD (all the details about thread created for connection) โ HASH and hash tables in MySQL โ Maybe some more... โ Using gdb to study InnoDB locks, table locks and metadata locks โ Using gdb to study server variables and user variables at session level โ A couple of real life use cases, working with core dump and alive mysqld โ Few details on using Python in gdb, https://github.com/vuvova/gdb-tools etc โ Discussion 3
- 4. www.percona.com Usually gdb is used by developers, to study core dumps... โ Mostly like this: gdb /path/to/mysqld /path/to/coredump โ Bug #76432 - โhandle_fatal_signal (sig=11) in __strlen_sse2_pminub on CHANGE MASTERโ โ Bug #69898 - โchange_master() invokes ha_innobase::truncate() in a DML transactionโ - a lot of useful gdb-related reading inside (check how Marko uses call rec_print_old(stderr,$8.frame+0x16e) etc) See also related Bug #69825 and how bug reporter attached full backtrace in related Bug #73155 4
- 5. www.percona.com ...or (surprise!) to debug their code โ Running โunder gdbโ: gdb --args bin/mysqlcheck -u root -p -S/tmp/mysql.sock --all-databases --optimize (gdb) thread apply all bt โ Attaching gdb to the process already running: gdb -p `pidof mysqld` โ Some examples: โ Percona Server Bug #1483251 - โsavepoints and replicationโ. Check how Vlad Lesin uses backtrace to understand the reason of the bug โ Percona Server Bug #1426345 - โPrepared statements in stored procedures crash query response time pluginโ. Check how Nickolay Ihalainen pinpoint the root cause of the bug by comparing values of various variables in gdb 5
- 6. www.percona.com More examples here on how MariaDB developers use gdb โ MDEV-13797 - InnoDB may hang if shutdown is initiated soon after startup, while rolling back recovered incomplete transactions โ MDEV-12052 - our buildbot tries to get backtrace for all threads for crash โ MDEV-12413 - be ready to run some gdb commands when you report bugs โ MDEV-14051 - this is how developers use โadvancedโ gdb. See Bug #88150 โ MDEV-13787 - real crash (fixed) โ MDEV-11044 - dumping pages etc, can't repeat, but still some useful details 6
- 7. www.percona.com Disassembling in gdb Reading symbols from mariadb-10.1.19-linux-x86_64/bin/mysqld...done. (gdb) info line row_sel_store_mysql_rec Line 2945 of "/home/buildbot/buildbot/build/storage/xtradb/row/row0sel.cc" starts at address 0x9c5060 <row_sel_store_mysql_rec(unsigned char*, row_prebuilt_t*, rec_t const*, ulint, dict_index_t const*, ulint const*)> and ends at 0x9c5077 <row_sel_store_mysql_rec(unsigned char*, row_prebuilt_t*, rec_t const*, ulint, dict_index_t const*, ulint const*)+23>. (gdb) disassemble row_sel_store_mysql_rec Dump of assembler code for function row_sel_store_mysql_rec(unsigned char*, row_prebuilt_t*, rec_t const*, ulint, dict_index_t const*, ulint const*): 0x00000000009c5060 <+0>: push %rbp .. 0x00000000009c525d <+509>: callq 0x964300 <mem_heap_block_free(mem_block_info_t*, mem_block_info_t*)> 0x00000000009c5262 <+514>: jmpq 0x9c5131 <row_sel_store_mysql_rec(unsigned char*, row_prebuilt_t*, rec_t const*, ulint, dict_index_t const*, ulint const*)+209> End of assembler dump. (gdb) list *0x9c51fe 0x9c51fe is in row_sel_store_mysql_rec(unsigned char*, row_prebuilt_t*, rec_t const*, ulint, dict_index_t const*, ulint const*) (/home/buildbot/buildbot/build/storage/xtradb/row/row0sel.cc:2988). 7
- 8. www.percona.com But production DBAs also may benefit from gdb! โ First of all, gdb allows to inspect the values of variables in the mysqld process memory, and thus you can check some details about user threads and statements executed that may not be easily available via SQL (missing feature, canโt connect, hangs, bug) โ Also gdb allows to change the values of variables, both global and session ones (missing feature, read only ones) directly or indirectly (by calling functions in the code) โ Finally, attaching gdb allows to get a backtrace for further study of the root cause of the problem 8
- 9. www.percona.com Domas is famous for these tricks... โ http://dom.as/2009/02/15/poor-mans-contention-profiling/ - this is what ended up as http://poormansprofiler.org/ and pt-pmp โ http://dom.as/2009/07/30/evil-replication-management/ - mysql> system gdb -p $(pidof mysqld) -ex "set opt_log_slave_updates=1" -batch โ http://dom.as/2010/01/02/read-ahead/ - gdb -ex "set srv_startup_is_before_trx_rollback_phase=1" -batch -p $(pidof mysqld) โ http://dom.as/2009/12/29/when-bad-things-happen/ 9
- 10. www.percona.com More examples of gdb use for MySQL DBAs โ Remember the names: Domas Mituzas, Shane Bester, Roel Van De Paar, Mark Callaghan, Aurimas Mikalauskas, Zhai Weixiang, ... โ http://www.percona.com/blog/2012/09/09/obtain-last-executed-statement-from- optimized-core-dump/ โ http://www.percona.com/blog/2013/11/11/how-to-extract-all-running-queries-inc luding-the-last-executed-statement-from-a-core-file/ โ http://mysqlbugs.blogspot.com.au/2012/09/how-to-obtain-all-executing-queries. html โ http://www.percona.com/blog/2010/03/23/too-many-connections-no-problem/ 10
- 11. www.percona.com What MySQL DBA can do with gdb โ Check stack traces (and variables), per thread: thread apply all bt [full] โ Print variables, up to complex one: thread 1 print do_command::thd->query_string.string.str โ Set new values for variables (global and per thread, even those formally read-only in MySQL while itโs running): set max_connections=5000 set opt_log_slave_updates=1 โ Call functions (that may do complex changes): call rpl_filter->add_do_db(strdup("hehehe")) โ Set breakpoints and watchpoints โ Work interactively or use gdb as a command line utility (-batch) โ Use macros/Python scripting, moreโฆ โ All these may not work, fail, hang, crash, produce obscure errorsโฆ โ You have to read and understand the source code 11
- 12. www.percona.com pt-pmp (Poor Manโs Profiler) โ http://www.percona.com/doc/percona-toolkit/2.2/pt-pmp.html pt-pmp [-i 1] [-s 0] [-b mysqld] [-p pidofmysqld] [-l 0] [-k file] [--version] โ It is based on original idea by Domas, http://poormansprofiler.org/ โ One of the recent examples how it is used: Bug #78277 - InnoDB deadlock, thread stuck on kernel calls from transparent page compression, โOpenโ โ When mysqld hangs or is slow, you can get some insight quickly: for example, Bug #86902 (MySQL server may hang when we turn off binlog...) โ When there are stalls, use pt-pmp to find out why (or what threads mostly do at the moment): Bug #69810 โ Use in production as a last resort (may hang mysqld, --SIGCONT) โ pt-pmp surely slows server down :) Hints: โ https://bugs.launchpad.net/percona-toolkit/+bug/1320168 - partial workaround โ Use quickstack instead of gdb (check this discussion) 12
- 13. www.percona.com Multi-threaded mysqld process and gdb โ process/thread/frame concepts: (gdb) thread 2 [Switching to thread 2 (Thread 0x7fe771550700 (LWP 2544))] #0 0x0000000000605774 in Item_func_numhybrid::val_int ( this=<value optimized out>) at /home/openxs/bzr2/percona-5.6/sql/item_func.cc:1013 1013 } (gdb) bt ... #12 0x00000000006f8a45 in dispatch_command (command=COM_QUERY, thd=0x7fe760f94000, packet=0x7fe77154fac0 "", packet_length=0) at /home/openxs/bzr2/percona-5.6/sql/sql_parse.cc:1434 ... (gdb) frame 12 #12 0x00000000006f8a45 in dispatch_command (command=COM_QUERY, thd=0x7fe760f94000, packet=0x7fe77154fac0 "", packet_length=0) at /home/openxs/bzr2/percona-5.6/sql/sql_parse.cc:1434 warning: Source file is more recent than executable. 1434 mysql_parse(thd, thd->query(), thd->query_length(), &parser_state); (gdb) p thd->query_string.string.str $2 = 0x7fe75301d010 "select benchmark(5", '0' <repeats 13 times>, ", 2*2)" โ https://sourceware.org/gdb/onlinedocs/gdb/Variables.html 13
- 14. www.percona.com THD structure grep -rn THD sql/sql_class.h class THD :public MDL_context_owner, public Statement, public Open_tables_state HASH user_vars; // hash for user vars struct system_variables variables; // Changeable local vars struct system_status_var status_var;// Per thread stat vars struct system_status_var *initial_status_var; /* used by show status */ Security_context main_security_ctx; ... CSET_STRING query_string; // inherited from Statementโฆ ... 14
- 15. www.percona.com THD structure (continued) (gdb) p thd->main_security_ctx->user $7 = 0x7fe753019058 "root" (gdb) p thd->main_security_ctx->host $8 = {Ptr = 0xc16759 "localhost", str_length = 9, Alloced_length = 0, alloced = false, str_charset = 0x1393de0} 15
- 16. www.percona.com Real life case: checking core dump gdb -ex 'set pagination 0' โฆ -ex 'thread apply all bt full' /path/to/mysqld /var/tmp/core.<pid> | tee core.<pid>.bt โ Make sure you know how to get core when mysqld crashes: http://www.percona.com/blog/2011/08/26/getting-mysql-core-file-on-linux/ โ Letโs check one example, we need crashing bug for this: There is one that affects MySQL < 5.7.20, and we may have some hint in MariaDB changelog 16
- 17. www.percona.com Real life case: attaching to alive mysqld This is how it goes: [root@centos openxs]# mysql -uroot -e "show variables like 'innodb_autoinc_lock_mode'" +--------------------------+-------+ | Variable_name | Value | +--------------------------+-------+ | innodb_autoinc_lock_mode | 0 | +--------------------------+-------+ [root@centos openxs]# mysql -uroot -e "set global innodb_autoinc_lock_mode=1" ERROR 1238 (HY000) at line 1: Variable 'innodb_autoinc_lock_mode' is a read only variable [root@centos openxs]# gdb -ex "set innobase_autoinc_lock_mode=1" -batch -p `pidof mysqld` โฆ [Thread debugging using libthread_db enabled] 0x00007ff31d6830d3 in poll () from /lib64/libc.so.6 โฆ check the variable value again now [root@centos openxs]# ps aux | grep mysqld [root@centos openxs]# kill -SIGCONT `pidof mysqld` 17
- 18. www.percona.com How to study InnoDB locks with gdb โ Read the code (or blogs, or backtraces) to find out what functions are called when InnoDB locks are requested: โ lock_table - table level locks โ lock_rec_lock - row level locks โ Make sure there is debug info for mysqld binary you use โ Attach gdb to running mysqld process in test env: [root@centos ~]# gdb -p `pidof mysqld` ... (gdb) b lock_table ... (gdb) b lock_rec_lock ... (gdb) c โ Run SQL you want to study and check sequence of calls, backtraces, variables... 18
- 19. www.percona.com How to study metadata locks with gdb โ Read the code (or blogs, or backtraces) to find out what functions are called when metadata locks are requested: โ MDL_request::init - metadata lock request โ MDL_context::aquire_lock - attempt to acquire lock โ Attach gdb to running mysqld process in test env: [root@centos ~]# gdb -p `pidof mysqld` ... (gdb) b MDL_request::init ... (gdb) c โ Run SQL you want to study and check sequence of calls, backtraces, variables... 19
- 20. www.percona.com How to find processlist thread id with gdb http://mysqlentomologist.blogspot.fi/2017/07/how-to-find-pro cesslist-thread-id-in-gdb.html โ It may depend on MySQL version (changes in 5.7+) โ Basic idea - check threads one by one, find frame with thd is defined, print: (gdb) thread 2 (gdb) p do_command::thd->thread_id โ In 5.7+ there is some difference: (gdb) thread 7 (gdb) p do_command::thd->m_thread_id (gdb) p do_command::thd->m_main_security_ctx โ Even more difference if you want to automate looping through threadsโฆ (more C++, singletons vs variables) 20
- 21. www.percona.com How to find SQL statement executing by thread with gdb https://www.percona.com/blog/2012/09/09/obtain-last-execut ed-statement-from-optimized-core-dump/ โ Basic idea is simple - in a frame with thd defined do: (gdb) p thd->query_string.string.str โ But how to find such a frame? โ Also, how to navigate through all threads in core dump? โ One of the answers is here: https://www.percona.com/blog/2013/11/11/how-to-extract-all-running-queries -including-the-last-executed-statement-from-a-core-file/ โ Prepare file with gdb commands like: ... thread N print do_command::thd->query_string.string.str 21
- 22. www.percona.com How to study session variables with gdb http://mysqlentomologist.blogspot.fi/2017/08/how-to-find-valu es-of-session-variables.html โ It started with a โsimpleโ question: how to find out from the core dump if the session behind the crashing thread had mrr=ON in the optimizer_switch? โ Basic idea is simple, itโs in thd->variables, somehow: (gdb) p do_command::thd->variables->optimizer_switch (gdb) p global_system_variables->optimizer_switch โ Then see defines in sql/sql_const.h: #define OPTIMIZER_SWITCH_MRR (1ULL << 6) โ Then we can print it better: p do_command::thd->variables->optimizer_switch & (1<<6) p /t do_command::thd->variables->optimizer_switch 22
- 23. www.percona.com How to study user variables with gdb http://mysqlentomologist.blogspot.com/2017/08/how-to-find-values-of-user-variab les.html โ Basically itโs somewhere there, in thd: p do_command::thd->user_vars โ But itโs not a simple array, itโs a HASH: (gdb) p my_hash_element(&(do_command::thd->user_vars), 1) (gdb) set $uvar = (user_var_entry *)(my_hash_element(&(do_command::thd->user_vars), 1)) (gdb) p $uvar (gdb) p *$uvar โ We can also get element by name: (gdb) set $uvar=(user_var_entry *)(my_hash_search(&(do_command::thd->user_vars), "e", strlen("e"))) (gdb) p *((my_decimal *)$uvar->m_ptr) 23
- 24. www.percona.com HASH structures in MySQL http://mysqlentomologist.blogspot.fi/2017/08/more-on-studying-mysql-hashes-in- gdb.html โ HASH structure is used everywhere in MySQL, from keyring to UDFs and table cache, to replication and NDB Cluster, with everything in between โ Check include/hash.h: typedef struct st_hash { ... ulong records; DYNAMIC_ARRAY array; ... } HASH; โ This gives us a way eventually to dump data without calling functions: (gdb) set $uvars=&(do_command::thd->user_vars) ... (gdb) p *(user_var_entry *) (((HASH_LINK*)((&($uvars->array))->buffer) + (0))->data) 24
- 25. www.percona.com How to find what thread had executed FTWRL http://mysqlentomologist.blogspot.fi/2017/04/how-to-find-wha t-thread-had-executed.html โ In MariaDB starting from 10.0.7 you can use METADATA_LOCK_INFO plugin. โ In MySQL starting from 5.7 you can use performance_schema.metadata_locks table. โ In MySQL starting from 5.6 (or MariaDB 10.x.y) you can use performance_schema.events_statements_history table. โ In all versions of MySQL or MariaDB you can attach gdb and check threads one by one: (gdb) set $thd=(THD *)(threads->first) (gdb) p $thd (gdb) p $thd->thread_id (gdb) p $thd->global_read_lock 25 Really? What about 5.7? Any workarounds?
- 26. www.percona.com How to study table level locks in gdb http://mysqlentomologist.blogspot.fi/2017/07/why-thread-may-hang-in-waiting-for.html โ How many of you know what mysqladmin debug does (sends COM_DEBUG, and what in reply)? โ How to get similar information in gdb? Find and study the code of display_table_locks(void) function, check what LIST, THR_LOCK and TABLE_SHARE structures are! โ Then use the force: (gdb) set $list=(LIST *)thr_lock_thread_list (gdb) set $lock=(THR_LOCK*) $list->data (gdb) p *($lock) (gdb) p *($lock)->write.data.owner (gdb) set $table=(TABLE *) &(*($lock)->write.data->debug_print_param) (gdb) p $table->s->path 26
- 27. www.percona.com Some gdb versions have Python, and this helps โ If you like and know Python and have gdb linked with it (try py print(1+1))... โ Use ~/.gdbinit file for complex Python macros โ http://mysqlbugs.blogspot.com/2012/09/how-to-obtain-all-executing-queries. html - Shane Bester on simplified navigation over threads, nice printing of selected values etc, โ https://mariadb.org/duel-gdb-vs-linked-lists-trees-hash-tables/ - โDuel: gdb vs. linked lists, trees, and hash tablesโ. Sergei Golubchik on simplified way to apply โsomethingโ (like print) to all/selected items of arrays, linked lists etc. Check https://github.com/vuvova/gdb-tools โ https://mariadb.org/making-life-prettier-gdb-prettyprinting-api/ - โMaking life prettier with gdb PrettyPrinting APIโ. Sergei Golubchik on how to use Python classes to pretty print almost anything inside MySQL code in gdb โ Make sure to check if you have Python 2 or 3 in gdb! (pip vs pip3 etc): [openxs@fc23 ~]$ ldd `which gdb` | grep pyt libpython3.5m.so.1.0 => /lib64/libpython3.5m.so.1.0 (0x00007fee3957d000) 27
- 28. www.percona.com Some things to check before relying on gdb โ Check that gdb is installed and works โ Check that MySQL/Percona/MariaDB server you use has symbolic information for gdb. See MDEV-13027 also. If you build from source: cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo โ DBA may need to get sudo/root access โ Make sure you know how to enable core dumps on your Linux, and know where they are located (it may become complicated) โ Install pt-pmp (or entire Percona Toolkit) - https://www.percona.com/get/pt-pmp - and check it โ Itโs probably a good idea to create useful ~/.gdbinit 28
- 29. www.percona.com Results of using gdb to study MySQL internals โ Immediate DBA problems solved without restart etc โ Better understanding of how MySQL works! โ Blog posts, talks, presentations: โ http://mysqlentomologist.blogspot.com/2016/01/exploring-metadata-locks-with-gdb-first.html โ http://mysqlentomologist.blogspot.com/2016/01/exploring-metadata-locks-with-gdb.html โ http://mysqlentomologist.blogspot.com/2016/01/exploring-metadata-locks-with-gdb-how.html โ http://mysqlentomologist.blogspot.com/2015/03/using-gdb-to-understand-what-locks-and_31.html โ http://mysqlentomologist.blogspot.com/2015/04/using-gdb-to-understand-what-locks-and.html โ http://www.slideshare.net/valeriikravchuk1/understanding-innodb-locks-and-deadlocks โ Bug reports and documentation requests to make MySQL and its manual better: โ Bug #79665 - Manual does NOT explain locks set by INSERT ... ON DUPLICATE KEY UPDATE properly โ Bug #77390 - Manual does not explain a "deadlock" case of online ALTER โ Bug #76588 - Metadata lock is NOT released when SELECT completes in case of autocommit=0 โ Bug #76563 - Manual does NOT explain when exactly AUTO-INC lock is set for "bulk inserts" โ Bug #76533 - AUTO_INC lock seems to be NOT set for INSERT INTO t(val) SELECT val FROM t 29
- 30. www.percona.com Is gdb an ultimate answer for MySQL DBA? No, usually it is a temporary, one time solution or last resort Instead you may (or should, whenever possible): โ Use real profilers at OS level (like perf) โ Use troubleshooting tools at MySQL level (like P_S) โ Implement missing feature (like setting some variable dynamically) or request it from developers โ Consider upgrade to version or fork that already has a feature you miss โ Plan your work and do maintenance properly โ Read the manual and source code 30
- 31. www.percona.com Thank you! Questions and Answers? Please, report bugs at: http://bugs.mysql.com https://jira.mariadb.org 31