Basic MySQL Troubleshooting for Oracle Database Administrators
0 likes1,116 views
This document provides an overview of basic MySQL troubleshooting techniques for Oracle database administrators. It covers MySQL server architecture including connectors, clients, APIs, storage engines, and plugins. It then discusses basic troubleshooting techniques such as error processing, access privileges, and using system variables, performance schema, and EXPLAIN to analyze query execution plans. The document is intended to help Oracle DBAs understand fundamental aspects of MySQL administration.
![More information about errors
[sveta@delly ~]$ perror 1630
MySQL error code 1630 (ER_FUNC_INEXISTENT_NAME_COLLISION):
FUNCTION %s does not exist. Check the ’Function Name Parsing and Resolution’
section in the Reference Manual
[sveta@delly ~]$ perror 1292
MySQL error code 1292 (ER_TRUNCATED_WRONG_VALUE):
Truncated incorrect %.32s value: ’%.128s’
[sveta@delly ~]$ perror 2
OS error code 2: No such file or directory
[sveta@delly ~]$ perror 150
MySQL error code 150: Foreign key constraint is incorrectly formed
14 www.percona.com](https://image.slidesharecdn.com/troubleshootingmysql-151102184013-lva1-app6891/85/Basic-MySQL-Troubleshooting-for-Oracle-Database-Administrators-14-320.jpg)
![Wrong access checklist
• SHOW GRANTS [FOR user@host]
• Grant tables
• mysql.db
• mysql.tables priv
• mysql.columns priv
• mysql.procs priv
• mysql.proxies priv
• SELECT USER(), CURRENT USER()
18 www.percona.com](https://image.slidesharecdn.com/troubleshootingmysql-151102184013-lva1-app6891/85/Basic-MySQL-Troubleshooting-for-Oracle-Database-Administrators-18-320.jpg)
![System Variables: how to set
• SET [GLOBAL] var name = NEW VAL
• Command-line option
• Configuration file
• In default location
• Specified by option –defaults-file
24 www.percona.com](https://image.slidesharecdn.com/troubleshootingmysql-151102184013-lva1-app6891/85/Basic-MySQL-Troubleshooting-for-Oracle-Database-Administrators-25-320.jpg)
![System variables: control before 5.7
• SHOW [GLOBAL] STATUS
• Tables in Information Schema
• GLOBAL ,SESSION VARIABLES
• GLOBAL ,SESSION STATUS
• GLOBAL
• Since server start
• SESSION
• For operations in current session
• Can be reset
• FLUSH STATUS
27 www.percona.com](https://image.slidesharecdn.com/troubleshootingmysql-151102184013-lva1-app6891/85/Basic-MySQL-Troubleshooting-for-Oracle-Database-Administrators-28-320.jpg)
![System variables: best practices
• Record currently used variables
• SHOW [GLOBAL] VARIABLES
• Make change dynamically if possible
• SET [GLOBAL] var name=NEW VAL
• Test in one session first
• Then change global variable
• Change configuration file after you are happy with
results
31 www.percona.com](https://image.slidesharecdn.com/troubleshootingmysql-151102184013-lva1-app6891/85/Basic-MySQL-Troubleshooting-for-Oracle-Database-Administrators-32-320.jpg)
![When affecting option is not known
• Record currently used variables
• SHOW [GLOBAL] VARIABLES
• Start mysqld with option –no-defaults
• This option must be first one!
• Check if problem is solved
• Change variable values one-by-one until you find one
which leads to the problem
32 www.percona.com](https://image.slidesharecdn.com/troubleshootingmysql-151102184013-lva1-app6891/85/Basic-MySQL-Troubleshooting-for-Oracle-Database-Administrators-33-320.jpg)
![EXPLAIN FORMAT=JSON
mysql> EXPLAIN FORMAT=JSON SELECT user, host FROM mysql.userG
*************************** 1. row ***************************
EXPLAIN: {
"query_block": {
"select_id": 1,
"table": {
"table_name": "user",
"access_type": "index",
"key": "PRIMARY",
"used_key_parts": [
"Host",
"User"
],
"key_length": "228",
"rows": 8,
"filtered": 100,
"using_index": true
}
}
}
37 www.percona.com](https://image.slidesharecdn.com/troubleshootingmysql-151102184013-lva1-app6891/85/Basic-MySQL-Troubleshooting-for-Oracle-Database-Administrators-38-320.jpg)
![Locks diagnostic
• SHOW [FULL] PROCESSLIST
• SHOW ENGINE INNODB STATUS
• INFORMATION SCHEMA
• PROCESSLIST
• InnoDB table
• Performance Schema
• METADATA LOCKS
• TABLE HANDLES
• EVENTS TRANSACTIONS *
• Both server-level and engine-level
• Contain GTID information
45 www.percona.com](https://image.slidesharecdn.com/troubleshootingmysql-151102184013-lva1-app6891/85/Basic-MySQL-Troubleshooting-for-Oracle-Database-Administrators-50-320.jpg)
![Locks diagnostic
• SHOW [FULL] PROCESSLIST
mysql> select id, state, info from information_schema.processlistG
******************** 1. row ********************
id: 5
state:
info: NULL
******************** 2. row ********************
id: 4
state: Waiting for table metadata lock
info: alter table titles add column f1 int
******************** 3. row ********************
id: 2
state: executing
info: select id, state, info from information_schema.processlist
45 www.percona.com](https://image.slidesharecdn.com/troubleshootingmysql-151102184013-lva1-app6891/85/Basic-MySQL-Troubleshooting-for-Oracle-Database-Administrators-51-320.jpg)
Basic MySQL Troubleshooting for Oracle Database Administrators
- 1. Basic MySQL Troubleshooting for Oracle Database Administrators Sveta Smirnova Principal Support Engineer October, 29, 2015
- 2. Table of Contents •MySQL Server overview •Basic troubleshooting techniques •High concurrency issues •Replication 2 www.percona.com
- 3. MySQL Server overview 3 www.percona.com
- 4. MySQL architecture Connectors: C, JDBC, ODBC, Python, ... Connection Pool: Authentication, Caches SQL interface Parser Optimizer Caches and Buffers: Global Engine-specific Storage engines: InnoDB, TokuDB, ... File system: Data, Index, logs, other files • Base • Installation layout • Log files • Connectors • Clients, APIs • Optimizer • Cache and buffers • Storage engines • Management 4 www.percona.com
- 5. Typical installation layout • datadir • Schema directory • Table and trigger files: *.frm, *.ibd, *.MYD, *.par, etc. • Schema • ... • InnoDB shared tablespace • Log files • InnoDB redo log files • Binary, relay log files • Error log • Slow query log • General query log • Configurable • You can setup custom path for each component • Including custom paths for tables 5 www.percona.com
- 6. Connectors for MySQL server • Clients • MySQL CLI • MySQL Workbench • Many other graphical and web-based • APIs • Exist for most popular programming languages • C, C++, JDBC, PHP, Python, Net, ODBC, more 6 www.percona.com
- 7. Monitoring solutions • Command-line • Percona Toolkit • MySQL Utilities • With GUI • MySQL Enterprise Monitor (MEM) • MEM plugin for Oracle Enterprise Manager • VividCortex • Many others 7 www.percona.com
- 8. Plugins • Storage engines • Authentication • Audit • Query rewrite • More 8 www.percona.com
- 9. Storage engines • Own data • Own index format • Own locking model • Own diagnostic • Own log files • CHECK TABLE 9 www.percona.com
- 11. Error processing • Warnings, errors, and notes mysql> select max (f1) from t1; ERROR 1630 (42000): FUNCTION test.max does not exist. Check the ’Function Name Parsing and Resolution’ section in the Reference Manual mysql> select * from t1 where "f1"=1; Empty set, 1 warning (0.05 sec) mysql> show warnings; +-----------+--------+----------------------------------------------------+ | Level | Code | Message | +-----------+--------+----------------------------------------------------+ | Warning | 1292 | Truncated incorrect DOUBLE value: ’f1’ | +-----------+--------+----------------------------------------------------+ 1 row in set (0.00 sec) 11 www.percona.com
- 12. Error handling in applications • Error information • mysql error • mysql errno • Warnings and notes • mysql info • mysql sqlstate • mysql warning count 12 www.percona.com
- 13. Error handling in stored routines • GET DIAGNOSTICS • GET DIAGNOSTICS rows = ROW COUNT, conditions = NUMBER; • GET DIAGNOSTICS CONDITION 1 code = RETURNED SQLSTATE, msg = MESSAGE TEXT; • SIGNAL/RESIGNAL • Can be used for custom errors 13 www.percona.com
- 14. More information about errors [sveta@delly ~]$ perror 1630 MySQL error code 1630 (ER_FUNC_INEXISTENT_NAME_COLLISION): FUNCTION %s does not exist. Check the ’Function Name Parsing and Resolution’ section in the Reference Manual [sveta@delly ~]$ perror 1292 MySQL error code 1292 (ER_TRUNCATED_WRONG_VALUE): Truncated incorrect %.32s value: ’%.128s’ [sveta@delly ~]$ perror 2 OS error code 2: No such file or directory [sveta@delly ~]$ perror 150 MySQL error code 150: Foreign key constraint is incorrectly formed 14 www.percona.com
- 15. MySQL Access Privilege System • No roles by default, limited user limits • All records are in the mysql database (schema) • Pluggable authentication since version 5.5 • Connections • TCP/IP with login-password • Socket (Unix) • Named pipe (Windows) 15 www.percona.com
- 16. Common user access issues • Privileged client cannot connect • Unprivileged client can connect • Privileged user cannot perform operation • Unprivileged user has undesired access 16 www.percona.com
- 17. Common queries for access issues mysql> select user, host from mysql.user order by user desc, host desc; +------+------------+ | user | host | +------+------------+ | root | localhost | | root | delly | | root | ::1 | | root | 127.0.0.1 | | foo | % | | | localhost | +------+------------+ 6 rows in set (0.00 sec) • Most descriptive host first, then wildcard • Socket connection by default on Unix 17 www.percona.com
- 18. Wrong access checklist • SHOW GRANTS [FOR user@host] • Grant tables • mysql.db • mysql.tables priv • mysql.columns priv • mysql.procs priv • mysql.proxies priv • SELECT USER(), CURRENT USER() 18 www.percona.com
- 19. Wrong access example mysql> show grants; +--------------------------------------------------------------------+ | Grants for root@localhost | +--------------------------------------------------------------------+ | GRANT ALL PRIVILEGES ON *.* TO ’root’@’localhost’ WITH GRANT OPTION| | GRANT PROXY ON ’’@’’ TO ’root’@’localhost’ WITH GRANT OPTION | +--------------------------------------------------------------------+ 2 rows in set (0.02 sec) mysql> show grants for foo@’%’; +-----------------------------------------------------+ | Grants for foo@% | +-----------------------------------------------------+ | GRANT USAGE ON *.* TO ’foo’@’%’ | | GRANT ALL PRIVILEGES ON ‘test‘.* TO ’foo’@’%’ | +-----------------------------------------------------+ 2 rows in set (0.02 sec) 19 www.percona.com
- 20. Connection statistics in P S • Tables accounts, users, hosts mysql> select user, host, current_connections as cur, -> total_connections as total from performance_schema.accounts; +------+-----------+-----+-------+ | user | host | cur | total | +------+-----------+-----+-------+ | foo | localhost | 0 | 3 | | root | localhost | 1 | 3 | | NULL | NULL | 14 | 17 | +------+-----------+-----+-------+ 3 rows in set (0.01 sec) • HOST CACHE 20 www.percona.com
- 21. Connection statistics in P S • Tables accounts, users, hosts • HOST CACHE • Content of DNS cache • Errors from: • Name Server • Connection • Authentication • max connect errors, max user errors, etc. • Your first assistant in case of connection issue 20 www.percona.com
- 22. Performance Schema • Monitors interval operations • Statements • Stages • Locks • Memory • Variables • Replication • IO • Mutexes and waits • Similar to Oracle wait interface 21 www.percona.com
- 23. What can affect query execution? • You run a query, it does not return an error, but still behaves not as expected • It can be: • Startup options or system variables • How optimizer creates query plan • Storage engine used • Parallel execution - next big section 22 www.percona.com
- 24. System variables and options: scope • Global • Control parameters, necessary for all server processes • Location of server files: datadir etc. • Shared buffers • More • Session • Control connection-specific parameters • MySQL option tables 23 www.percona.com
- 25. System Variables: how to set • SET [GLOBAL] var name = NEW VAL • Command-line option • Configuration file • In default location • Specified by option –defaults-file 24 www.percona.com
- 26. System Variables: who can change • Global options and few session options • A user with privilege SUPER • Session options • Anybody • There are no limits! 25 www.percona.com
- 27. Buffers: when allocated • Those which control behavior of whole server • Once at server startup • Can start with low values, then grow to specified • Connection options • For every connection when connection opens • Operation-specific • For every operation when needed • Can be allocated more than once per query 26 www.percona.com
- 28. System variables: control before 5.7 • SHOW [GLOBAL] STATUS • Tables in Information Schema • GLOBAL ,SESSION VARIABLES • GLOBAL ,SESSION STATUS • GLOBAL • Since server start • SESSION • For operations in current session • Can be reset • FLUSH STATUS 27 www.percona.com
- 29. System status variables: example mysql> show global status like ’Handler_read_rnd_next’G *************************** 1. row *************************** Variable_name: Handler_read_rnd_next Value: 27 1 row in set (0.00 sec) mysql> show status like ’Handler_read_rnd_next’G *************************** 1. row *************************** Variable_name: Handler_read_rnd_next Value: 7 1 row in set (0.00 sec) 28 www.percona.com
- 30. Information Schema • Contains metadata information • Tables • Indexes • Other • Allows to create plugins • InnoDB plugins • Similar to Data Dictionary Views in Oracle 29 www.percona.com
- 31. System variables: control in 5.7 • Performance Schema tables • variabes by* • user variables by* • status by* • Statistics grouped by • Global • Session • Thread • Account/Host/User 30 www.percona.com
- 32. System variables: best practices • Record currently used variables • SHOW [GLOBAL] VARIABLES • Make change dynamically if possible • SET [GLOBAL] var name=NEW VAL • Test in one session first • Then change global variable • Change configuration file after you are happy with results 31 www.percona.com
- 33. When affecting option is not known • Record currently used variables • SHOW [GLOBAL] VARIABLES • Start mysqld with option –no-defaults • This option must be first one! • Check if problem is solved • Change variable values one-by-one until you find one which leads to the problem 32 www.percona.com
- 34. MySQL Optimizer • EXPLAIN is less powerful if compare with Oracle • Visual EXPLAIN in MySQL Workbench • EXPLAIN EXTENDED • Should be followed by SHOW WARNINGS • EXPLAIN PARTITIONS • EXPLAIN FORMAT=JSON • INFORMATION SCHEMA.TRACE • Status variables ’Handler %’ 33 www.percona.com
- 35. EXPLAIN in Oracle • http://docs.oracle.com/cd/B10500_01/server. 920/a96533/ex_plan.htm EXPLAIN PLAN SET statement_id = ’example_plan4’ FOR SELECT h.order_number, l.revenue_amount, l.ordered_quantity FROM so_headers_all h, so_lines_all l WHERE h.customer_id = :b1 AND h.date_ordered > SYSDATE30 AND l.header_id = h.header_id ; Plan -------------------------------------------------- SELECT STATEMENT NESTED LOOPS TABLE ACCESS BY INDEX ROWID SO_HEADERS_ALL INDEX RANGE SCAN SO_HEADERS_N1 TABLE ACCESS BY INDEX ROWID SO_LINES_ALL INDEX RANGE SCAN SO_LINES_N1 34 www.percona.com
- 36. EXPLAIN in MySQL mysql> EXPLAIN EXTENDED SELECT user, host FROM mysql.userG *************************** 1. row *************************** id: 1 select_type: SIMPLE table: user type: index possible_keys: NULL key: PRIMARY key_len: 228 ref: NULL rows: 4 filtered: 100.00 Extra: Using index 1 row in set, 1 warning (0.00 sec) mysql> SHOW WARNINGSG ******* 1. row ******* Level: Note Code: 1003 Message: /* select#1 */ select ‘mysql‘.‘user‘.‘User‘ AS ‘user‘, ‘mysql‘.‘user‘.‘Host‘ AS ‘host‘ from ‘mys 35 www.percona.com
- 37. EXPLAIN: overview mysql> explain extended select * from t1 join t2 where t1.int_key=1; +----+-------------+-------+-------+---------------+---------+---------+-------+------+------+----------- | id | select_type | table | type | possible_keys | key | key_len | ref | rows | f... | Extra +----+-------------+-------+-------+---------------+---------+---------+-------+------+------+----------- | 1 | SIMPLE | t1 | ref | int_key,ik | int_key | 5 | const | 4 | 100. | NULL | 1 | SIMPLE | t2 | index | NULL | pk | 9 | NULL | 6 | 100. | Using inde Using join (Block Nes +----+-------------+-------+-------+---------------+---------+---------+-------+------+------+----------- 2 rows in set, 1 warning (0.00 sec) Note (Code 1003): /* select#1 */ select ‘test‘.‘t1‘.‘pk‘ AS ‘pk‘,‘test‘.‘t1‘.‘int_key‘ AS ‘int_key‘,‘test AS ‘pk‘,‘test‘.‘t2‘.‘int_key‘ AS ‘int_key‘ from ‘test‘.‘t1‘ join ‘test‘.‘t2‘ where (‘test‘.‘t1‘.‘int_key‘ Number of select Select type Tables, for which information is printed How data is accessed Possible keys Key, which was actually used Length of the key Which columns were compared with the index Number of examined rows % of filtered rows rows x filtered / 100 — number of rows, which will be joined with another table Additional information Table, for which information is printed Product of rows here: how many rows in all tables will be accessed For this example estimated value is 4*6 = 24 Actual (optimized) query as executed by MySQL Server 36 www.percona.com
- 38. EXPLAIN FORMAT=JSON mysql> EXPLAIN FORMAT=JSON SELECT user, host FROM mysql.userG *************************** 1. row *************************** EXPLAIN: { "query_block": { "select_id": 1, "table": { "table_name": "user", "access_type": "index", "key": "PRIMARY", "used_key_parts": [ "Host", "User" ], "key_length": "228", "rows": 8, "filtered": 100, "using_index": true } } } 37 www.percona.com
- 39. When EXPLAIN lies: Handler % variables mysql> flush status; Query OK, 0 rows affected (0.00 sec) mysql> select count(*) from employees join titles using(emp_no) -> where title=’Senior Engineer’G *************************** 1. row *************************** count(*): 97750 1 row in set (3.24 sec) mysql> SHOW STATUS LIKE ’Handler_read_%’; +----------------------------+--------+ | Variable_name | Value | +----------------------------+--------+ | Handler_read_first | 1 | | Handler_read_key | 300027 | | Handler_read_last | 0 | | Handler_read_next | 397774 | ... 38 www.percona.com
- 40. MySQL Optimizer: other tools • INFORMATION SCHEMA.OPTIMIZER TRACE • join preparation, join optimization, join execution • considered execution plans, refine plan, more • Query Analyzer in MEM • Visual EXPLAIN in MySQL Workbench 39 www.percona.com
- 41. MySQL Storage Engines • Own way to handle • Corruption • Index statistics • CHECK TABLE to check for errors • They care about physical data, so all data information is on their level • Options usually start from engine name • myisam *, innodb *, custom * 40 www.percona.com
- 42. Storage Engines explained • InnoDB • TokuDB • Popular non-transactional engines • MyISAM - Previous default engine • Memory • Federated • CSV • Blackhole 41 www.percona.com
- 43. Storage Engines explained • InnoDB • Most used storage engine • Physical layout • *frm file – table definition • Shared tablespace • *ibd file – tablespace for individual table - –innodb file per table = 1 • Redo log files • Automatic startup check 41 www.percona.com
- 44. Storage Engines explained • InnoDB • Own locking model: row-based • Troubleshooting tools • InnoDB Monitors (SHOW ENGINE InnoDB STATUS) • Tables in Information Schema • Instrumentation in Performance Schema 41 www.percona.com
- 45. Storage Engines explained • TokuDB • Fractal tree • Optimized for compression and writes • Troubleshooting tools • SHOW ENGINE tokudb STATUS • Information schema tables • Status variables 41 www.percona.com
- 46. Storage Engines explained • Popular non-transactional engines • Server provides locking model • Table-level locks • Can have own instrumentation • MyISAM • Status variables • Performance Schema • Set of utilities, names start from myisam * 41 www.percona.com
- 47. High concurrency issues 42 www.percona.com
- 48. Common concurrency issues • Query or transaction waits a lock, held by another one • Fight for system resources • Resource overload • Resource underload 43 www.percona.com
- 49. Lock types and transactions • Lock types • Levels • MDL • Table-level • Row-level • What do they lock • Read locks • Block writes • Write locks • Block reads and writes • Transactions • Server-level • MDL locks • Table locks • Engine-level • Table locks • Row locks • AUTOCOMMIT • supported 44 www.percona.com
- 50. Locks diagnostic • SHOW [FULL] PROCESSLIST • SHOW ENGINE INNODB STATUS • INFORMATION SCHEMA • PROCESSLIST • InnoDB table • Performance Schema • METADATA LOCKS • TABLE HANDLES • EVENTS TRANSACTIONS * • Both server-level and engine-level • Contain GTID information 45 www.percona.com
- 51. Locks diagnostic • SHOW [FULL] PROCESSLIST mysql> select id, state, info from information_schema.processlistG ******************** 1. row ******************** id: 5 state: info: NULL ******************** 2. row ******************** id: 4 state: Waiting for table metadata lock info: alter table titles add column f1 int ******************** 3. row ******************** id: 2 state: executing info: select id, state, info from information_schema.processlist 45 www.percona.com
- 52. Locks diagnostic • Performance Schema mysql> select processlist_id, object_type, lock_type, lock_status, source -> from metadata_locks join threads on (owner_thread_id=thread_id) -> where object_schema=’employees’ and object_name=’titles’G *************************** 1. row *************************** processlist_id: 4 object_type: TABLE lock_type: EXCLUSIVE lock_status: PENDING -- waits source: mdl.cc:3263 *************************** 2. row *************************** processlist_id: 5 object_type: TABLE lock_type: SHARED_READ lock_status: GRANTED -- holds source: sql_parse.cc:5707 45 www.percona.com
- 53. Locks diagnostic • SHOW ENGINE INNODB STATUS ------------- TRANSACTIONS ------------- Trx id counter 0 26243837 Purge done for trx’s n:o < 0 26243834 undo n:o < 0 0 History list length 2 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 0 26243836, ACTIVE 4 sec, OS thread id 101514240 starting index read mysql tables in use 1, locked 1 LOCK WAIT 2 lock struct(s), heap size 320, 1 row lock(s) MySQL thread id 4485, query id 25022137 localhost root Updating update t set a=36 where a=6 45 www.percona.com
- 54. InnoDB Monitors • SHOW ENGINE INNODB STATUS • Permanent output • innodb status output • innodb status output locks • innodb-status-file - Deleted on normal shutdown! 46 www.percona.com
- 55. Other InnoDB diagnostic • INFORMATION SCHEMA • INNODB TRX • INNODB LOCKS • INNODB LOCK WAITS • INNODB METRICS • Options innodb monitor * • Option innodb print all deadlocks 47 www.percona.com
- 56. Locks diagnostic summary • Table-level • PROCESSLIST: ”Waiting for table lock” • P S.TABLE HANDLES • Row-level • InnoDB monitors • SHOW ENGINE INNODB STATUS • Tables in INFORMATION SCHEMA • Option –innodb print all deadlocks • MDL • PROCESSLIST • ”Waiting for metadata lock” • P S.METADATA LOCKS 48 www.percona.com
- 58. MySQL Replication Overview • Always available, needs to be setup • Asynchronous master-slave • Master • Keeps all updates in binary log file • Two formats: ROW and STATEMENT • Slave • IO thread read updates from master and stores in relay log file • SQL thread executes updates • Multiple SQL threads since 5.6 • Multiple masters since 5.7 • GTIDs since 5.6 50 www.percona.com
- 59. Main troubleshooting tools • SHOW SLAVE STATUS • Tables in Performance Schema and system database • Multi-threaded slave (MTS) has additional output for each worker • Error log file 51 www.percona.com
- 60. Main troubleshooting tools • SHOW SLAVE STATUS mysql> show slave status G ************************ 1. row ************************ Slave_IO_State: Queueing master event to the relay log ... Master_Log_File: master-bin.000003 Read_Master_Log_Pos: 191 Relay_Log_File: slave-relay-bin.000006 Relay_Log_Pos: 4 Relay_Master_Log_File: master-bin.000002 Slave_IO_Running: Yes Slave_SQL_Running: Yes ... Last_Errno: 0 Last_Error: ... Exec_Master_Log_Pos: 319 ... 51 www.percona.com
- 61. Main troubleshooting tools • Tables in Performance Schema and system database • replication connection configuration • replication applier configuration • replication connection status • replication applier status • replication applier status by coordinator • replication applier status by worker - MTS only • mysql.slave master info • mysql.slave relay log info • mysql.slave worker info 51 www.percona.com
- 62. Main troubleshooting tools • Multi-threaded slave (MTS) mysql> select * from replication_applier_status_by_workerG *************************** 1. row *************************** CHANNEL_NAME: WORKER_ID: 1 THREAD_ID: 25 SERVICE_STATE: ON LAST_SEEN_TRANSACTION: LAST_ERROR_NUMBER: 0 LAST_ERROR_MESSAGE: LAST_ERROR_TIMESTAMP: 0000-00-00 00:00:00 *************************** 2. row *************************** CHANNEL_NAME: WORKER_ID: 2 THREAD_ID: 26 SERVICE_STATE: ON LAST_SEEN_TRANSACTION: d0753e78-14ec-11e5-b3fb-28b2bd7442fd:770 ... 51 www.percona.com
- 63. IO thread: communication issues • Check slave error log • SHOW SLAVE STATUS • P S.replication connection status • Access error • Try to connect using normal MySQL client using slave credentials • SHOW GRANTS • Fix privileges on master • Restart slave 52 www.percona.com
- 64. SQL thread: typical issues • Simple master-slave • Data is different on master and slave • Replication event can not be applied • Different errors on master and slave • Slave lags far behind the master • Circular replication or other writes in addition to slave SQL thread • Data is different on master and slave 53 www.percona.com
- 65. Data is different on master and slave • Was the table modified besides the SQL thread? • How? • Can it affect content of the table in the wrong way? • Are the table definitions same on master and slave? • MySQL Utilities • mysqlrplsync, mysqldbcompare, mysqldiff • Maybe master events were applied in wrong order? • Use mysqlbinlog to find queries caused the issue • Check application, running on master 54 www.percona.com
- 66. Events applied in different order • Happens only with STATEMENT format • Row lock issues • Triggers • SET GLOBAL slave skip counter - No GTIDs! • Skip transaction - GTIDs • Synchronize tables! • Different options • Start slave with master’s options, then check • Happens in older versions 55 www.percona.com
- 67. Slave lags behind master • Threads • Master runs in multiple update threads • Slave uses single • Seconds behind master is growing - Not 100 % reliable! • Tune slave performance • Multi-threaded slave • One thread per database in 5.6 • Affected by concurrency issues • Indexes on slave only • For statement-based replication 56 www.percona.com
- 69. Further reading • MySQL Troubleshooting book • Planet MySQL • MySQL User Reference Manual • Bug trackers • http://bugs.mysql.com • https://bugs.launchpad.net/percona-server/ 58 www.percona.com
- 70. Place for your questions ??? 59 www.percona.com