20190817 coscup-oracle my sql innodb cluster sharing

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
MySQL InnoDB Cluster
Experience Sharing
實戰分享
Ivan Ma 馬楚成
MySQL Solution Engineer, APAC
2019-08-17

Agenda
• MySQL Replication Components
• MySQL InnoDB Cluster in action
• Network Stability
• MySQL InnoDB Cluster Operations
• Backup & Recovery & Member Provisioning
• GTID Consistency
• MySQL InnoDB Cluster to MySQL InnoDB Cluster Replication
• Troubleshooting
• GTID Consistency Walkthru with MySQL InnoDB Cluster

MySQL Innovation: 5.7 -> 8.0
- 3x Better Performance
- Replication Enhancements
- Optimizer Cost Model
- JSON Support
- Improved Security
- Sys & Performance Schema
- GIS
MySQL 5.7
- MySQL Group Replication
- MySQL Router
- MySQL Shell
MySQL 8.0
- 2x Better Performance
- NoSQL Document Store
- JSON
- CTEs
- Window Functions
- Data Dictionary
- InnoDB
- Replication
- Roles
- Unicode
- GIS
2 Years in Development
400+ Worklogs
5000+ Bugs Fixed
500 New Tests
Oct - 2015
Dec 2016
April 2018 GA

Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 4
Group
Replication
•Configuration
Based
InnoDB Cluster
•MySQL GR
•MySQL Shell
•MySQL Router
MySQL Shell
•Admin API to
set Primary
Instance, …
•Cluster options
: member
weight, etc.
Network
Reliability
•Expel Time out
•autorejoin tries
•unreachable
majority
timeout
GR Consistency
•eventual /
before / after
•before on
primary
consisdentcy
MySQL Router
•log rotation
•BEFORE_ON_P
RIMARY_FAILO
VER
InnoDB Cluster
Privsioning with
CLONE
Stable  Consistency  Easy to Use

High Availability- MySQL InnoDB Cluster
5

“Cluster”
6

MySQL
Master /
Slave
Replication
OS or VM
vendor
active/passive
shared disk
solutions
MySQL
Group
Replication /
MySQL
InnoDB
Cluster
MySQL
NDB Cluster
High Availability: MySQL Solutions
9 9 . 9 9 9 %
7
Continuous
Availability

Slave Server
• Session thread updates requests to Storage
Engine from application, and changes are
written to binlog before apply to storage engine
• Dump thread reads event from binlog and
propagate them to slave server
• I/O thread read replication events, stores them
to relay log
• SQL thread: reads relay log and applies them to
storage engines
MySQL Replication – Asynchronized Replication
Session
Binary
Log
Master Server
Dump I/O
Relay
Log
SQL
8
Transaction Commit
Ack

Slave Server
• Session thread updates requests to SE from application, and
changes are written to binlog before apply to storage engine
• Dump thread reads event from binlog and propagate them to
slave server
• I/O thread read replication events, stores them to relay log
• SQL thread: reads relay log and applies them to
storage engines
MySQL Replication – SEMI-Synchronized Replication
Session
Binary
Log
Master Server
Dump I/O
Relay
Log
SQL
9
Transaction Commit
Ack

What if
• Master & Many Slaves
– Master is DOWN
• Server Promotion to MASTER
• Other Slaves to Connect to NEW MASTER
10
DOWN

What if
• Master & Many Slaves
– Master is DOWN
• Server Promotion to MASTER
– Other Slaves to Connect to NEW MASTER
• Manually (or external script/tools)
to configure Server
– CHANGE MASTER to …..
• Application to CONNECT to NEW Servers
– VIP
– Change Application
11
DOWN
DOWN
New Master
ACTIVE &
PASSIVE
with
Shared DISK

MySQL InnoDB Cluster: Architecture
M
M M
MySQL Connector
Application
MySQL Router
MySQL Connector
Application
MySQL Router
MySQL Shell
HA
Group Replication
12
MySQL Enterprise Monitor

Creating MySQL InnoDB Cluster
• All MySQL Database Instances are NEW
– 3 New Instances
– Forming a Cluster
• One Instance (with data)
– Provision New Instance(s)
– Forming a Cluster

Simple 3 Nodes InnoDB Cluster
• Node 1 (3310) + Node2 (3320) + Node3 (3330) in 1 VM
• dba.configureInstance( ‘root@127.0.0.1:3310’, {clusterAdmin:’gradmin’, clusterAdmin:’grpass’})
• connect gradmin:grpass@hostname:3310
• var x = dba.createCluster( ‘mycluster’)
• x.addInstance(‘gradmin:grpass@hostname:3320’)
• x.addInstance(‘gradmin:grpass@hostname:3330’)
• x.status()
14

3 Nodes Cluster in Docker
15
Machine
Docker Engine
mysqld1 - Container mysqld2 - Container mysqld3 - Container
mysqlshell Container – start.sh  my.js

Cloning
• Instance Clone is NEW feature in MySQL 8.0.17
• How to Clone
16

InnoDB Cluster - Options
• 3 parameters  Network Stability
– MySQL Shell dba.createCluster ( <clusterName>, <options>)
• expelTimeout: integer value to define the time period in seconds that cluster members should wait for a non-responding
member before evicting it from the cluster.
• autoRejoinTries: integer value to define the number of times an instance will attempt to rejoin the cluster after being
expelled.
– group_replication_unreachable_majority_timeout - Specifies a number of seconds for
which members that suffer a network partition and cannot connect to the majority
wait before leaving the group.
• The consequence is that it determines also how long a PRIMARY switching can be taken place
• group_replication_exit_state_action – Default (8.0.12  8.015 = ABORT
SERVER), others  READ_ONLY
– Recommend to SPECIFY No matter what the DEFAULT is
17

MySQL InnoDB Cluster - Options
• Member Weight group_replication_member_weight
– Assign Priority to Member for PRIMARY election
– Why do we need to assign Member Weight
• Get a RELATIVE IDLE server (e.g. the 3rd Server to handle LARGE JOB or BACKUP
• IP White List (group_replication_ip_whitelist), Local Address
(group_replication_local_address) and Seeds (group_replication_group_seeds)
– Internal Connect Network for Cluster
– ip whilelist – e.g. Host IP/24 (Not Subnet)
• Large Message Size - group_replication_communication_max_message_size
– Feature in 8.0.16
– Large messages sent between Group Replication group members can now be split into multiple
messages when they exceed a user-defined threshold size. Sending an abnormally large message can
result in some group members being reported as failed and expelled from the group.
18

Group Replication Consistency
• SESSION variables
– group_replication_consistency = ‘EVENTUAL’ # DEFAULT
– group_replication_consistency = ‘BEFORE’ # for READ connection
– group_replication_consistency = ‘AFTER’ # for WRITE connection
• MySQL ROUTER
– Server settings : group_replication_consistency = ‘BEFORE_ON_PRIMARY_FAILOVER’
– mysqlsh>{cluster}.setOption({failoverConsistency:’BEFORE_ON_PRIMARY_FAILOVER’})
• Settings
– EVENTUAL, BEFORE, AFTER, BEFORE_AND_AFTER, BEFORE_ON_PRIMARY_FAILOVER
19

BEFORE_ON_PRIMARY_FAILOVER
20
MySQL
Primary
MySQL
Secondary
MySQL
Secondary
(1)
RELAY RELAY
(1) (1)
MySQL Router
T1 : INSERT (1)

21
MySQL
Primary
MySQL
Secondary
MySQL
Secondary
(1)
RELAY RELAY
(1) (1)
MySQL Router
T1 : INSERT (1)
Crashed

22
MySQL
Primary
MySQL
Primary
MySQL
Secondary
(1)
RELAY RELAY
(1) (1)
MySQL Router
T1 : INSERT (1)
Crashed
Promoted
But Not Yet Online

23
MySQL
Primary
MySQL
Primary
MySQL
Secondary
(1)
RELAY RELAY
(1) (1)
MySQL Router
T1 : INSERT (1)
Crashed
APPLIED
ONLINE

MySQL InnoDB Operations
• Switching Primary
– dba.getCluster(‘<clustername’).setPrimaryInstance(‘host:port’)
• MySQL Shell
– Check Status (extended:true) (membersQuery:true)
– Showing & Setting Options
• setOption(option, value)
• setInstanceOption(instance, option, value)
• MySQL Rouer
– Log Rotation
• To try : mv the <MySQLRouter Config>/logs/<the log file> to another file
• kill (SIGHUP) on pid <myqlrouter> , you will see new log file
– Check logrotate - https://www.linode.com/docs/uptime/logs/use-logrotate-to-manage-log-files/
24

MySQL InnoDB Operations
• Restarting / Rebooting MySQL InnoDB Cluster
– dba.rebootClusterFromCompleteOutage( … )
• Stop ONE Instance from MySQL InnoDB Cluster
– STOP GROUP_REPLICATION
• Upgrade MySQL InnoDB Cluster
– Rolling Upgrade
25

MySQL InnoDB Cluster Backup & Recovery
• Which Server to Backup
– Data
– Configuration
• my.cnf / auto.cnf / mysqld-auto.cnf
• Logical VS Physical
– myqsldump
– MySQL Enterprise Backup
• Backup
– Where is the PERSISTED VARIABLES
• mysqld-auto.cnf
– The SERVER UUID
• auto.cnf
26

MySQL InnoDB Cluster Backup & Recovery Scenarios
• Node A (Primary), B, C in MySQL InnoDB Cluster ==? Node B is crashed
– Recovery from Node C
• Backup Node C (Data)
• Backup Node B (my.cnf, mysqld-auto.cnf, auto.cnf)
• Recovery Node C Data on Node B
– COPY back mysqld-auto.cnf and auto.cnf to DATADIR
– Using Node B (my.cnf)
– Restart Node B
• Provisioning New
• DR  Recovery to Production (Failback)
27

MySQL InnoDB Cluster to MySQL InnoDB Cluster
• What is special in InnoDB Cluster vs Group Replication
– mysql_innodb_cluster_metadata : The InnoDB Cluster Data Dictionary Database
• The DC1 and DC2 MySQL InnoDB Cluster
– 2 metadata Database (Same name)
– DC1 InnoDB State change propagates to DC2
• Cluster1 and Cluster2 entries in METADATA
• Then HOW
28

• DC1 • DC2
29
MySQL InnoDB Cluster [ DC1  DC2 ]
MySQL MySQL MySQL MySQL MySQL MySQL
ClusterDC1
mysql_innodb_cluster_metadata
- clusterDC1
- clusterDC2
ClusterDC2
mysql_innodb_cluster_metadata
- clusterDC1
- clusterDC2
Replication
Replication

DC1 and DC2 with MySQL Replication and MySQL Router
MySQL Router
P
S S
MySQL Replication
Production Env DR Env

DC1 and DC2 with MySQL Replication and MySQL Router
MySQL Router
X
S S
MySQL Replication
MySQL Router
X
S P
MySQL Replication

Rolling Upgrade MySQL InnoDB Cluster
• Steps
– Upgrade Secondary Nodes and then Finally Upgrade Primary
– For each node
• SQL Mode :
mysql>set persist group_replication_start_on_boot=0;
mysql> shutdown;
• Start up the MySQL Server
# mysqld_safe --defaults-file=my.cnf &
• Upgrade the node using mysql_upgrade
# mysql_upgrade -uroot -h127.0.0.1 -P3306 –p
• SQL Mode
mysql>set persist group_replication_start_on_boot=1;
mysql> restart ;
32
https://youtu.be/_jbqvGlWExYVideo on Youtube

Troubleshooting MySQL InnoDB Cluster
• Node MISSING status
– The server has more Transaction than others – exit (Next Slide)
– Is it Network Stability
– Is it Long Transaction and Server is NOT responsive
– Regular VM Backup (like VM Consolidated Backup VCB) Freeze One Node (The node does not response
in 5 seconds and being kicked out)
• MySQL Router – Too many Connections
– Limitation in OLD Version of MySQL Router (e.g. MySQL 2.x)
– Upgrade to latest Version of MySQL Router
• O/S limits
– ulimit -a
33

Troubleshooting – GTID Inconsistency (example)
• GTID Inconsistency
– http://mysqlhk.blogspot.com/2019/07/troubleshooting-mysql-innodb-cluster.html
• 2019-07-04T14:34:11.465132Z 0 [ERROR] [MY-011526] [Repl] Plugin group_replication reported: 'This member has more executed
transactions than those present in the group. Local transactions: 0e632a11-9e24-11e9-a2ca-0800271b198a:1,
115217b5-9e24-11e9-8686-0800271b198a:1-17,
a7a6cc05-9e0f-11e9-ad3e-0800271b198a:1-20034 > Group transactions: 115217b5-9e24-11e9-8686-0800271b198a:1-17,
a7a6cc05-9e0f-11e9-ad3e-0800271b198a:1-20034'
2019-07-04T14:34:11.465213Z 0 [ERROR] [MY-011522] [Repl] Plugin group_replication reported: 'The member contains
transactions not present in the group. The member will now exit the group.'
34
0e632a11-9e24-11e9-a2ca-0800271b198a:1,
115217b5-9e24-11e9-8686-0800271b198a:1-17,
a7a6cc05-9e0f-11e9-ad3e-0800271b198a:1-20034
115217b5-9e24-11e9-8686-0800271b198a:1-17,
a7a6cc05-9e0f-11e9-ad3e-0800271b198a:1-20034>

• mysql> show binlog events in 'mysqllog.000005';+-----------------+-----+----------------+-----------+-------------+-----------------------------------------------------------------------------------------+
| Log_name | Pos | Event_type | Server_id | End_log_pos | Info |
+-----------------+-----+----------------+-----------+-------------+-----------------------------------------------------------------------------------------+
| mysqllog.000005 | 4 | Format_desc | 103 | 124 | Server ver: 8.0.16-commercial, Binlog ver: 4 |
| mysqllog.000005 | 124 | Previous_gtids | 103 | 231 | 115217b5-9e24-11e9-8686-0800271b198a:1-17,
a7a6cc05-9e0f-11e9-ad3e-0800271b198a:1-20034 |
| mysqllog.000005 | 231 | Gtid | 103 | 304 | SET @@SESSION.GTID_NEXT= '0e632a11-9e24-11e9-a2ca-0800271b198a:1' |
| mysqllog.000005 | 304 | Query | 103 | 383 | flush tables |
| mysqllog.000005 | 383 | Stop | 103 | 402 | |
+-----------------+-----+----------------+-----------+-------------+-----------------------------------------------------------------------------------------+
5 rows in set (0.00 sec)
35
Then you might try to identify Why there is a FLUSH TABLE

Fix it by adding the GTID to other server nodes – to make ALL nodes having the same SET of GTIDs
Login to the Node 3306 (which is the PRIMARY node), adding back the transaction :
mysql -uroot -h127.0.0.1 -P3306 -pmysql
mysql >SET @@SESSION.GTID_NEXT= '0e632a11-9e24-11e9-a2ca-0800271b198a:1';
mysql> flush tables;
mysql> exit

Summary
• MySQL InnoDB Cluster in action
• Network Stability
• MySQL InnoDB Cluster Operations
• Backup & Recovery & Member Provisioning
• GTID Consistency
• MySQL InnoDB Cluster to MySQL InnoDB Cluster Replication
• Troubleshooting
• GTID Consistency Walkthru with MySQL InnoDB Cluster
37

20190817 coscup-oracle my sql innodb cluster sharing

More Related Content

What's hot (20)

Similar to 20190817 coscup-oracle my sql innodb cluster sharing (20)

More from Ivan Ma (9)

Recently uploaded (20)

20190817 coscup-oracle my sql innodb cluster sharing