Running MariaDB in multiple data centers

Editor's Notes

• #2: Specific slide for use on presentations used at OpenWorks 2019 Note: the DB of MariaDB is not bolded
• #5: Let’s start our discussion today by trying to answer a few simple questions on the need an enterprise would consider a DB architecture that supports multi Data Center operations. As we’ll see today the reason can vary, the architectures can vary and most importantly the complexity and the ability to manage it will drive the choices one will make. Discuss Briefly, spend about 60 seconds just a few of the sub topics in general. We’ll get to greater detail later. Topology Choices for determined use cases Traditional Disaster Recovery/Secondary Site – HA/FO Asynchronous/Semi-Synchronous (Warm Site, Read/Write Split, Read Scale, Disbursed Backup Node) Explain Master/Slave Basics Geo-Synchronous Distributed – Multi-Master/Active-Active Synchronous(Hot Site, No Switch over) Application/Location Affinity(Application and DB partitioning) Explain Galera Basics Technical Architectures – How they work, How to Manage MDB Standard Master/Slave HA/FO/Read Scale/Read/Write Splitting Binlog and Relay Log MDB Cluster – Galera Cluster Multi Master/Active-Active WSREP, Certify, Apply flow Application/Location Affinity Split Brain(What it is and how to mitigate), Galera Arbitrator MaxScale – Simplicity and Management(Should we include ClusterControl ?) Query Routing Load Balancing Automated FO MaxScale Redunancy/SPF(keepalived) Monitoring SqlDM or some network tools ClusterControl ? Command line
• #6: Let’s start our discussion today by trying to answer a few simple questions on the need an enterprise would consider a DB architecture that supports multi Data Center operations. Today, companies are undergoing a digital transformation: offline operations are becoming online operations, enterprise applications are becoming customer-facing applications, and engagement is happening anywhere and everywhere via web, mobile and Internet of Things (IoT) applications – and when it comes to customer experience, availability is not a preference, it is a requirement. As we’ll see today the reason can vary, the architectures can vary and most importantly the complexity and the ability to manage it will drive the choices one will make.
• #7: Many reasons why we would run across multiple datacenters ! However, when it comes any of the 4 high level reasons, the trade-offs between performance, durability and consistency have to be considered. There are times when durability and consistency is more important than performance. There are times when performance is more important. The right trade-offs depends on the business needs, the use case and the technical requirements.
• #8: So what we’ll look at is how I can take MDB as illustrated above, assembly the necessary components and deploy across multiple DC’s. Basically – what are the capabilities with MDB that can be used in multi-DC architecture, what can I use ? MariaDB TX uses local storage and replication (with or without clustering) to provide high availability via multiple database servers. There is no single point of failure (SPOF). In fact, when MariaDB TX is configured for high availability, downtime due to an unplanned infrastructure failure is all but removed. MariaDB TX, with a history of proven enterprise reliability and community-led innovation, is a complete database solution for any and every enterprise. MariaDB TX, when deployed, is comprised of MariaDB connectors (e.g., JDBC/ODBC), MariaDB MaxScale (a database proxy and firewall), MariaDB Server, MariaDB Cluster (multi-master replication), MariaDB tools and access to MariaDB services – and is available via an enterprise open source subscription.
• #11: Clustering In the example above, a three-node cluster is deployed across two data centers with two nodes in the active data center (DC1) and one node in the passive data center (DC2). The configuration for the database proxy in DC1, Proxy 1, assigns a priority value of 1 to Node 1, 2 to Node 2 and 3 to Node 3. It assigns the role of master to Node 1 because Node 1 has the highest priority value. Proxy 1 uses a basic router to route all reads and writes to any node assigned the master role, Node 1 by default. If Node 1 fails, Proxy 1 will assign the role of master to Node 2 – in the same data center, DC1. If DC1 fails, applications can connect to the database proxy in DC2, Proxy 2. The configuration for Proxy 2 assigns a priority value of 1 to Node 3, 2 to Node 2 and 3 to Node 1. It assigns the role of master to Node 3 because it has the lowest priority value. Proxy 2 uses a basic router to route all reads and writes to any node assigned the master role, Node 3 by default. A cluster can be deployed across multiple data centers if a) there is enough network bandwidth between them to minimize latency and b) the write sets are small (i.e., transactions to not change a lot of rows). Partitioning the data.
• #12: Multiple DC’s Master/slave replication In the example below, circular replication (e.g., bidirectional master/slave replication) can be used to synchronize data between an active data center (DC1) and a passive data center (DC2). The master in DC1 is configured as a slave to the master in DC2. The master in DC2 is configured as a slave to the master in DC1. If DC1 fails, applications can connect to the database proxy, MariaDB MaxScale, in DC2.
• #13: Read scalability Master/slave replication In the example above, a second database proxy is configured and deployed as a binlog server to relay transactions from the master to many slaves for read scaling. The binlog server reduces the replication overhead on the master – instead of many slaves replicating from the master, a single binlog server replicates from it. The master is configured for semi-synchronous replication, for high availability and durability, with Slave M1 and Slave M2, and for asynchronous replication, for read scalability, with the binlog server. The database proxy is configured with two routers, one for each cluster, with each router having a different port. The first will route all writes to the master in Cluster 1. The second will route all reads to the slaves in Cluster 2 (Slave R2 to Slave R100). The Binlog Server MAXSCALE AS A REPLICATION PROXY, AKA THE BINLOG SERVER In database setups with a large number of users reading data, the Binlog Server can be used to offload traffic to the Master, make Master failover easier to handle and in general simplify replication. In this blog I will describe the benefits of Binlog Server and how to set up MaxScale as a Binlog Server. In a traditional MariaDB/MySQL replication setup a single master server is created and a set of slaves of MariaDB/MySQL servers are configured to pull the binlog files from the master, putting a lot of load on the master. Introducing a layer between the master server and the slave servers can reduce the load on the master by only serving MaxScale’s Binlog Server instead of all the slaves. The slaves will only need to be aware of the Binlog Server and not the real master server. Removing the requirement for the slaves to have knowledge of the master also simplifies the process of replacing a failed master within a replication environment. MaxScale, with Binlog Server, can act as a slave to the real master and as a master to the slaves in the same way as an intermediate MySQL master does, however it does not implement any re-execution of the statements within the binary log. The latency that is introduced is mostly added network latency associated with adding the extra network hop. There is no appreciable processing performed at the MaxScale level, other than for managing the local cache of the binlog files. In addition every MaxScale that is acting as a proxy of the master will have exactly the same binlog events as the master itself. This means that a slave can be moved between any of the MaxScale servers or to the real master without the need to perform any special processing. The result is much simpler behavior for failure recovery and the ability to have a very simple and redundant proxy layer for the slave servers. THE BINLOG SERVER’S MAIN FEATURES ARE: The Binlog Server requests and receives binlog records from the master server autonomously of any slave activity. Stored binlogs are identical to the one stored in the Master server. Binlog records received from the master must be relayed to the slaves that are able to accept them. The slave servers must be able to request historical binlog records without sending any additional traffic to the master server.
• #18: Master/slave replication The master assigns a transaction, a global transaction ID (GTID) and writes the transaction to its binary log. A slave requests the next transaction from the master (sending its current GTID), writes the transaction to its relay log and executes it.
• #19: Automatic failover The database proxy, MariaDB MaxScale, has built-in automatic failover. If it is enabled, and the master fails, it will promote the most up-to-date slave (based on GTID) to master and reconfigure the remaining slaves (if any) to replicate from it. In addition, if automatic rejoin is enabled and the failed master is recovered, it will be reconfigured as a slave. Asynchronous replication With asynchronous replication, transactions are replicated after being committed. The master does not wait for any of the slaves to acknowledge the transaction before committing it. It does not affect write performance. However, if the master fails and automatic failover is enabled, there will be data loss if one or more transactions have not been replicated to a slave. In the example above, the database proxy would promote Slave 1 to master because it is the slave with the highest GTID. However, the most recent transaction (GTID = 3) had not been replicated before the master failed. There would be data loss. Asynchronous replication is recommended for read-intensive workloads or mixed/write-intensive workloads where the highest write performance is required. Asynchronous replication is recommended for read-intensive workloads or mixed/write-intensive workloads where the highest write performance is required.
• #20: Semi-synchronous replication With semi-synchronous replication, a transaction is not committed until it has been replicated to a slave. It affects write performance, but the effect is minimized by waiting for transactions to replicate to one slave rather than every slave. However, if the master fails and automatic failover is enabled, there will be no data loss because every transaction has been replicated to a slave. In the example below, the database proxy would promote Slave 2 to master because it is the slave with the highest GTID. With semi-synchronous replication, there would be no data loss because at least one of the slaves will have every transaction written to its relay log. NOTE: The master will wait for up to 10 seconds (default) for a transaction to be replicated to a slave before it reverts to asynchronous replication. If it does, and one of the slaves catches up, the master will restore semi-synchronous replication. If all of the slaves are slow, the timeout can be reduced to maintain write performance (but with less durability), or increased to maintain durability (but with less write performance). Semi-synchronous replication is recommended for mixed/write-intensive workloads where high write performance and strong durability are required.
• #22: Clustering In the example above, a three-node cluster is deployed across two data centers with two nodes in the active data center (DC1) and one node in the passive data center (DC2). The configuration for the database proxy in DC1, Proxy 1, assigns a priority value of 1 to Node 1, 2 to Node 2 and 3 to Node 3. It assigns the role of master to Node 1 because Node 1 has the lowest priority value. Proxy 1 uses a basic router to route all reads and writes to any node assigned the master role, Node 1 by default. If Node 1 fails, Proxy 1 will assign the role of master to Node 2 – in the same data center, DC1. If DC1 fails, applications can connect to the database proxy in DC2, Proxy 2. The configuration for Proxy 2 assigns a priority value of 1 to Node 3, 2 to Node 2 and 3 to Node 1. It assigns the role of master to Node 3 because it has the lowest priority value. Proxy 2 uses a basic router to route all reads and writes to any node assigned the master role, Node 3 by default. Multi-master clustering MariaDB TX supports multi-master clustering via MariaDB Cluster (i.e., Galera Cluster). The originating node assigns a transaction a GTID, and during the commit phase, sends all of the rows modified by it (i.e., writes) to every node within the cluster, including itself. If the writes are accepted by every node within the cluster, the originating node applies the writes and commits the transaction. The other nodes will apply the writes and commit the transaction asynchronously. Automatic failover If there is a node failure, the cluster will automatically remove it and the database proxy, MariaDB MaxScale, will stop routing queries to it. If the database proxy was routing reads and writes to the failed node, and because every node can accept reads and writes, the database proxy will select a different node and begin routing reads and writes to it. Synchronous replication With synchronous replication, a transaction is not committed until its changes (i.e., modified rows) have been replicated to every node within the cluster. The write performance is limited by the slowest node within the cluster. However, if the node a write was routed to fails, there will be no data loss because the changes for every transaction will have been replicated to every node within the cluster. In the example below, the database proxy would be routing reads and writes to Node 2 because it has the lowest priority value of the remaining nodes. There would be no data loss because with synchronous replication, every node has the changes of every transaction. The database proxy can be configured so automatic failover is deterministic (e.g., based on priority value) by setting the use_priority parameter to “true” in the Galera Cluster monitor configuration and the priority parameter in the database server configurations. Clustering In the example below, a three-node cluster is deployed across two data centers with two nodes in the active data center (DC1) and one node in the passive data center (DC2). The configuration for the database proxy in DC1, Proxy 1, assigns a priority value of 1 to Node 1, 2 to Node 2 and 3 to Node 3. It assigns the role of master to Node 1 because Node 1 has the lowest priority value. Proxy 1 uses a basic router to route all reads and writes to any node assigned the master role, Node 1 by default. If Node 1 fails, Proxy 1 will assign the role of master to Node 2 – in the same data center, DC1. If DC1 fails, applications can connect to the database proxy in DC2, Proxy 2. The configuration for Proxy 2 assigns a priority value of 1 to Node 3, 2 to Node 2 and 3 to Node 1. It assigns the role of master to Node 3 because it has the lowest priority value. Proxy 2 uses a basic router to route all reads and writes to any node assigned the master role, Node 3 by default.
• #23: In a Galera cluster, an uneven number of nodes, e.g. 5 or 7, is strongly advised in order to avoid split-brain situations. MaxScale can be used for load balancing.
• #31: Specific slide for use on presentations used at OpenWorks 2019
• #46: In a Galera cluster, an uneven number of nodes, e.g. 5 or 7, is strongly advised in order to avoid split-brain situations. MaxScale can be used for load balancing.
• #49: One Service with - One listener port(readwritesplit) or two Service with a read and a write listener port(readconnrouter) traffic separated by queries - Each service configured with NamedServerFilter with pattern match on the queries to be separated to specific server [NamedServerFilter]type=filtermodule=namedserverfiltermatch= *from *usersoptions=ignorecaseserver=server3[MyService]type=servicerouter=readwritesplitservers=server1,server2,server3user=myuserpassword=mypasswordfilters=NamedServerFilter