Distributed Applications with Apache Zookeeper

Editor's Notes

• #3: Wears a lot of hats. Can serve multiple purposes, all related to coordination of a distributed system. More concisely, Zookeeper is a coordination service for distributed applications. A distributed system is a software system in which components located on networked computers communicate and coordinate their actions by passing messages. The components interact with each other in order to achieve a common goal. Wikipedia
• #4: Digging a bit deeper: Simply put, it backs the nodes of your distributed system through a tree structure of znodes (more in upcoming slides) Central nervous system for your distributed application Centralized and replicated across an odd number of hosts to make up an ensemble Data is kept in memory and is backed up to a log for reliability. By using memory ZooKeeper is very fast and can handle the high loads typically seen in chatty coordination protocols across huge numbers of processes. Prefers read-heavy based applications Clients connect to any zk node in the ensemble and maintain that connection API of create, read, update, delete (but also watches, more in later slides) Replicated hosts make up an “ensemble”
• #5: Writes are bubbled up to one elected leader (which is why there should be an odd # of instances). A quorum must confirm the update. Updates are submitted concurrently by clients and committed by FIFO order. After update, incremental state changes are broadcast to replicas using Zookeeper Atomic Broadcast (ZAB). Each state change is incremental with respect to the previous state, so there is an implicit dependence on the order of the state changes. This guarantees: Sequential Consistency - Updates from a client will be applied in the order that they were sent. Atomicity - Updates either succeed or fail. No partial results.
• #6: Znodes maintain a stat structure that includes version numbers for data changes, acl changes. The stat structure also has timestamps. The version number, together with the timestamp allow ZooKeeper to validate the cache and to coordinate updates. Each time a znode's data changes, the version number increases. Watches Clients can set watches on znodes. Changes to that znode trigger the watch and then clear the watch. When a watch triggers, ZooKeeper sends the client a notification. Ephemeral Nodes ZooKeeper also has the notion of ephemeral nodes. These znodes exists as long as the session that created the znode is active. When the session ends the znode is deleted. Because of this behavior ephemeral znodes are not allowed to have children. Time in ZooKeeper ZooKeeper tracks time multiple ways: Zxid Every change to the ZooKeeper state receives a stamp in the form of a zxid (ZooKeeper Transaction Id). This exposes the total ordering of all changes to ZooKeeper. Each change will have a unique zxid and if zxid1 is smaller than zxid2 then zxid1 happened before zxid2. Version numbers Every change to a a node will cause an increase to one of the version numbers of that node. The three version numbers are version (number of changes to the data of a znode), cversion (number of changes to the children of a znode), and aversion (number of changes to the ACL of a znode). Ticks When using multi-server ZooKeeper, servers use ticks to define timing of events such as status uploads, session timeouts, connection timeouts between peers, etc. The tick time is only indirectly exposed through the minimum session timeout (2 times the tick time); if a client requests a session timeout less than the minimum session timeout, the server will tell the client that the session timeout is actually the minimum session timeout. Real time ZooKeeper doesn't use real time, or clock time, at all except to put timestamps into the stat structure on znode creation and znode modification. A critical component of ZooKeeper is Zab, the ZooKeeper Atomic Broadcast algorithm, which is the protocol that manages atomic updates to the replicas. It is responsible for agreeing on a leader in the ensemble, synchronizing the replicas, managing update transactions to be broadcast, as well as recovering from a crashed state to a valid state.
• #7: Why not simply use a database? Because of the guarantees
• #8: Fast especially in read-heavy workload (10:1 performance against writes) More performance data in backup slides
• #9: Service Discovery - watch thru heartbeat Let ELECTION be a path of choice of the application. To volunteer to be a leader: Create znode z with path "ELECTION/guid-n_" with both SEQUENCE and EPHEMERAL flags; Let C be the children of "ELECTION", and i be the sequence number of z; Watch for changes on "ELECTION/guid-n_j", where j is the largest sequence number such that j < i and n_j is a znode in C; Upon receiving a notification of znode deletion: Let C be the new set of children of ELECTION; If z is the smallest node in C, then execute leader procedure; Otherwise, watch for changes on "ELECTION/guid-n_j", where j is the largest sequence number such that j < i and n_j is a znode in C;
• #11: http://engineering.pinterest.com/post/55272557617/building-a-follower-model-from-scratch When a Redis machine reaches either the memory or CPU thresholds, we split it either horizontally orvertically. Vertical sharding a Redis machine is simply cutting the number of running Redis instances on the machine by half. We bring up a new master as a slave of the existing master and once the slaving is complete, we make it the new master for half of the Redis instances leaving the old master as the master for the other half. The entire user id space is split into 8192 virtual shards. We place one virtual shard per Redis DB, and run multiple Redis instances (ranging from 8 to 32) on each machine depending on the memory and CPU consumption of the shards on those instances. Similarly, we run multiple Redis DBs per Redis instance.
• #12: Once a part of Hadoop Currently maintained by Yahoo! and the Apache Software Foundation HBase is the Hadoop database, a distributed, scalable, big data store. HBase supports hosting of very large tables -- billions of rows X millions of columns -- atop clusters of commodity hardware. Currently, hbase clients find the cluster to connect to by asking zookeeper. The only configuration a client needs is the zk quorum to connect to. Masters and hbase slave nodes (regionservers) all register themselves with zk. If their znode evaporates, the master or regionserver is consided lost and repair begins. HBase currently will default to manage the zookeeper cluster. It does this in an attempt at not burdening users with yet another technology to figure; things are bad enough for the hbase noob what with hbase, hdfs, and mapreduce. Part of hbase's management of zk includes being able to see zk configuration in the hbase configuration files. Anything that has the hbase.zookeeper prefix will have its suffix mapped to the corresponding zoo.cfg setting (HBase parses its config. and feeds the relevant zk configurations to zk on start).
• #13: Using Java as it’s very readable and shows type information. The same can easily be achieved with Node.js, Perl, or your language of choice. Create - First we create a persistent node to act as a group (parent) for child nodes. Join - Second we create and join ephemeral child nodes to that parent node. Each example creates with “unsafe acl”, so anyone who interacts with ZK can perform an update to that node. An alternative is CREATOR_ALL_ACL where only the creator can control the node.
• #14: List Think of use cases (active sessions, active system workers, etc) Delete -1 deletes unconditionally. Can also put a version in here to delete, so it will not delete a node that does not match the version specified. Will broadcast to any clients watching that node. Because of the simplicity of operations on paths, each must catch errors if that node does not exist.
• #20: http://www.tcs.hut.fi/Studies/T-79.5001/reports/2012-deSouzaMedeiros.pdf http://web.stanford.edu/class/cs347/reading/zab.pdf