The Computer Science Behind a modern Distributed Database
2 likes429 views
The document discusses the complexities of implementing modern distributed databases, focusing on concepts such as resilience, consensus protocols (Paxos and Raft), sorting algorithms, log-structured merge trees (LSM-trees), hybrid logical clocks (HLC), and distributed ACID transactions. It emphasizes the importance of understanding these computer science principles to ensure data consistency and performance in distributed systems. Several methodologies and their challenges are outlined, as well as various databases that implement these concepts.
The Computer Science Behind a modern Distributed Database
- 1. The Computer Science behind a Modern Distributed Database Dan Larkin-York Chicago / February 20, 2018 www.arangodb.com
- 2. Overview Topics Resilience and Consensus Sorting Log-structured Merge Trees Hybrid Logical Clocks Distributed ACID Transactions Bottom line: You need CompSci to implement a modern data store
- 3. Resilience and Consensus The Problem A modern data store is distributed,
- 4. Resilience and Consensus The Problem A modern data store is distributed, because it needs to scale out and/or be resilient.
- 5. Resilience and Consensus The Problem A modern data store is distributed, because it needs to scale out and/or be resilient. Different parts of the system need to agree on things.
- 6. Resilience and Consensus The Problem A modern data store is distributed, because it needs to scale out and/or be resilient. Different parts of the system need to agree on things. Consensus is the art to achieve this as well as possible in software. This is relatively easy, if things are good, but very hard, if:
- 7. Resilience and Consensus The Problem A modern data store is distributed, because it needs to scale out and/or be resilient. Different parts of the system need to agree on things. Consensus is the art to achieve this as well as possible in software. This is relatively easy, if things are good, but very hard, if: the network has outages, the network has dropped, delayed or duplicated packets, disks fail (and come back with corrupt data), machines fail (and come back with old data), racks fail (and come back with or without data).
- 8. Resilience and Consensus The Problem A modern data store is distributed, because it needs to scale out and/or be resilient. Different parts of the system need to agree on things. Consensus is the art to achieve this as well as possible in software. This is relatively easy, if things are good, but very hard, if: the network has outages, the network has dropped, delayed or duplicated packets, disks fail (and come back with corrupt data), machines fail (and come back with old data), racks fail (and come back with or without data). (And we have not even talked about malicious attacks and enemy action.)
- 9. Paxos and Raft Traditionally, one uses the Paxos Consensus Protocol (1989 . . . 1998). More recently, Raft (2013) has been proposed. Paxos is a challenge to understand and to implement efficiently. Various variants exist. Raft is designed to be understandable.
- 10. Paxos and Raft Traditionally, one uses the Paxos Consensus Protocol (1989 . . . 1998). More recently, Raft (2013) has been proposed. Paxos is a challenge to understand and to implement efficiently. Various variants exist. Raft is designed to be understandable. My advice: First try to understand Paxos for some time (do not implement it!), then enjoy the beauty of Raft,
- 11. Paxos and Raft Traditionally, one uses the Paxos Consensus Protocol (1989 . . . 1998). More recently, Raft (2013) has been proposed. Paxos is a challenge to understand and to implement efficiently. Various variants exist. Raft is designed to be understandable. My advice: First try to understand Paxos for some time (do not implement it!), then enjoy the beauty of Raft, but do not implement it either!
- 12. Paxos and Raft Traditionally, one uses the Paxos Consensus Protocol (1989 . . . 1998). More recently, Raft (2013) has been proposed. Paxos is a challenge to understand and to implement efficiently. Various variants exist. Raft is designed to be understandable. My advice: First try to understand Paxos for some time (do not implement it!), then enjoy the beauty of Raft, but do not implement it either! Use some battle-tested implementation you trust!
- 13. Paxos and Raft Traditionally, one uses the Paxos Consensus Protocol (1989 . . . 1998). More recently, Raft (2013) has been proposed. Paxos is a challenge to understand and to implement efficiently. Various variants exist. Raft is designed to be understandable. My advice: First try to understand Paxos for some time (do not implement it!), then enjoy the beauty of Raft, but do not implement it either! Use some battle-tested implementation you trust! But most importantly: DO NOT TRY TO INVENT YOUR OWN!
- 14. Raft in a slide An odd number of servers each keep a persisted log of events.
- 15. Raft in a slide An odd number of servers each keep a persisted log of events. Everything is replicated to everybody.
- 16. Raft in a slide An odd number of servers each keep a persisted log of events. Everything is replicated to everybody. They democratically elect a leader with absolute majority.
- 17. Raft in a slide An odd number of servers each keep a persisted log of events. Everything is replicated to everybody. They democratically elect a leader with absolute majority. Only the leader may append to the replicated log.
- 18. Raft in a slide An odd number of servers each keep a persisted log of events. Everything is replicated to everybody. They democratically elect a leader with absolute majority. Only the leader may append to the replicated log. An append only counts when a majority has persisted and confirmed it.
- 19. Raft in a slide An odd number of servers each keep a persisted log of events. Everything is replicated to everybody. They democratically elect a leader with absolute majority. Only the leader may append to the replicated log. An append only counts when a majority has persisted and confirmed it. Very smart logic to ensure a unique leader and automatic recovery from failure.
- 20. Raft in a slide An odd number of servers each keep a persisted log of events. Everything is replicated to everybody. They democratically elect a leader with absolute majority. Only the leader may append to the replicated log. An append only counts when a majority has persisted and confirmed it. Very smart logic to ensure a unique leader and automatic recovery from failure. It is all a lot of fun to get right, but it is proven to work.
- 21. Raft in a slide An odd number of servers each keep a persisted log of events. Everything is replicated to everybody. They democratically elect a leader with absolute majority. Only the leader may append to the replicated log. An append only counts when a majority has persisted and confirmed it. Very smart logic to ensure a unique leader and automatic recovery from failure. It is all a lot of fun to get right, but it is proven to work. One puts a key/value store on top, the log contains the changes.
- 22. Raft demo Demo http://raft.github.io/raftscope/index.html (by Diego Ongaro)
- 23. Sorting The Problem Data stores need indexes. In practice, we need to sort things.
- 24. Sorting The Problem Data stores need indexes. In practice, we need to sort things. Most published algorithms are rubbish on modern hardware.
- 25. Sorting The Problem Data stores need indexes. In practice, we need to sort things. Most published algorithms are rubbish on modern hardware. The problem is no longer the comparison computations but the data movement.
- 26. Sorting The Problem Data stores need indexes. In practice, we need to sort things. Most published algorithms are rubbish on modern hardware. The problem is no longer the comparison computations but the data movement. Since 1983 and the Apple IIe, compute power in one core has increased by about ×20000 and now we have 32 cores in some CPUs a single memory access only by about ×40 this means computation has outpaced memory access by ×16000!
- 27. Idea for a parallel sorting algorithm: Merge Sort Min−Heap: sorted merged
- 28. Idea for a parallel sorting algorithm: Merge Sort Min−Heap: sorted merged Nearly all comparisons hit the L2 cache!
- 29. Log structured merge trees (LSM-trees) The Problem People rightfully expect from a data store, that it can hold more data than the available RAM, works well with SSDs and spinning rust, allows fast bulk inserts into large data sets, and provides fast reads in a hot set that fits into RAM.
- 30. Log structured merge trees (LSM-trees) The Problem People rightfully expect from a data store, that it can hold more data than the available RAM, works well with SSDs and spinning rust, allows fast bulk inserts into large data sets, and provides fast reads in a hot set that fits into RAM. Traditional B-tree based structures often fail to deliver with the last 2.
- 31. Log structured merge trees (LSM-trees) (Source: http://www.benstopford.com/2015/02/14/log-structured-merge-trees/, Author: Ben Stopford, License: Creative Commons)
- 32. Log structured merge trees (LSM-trees) LSM-trees — summary writes first go into memtables, all files are sorted and immutable, compaction happens in the background, efficient merge sort can be used, all writes use sequential I/O, Bloom filters or Cuckoo filters for fast negatives, =⇒ good write throughput and reasonable read performance, used in ArangoDB, BigTable, Cassandra, FaunaDB, HBase, InfluxDB, LevelDB, MarkLogic, MongoDB, MySQL, RocksDB, SQLite4, WiredTiger, etc.
- 33. Hybrid Logical Clocks (HLC) The Problem Clocks in different nodes of distributed systems are not in sync.
- 34. Hybrid Logical Clocks (HLC) The Problem Clocks in different nodes of distributed systems are not in sync. general relativity poses fundamental obstructions to synchronicity, in practice, clock skew happens, Google can use atomic clocks, even with NTP (network time protocol) we have to live with ≈ 20ms.
- 35. Hybrid Logical Clocks (HLC) The Problem Clocks in different nodes of distributed systems are not in sync. general relativity poses fundamental obstructions to synchronicity, in practice, clock skew happens, Google can use atomic clocks, even with NTP (network time protocol) we have to live with ≈ 20ms. Therefore, we cannot compare time stamps from different nodes!
- 36. Hybrid Logical Clocks (HLC) The Problem Clocks in different nodes of distributed systems are not in sync. general relativity poses fundamental obstructions to synchronicity, in practice, clock skew happens, Google can use atomic clocks, even with NTP (network time protocol) we have to live with ≈ 20ms. Therefore, we cannot compare time stamps from different nodes! Why would this help? establish “happened after” relationship between events, e.g. for conflict resolution, log sorting, detecting network delays, time to live could be implemented easily.
- 37. Hybrid Logical Clocks (HLC) The Idea Every computer has a local clock, and we use NTP to synchronize.
- 38. Hybrid Logical Clocks (HLC) The Idea Every computer has a local clock, and we use NTP to synchronize. If two events on different machines are linked by causality, the cause should have a smaller time stamp than the effect.
- 39. Hybrid Logical Clocks (HLC) The Idea Every computer has a local clock, and we use NTP to synchronize. If two events on different machines are linked by causality, the cause should have a smaller time stamp than the effect. causality ⇐⇒ a message is sent Send a time stamp with every message. The HLC always returns a value > max(local clock, largest time stamp ever seen).
- 40. Hybrid Logical Clocks (HLC) The Idea Every computer has a local clock, and we use NTP to synchronize. If two events on different machines are linked by causality, the cause should have a smaller time stamp than the effect. causality ⇐⇒ a message is sent Send a time stamp with every message. The HLC always returns a value > max(local clock, largest time stamp ever seen). Causality is preserved, time can “catch up” with logical time eventually. http://muratbuffalo.blogspot.com.es/2014/07/ hybrid-logical-clocks.html
- 41. Distributed ACID Transactions Atomic either happens in its entirety or not at all Consistent reading sees a consistent state, writing pre- serves consistency Isolated concurrent transactions do not see each other Durable committed writes are preserved after shut- down and crashes
- 42. Distributed ACID Transactions Atomic either happens in its entirety or not at all Consistent reading sees a consistent state, writing pre- serves consistency Isolated concurrent transactions do not see each other Durable committed writes are preserved after shut- down and crashes (All relatively doable when transactions happen one after another!)
- 43. Distributed ACID Transactions The Problem In a distributed system: How to make sure, that all nodes agree on whether the transaction has happened? (Atomicity)
- 44. Distributed ACID Transactions The Problem In a distributed system: How to make sure, that all nodes agree on whether the transaction has happened? (Atomicity) How to create a consistent snapshot across nodes? (Consistency)
- 45. Distributed ACID Transactions The Problem In a distributed system: How to make sure, that all nodes agree on whether the transaction has happened? (Atomicity) How to create a consistent snapshot across nodes? (Consistency) How to hide ongoing activities until commit? (Isolation)
- 46. Distributed ACID Transactions The Problem In a distributed system: How to make sure, that all nodes agree on whether the transaction has happened? (Atomicity) How to create a consistent snapshot across nodes? (Consistency) How to hide ongoing activities until commit? (Isolation) How to handle lost nodes? (Durability)
- 47. Distributed ACID Transactions The Problem In a distributed system: How to make sure, that all nodes agree on whether the transaction has happened? (Atomicity) How to create a consistent snapshot across nodes? (Consistency) How to hide ongoing activities until commit? (Isolation) How to handle lost nodes? (Durability) We have to take replication, resilience and failover into account.
- 48. Distributed ACID Transactions WITHOUT Distributed databases without ACID transactions: ArangoDB, BigTable, Couchbase, Datastax, Dynamo, Elastic, HBase, MongoDB, RethinkDB, Riak, and lots more . . . WITH Distributed databases with ACID transactions: CockroachDB, FaunaDB, FoundationDB, MarkLogic, Spanner
- 49. Distributed ACID Transactions WITHOUT Distributed databases without ACID transactions: ArangoDB, BigTable, Couchbase, Datastax, Dynamo, Elastic, HBase, MongoDB, RethinkDB, Riak, and lots more . . . WITH Distributed databases with ACID transactions: CockroachDB, FaunaDB, FoundationDB, MarkLogic, Spanner =⇒ Very few distributed engines promise ACID, because this is hard!
- 50. Distributed ACID Transactions Basic Idea Use Multi Version Concurrency Control (MVCC), i.e. multiple revisions of a data item are kept.
- 51. Distributed ACID Transactions Basic Idea Use Multi Version Concurrency Control (MVCC), i.e. multiple revisions of a data item are kept. Do writes and replication decentrally and distributed, without them becoming visible from other transactions.
- 52. Distributed ACID Transactions Basic Idea Use Multi Version Concurrency Control (MVCC), i.e. multiple revisions of a data item are kept. Do writes and replication decentrally and distributed, without them becoming visible from other transactions. Then have some place, where there is a switch, which decides when the transaction becomes visible.
- 53. Distributed ACID Transactions Basic Idea Use Multi Version Concurrency Control (MVCC), i.e. multiple revisions of a data item are kept. Do writes and replication decentrally and distributed, without them becoming visible from other transactions. Then have some place, where there is a switch, which decides when the transaction becomes visible. These “switches” need to be persisted somewhere (durability), scale out (no bottleneck for commit/abort), be replicated (no single point of failure), be resilient in case of fail-over (fault-tolerance).
- 54. Distributed ACID Transactions Basic Idea Use Multi Version Concurrency Control (MVCC), i.e. multiple revisions of a data item are kept. Do writes and replication decentrally and distributed, without them becoming visible from other transactions. Then have some place, where there is a switch, which decides when the transaction becomes visible. These “switches” need to be persisted somewhere (durability), scale out (no bottleneck for commit/abort), be replicated (no single point of failure), be resilient in case of fail-over (fault-tolerance). Transaction visibility needs to be implemented (MVCC), so comparing time stamps play a crucial role.
- 55. Thank you! Further questions? Follow us on twitter: @arangodb Join our slack: slack.arangodb.com Download and documentation: https://arangodb.com Issues and source (Star us!): https://github.com/arangodb/arangodb Info and slides: https://arangodb.com/speakers/daniel-larkin-york