SlideShare a Scribd company logo

CockroachDB: Architecture of a Geo-Distributed SQL Database

C4Media, profile picture
C4Media

This document presents an overview of CockroachDB, a geo-distributed SQL database focused on scalability, fault tolerance, and efficient data management. It discusses key concepts such as data ranges, replication, transactions, and performance optimizations, emphasizing its ability to handle data center failures and place data close to users. The architecture allows for automatic range splitting, load balancing, and resilience through the Raft consensus algorithm, making it suitable for applications that require high availability and strong consistency.

Technology
Related topics:
architecture-design • Distributed Systems Overview
1 of 111
1
2
3
4
5
Most read
6
Most read
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
Most read
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
Architecture of a Geo-Distributed SQL Database CockroachDB Peter Mattis (@petermattis), Co-founder & CTO
InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ cockroachdb-distributed-sql/
Presented at QCon New York www.qconnewyork.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
CockroachDB: Geo-distributed SQL Database Make Data Easy • Distributed ○ Horizontally scalable to grow with your application • Geo-distributed ○ Handle datacenter failures ○ Place data near usage ○ Push computation near data • SQL ○ Lingua-franca for rich data storage ○ Schemas, indexes, and transactions make app development easier
AGENDA ● Introduction ● Ranges and Replicas ● Transactions ● SQL Data in a KV World ● SQL Execution ● SQL Optimization
Distributed, Replicated, Transactional KV* • Keys and values are strings ○ Lexicographically ordered by key • Multi-version concurrency control (MVCC) ○ Values are never updated “in place”, newer versions shadow older versions ○ Tombstones are used to delete values ○ Provides snapshot to each transaction • Monolithic key-space * Not exposed for external usage
Monolithic Key Space DOGS carl dagne figment jack pinetop sooshi stella zee muddy peetey lula lady Monolithic logical key space ● Ordered lexicographically by key
Ranges DOGS carl dagne figment jack pinetop sooshi stella zee muddy peetey lula lady carl dagne figment jack muddy peetey lula lady pinetop sooshi stella zee Key space divided into contiguous ~64MB ranges Ranges are small enough to be moved/split quickly Ranges are large enough to amortize indexing overhead
Range Indexing DOGS carl dagne figment jack pinetop sooshi stella zee muddy peetey lula lady Index structure used to locate ranges (very much like a B-tree) 1 2 3 carl - jack lady - peetey pinetop - zee carl dagne figment jack muddy peetey lula lady pinetop sooshi stella zee
Ordered Range Scans DOGS carl dagne figment jack pinetop sooshi zee peetey lula lady Ordered keys enable efficient range scans dogs >= “muddy” AND <= “stella” 1 2 3 carl - jack lady - peetey pinetop - zee carl dagne figment jack peetey lula lady pinetop sooshi zee muddy stella stella muddy
Transactional Updates DOGS carl dagne figment jack pinetop sooshi zee peetey lula lady Transactions used to insert records into ranges 1 2 3 carl - jack lady - peetey pinetop - zee stella muddy INSERT[sunny] INSERT[sunny] Space available in range? - YES carl dagne figment jack muddy peetey lula lady pinetop sooshi stella zee pinetop sooshi stella zee muddy peetey lula lady ✓?
Transactional Updates DOGS carl dagne figment jack pinetop sooshi zee peetey lula lady 1 2 3 carl - jack lady - peetey pinetop - zee stella muddy INSERT[sunny] carl dagne figment jack muddy peetey lula lady pinetop sooshi stella sunny zee pinetop sooshi stella zee muddy peetey lula lady ✓ Transactions used to insert records into ranges INSERT[sunny]
Range Splits DOGS carl dagne figment jack pinetop sooshi zee peetey lula lady 1 2 3 carl - jack lady - peetey pinetop - zee stella muddy INSERT[rudy] carl dagne figment jack muddy peetey lula lady pinetop sooshi stella sunny zee pinetop sooshi stella zee muddy peetey lula lady BUT… what happens when a range is full? ✓? INSERT[rudy] Space available in range? - NO
Range Splits DOGS carl dagne figment jack pinetop sooshi zee peetey lula lady stella muddy INSERT[rudy] carl dagne figment jack muddy peetey lula lady pinetop rudy sooshi pinetop sooshi stella zee muddy peetey lula lady Ranges are automatically split, a new range index is created & order maintained ✓ INSERT[rudy] split range and insert stella sunny zee 1 2 3 carl - jack lady - peetey pinetop - sooshi 4 stella - zee
Raft and Replication Ranges (~64MB) are the unit of replication Each range is a Raft group (Raft is a consensus replication protocol) Default to 3 replicas, though this is configurable • Important system ranges default to 5 replicas • Note: 2 replicas doesn’t make sense in consensus replication Raft group
Raft and Replication Raft provides “atomic replication” of commands Commands are proposed by the leaseholder replica and distributed to the follower replicas, but only accepted when a quorum of replicas have acknowledged receipt * Leaseholder == Raft leader Raft group LEASEHOLDER
node1 node2 node4 node3 Range Leases muddy peetey lula lady pinetop sooshi stella zee carl dagne figment jack Reads with consensus Reads must talk to a quorum of replicas READ[carl]
node1 node2 node4 node3 Range Leases muddy peetey lula lady pinetop sooshi stella zee carl dagne figment jack Reads without consensus One replica is chosen as the leaseholder READ[carl] leaseholder
node1 node2 node4 node3 Range Leases muddy peetey lula lady pinetop sooshi stella zee carl dagne figment jack Reads without consensus One replica is chosen as the leaseholder ● Coordinates writes (proposal, key locking) ● Performs reads READ[carl] leaseholder
node1 node2 node4 node3 Replica Placement muddy peetey lula lady pinetop sooshi stella zee ● Space ● Diversity ● Load ● Latency carl dagne figment jack Each Range is a Raft state machine A Range has 1 or more Replicas
node1 node2 node4 node3 Replica Placement: Diversity muddy peetey lula lady carl dagne figment jack Diversity optimizes placement of replicas across “failure domains” ● Disk ● Single machine ● Rack ● Datacenter ● Region pinetop sooshi stella zee
node1 node2 node6 node4 node5 Replica Placement: Load muddy peetey lula lady pinetop sooshi stella zee carl dagne figment jack Load Balances placement using heuristics that considers real-time usage metrics of the data itself This range is high load as it is accessed more than others While we show this for ranges within a single table, this is also applicable across all ranges across ALL tables, which is the more typical situation node1node3
Replica Placement: Latency & Geo-partitioning muddy peetey lula lady carl dagne figment jack pinetop sooshi stella zee USE/muddy USE/stella USE/figment USE/dagne USW/jack USW/lady USW/peetey USW/pinetop EU/carl EU/lula EU/sooshi EU/zee We apply a constraint that indicates regional placement so we can ensure low latency access or jurisdictional control of data
Rebalancing Replicas node1 node5 node4 node2 node3 NEW Scale: Add a node If we add a node to the cluster, CockroachDB automatically redistributed replicas to even load across the cluster Uses the replica placement heuristics from previous slides to decide which node to add to and which to remove from
Rebalancing Replicas node1 node5 node4 node2 node3 NEW Scale: Add a node If we add a node to the cluster, CockroachDB automatically redistributed replicas to even load across the cluster Uses the replica placement heuristics from previous slides Movement is decomposed into adding a replica followed by removing a replica
Rebalancing Replicas node1 node5 node4 node2 node3 NEW Scale: Add a node If we add a node to the cluster, CockroachDB automatically redistributed replicas to even load across the cluster Uses the replica placement heuristics from previous slides Movement is decomposed into adding a replica followed by removing a replica
Rebalancing Replicas node1 node5 node4 node2 node3 Loss of a node Permanent Failure If a node goes down, the Raft group realizes a replica is missing and replaces it with a new replica on an active node Uses the replica placement heuristics from previous slides
Rebalancing Replicas node1 node5 node4 node2 node3 Loss of a node Permanent Failure If a node goes down, the Raft group realizes a replica is missing and replaces it with a new replica on an active node Uses the replica placement heuristics from previous slides The failed replica is removed from the Raft group and a new replica created. The leaseholder sends a snapshot of the Range’s state to bring the new replica up to date.
Rebalancing Replicas node1 node5 node4 node2 Loss of a node Temporary Failure If a node goes down for a moment, the leaseholder can “catch up” any replica that is behind The leaseholder can send commands to be replayed OR it can send a snapshot of the current Range data. We apply heuristics to decide which is most efficient for a given failure. node3
AGENDA ● Introduction ● Ranges and Replicas ● Transactions ● SQL Data in a KV World ● SQL Execution ● SQL Optimization
Transactions Atomicity, Consistency, Isolation, Durability Serializable Isolation • As if the transactions are run in a serial order • Gold standard isolation level • Make Data Easy - weaker isolation levels are too great a burden Transactions can span arbitrary ranges Conversational • The full set of operations is not required up front
Transactions Raft provides atomic writes to individual ranges Bootstrap transaction atomicity using Raft atomic writes Transaction record atomically flipped from PENDING to COMMIT
Distributed Transactions node1 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady node2 node3 carl peetey dagne lady lula muddy peetey lady node4 carl dagne figment jack pinetop sooshi stella zee pinetop sooshi stella zee pinetop sooshi stella zee carl peetey dagne lady lula muddy peetey lady carl dagne figment jack INSERT INTO dogs VALUES (sunny, ozzie)
Distributed Transactions node1 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady node2 node3 carl peetey dagne lady lula muddy peetey lady node4 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady carl dagne figment jack pinetop sooshi stella zee pinetop sooshi stella zee BEGIN TXN1 WRITE[sunny] GATEWAY INSERT INTO dogs VALUES (sunny, ozzie) transactions TXN1: PENDING pinetop sooshi stella zee
Distributed Transactions node1 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady node2 node3 carl peetey dagne lady lula muddy peetey lady node4 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady carl dagne figment jack pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee BEGIN TXN1 WRITE[sunny] GATEWAY INSERT INTO dogs VALUES (sunny, ozzie) transactions TXN1: PENDING pinetop sooshi stella sunny zee
Distributed Transactions node1 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady node2 node3 carl peetey dagne lady lula muddy peetey lady node4 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady carl dagne figment jack pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee BEGIN TXN1 WRITE[sunny] GATEWAY INSERT INTO dogs VALUES (sunny, ozzie) transactions TXN1: PENDING pinetop sooshi stella sunny zee ACK pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee
Distributed Transactions node1 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady node2 node3 carl peetey dagne lady lula muddy peetey lady node4 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady carl dagne figment jack pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee BEGIN TXN1 WRITE[sunny] GATEWAY INSERT INTO dogs VALUES (sunny, ozzie) transactions TXN1: PENDING pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee
Distributed Transactions node1 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady node2 node3 carl peetey dagne lady lula muddy peetey lady node4 carl dagne figment jack carl dagne figment jack pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee BEGIN TXN1 WRITE[sunny] WRITE[ozzie] GATEWAY INSERT INTO dogs VALUES (sunny, ozzie) transactions TXN1: PENDING pinetop sooshi stella sunny zee carl peetey dagne lady lula muddy ozzie peetey lady ACK pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee
Distributed Transactions node1 carl dagne figment jack carl peetey dagne lady lula muddy ozzie peetey lady node2 node3 carl peetey dagne lady lula muddy ozzie peetey lady node4 carl dagne figment jack carl dagne figment jack BEGIN TXN1 WRITE[sunny] WRITE[ozzie] GATEWAY INSERT INTO dogs VALUES (sunny, ozzie) transactions TXN1: PENDING carl peetey dagne lady lula muddy ozzie peetey lady pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee
Distributed Transactions node1 carl dagne figment jack node2 node3 node4 carl dagne figment jack carl dagne figment jack BEGIN TXN1 WRITE[sunny] WRITE[ozzie] GATEWAY INSERT INTO dogs VALUES (sunny, ozzie) transactions TXN1: PENDING ACK pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee carl peetey dagne lady lula muddy ozzie peetey lady carl peetey dagne lady lula muddy ozzie peetey lady carl peetey dagne lady lula muddy ozzie peetey lady
Distributed Transactions node1 carl dagne figment jack node2 node3 node4 carl dagne figment jack carl dagne figment jack BEGIN TXN1 WRITE[sunny] WRITE[ozzie] GATEWAY INSERT INTO dogs VALUES (sunny, ozzie) transactions TXN1: PENDING ACK pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee carl peetey dagne lady lula muddy ozzie peetey lady carl peetey dagne lady lula muddy ozzie peetey lady carl peetey dagne lady lula muddy ozzie peetey lady
Distributed Transactions node1 carl dagne figment jack carl peetey dagne lady lula muddy ozzie peetey lady node2 node3 carl peetey dagne lady lula muddy ozzie peetey lady node4 carl dagne figment jack carl dagne figment jack pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee BEGIN TXN1 WRITE[sunny] WRITE[ozzie] COMMIT GATEWAY INSERT INTO dogs VALUES (sunny, ozzie) transactions TXN1: COMMIT pinetop sooshi stella sunny zee carl peetey dagne lady lula muddy ozzie peetey lady
Distributed Transactions node1 carl dagne figment jack carl peetey dagne lady lula muddy ozzie peetey lady node2 node3 carl peetey dagne lady lula muddy ozzie peetey lady node4 carl dagne figment jack carl dagne figment jack pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee BEGIN TXN1 WRITE[sunny] WRITE[ozzie] COMMIT GATEWAY INSERT INTO dogs VALUES (sunny, ozzie) pinetop sooshi stella sunny zee carl peetey dagne lady lula muddy ozzie peetey lady ACK
Transactions: Pipelining Serial Pipelined
Transactions: Pipelining Serial Pipelined sunny sunny BEGIN WRITE[sunny] txn:sunny (pending)
Transactions: Pipelining Serial Pipelined txn:sunny (pending) sunny ozzie sunny ozzie BEGIN WRITE[sunny] WRITE[ozzie]
Transactions: Pipelining Serial Pipelined txn:sunny (pending) sunny ozzie txn:sunny (commit)[keys: sunny, ozzie] txn:sunny (staged)[keys: sunny, ozzie] sunny ozzie BEGIN WRITE[sunny] WRITE[ozzie] COMMIT
Transactions: Pipelining Serial Pipelined txn:sunny (pending) sunny ozzie txn:sunny (commit)[keys: sunny, ozzie] BEGIN WRITE[sunny] WRITE[ozzie] COMMIT Committed once all operations complete We replaced the centralized commit marker with a distributed one t sunny ozzie txn:sunny (staged)[keys: sunny, ozzie] * “Proved” with TLA+
AGENDA ● Introduction ● Ranges and Replicas ● Transactions ● SQL Data in a KV World ● SQL Execution ● SQL Optimization
SQL Structured Query Language Declarative, not imperative • These are the results I want vs perform these operations in this sequence Relational data model • Typed: INT, FLOAT, STRING, ... • Schemas: tables, rows, columns, foreign keys
SQL: Tabular Data in a KV World SQL data has columns and types?!? How do we store typed and columnar data in a distributed, replicated, transactional key-value store? • The SQL data model needs to be mapped to KV data • Reminder: keys and values are lexicographically sorted
CREATE TABLE inventory ( id INT PRIMARY KEY, name STRING, price FLOAT ) SQL Data Mapping: Inventory Table ID Name Price 1 Bat 1.11 2 Ball 2.22 3 Glove 3.33 Key Value /1 “Bat”,1.11 /2 “Ball”,2.22 /3 “Glove”,3.33
CREATE TABLE inventory ( id INT PRIMARY KEY, name STRING, price FLOAT ) SQL Data Mapping: Inventory Table ID Name Price 1 Bat 1.11 2 Ball 2.22 3 Glove 3.33 Key Value /<Table>/<Index>/1 “Bat”,1.11 /<Table>/<Index>/2 “Ball”,2.22 /<Table>/<Index>/3 “Glove”,3.33
CREATE TABLE inventory ( id INT PRIMARY KEY, name STRING, price FLOAT ) SQL Data Mapping: Inventory Table ID Name Price 1 Bat 1.11 2 Ball 2.22 3 Glove 3.33 Key Value /inventory/primary/1 “Bat”,1.11 /inventory/primary/2 “Ball”,2.22 /inventory/primary/3 “Glove”,3.33
CREATE TABLE inventory ( id INT PRIMARY KEY, name STRING, price FLOAT, INDEX name_idx (name) ) SQL Data Mapping: Inventory Table ID Name Price 1 Bat 1.11 2 Ball 2.22 3 Glove 3.33 Key Value /inventory/name_idx/”Bat”/1 ∅ /inventory/name_idx/”Ball”/2 ∅ /inventory/name_idx/”Glove”/3 ∅
CREATE TABLE inventory ( id INT PRIMARY KEY, name STRING, price FLOAT, INDEX name_idx (name) ) SQL Data Mapping: Inventory Table ID Name Price 1 Bat 1.11 2 Ball 2.22 3 Glove 3.33 4 Bat 4.44 Key Value /inventory/name_idx/”Bat”/1 ∅ /inventory/name_idx/”Ball”/2 ∅ /inventory/name_idx/”Glove”/3 ∅
CREATE TABLE inventory ( id INT PRIMARY KEY, name STRING, price FLOAT, INDEX name_idx (name) ) SQL Data Mapping: Inventory Table ID Name Price 1 Bat 1.11 2 Ball 2.22 3 Glove 3.33 4 Bat 4.44 Key Value /inventory/name_idx/”Bat”/1 ∅ /inventory/name_idx/”Ball”/2 ∅ /inventory/name_idx/”Glove”/3 ∅ /inventory/name_idx/”Bat”/4 ∅
AGENDA ● Introduction ● Ranges and Replicas ● Transactions ● SQL Data in a KV World ● SQL Execution ● SQL Optimization
SQL Execution Relational operators • Projection (SELECT <columns>) • Selection (WHERE <filter>) • Aggregation (GROUP BY <columns>) • Join (JOIN), union (UNION), intersect (INTERSECT) • Scan (FROM <table>) • Sort (ORDER BY) ○ Technically, not a relational operator
SQL Execution • Relational expressions have input expressions and scalar expressions ○ For example, a “filter” expression has 1 input expression and a scalar expression that filters the rows from the child ○ The scan expression has zero inputs • Query plan is a tree of relational expressions • SQL execution takes a query plan and runs the operations to completion
SQL Execution: Example SELECT name FROM inventory WHERE name >= “b” AND name < “c”
SQL Execution: Scan SELECT name FROM inventory WHERE name >= “b” AND name < “c” Scan inventory
SQL Execution: Filter SELECT name FROM inventory WHERE name >= “b” AND name < “c” Scan inventory Filter name >= “b” AND name < “c”
SQL Execution: Project SELECT name FROM inventory WHERE name >= “b” AND name < “c” Scan inventory Filter name >= “b” AND name < “c” Project name
SQL Execution: Project SELECT name FROM inventory WHERE name >= “b” AND name < “c” Scan inventory Filter name >= “b” AND name < “c” Project name Results
SQL Execution: Index Scans SELECT name FROM inventory WHERE name >= “b” AND name < “c” Scan inventory@name [“b” - “c”) The filter gets pushed into the scan
SQL Execution: Index Scans SELECT name FROM inventory WHERE name >= “b” AND name < “c” Scan inventory@name [“b” - “c”) Project name Results
SQL Execution: Correctness Correct SQL execution involves lots of bookkeeping • User defined tables, and indexes • Queries refer to table and column names • Execution uses table and column IDs • NULL handling
SQL Execution: Performance Performant SQL execution • Tight, well written code • Operator specialization ○ hash group by, stream group by ○ hash join, merge join, lookup join, zig-zag join • Distributed execution
SQL Execution: Group By SELECT COUNT(*), country FROM customers GROUP BY country Name Country Bob United States Hans Germany Jacques France Marie France Susan United States
SQL Execution: Hash Group By SELECT COUNT(*), country FROM customers GROUP BY country Name Country Bob United States Hans Germany Jacques France Marie France Susan United States
SQL Execution: Hash Group By SELECT COUNT(*), country FROM customers GROUP BY country Name Country Bob United States Hans Germany Jacques France Marie France Susan United States United States 1
SQL Execution: Hash Group By SELECT COUNT(*), country FROM customers GROUP BY country Name Country Bob United States Hans Germany Jacques France Marie France Susan United States United States 1 Germany 1
SQL Execution: Hash Group By SELECT COUNT(*), country FROM customers GROUP BY country Name Country Bob United States Hans Germany Jacques France Marie France Susan United States United States 1 Germany 1 France 1
SQL Execution: Hash Group By SELECT COUNT(*), country FROM customers GROUP BY country Name Country Bob United States Hans Germany Jacques France Marie France Susan United States United States 1 Germany 1 France 2
SQL Execution: Hash Group By SELECT COUNT(*), country FROM customers GROUP BY country Name Country Bob United States Hans Germany Jacques France Marie France Susan United States United States 2 Germany 1 France 2
SQL Execution: Group By Revisited SELECT COUNT(*), country FROM customers GROUP BY country Name Country Bob United States Hans Germany Jacques France Marie France Susan United States
SQL Execution: Sort on Grouping Column(s) SELECT COUNT(*), country FROM customers GROUP BY country Name Country Jacques France Marie France Hans Germany Bob United States Susan United States
SQL Execution: Streaming Group By SELECT COUNT(*), country FROM customers GROUP BY country Name Country Jacques France Marie France Hans Germany Bob United States Susan United States France 1
SQL Execution: Streaming Group By SELECT COUNT(*), country FROM customers GROUP BY country Name Country Jacques France Marie France Hans Germany Bob United States Susan United States France 2
SQL Execution: Streaming Group By SELECT COUNT(*), country FROM customers GROUP BY country Name Country Jacques France Marie France Hans Germany Bob United States Susan United States France 2 Germany 1
SQL Execution: Streaming Group By SELECT COUNT(*), country FROM customers GROUP BY country Name Country Jacques France Marie France Hans Germany Bob United States Susan United States France 2 Germany 1 United States 1
SQL Execution: Streaming Group By SELECT COUNT(*), country FROM customers GROUP BY country Name Country Jacques France Marie France Hans Germany Bob United States Susan United States France 2 Germany 1 United States 2
Distributed SQL Execution Network latencies and throughput are important considerations in geo-distributed setups Push fragments of computation as close to the data as possible
Distributed SQL Execution: Streaming Group By SELECT COUNT(*), country FROM customers GROUP BY country Scan customers Scan customers Scan customers scan scan scan
Distributed SQL Execution: Streaming Group By SELECT COUNT(*), country FROM customers GROUP BY country Scan customers Scan customers Scan customers Group-By “country” Group-By “country” Group-By “country” group-by group-by group-by
Distributed SQL Execution: Streaming Group By SELECT COUNT(*), country FROM customers GROUP BY country Scan customers Scan customers Scan customers Group-By “country” Group-By “country” Group-By “country” Group-By “country” group-by
AGENDA ● Introduction ● Ranges and Replicas ● Transactions ● SQL Data in a KV World ● SQL Execution ● SQL Optimization
SQL Optimization An optimizer explores many plans that are logically equivalent to a given query and chooses the best one Parse ExecuteSearch Memo Prep AST Plan Fold Constants Check Types Resolve Names Report Semantic Errors Compute properties Retrieve and attach stats Cost-independent transformations Cost-based transformationsParse SQL
SQL Optimization: Cost-Independent Transformations • Some transformations always make sense ○ Constant folding ○ Filter push-down ○ Decorrelating subqueries* ○ ... • These transformations are cost-independent ○ If the transformation can be applied to the query, it is applied • Domain Specific Language for transformations ○ Compiled down to code which efficiently matches query fragments in the memo ○ ~200 transformations currently defined * Actually cost-based, but we’re treating it as cost-independent right now
SQL Optimization: Filter Push-Down SELECT * FROM a JOIN b WHERE x > 10 Scan a@primary Filter x > 10 Results Scan b@primary Join Initial plan
SQL Optimization: Filter Push-Down SELECT * FROM a JOIN b WHERE x > 10 Scan a@primary Filter x > 10 Results Scan b@primary Join Filter x > 10 After filter push-down
SQL Optimization: Cost-Based Transformations • Some transformations are not universally good ○ Index selection ○ Join reordering ○ ... • These transformations are cost-based ○ When should the transformation be applied? ○ Need to try both paths and maintain both the original and transformed query ○ State explosion: thousands of possible query plans ■ Memo data structure maintains a forest of query plans ○ Estimate cost of each query, select query with lowest cost • Costing ○ Based on table statistics and estimating cardinality of inputs to relational expressions
SQL Optimization: Cost-based Index Selection The index to use for a query is affected by multiple factors • Filters and join conditions • Required ordering (ORDER BY) • Implicit ordering (GROUP BY) • Covering vs non-covering (i.e. is an index-join required) • Locality
SQL Optimization: Cost-based Index Selection SELECT * FROM a WHERE x > 10 ORDER BY y Required orderings affect index selection Sorting is expensive if there are a lot of rows Sorting can be the better option if there are few rows
SQL Optimization: Cost-based Index Selection Required orderings affect index selection Sorting is expensive if there are a lot of rows Sorting can be the better option if there are few rows Scan a@primary Filter x > 10 Sort y SELECT * FROM a WHERE x > 10 ORDER BY y
SQL Optimization: Cost-based Index Selection Required orderings affect index selection Sorting is expensive if there are a lot of rows Sorting can be the better option if there are few rows Scan a@primary Scan a@x [10 - ) Filter x > 10 Sort y Sort y SELECT * FROM a WHERE x > 10 ORDER BY y
SQL Optimization: Cost-based Index Selection Required orderings affect index selection Sorting is expensive if there are a lot of rows Sorting can be the better option if there are few rows Scan a@primary Scan a@x [10 - ) Filter x > 10 Scan a@y Sort y Sort y Filter x > 10 SELECT * FROM a WHERE x > 10 ORDER BY y
SQL Optimization: Cost-based Index Selection Required orderings affect index selection Sorting is expensive if there are a lot of rows Sorting can be the better option if there are few rows Scan a@primary Scan a@x [10 - ) Filter x > 10 Scan a@y Sort y Sort y Filter x > 10 SELECT * FROM a WHERE x > 10 ORDER BY y 10 100,000 10 10 Lowest Cost
SQL Optimization: Cost-based Index Selection Required orderings affect index selection Sorting is expensive if there are a lot of rows Sorting can be the better option if there are few rows Scan a@primary Scan a@x [10 - ) Filter x > 10 Scan a@y Sort y Sort y Filter x > 10 SELECT * FROM a WHERE x > 10 ORDER BY y 50,000 100,000 50,000 50,000 Lowest Cost
Locality-Aware SQL Optimization Network latencies and throughput are important considerations in geo-distributed setups Duplicate read-mostly data in each locality Plan queries to use data from the same locality
Locality-Aware SQL Optimization Three copies of the postal_codes table data Use replication constraints to pin the copies to different geographic regions (US-East, US-West, EU) CREATE TABLE postal_codes ( id INT PRIMARY KEY, code STRING, INDEX idx_eu (id) STORING (code), INDEX idx_usw (id) STORING (code) )
Locality-Aware SQL Optimization Optimizer includes locality in cost model Automatically selects index from same locality: primary, idx_eu, or idx_usw CREATE TABLE postal_codes ( id INT PRIMARY KEY, code STRING, INDEX idx_eu (id) STORING (code), INDEX idx_usw (id) STORING (code) ) SELECT * FROM postal_codes
Conclusion ● Distributed, replicated, transactional key-value store ● Monolithic key space ● Raft replication of ranges (~64MB) ● Replica placement signals: space, diversity, load, latency ● Pipelined transaction operations ● Mapping SQL data to KV storage ● Distributed SQL execution ● Distributed SQL optimization
www.cockroachlabs.com github.com/cockroachdb/cockroach Thank You
A Simple Transaction node1 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady node1 node1 carl peetey dagne lady lula muddy peetey lady node1 carl dagne figment jack pinetop sooshi stella zee pinetop sooshi stella zee pinetop sooshi stella zee carl peetey dagne lady lula muddy peetey lady carl dagne figment jack INSERT INTO DOGS (sunny);
A Simple Transaction: One Range node1 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady node1 node1 carl peetey dagne lady lula muddy peetey lady node1 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady carl dagne figment jack pinetop sooshi stella zee pinetop sooshi stella zee pinetop sooshi stella sunny zee BEGIN WRITE[sunny] COMMIT GATEWAY INSERT INTO DOGS (sunny); NOTE: a gateway can be ANY CockroachDB instance. It can find the leaseholder for any range and execute a transaction
A Simple Transaction: One Range node1 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady node1 node1 carl peetey dagne lady lula muddy peetey lady node1 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady carl dagne figment jack pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee BEGIN WRITE[sunny] COMMIT GATEWAY INSERT INTO DOGS (sunny); ACK
A Simple Transaction: One Range node1 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady node1 node1 carl peetey dagne lady lula muddy peetey lady node1 carl dagne figment jack carl peetey dagne lady lula muddy peetey lady carl dagne figment jack pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee pinetop sooshi stella sunny zee BEGIN WRITE[sunny] COMMIT GATEWAY INSERT INTO DOGS (sunny); ACK
Ranges CockroachDB implements order-preserving data distribution • Automates sharding of key/value data into “ranges” • Supports efficient range scans • Requires an indexing structure Foundational capability that enables efficient distribution of data across nodes within a CockroachDB cluster * This approach is also used by Bigtable (tablets), HBase (regions) & Spanner (ranges)
Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ cockroachdb-distributed-sql/

More Related Content

PDF
Enterprise Architecture vs. Data Architecture
DATAVERSITY
 
PPT
Consistency protocols
ZongYing Lyu
 
PPTX
NoSQL Graph Databases - Why, When and Where
Eugene Hanikblum
 
PDF
Smart Sensors for Infrastructure and Structural Health Monitoring
Jeffrey Funk
 
PPTX
OpenTelemetry For Architects
Kevin Brockhoff
 
PPTX
Replication in Distributed Systems
Kavya Barnadhya Hazarika
 
PPTX
Introduction to couchbase
Dipti Borkar
 
PPTX
Azure DevOps
Juan Fabian
 
Enterprise Architecture vs. Data Architecture
DATAVERSITY
 
Consistency protocols
ZongYing Lyu
 
NoSQL Graph Databases - Why, When and Where
Eugene Hanikblum
 
Smart Sensors for Infrastructure and Structural Health Monitoring
Jeffrey Funk
 
OpenTelemetry For Architects
Kevin Brockhoff
 
Replication in Distributed Systems
Kavya Barnadhya Hazarika
 
Introduction to couchbase
Dipti Borkar
 
Azure DevOps
Juan Fabian
 

What's hot (20)

PPTX
CockroachDB
andrei moga
 
PDF
MinIO January 2020 Briefing
Jonathan Symonds
 
PDF
ProxySQL High Avalability and Configuration Management Overview
René Cannaò
 
PPTX
kafka
Amikam Snir
 
PPTX
Ceph Performance and Sizing Guide
Jose De La Rosa
 
PPTX
PostgreSQL and CockroachDB SQL
CockroachDB
 
PDF
Producer Performance Tuning for Apache Kafka
Jiangjie Qin
 
PDF
All about Zookeeper and ClickHouse Keeper.pdf
Altinity Ltd
 
PDF
Apache Kafka Fundamentals for Architects, Admins and Developers
confluent
 
PDF
Getting Started with Kubernetes
VMware Tanzu
 
PDF
Getting Started with Confluent Schema Registry
confluent
 
PDF
Handle Large Messages In Apache Kafka
Jiangjie Qin
 
PDF
Apache Kafka Introduction
Amita Mirajkar
 
PPTX
Introduction to Apache Kafka
Jeff Holoman
 
PDF
Yahoo! JAPANのプライベートRDBクラウドとマルチライター型 MySQL #dbts2017 #dbtsOSS
Yahoo!デベロッパーネットワーク
 
PDF
Kafka 101 and Developer Best Practices
confluent
 
PDF
Implementing Observability for Kubernetes.pdf
Jose Manuel Ortega Candel
 
PDF
Autoscaling Kubernetes
craigbox
 
PPTX
Ceph Introduction 2017
Karan Singh
 
PDF
Linux Container Technology 101
inside-BigData.com
 
CockroachDB
andrei moga
 
MinIO January 2020 Briefing
Jonathan Symonds
 
ProxySQL High Avalability and Configuration Management Overview
René Cannaò
 
kafka
Amikam Snir
 
Ceph Performance and Sizing Guide
Jose De La Rosa
 
PostgreSQL and CockroachDB SQL
CockroachDB
 
Producer Performance Tuning for Apache Kafka
Jiangjie Qin
 
All about Zookeeper and ClickHouse Keeper.pdf
Altinity Ltd
 
Apache Kafka Fundamentals for Architects, Admins and Developers
confluent
 
Getting Started with Kubernetes
VMware Tanzu
 
Getting Started with Confluent Schema Registry
confluent
 
Handle Large Messages In Apache Kafka
Jiangjie Qin
 
Apache Kafka Introduction
Amita Mirajkar
 
Introduction to Apache Kafka
Jeff Holoman
 
Yahoo! JAPANのプライベートRDBクラウドとマルチライター型 MySQL #dbts2017 #dbtsOSS
Yahoo!デベロッパーネットワーク
 
Kafka 101 and Developer Best Practices
confluent
 
Implementing Observability for Kubernetes.pdf
Jose Manuel Ortega Candel
 
Autoscaling Kubernetes
craigbox
 
Ceph Introduction 2017
Karan Singh
 
Linux Container Technology 101
inside-BigData.com
 
Ad

Similar to CockroachDB: Architecture of a Geo-Distributed SQL Database (20)

PPT
HPTS talk on micro-sharding with Katta
Ted Dunning
 
PDF
Using OpenStack Swift for Extreme Data Durability
Christian Schwede
 
PDF
Ippevent : openshift Introduction
kanedafromparis
 
PPTX
Scylla Summit 2018: Consensus in Eventually Consistent Databases
ScyllaDB
 
PDF
5 levels of high availability from multi instance to hybrid cloud
Rafał Leszko
 
PPTX
SolrCloud in Public Cloud: Scaling Compute Independently from Storage - Ilan ...
Lucidworks
 
PDF
5 Levels of High Availability: From Multi-instance to Hybrid Cloud
Rafał Leszko
 
PDF
MSR 2009
swy351
 
PPT
HPTS talk on micro sharding with Katta
MapR Technologies
 
PDF
Percon XtraDB Cluster in a nutshell
Frederic Descamps
 
PPTX
Taking Splunk to the Next Level - Architecture Breakout Session
Splunk
 
PPTX
Leveraging Endpoint Flexibility in Data-Intensive Clusters
Ran Ziv
 
PPTX
Ten tools for ten big data areas 03_Apache Spark
Will Du
 
PPTX
Using Kubernetes to deliver a “serverless” service
DoKC
 
PDF
NetflixOSS Open House Lightning talks
Ruslan Meshenberg
 
PDF
DEVIEW 2013
Patrick McGarry
 
PDF
Scalable Persistent Storage for Erlang: Theory and Practice
Amir Ghaffari
 
PDF
Elasticsearch
Christophe Marchal
 
PPTX
Lessons learned from running Spark on Docker
DataWorks Summit
 
PDF
Elastic Data Analytics Platform @Datadog
C4Media
 
HPTS talk on micro-sharding with Katta
Ted Dunning
 
Using OpenStack Swift for Extreme Data Durability
Christian Schwede
 
Ippevent : openshift Introduction
kanedafromparis
 
Scylla Summit 2018: Consensus in Eventually Consistent Databases
ScyllaDB
 
5 levels of high availability from multi instance to hybrid cloud
Rafał Leszko
 
SolrCloud in Public Cloud: Scaling Compute Independently from Storage - Ilan ...
Lucidworks
 
5 Levels of High Availability: From Multi-instance to Hybrid Cloud
Rafał Leszko
 
MSR 2009
swy351
 
HPTS talk on micro sharding with Katta
MapR Technologies
 
Percon XtraDB Cluster in a nutshell
Frederic Descamps
 
Taking Splunk to the Next Level - Architecture Breakout Session
Splunk
 
Leveraging Endpoint Flexibility in Data-Intensive Clusters
Ran Ziv
 
Ten tools for ten big data areas 03_Apache Spark
Will Du
 
Using Kubernetes to deliver a “serverless” service
DoKC
 
NetflixOSS Open House Lightning talks
Ruslan Meshenberg
 
DEVIEW 2013
Patrick McGarry
 
Scalable Persistent Storage for Erlang: Theory and Practice
Amir Ghaffari
 
Elasticsearch
Christophe Marchal
 
Lessons learned from running Spark on Docker
DataWorks Summit
 
Elastic Data Analytics Platform @Datadog
C4Media
 
Ad

More from C4Media (20)

PDF
Streaming a Million Likes/Second: Real-Time Interactions on Live Video
C4Media
 
PDF
Next Generation Client APIs in Envoy Mobile
C4Media
 
PDF
Software Teams and Teamwork Trends Report Q1 2020
C4Media
 
PDF
Understand the Trade-offs Using Compilers for Java Applications
C4Media
 
PDF
Kafka Needs No Keeper
C4Media
 
PDF
High Performing Teams Act Like Owners
C4Media
 
PDF
Does Java Need Inline Types? What Project Valhalla Can Bring to Java
C4Media
 
PDF
Service Meshes- The Ultimate Guide
C4Media
 
PDF
Shifting Left with Cloud Native CI/CD
C4Media
 
PDF
CI/CD for Machine Learning
C4Media
 
PDF
Fault Tolerance at Speed
C4Media
 
PDF
Architectures That Scale Deep - Regaining Control in Deep Systems
C4Media
 
PDF
ML in the Browser: Interactive Experiences with Tensorflow.js
C4Media
 
PDF
Build Your Own WebAssembly Compiler
C4Media
 
PDF
User & Device Identity for Microservices @ Netflix Scale
C4Media
 
PDF
Scaling Patterns for Netflix's Edge
C4Media
 
PDF
Make Your Electron App Feel at Home Everywhere
C4Media
 
PDF
The Talk You've Been Await-ing For
C4Media
 
PDF
Future of Data Engineering
C4Media
 
PDF
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
C4Media
 
Streaming a Million Likes/Second: Real-Time Interactions on Live Video
C4Media
 
Next Generation Client APIs in Envoy Mobile
C4Media
 
Software Teams and Teamwork Trends Report Q1 2020
C4Media
 
Understand the Trade-offs Using Compilers for Java Applications
C4Media
 
Kafka Needs No Keeper
C4Media
 
High Performing Teams Act Like Owners
C4Media
 
Does Java Need Inline Types? What Project Valhalla Can Bring to Java
C4Media
 
Service Meshes- The Ultimate Guide
C4Media
 
Shifting Left with Cloud Native CI/CD
C4Media
 
CI/CD for Machine Learning
C4Media
 
Fault Tolerance at Speed
C4Media
 
Architectures That Scale Deep - Regaining Control in Deep Systems
C4Media
 
ML in the Browser: Interactive Experiences with Tensorflow.js
C4Media
 
Build Your Own WebAssembly Compiler
C4Media
 
User & Device Identity for Microservices @ Netflix Scale
C4Media
 
Scaling Patterns for Netflix's Edge
C4Media
 
Make Your Electron App Feel at Home Everywhere
C4Media
 
The Talk You've Been Await-ing For
C4Media
 
Future of Data Engineering
C4Media
 
Automated Testing for Terraform, Docker, Packer, Kubernetes, and More
C4Media
 

Recently uploaded (20)

PDF
NewMind AI Weekly Chronicles - August'25-Week II
NewMind AI
 
PDF
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
PDF
gpt5_lecture_notes_comprehensive_20250812015547.pdf
Chinchu40
 
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
PDF
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Safe Software
 
PDF
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
PPTX
A Presentation on Artificial Intelligence
memembruh336
 
PDF
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
PDF
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
PDF
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
PDF
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
PDF
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
PDF
Approach and Philosophy of On baking technology
30anthuong17
 
PPT
Teaching material agriculture food technology
LiaRayya
 
PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
NewMind AI Weekly Chronicles - August'25-Week II
NewMind AI
 
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
gpt5_lecture_notes_comprehensive_20250812015547.pdf
Chinchu40
 
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
Peak of Data & AI Encore- AI for Metadata and Smarter Workflows
Safe Software
 
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
A Presentation on Artificial Intelligence
memembruh336
 
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
Electronic commerce courselecture one. Pdf
OmerMohamed64
 
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
Approach and Philosophy of On baking technology
30anthuong17
 
Teaching material agriculture food technology
LiaRayya
 
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 

CockroachDB: Architecture of a Geo-Distributed SQL Database