![Design Concepts
NQuad :
Subject P r e d i c a t e Object .
<a l i c e > <f r i e n d > <bob> .
<a l i c e > <f r i e n d > <eve> .
<bob> <f r i e n d > <eve> .
Posting L i s t :
Subject P r e d i c a t e => ∗
<a l i c e > <f r i e n d > => [<bob>, <eve >]
Shard :
A l l p o s t i n g l i s t s f o r one p r e d i c a t e
<f r i e n d > => [< a l i c e f r i e n d >, <bob f r i e n d >]
Group :
Shards of d i f f e r e n t p r e d i c a t e s form a group
[ ∗ , P1 , ∗ ] , [ ∗ , P2 , ∗ ] , . . .
Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 7 / 23](https://image.slidesharecdn.com/dgraph-graphdatabaseforproductionenvironment-180115150317/85/Dgraph-Graph-database-for-production-environment-11-320.jpg)
![Design Query Execution
Query Execution
Tree-like query structure.
Each branch is handled by one goroutine, allowing concurrent
execution of sibling branches.
Rough algorithm:
1 Retrieving results from network
2 Apply Filters, if any [one goroutine per child filter]
3 If required, do sort over network and run pagination
4 If only count required, do count and return
5 Process children, one child per goroutine recursively. Each child again
starts at 1.
Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 9 / 23](https://image.slidesharecdn.com/dgraph-graphdatabaseforproductionenvironment-180115150317/85/Dgraph-Graph-database-for-production-environment-13-320.jpg)
Dgraph: Graph database for production environment
- 1. Dgraph: Graph database for production environment Tomasz Zdybał, Dgraph Labs https://dgraph.io Feb 08, 2017 Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 1 / 23
- 2. Overview Overview 1 Introduction 2 Design 3 Highly debated decisions 4 GraphQL+- query language 5 Benchmarks 6 Q & A Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 2 / 23
- 3. Introduction Introduction Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 3 / 23
- 4. Introduction What is Dgraph? Dgraph is an open source graph database built for web-scale production environments written entirely in Go. Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 4 / 23
- 5. Introduction What is Dgraph? Dgraph is an open source graph database built for web-scale production environments written entirely in Go. Goals Sharded and Distributed Automatic Replication Consistent Highly Available by design Fault Tolerant Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 4 / 23
- 6. Introduction Why build it? Graph technology is hidden, but everywhere. Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 5 / 23
- 7. Introduction Why build it? Graph technology is hidden, but everywhere. Any company doing anything smart is using graphs. Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 5 / 23
- 8. Introduction Why build it? Graph technology is hidden, but everywhere. Any company doing anything smart is using graphs. Existing graphs have limited scalability (Neo4j), or graph layers on top of other databases (Titan/DSE). Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 5 / 23
- 9. Introduction Why build it? Graph technology is hidden, but everywhere. Any company doing anything smart is using graphs. Existing graphs have limited scalability (Neo4j), or graph layers on top of other databases (Titan/DSE). With no existing standard, companies have to build their own graph systems (Facebook Tao, Google Knowledge Graph, Twitter FlockDB, Dropbox, Pinterest, numerous others). Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 5 / 23
- 10. Design Design Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 6 / 23
- 11. Design Concepts NQuad : Subject P r e d i c a t e Object . <a l i c e > <f r i e n d > <bob> . <a l i c e > <f r i e n d > <eve> . <bob> <f r i e n d > <eve> . Posting L i s t : Subject P r e d i c a t e => ∗ <a l i c e > <f r i e n d > => [<bob>, <eve >] Shard : A l l p o s t i n g l i s t s f o r one p r e d i c a t e <f r i e n d > => [< a l i c e f r i e n d >, <bob f r i e n d >] Group : Shards of d i f f e r e n t p r e d i c a t e s form a group [ ∗ , P1 , ∗ ] , [ ∗ , P2 , ∗ ] , . . . Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 7 / 23
- 12. Design Raft and Replication Raft and Replication Groups are replicated across servers. Reads and writes go via Raft consensus algorithm. Each server can handle many groups. Ideally 1, 3 or 5 servers handling a group (odd number for consensus). Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 8 / 23
- 13. Design Query Execution Query Execution Tree-like query structure. Each branch is handled by one goroutine, allowing concurrent execution of sibling branches. Rough algorithm: 1 Retrieving results from network 2 Apply Filters, if any [one goroutine per child filter] 3 If required, do sort over network and run pagination 4 If only count required, do count and return 5 Process children, one child per goroutine recursively. Each child again starts at 1. Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 9 / 23
- 14. Highly debated decisions Highly debated decisions Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 10 / 23
- 15. Highly debated decisions Serialization Protocol Buffers vs Flatbuffers Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 11 / 23
- 16. Highly debated decisions Key-Value Store RocksDB vs BoltDB BoltDB acquires a global mutex lock around all reads and writes; which skews it towards read-only system. Going to Cgo for RocksDB is as fast as Bolt; but performs a whole lot better under read-write workload. Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 12 / 23
- 17. GraphQL+- query language GraphQL+- query language Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 13 / 23
- 18. GraphQL+- query language Introduction GraphQL+- used as primary language of communication feature-rich variation of GraphQL added support for graph operations removed features not fitting well for a graph database Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 14 / 23
- 19. GraphQL+- query language Introduction GraphQL+- used as primary language of communication feature-rich variation of GraphQL added support for graph operations removed features not fitting well for a graph database Why GraphQL as a base? graph-like query syntax schema validation subgraph shaped response Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 14 / 23
- 20. GraphQL+- query language Features GraphQL+- features pagination (using first, offset and after arguments) aliasing of predicate names counting number of entities filtering (at root and predicate level) various functions (comparison, term matching, geolocation) sorting/ordering variables result normalization Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 15 / 23
- 21. GraphQL+- query language Example { me( i d : m.06 pj8 ) { name . en d i r e c t o r . f i l m @ f i l t e r ( a l l o f (name . en , ” j o n e s i n d i a n a ” ) OR a l l o f (name . en , ” j u r a s s i c park ” ) ) { u i d name . en } } } Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 16 / 23
- 22. Benchmarks Benchmarks Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 17 / 23
- 23. Benchmarks Cayley Cayley Cayley is a graph layer, not a graph database. Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 18 / 23
- 24. Benchmarks Cayley Cayley Cayley is a graph layer, not a graph database. Data handling is done outside of Cayley by other databases. Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 18 / 23
- 25. Benchmarks Cayley Cayley Cayley is a graph layer, not a graph database. Data handling is done outside of Cayley by other databases. Results (Cayley with Bolt on Macbook) Loading 21M RDFs – Dgraph is 9.7x faster Queries (Gremlin) – Dgraph is 36.6x faster Queries (MQL) – Dgraph is 5x faster Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 18 / 23
- 26. Benchmarks Cayley Cayley Cayley is a graph layer, not a graph database. Data handling is done outside of Cayley by other databases. Results (Cayley with Bolt on Macbook) Loading 21M RDFs – Dgraph is 9.7x faster Queries (Gremlin) – Dgraph is 36.6x faster Queries (MQL) – Dgraph is 5x faster Dgraph would perform even better on distributed setup, when distributed joins come into play. Link to full benchmark code: https://github.com/ankurayadav/graphdb-benchmarks Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 18 / 23
- 27. Benchmarks Neo4j Neo4j Probably the most popular graph database Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 19 / 23
- 28. Benchmarks Neo4j Neo4j Probably the most popular graph database Results (on Thinkpad T460) Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 19 / 23
- 29. Benchmarks Neo4j Neo4j Probably the most popular graph database Results (on Thinkpad T460) Loading 1.1M RDFs – Dgraph is 100x faster Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 19 / 23
- 30. Benchmarks Neo4j Neo4j Probably the most popular graph database Results (on Thinkpad T460) Loading 1.1M RDFs – Dgraph is 100x faster Queries – Dgraph is 3x-6x faster on a read-write workload and at least as fast on read only Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 19 / 23
- 31. Benchmarks Neo4j Neo4j Probably the most popular graph database Results (on Thinkpad T460) Loading 1.1M RDFs – Dgraph is 100x faster Queries – Dgraph is 3x-6x faster on a read-write workload and at least as fast on read only Benchmark description: https://open.dgraph.io/post/benchmark-neo4j/ Link to full benchmark code: https://github.com/dgraph-io/benchmarks/tree/master/data/neo4j Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 19 / 23
- 32. Benchmarks Neo4j Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 20 / 23
- 33. Benchmarks Neo4j How Dgraph can load up data at least 100 times faster than Neo4j (clickable link) Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 21 / 23
- 34. Q & A Q & A Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 22 / 23
- 35. Q & A Questions and Answers Tomasz Zdybał, Dgraph Labs Dgraph: Graph database for production environment 23 / 23