Graph Processing with Apache TinkerPop
- 1. Graph Processing with Apache TinkerPop (incubating) Jason Plurad Software Engineer, IBM | Committer, Apache TinkerPop
- 2. โข Project Update โข Graph Landscape โข A Graph Problem โข Hands-On Graph http://tinkerpop.apache.org
- 3. About Me โข Twitter @pluradj โข GitHub @pluradj โข Open channels โ TinkerPop mailing lists โ Titan mailing list โ Stack Overflow
- 4. (Apache) TinkerPop (incubating) โข 2009: Inception โข 2012: TinkerPop 2 โข 2015: Apache Incubator โข 2016: Top Level Project? โ TLP VOTE passed! โ Waiting on board meeting to establish TLP
- 5. Podling Releases โข 3.0 โ Major refactor, Java 8 lambda expressions, Gremlin Server, OLAP graph computers โข 3.1 โ Hadoop 2 support, persisted RDDs โข 3.2 โ OLAP job chaining, OLAP graph filters, performance improvements
- 6. Common graph data domains โข Social Network Analysis โข Configuration Management Database โข Master Data Management โข Recommendation Engines โข Knowledge Graphs โข Internet of Things
- 7. Property Graph and Gremlin โข Structure โ Vertex โ Edge โ Properties โข Gremlin โ Domain specific language (DSL) for graph โ Data flow: forward and backward โ Traversal Steps โ Bindings for non-JVM languages
- 9. Graph Landscape โข Graph database vs Graph processor โ OLTP vs OLAP โ Neighborhood vs whole graph โข Multi-model: not the only store in your app
- 10. IBM Graph (Beta) โข Managed Graph-as-a-Service (OLTP) โข Focus on your data, not install and operations โข #sleepMore http://ibm.biz/IBMGraph
- 11. What is this? module.exports = xxxxxxx; function xxxxxxx (str, len, ch) { str = String(str); var i = -1; if (!ch && ch !== 0) ch = ' '; len = len - str.length; while (++i < len) { str = ch + str; } return str; }
- 12. A Graph Problem: Dependency Management โข On March 22, 2016 npm broke the Internet โข Left-pad was unpublished โ 11 lines of code โ WTFPL license โ Hundreds of breaking builds per minute โ http://blog.npmjs.org/post/141577284765/kik-left-pad-and-npm โข Are we safe with Apache?
- 13. Questions for the graph โข Which dependencies are at risk? โข Which ones should be refactored to avoid? โข Risk factors โ Unsuitable license โ Single developer โ Too little code / Too much code โ Changes too frequently / Code is stagnant โ Nobody else is using it
- 15. Titan (Aurelius) โข Pick a graph database for OLTPโฆ โ Apache license but not in ASF โข Code has stagnated in the open โ DataStax Enterprise (DSE) Graph โ Wide open opportunities โข Genesis Graph is up next! โข Apache S2Graph (incubating) โข Apache Flink (Gelly) โข Apache Solr (GraphQuery)
- 16. Apache Spark or Apache Giraph โข Pick a graph processor for OLAPโฆ โ Spark is the new hotness โ Giraph is better suited for gigantic graphs โข By using Apache TinkerPop and Gremlin, we can use either one seamlessly
- 17. Vagrant and Virtualbox โข Developers donโt always get keys to the cloud โข Virtual machines to the rescue โ Host: 16 GB RAM or more โ 3-4 VMs with 3 GB RAM โข Prove out your graph algorithms on a small data set before wasting time on a big data set
- 18. Apache Ambari โข Simple install for Apache Hadoop and related Apache big data packages โ HDFS, YARN, MapReduce, HBase, Spark, etc โข Management and monitoring dashboard โข Enables integration of other software
- 19. Getting the data โข NPM registry runs on Apache CouchDB โข Replication in Apache CouchDB is awesome โ https://skimdb.npmjs.com/registry
- 20. Transform the data โข Apache CouchDB is a document store โข Dependencies are graph data โข Other things can be too โ Users โ Keywords โ License โข Graph model depends on the questions you want to ask of the graph
- 22. Hands-On: Gremlin Console https://asciinema.org/a/21qk1rn9yt6tt7sour9w9ynxn
- 25. Out of the box Vertex Programs โข Traversal โข BulkLoader โข BulkDumper โข PageRank โข PeerPressure
- 27. OLAP Traversal Sources > graph = GraphFactory.open('conf/npmgraph- olap.properties') > g = graph.traversal().withComputer(SparkGraphComputer) > g = graph.traversal().withComputer(GiraphGraphComputer) Graph Statistics via TraversalVertexProgram > g.V().count() // vertex count > g.E().count() // edge count > g.V().label().groupCount() // vertex label distribution > g.E().label().groupCount() // edge label distribution > g.V().properties().key().groupCount() // vertex property distribution
- 28. Next stop? More data! โข Graphs are for connecting data! โข Consume data from GitHub โ User data โ Static code analysis โ Code usage analysis โข Consume data from Twitter โ Trending news โ Security alerts
- 29. Summary โข Apache TinkerPop is for graph computing โข OLTP vs OLAP is an important distinction โ Gremlin allows you to seamless bridge the two โข Graph thinking is different than relational โ Is the future multi-model? โข Many opportunities to innovate in this space
- 30. Acknowledgements โข Marko Rodriguez โ Gremlin language, Gremlin OLAP โข Ketrina Yim โ Illustrator, creator of Gremlin and friends โข Stephen Mallette โ TinkerPop release manager, Gremlin applications โข Daniel Kuppitz โ Gremlin language guru โข David Robinson โ Big data, multi-model architect/developer
- 31. Questions?
- 32. Thank you!