Spark Concepts - Spark SQL, Graphx, Streaming
Download as PPTX, PDF17 likes3,426 views
This document provides an overview of Apache Spark modules including Spark SQL, GraphX, and Spark Streaming. Spark SQL allows querying structured data using SQL, GraphX provides APIs for graph processing, and Spark Streaming enables scalable stream processing. The document discusses Resilient Distributed Datasets (RDDs), SchemaRDDs, querying data with SQLContext, GraphX property graphs and algorithms, StreamingContext, and input/output operations in Spark Streaming.
Spark Concepts - Spark SQL, Graphx, Streaming
- 1. Apache Spark Concepts - Spark SQL, GraphX, Streaming Petr Zapletal Cake Solutions
- 2. Apache Spark and Big Data 1) History and market overview 2) Installation 3) MLlib and Machine Learning on Spark 4) Porting R code to Scala and Spark 5) Concepts - Spark SQL, GraphX, Streaming 6) Spark’s distributed programming model 7) Deployment
- 3. Table of contents ● Resilient Distributed Datasets ● Spark SQL ● GraphX ● Spark Streaming ● Q & A
- 5. Resilient Distributed Datasets ● Immutable, distributed collection of records ● Lazy evaluation, caching option, can be persisted ● Number of operations & transformations ● Can be created from data storage or different RDD
- 6. Spark SQL ● Spark’s interface to work with structured or semistructured data ● Structured data o known set of fields for each record - schema ● Main capabilities o load data from variety of structured sources o query the data with SQL o integration between Spark (Java, Scala and Python API) and SQL (joining RDDs and SQL tables, using SQL functionality)
- 7. More than SQL ● Unified interface for structured data
- 8. SchemaRDD ● RDD of row objects, each representing a record ● Known schema (i.e. data fields) of its rows ● Behaves like regular RDD, stored in more efficient manner ● Adds new operations, especially running SQL queries ● Can be created from o external data sources o results of queries o regular RDD ● Used in ML Pipeline API
- 9. SchemaRDD
- 10. Getting Started ● Entry points: o HiveContext superset functionality, Hive related o SQLContext
- 11. ● Loads input JSON file into SchemaRDD ● Uses context to execute query Query Example
- 12. Loading and Saving Data ● Supports number of structured data sources o Apache Hive data warehouse infrastructure on top of Hadoop summarization, querying (SQL-like interface) and analysis o Parquet column-oriented storage format in Hadoop ecosystem efficient storage of records with nested fields o JSON o RDDs o JDBC/ODBC Server connecting Business Intelligence tools remote access to Spark cluster
- 13. GraphX ● New Spark API for graphs and graph-parallel computation ● Resilient Distributed Property Graph (RDPG, extends RDD) o directed multigraph ( -> parallel edges) o properties attached to each vertex and edge ● Common graph operations (subgraph computation, joining vertices, ...) ● Growing collection of graph algorithms
- 14. Motivation ● Growing scale and importance of graph data ● Application of data-parallel algorithms to graph computation is inefficient ● Graph-parallel systems (Pregel, PowerGraph, ...) designed for efficient execution of graph algorithms o do not address graph construction & transformation o limited fault tolerance & data mining support
- 16. Property Graph ● Directed multigraph with user defined objects to each vertex and edge
- 17. Property Graph
- 18. Triplet View ● Logical join of vertex and edge properties
- 19. Graph Operations ● Basic information (numEdges, numVertices, inDegrees, ...) ● Views (vertices, edges, triplets) ● Caching (persist, cache, ...) ● Transformation (mapVertices, mapEdges, ...) ● Structure modification (reverse, subgraph, ...) ● Neighbour aggregation (collectNeighbours, aggregations, ...) ● Pregel API ● Graph builders (various I/O operations) ● ...
- 20. Graph Algorithms ● Built-in algorithms o PageRank, Connected Components, Triangle Count, ...
- 21. Demo
- 22. Spark Streaming ● Scalable, high-throughput, fault-tolerant stream processing
- 23. Architecture ● Streams are chopped up into batches ● Each batch is processed in Spark ● Results pushed out in batches
- 26. StreamingContext ● Entry point for all streaming functionality o define input sources o stream transformations o output operations to DStreams o starts & stops streaming process ● Limitations o once started, computations cannot be added o cannot be restarted o one active per JVM
- 27. Discretized Streams ● Basic abstraction, represents a continuous stream of data ● DStreams ● Implemented as series of RDDs
- 28. Stateless Transformations ● Processing of each batch does not depend on previous batches ● Transformation is separately applied to every batch o Map, flatMap, filter, reduce, groupBy, … ● Combining data from multiple DStreams o Join, cogroup, union, ...
- 29. Stateful Transformations ● Use data or intermediate results from previous batches to compute the result of the current batch ● Windowed operations o act over a sliding window of time periods ● UpdateStateByKey o maintain state while continuously updating it with new information ● Require checkpointing
- 30. Output Operations ● Specify what needs to be done with the final transformed data ● Pushing to external DB, printing, … ● If not performed, DStream is not evaluated
- 31. Input Sources ● Built-in support for a number of different data sources ● Often in additional libraries (i.e. spark-streaming-kafka) ● HDFS ● Akka Actor Stream ● Apache Kafka ● Apache Flume ● Twitter Stream ● Kinesis ● Custom Sources ● ...
- 32. Demo
- 33. Conclusion ● RDD repetition ● Spark Modules Overview o Spark SQL o GraphX o Spark Streaming
- 34. Questions
Editor's Notes
- #11: Apache Hive is a data warehouse infrastructure built on top of Hadoop for providing data summarization, query, and analysis.[2] While initially developed by Facebook, Apache Hive is now used and developed by other companies such as Netflix.[3][4] Amazon maintains a software fork of Apache Hive that is included in Amazon Elastic MapReduce on Amazon Web Services.[5]
- #16: Connected Components and PageRank algorithms https://amplab.cs.berkeley.edu/wp-content/uploads/2014/09/graphx.pdf For Spark we implemented the algorithms both using idiomatic dataflow operators (Naive Spark, as described in Section 3.2) and using an optimized implementation (Optimized Spark) that eliminates movement of edge data by pre-partitioning the edges to match the partitioning adopted by GraphX. We have excluded Giraph and Optimized Spark from Figure 7c because they were unable to scale to the larger web-graph in the allotted memory of the cluster. While the basic Spark implementation did not crash, it was forced to re-compute blocks from disk and exceeded 8000 seconds per iteration. We attribute the increased memory overhead to the use of edge-cut partitioning and the need to store bi-directed edges and messages for the connected components algorithm
- #17: https://spark.apache.org/docs/latest/graphx-programming-guide.html
- #29: cogroup - When called on DStream of (K, V) and (K, W) pairs, return a new DStream of (K, Seq[V], Seq[W]) tuples. join - When called on two DStreams of (K, V) and (K, W) pairs, return a new DStream of (K, (V, W)) pairs with all pairs of elements for each key. union - Return a new DStream that contains the union of the elements in the source DStream and otherDStream.