Introduction to Apache Spark
1 like1,251 views
The document introduces Apache Spark and its core concepts, including resilient distributed datasets (RDDs), transformations, and actions. It outlines Spark's architecture, data processing capabilities, and the functionality of Spark SQL and Spark Streaming. Additionally, it provides resources for further learning and practical workshops on the topic.
Introduction to Apache Spark
- 1. Introduction to Spark Lambda World
- 2. Who are we? Juan Pedro Moreno Scala Software Engineer at 47Degrees @juanpedromoreno Fran PĂ©rez Scala Software Engineer at 47Degrees @FPerezP Workshop repo: https://github.com/47deg/spark-workshop
- 3. Roadmap âą Intro Big Data and Spark âą Spark Architecture âą Resilient Distributed Datasets (RDDs) âą Transformations and Actions on Data using RDDs âą Overview Spark SQL and DataFrames âą Overview Spark Streaming âą Spark Architecture and Cluster Deployment
- 4. Apache Spark Overview https://github.com/apache/spark âą Fast and general engine for large-scale data processing âą Speed âą Ease of Use âą Generality âą Runs Everywhere http://spark.apache.org
- 5. Spark Architecture Scala Java Python R Spark SQL Spark Streaming MLlib GraphX DataFrames API RDD API Spark Core Hadoop HDFS Cassandra JSON MySQL ⊠DATA SOURCES
- 6. Spark Core Concepts Driver Program Worker Node Worker Node Cluster Manager SparkContext Executor Executor Cache Cache Task Task Task Task Hadoop YARN Standalone Apache Mesos
- 7. Spark Core Concepts âą Executor: A process launched for an application on a worker node. Each application has its own executors. âą Jobs: A parallel computation consisting of one or multiple stages that gets spawned in response to a Spark action. âą Stages: Smaller set of tasks that each job is divided into. âą Tasks: A unit of work that will be sent to one executor.
- 8. Resilient Distributed Datasets âą Immutable. âą Partitioned collection. âą Operates in parallel. âą Customizable.
- 9. RDDs - Partitions âą A Partition is one of the diïŹerent chunks that a RDD is splitted on and that is sent to a node âą The more partitions we have, the more parallelism we get âą Each partition is candidate to be spread out to diïŹerent worker nodes Error, ts, msg1 Warn, ts, msg2 Error, ts, msg1 Info, ts, msg8 Warn, ts, msg2 Info, ts, msg8 Error, ts, msg3 Info, ts, msg5 Info, ts, msg5 Error, ts, msg4 Warn, ts, msg9 Error, ts, msg1 RDD with 4 partitions
- 10. RDDs - Partitions RDD with 8 partitions P1 P2 P3 P4 P5 P6 P7 P8 Worker Node Executor Worker Node Executor Worker Node Executor Worker Node Executor P1 P5 P2 P6 P3 P7 P4 P8
- 11. RDDs - Operations Transformations âą Lazy operations. They donât return a value, but a pointer to a new RDD. Actions âą Non-lazy operations. They apply an operation to a RDD and return a value or write data to an external storage system.
- 12. RDDs - Transformations A set of some of the most popular Spark transformations: âą map âą ïŹatMap âą ïŹlter âą groupByKey âą reduceByKey
- 13. RDDs - Actions A set of some of the most popular Spark actions: âą reduce âą collect âą foreach âą saveAsTextFile
- 14. Transformations and Actions With Visual Mnemonics, better. Thanks to JeïŹrey Thompson âą http://data-frack.blogspot.com.es/2015/01/visual-mnemonics-for- pyspark-api.html âą https://github.com/jkthompson/pyspark-pictures âą http://nbviewer.ipython.org/github/jkthompson/pyspark-pictures/ blob/master/pyspark-pictures.ipynb
- 15. Practice - Part 1 && Part 2
- 16. Overview Spark SQL and DataFrames âą Works with structured and semistructured data âą DataFrame simpliïŹes working with structured data âą Read/Write from structure data like JSON, Hive tables, Parquet, etc. âą SQL inside your Spark App âą Best Performance and more powerful operations API
- 17. Practice - Part 3
- 18. Overview Spark Streaming âą Streaming Applications âą DStreams or Discretized Streams âą Continuous Series of RDDs, grouped by batches Kafka Spark StreamingR e c e i v e r s Flume HDFS batches of input data Spark Core HDFS/S3 Database Kinesis Dashboard Twitter
- 19. Resources âą OïŹcial docs - http://spark.apache.org/docs/latest âą Learning Spark - http://shop.oreilly.com/product/0636920028512.do âą Databricks Spark Knowledge Base - https://goo.gl/wMy7Se âą Community packages for Spark - http://spark-packages.org/ âą Apache Spark Youtube channel - https://www.youtube.com/user/ TheApacheSpark âą API through pictures - https://goo.gl/JMDeqJ âą 47 Degrees Blog - http://www.47deg.com/blog/tags/spark