Apache Spark Architecture | Apache Spark Architecture Explained | Apache Spark Tutorial |Simplilearn
12 likes2,213 views
This presentation on Spark Architecture will give an idea of what is Apache Spark, the essential features in Spark, the different Spark components. Here, you will learn about Spark Core, Spark SQL, Spark Streaming, Spark MLlib, and Graphx. You will understand how Spark processes an application and runs it on a cluster with the help of its architecture. Finally, you will perform a demo on Apache Spark. So, let's get started with Apache Spark Architecture. YouTube Video: https://www.youtube.com/watch?v=CF5Ewk0GxiQ What is this Big Data Hadoop training course about? The Big Data Hadoop and Spark developer course have been designed to impart an in-depth knowledge of Big Data processing using Hadoop and Spark. The course is packed with real-life projects and case studies to be executed in the CloudLab. What are the course objectives? Simplilearn’s Apache Spark and Scala certification training are designed to: 1. Advance your expertise in the Big Data Hadoop Ecosystem 2. Help you master essential Apache and Spark skills, such as Spark Streaming, Spark SQL, machine learning programming, GraphX programming and Shell Scripting Spark 3. Help you land a Hadoop developer job requiring Apache Spark expertise by giving you a real-life industry project coupled with 30 demos What skills will you learn? By completing this Apache Spark and Scala course you will be able to: 1. Understand the limitations of MapReduce and the role of Spark in overcoming these limitations 2. Understand the fundamentals of the Scala programming language and its features 3. Explain and master the process of installing Spark as a standalone cluster 4. Develop expertise in using Resilient Distributed Datasets (RDD) for creating applications in Spark 5. Master Structured Query Language (SQL) using SparkSQL 6. Gain a thorough understanding of Spark streaming features 7. Master and describe the features of Spark ML programming and GraphX programming Who should take this Scala course? 1. Professionals aspiring for a career in the field of real-time big data analytics 2. Analytics professionals 3. Research professionals 4. IT developers and testers 5. Data scientists 6. BI and reporting professionals 7. Students who wish to gain a thorough understanding of Apache Spark Learn more at https://www.simplilearn.com/big-data-and-analytics/apache-spark-scala-certification-training
![Spark SQL
Spark allows you to define custom SQL functions called User Defined Functions (UDFs)
SQL
def lowerRemoveAllWhiteSpaces(s: String): String = {
s.tolowerCase().replace(“S”, ‘’”)
}
val lowerRemoveAllWhiteSpacesUDF = udf[String, String]
(lowerRemoveAllWhiteSpaces)
val sourceDF = spark.createDF(
List(
(“ WELCOME “)
(“ SpaRk SqL “)
), List(
(“text”, StringType, true)
)
)
sourceDF.select(
lowerRemoveAllWhiteSpacesUDF(col(“text”)).as(“clean_text”)
).show()
UDF that removes all
the whitespace and
lowercases all the characters
in a string
clean_text
welcome
sparksql
Output](https://image.slidesharecdn.com/sparkarchitecture-190809072757/85/Apache-Spark-Architecture-Apache-Spark-Architecture-Explained-Apache-Spark-Tutorial-Simplilearn-24-320.jpg)
More Related Content
What's hot (20)
Similar to Apache Spark Architecture | Apache Spark Architecture Explained | Apache Spark Tutorial |Simplilearn (20)
More from Simplilearn (20)
Recently uploaded (20)
Apache Spark Architecture | Apache Spark Architecture Explained | Apache Spark Tutorial |Simplilearn
- 2. 1. What is Spark? 2. Components of Spark Spark Core Spark SQL Spark Streaming Spark MLlib GraphX 3. Apache Spark Architecture 4. Running a Spark Application What’s in it for you?
- 3. What is Apache Spark? Apache Spark is a top-level open-source cluster computing framework used for real-time processing and analysis of a large amount of data
- 4. What is Apache Spark? Apache Spark is a top-level open-source cluster computing framework used for real-time processing and analysis of a large amount of data Fast processing Spark processes data faster since it saves time in reading and writing operations
- 5. What is Apache Spark? Apache Spark is a top-level open-source cluster computing framework used for real-time processing and analysis of a large amount of data Fast processing Real-time streaming Spark processes data faster since it saves time in reading and writing operations Spark allows real-time streaming and processing of data
- 6. What is Apache Spark? Apache Spark is a top-level open-source cluster computing framework used for real-time processing and analysis of a large amount of data Fast processing Real-time streaming In-memory computation Spark processes data faster since it saves time in reading and writing operations Spark allows real-time streaming and processing of data Spark has DAG execution engine that provides in-memory computation
- 7. What is Apache Spark? Apache Spark is a top-level open-source cluster computing framework used for real-time processing and analysis of a large amount of data Fast processing Real-time streaming In-memory computation Fault tolerant Spark processes data faster since it saves time in reading and writing operations Spark allows real-time streaming and processing of data Spark has DAG execution engine that provides in-memory computation Spark is fault tolerant through RDDs which are designed to handle the failure of any worker node in the cluster
- 9. Spark Core Apache Spark Components
- 10. Spark Core Spark SQL SQL Apache Spark Components
- 11. Spark Streaming Spark Core Spark SQL SQL Streaming Apache Spark Components
- 12. MLlib Spark Streaming Spark Core Spark SQL SQL Streaming MLlib Apache Spark Components
- 13. MLlib Spark Streaming Spark Core Spark SQL GraphX SQL Streaming MLlib Apache Spark Components
- 14. Spark Core Spark is the core engine for large-scale parallel and distributed data processing
- 15. Spark Core Spark is the core engine for large-scale parallel and distributed data processing Memory management and fault recovery Scheduling, distributing and monitoring jobs on a cluster Interacting with storage system Performs the following:
- 16. Spark RDD Resilient Distributed Datasets (RDDs) are the building blocks of any Spark application Create RDD Transformations RDD Actions Results Transformations are Operations (such as map, filter, join, union) that are performed on an RDD that yields a new RDD containing the result Actions are operations (such as reduce, first, count) that return a value after running a computation on an RDD
- 17. Spark SQL Spark SQL is Apache Spark’s module for working with structured data SQL
- 18. Spark SQL Spark SQL is Apache Spark’s module for working with structured data SQL Integrated You can integrate Spark SQL with Spark programs and query structured data inside Spark programs Spark SQL features
- 19. Spark SQL Spark SQL is Apache Spark’s module for working with structured data SQL Integrated High Compatibility You can integrate Spark SQL with Spark programs and query structured data inside Spark programs You can run unmodified Hive queries on existing warehouses in Spark SQL. With existing Hive data, queries and UDFs, Spark SQL offers full compatibility Spark SQL features
- 20. Spark SQL Spark SQL is Apache Spark’s module for working with structured data SQL Integrated High Compatibility Scalability You can integrate Spark SQL with Spark programs and query structured data inside Spark programs You can run unmodified Hive queries on existing warehouses in Spark SQL. With existing Hive data, queries and UDFs, Spark SQL offers full compatibility Spark SQL leverages RDD model as it supports large jobs and mid- query fault tolerance. Moreover, for both interactive and long queries, it uses the same engine Spark SQL features
- 21. Spark SQL Spark SQL is Apache Spark’s module for working with structured data SQL Integrated Spark SQL features High Compatibility Scalability Standard Connectivity You can integrate Spark SQL with Spark programs and query structured data inside Spark programs You can run unmodified Hive queries on existing warehouses in Spark SQL. With existing Hive data, queries and UDFs, Spark SQL offers full compatibility Spark SQL leverages RDD model as it supports large jobs and mid- query fault tolerance. Moreover, for both interactive and long queries, it uses the same engine You can easily connect Spark SQL with JDBC or ODBC. For connectivity for business intelligence tools, both turned as industry norms
- 22. Spark SQL Spark SQL is Apache Spark’s module for working with structured data DataFrame DSLSpark SQL and HQL DataFrame API Data Source API CSV JSON JDBC SQL Architecture SQL
- 23. Spark SQL Spark SQL has three main layers Spark SQL is Apache Spark’s module for working with structured data Language API SchemaRDD Data Sources Spark is compatible and even supported by the languages like Python, HiveQL, Scala, and Java As Spark SQL works on schema, tables, and records, you can use SchemaRDD or data frame as a temporary table Data sources for Spark SQL are different like JSON document, HIVE tables, and Cassandra database SQL
- 24. Spark SQL Spark allows you to define custom SQL functions called User Defined Functions (UDFs) SQL def lowerRemoveAllWhiteSpaces(s: String): String = { s.tolowerCase().replace(“S”, ‘’”) } val lowerRemoveAllWhiteSpacesUDF = udf[String, String] (lowerRemoveAllWhiteSpaces) val sourceDF = spark.createDF( List( (“ WELCOME “) (“ SpaRk SqL “) ), List( (“text”, StringType, true) ) ) sourceDF.select( lowerRemoveAllWhiteSpacesUDF(col(“text”)).as(“clean_text”) ).show() UDF that removes all the whitespace and lowercases all the characters in a string clean_text welcome sparksql Output
- 25. Spark Streaming Spark Streaming an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams Streaming
- 26. Spark Streaming Spark Streaming an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams Data can be ingested from many sources and the processed data can be pushed out to different filesystems Streaming
- 27. Spark Streaming Spark Streaming an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams Data can be ingested from many sources and the processed data can be pushed out to different filesystems Streaming Streaming data sources Static data sources
- 28. Spark Streaming Spark Streaming an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams Data can be ingested from many sources and the processed data can be pushed out to different filesystems Streaming Streaming Streaming data sources Static data sources
- 29. Spark Streaming Spark Streaming an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams Data can be ingested from many sources and the processed data can be pushed out to different filesystems Streaming Streaming Streaming data sources Static data sources Data storage
- 30. Spark Streaming Spark Streaming an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams Spark Streaming receives live input data streams and divides the data into batches, which are then processed by the Spark engine to generate the final stream of results in batches Streaming Engine Input data stream Batches of input data Batches of processed data Streaming
- 31. Spark Streaming Spark Streaming an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams Streaming Here is an example of a basic RDD operation to extract individual words from lines of text in an input data stream Lines From Time 0 and 1 Lines From Time 1 and 2 Lines From Time 2 and 3 Lines From Time 3 and 4 Words From Time 0 and 1 Words From Time 1 and 2 Words From Time 2 and 3 Words From Time 3 and 4 Lines DStream Words DStream flatMap Operation
- 32. Spark MLlib MLlib is Spark’s machine learning library. Its goal is to make practical machine learning scalable and easy MLlib
- 33. MLlib is Spark’s machine learning library. Its goal is to make practical machine learning scalable and easy MLlib At a high level, it provides the following: ML Algorithms: classification, regression, clustering, and collaborative filtering Spark MLlib
- 34. MLlib is Spark’s machine learning library. Its goal is to make practical machine learning scalable and easy MLlib At a high level, it provides the following: ML Algorithms: classification, regression, clustering, and collaborative filtering Featurization: feature extraction, transformation, dimensionality reduction, and selection Spark MLlib
- 35. MLlib is Spark’s machine learning library. Its goal is to make practical machine learning scalable and easy MLlib At a high level, it provides the following: ML Algorithms: classification, regression, clustering, and collaborative filtering Featurization: feature extraction, transformation, dimensionality reduction, and selection Pipelines: tools for constructing, evaluating, and tuning ML pipelines Spark MLlib
- 36. MLlib is Spark’s machine learning library. Its goal is to make practical machine learning scalable and easy MLlib At a high level, it provides the following: ML Algorithms: classification, regression, clustering, and collaborative filtering Featurization: feature extraction, transformation, dimensionality reduction, and selection Pipelines: tools for constructing, evaluating, and tuning ML pipelines Utilities: linear algebra, statistics, data handling Spark MLlib
- 37. GraphX GraphX is a component in Spark for graphs and graph-parallel computation GraphX is used to model relations between objects. A graph has vertices (objects) and edges (relationships). Mathew Justin Edge Vertex Relationship: Friends
- 38. GraphX GraphX is a component in Spark for graphs and graph-parallel computation Provides a uniform tool for ETL Exploratory data analysis Interactive graph computations
- 39. GraphX is a component in Spark for graphs and graph-parallel computation Page Rank Fraud Detection Geographic information system Disaster management Following are the applications of GraphX GraphX
- 41. Spark Architecture Spark Architecture is based on 2 important abstractions
- 42. Spark Architecture Spark Architecture is based on 2 important abstractions Resilient Distributed Dataset (RDD) RDD’s are the fundamental units of data in Apache Spark that are split into partitions and can be executed on different nodes of a cluster Cluster RDD
- 43. Spark Architecture Spark Architecture is based on 2 important abstractions Resilient Distributed Dataset (RDD) Directed Acyclic Graph (DAG) RDD’s are the fundamental units of data in Apache Spark that are split into partitions and can be executed on different nodes of a cluster Cluster DAG is the scheduling layer of the Spark Architecture that implements stage-oriented scheduling and eliminates the Hadoop MapReduce multistage execution model RDD Stage 1 Parallelize Filter Map Stage 2 reduceByKey Map
- 44. Spark Architecture Master Node Driver Program SparkContext • Master Node has a Driver Program • The Spark code behaves as a driver program and creates a SparkContext which is a gateway to all the Spark functionalities Apache Spark uses a master-slave architecture that consists of a driver, that runs on a master node, and multiple executors which run across the worker nodes in the cluster
- 45. Spark Architecture Cluster Manager • Spark applications run as independent sets of processes on a cluster • The driver program & Spark context takes care of the job execution within the cluster Master Node Driver Program SparkContext
- 46. Spark Architecture Cache Task Task Executor Worker Node Cache Task Task Executor Worker Node • A job is split into multiple tasks that are distributed over the worker node • When an RDD is created in Spark context, it can be distributed across various nodes • Worker nodes are slaves that execute different tasks Cluster Manager Master Node Driver Program SparkContext
- 47. Spark Architecture Cache Task Task Executor Worker Node Cache Task Task Executor Worker Node • Executor is responsible for the execution of these tasks • Worker nodes execute the tasks assigned by the Cluster Manager and returns the resultback to the Spark Context Master Node Driver Program SparkContext Cluster Manager
- 48. Spark Architecture Cache Task Task Executor Worker Node Cache Task Task Executor Worker Node • Worker nodes execute the tasks assigned by the Cluster Manager and returns it back to the Spark Context • Executor is responsible for the execution of these tasks Master Node Driver Program SparkContext Cluster Manager
- 50. Spark Session Driver Program Application How a Spark application runs on a cluster? Spark applications run as independent processes, coordinated by the SparkSession object in the driver program
- 51. Spark Session Driver Program Application Resource Manager/ Cluster Manager How a Spark application runs on a cluster? The resource or cluster manager assigns tasks to workers, one task per partition
- 52. Spark Session Driver Program Application Worker Node Executor Task Task Cache Partition Partition Disk Data Data How a Spark application runs on a cluster? Resource Manager/ Cluster Manager • A task applies its unit of work to the dataset in its partition and outputs a new partition dataset • Because iterative algorithms apply operations repeatedly to data, they benefit from caching datasets across iterations
- 53. How a Spark application runs on a cluster? Spark Session Driver Program Application Executor Task Task Cache Partition Partition Disk Data Data Resource Manager/ Cluster Manager Results are sent back to the driver application or can be saved to disk Worker Node