Introduction to Spark SQL training workshop
2 likes545 views
The document outlines a training workshop on Spark SQL, covering an overview of Spark SQL, DataFrame queries, and additional functions. It highlights use cases for ETL and analytics, including an example of a restaurant finder app, and discusses the lazy execution and caching concepts. The document includes links to slides and notebooks for hands-on training, and encourages participants to familiarize themselves with Spark's programming interfaces using various languages.
![Use case: ETL & analytics
• Example: restaurant finder app
• Log data: Timestamp, UserID, Location, RestaurantType
• [ 4/24/2014 6:22:51 PM, 1000618, -85.5750, 42.2959, Pizza ]
• Analytics
• What time of day do users use the app?
• What is the most popular restaurant type in San Jose, CA?
Logs ETL Analytics
Spark SQL Spark SQL](https://image.slidesharecdn.com/sparksqltraining-201608-wibd-160805215336/85/Introduction-to-Spark-SQL-training-workshop-6-320.jpg)
![Use case: Feature Extraction for ML
• Example: restaurant finder app
• Log data: Timestamp, UserID, Location, RestaurantType
• [ 4/24/2014 6:22:51 PM, 1000618, -85.5750, 42.2959, Pizza ]
• Machine Learning to train a model of user preferences
• Use Spark SQL to extract features for the model
• Example features: hour of day, distance to a restaurant, restaurant
type
Logs ETL Features ML Training
Spark SQL Spark SQL
See Notebook …](https://image.slidesharecdn.com/sparksqltraining-201608-wibd-160805215336/85/Introduction-to-Spark-SQL-training-workshop-18-320.jpg)
![Dataset (new in 2.0)
• DataFrames are untyped
• df.select($”col1” + 3)
• Useful when exploring new data
• Datasets are typed
• Dataset[T]
• Associates an object of type T with each row
• Catches type mismatches at compile time
• DataFrame = Dataset[Row]
• A DataFrame is one specific type of Dataset[T]
case class FarmersMarket(FMID: Int, MarketName: String)
val ds : Dataset[FarmersMarket] …
Numerical type assumed, but
not checked at compile time](https://image.slidesharecdn.com/sparksqltraining-201608-wibd-160805215336/85/Introduction-to-Spark-SQL-training-workshop-20-320.jpg)
Introduction to Spark SQL training workshop
- 1. SPARK SQL Xinh Huynh Women in Big Data training workshop August, 2016
- 3. Outline • Part 1: Spark SQL Overview, SQL Queries • Part 2: DataFrame Queries • Part 3: Additional DataFrame Functions
- 4. Outline • Part 1: Spark SQL Overview, SQL Queries • Part 2: DataFrame Queries • Part 3: Additional DataFrame Functions
- 5. Why learn Spark SQL? • Most popular component in Spark • Spark Survey 2015 • Use cases • ETL • Analytics • Feature Extraction for machine learning % of users 0 18 35 53 70 Spark SQL DataFrames MLlib, GraphX Streaming
- 6. Use case: ETL & analytics • Example: restaurant finder app • Log data: Timestamp, UserID, Location, RestaurantType • [ 4/24/2014 6:22:51 PM, 1000618, -85.5750, 42.2959, Pizza ] • Analytics • What time of day do users use the app? • What is the most popular restaurant type in San Jose, CA? Logs ETL Analytics Spark SQL Spark SQL
- 7. How Spark SQL fits into Spark (2.0) Spark Core (RDD) Catalyst SQL DataFrame / Dataset ML Pipelines Structured Streaming GraphFrames Spark SQL http://www.slideshare.net/SparkSummit/deep-dive-into-catalyst-apache-spark-20s-optimizer-63071120
- 8. Spark SQL programming interfaces Catalyst SQL DataFrame / Dataset Spark SQL SQL Scala, Java, R, Python Scala, Java
- 9. SQL or DataFrame? • Use SQL if you are already familiar with SQL • Use DataFrame • To write queries in a general-purpose programming language (Scala, Python, …). • Use DataFrame to catch syntax errors earlier: SQL DataFrame Syntax Error Example “SELEECT id FROM table” df.seleect(“id”) Caught at Runtime Compile Time
- 10. Loading and examining a table, Query with SQL • See Notebook: http://tinyurl.com/spark-nb1
- 11. Setup for Hands-on Training 1. Sign on to WiFi with your assigned access code 1. See slip of paper in front of your seat 2. Sign in to https://community.cloud.databricks.com/ 3. Go to "Clusters" and create a Spark 2.0 cluster 1. This may take a minute. 4. Go to “Workspace” -> Users -> Home -> Create -> Notebook 1. Select Language = Scala 2. Create
- 12. Outline • Part 1: Spark SQL Overview, SQL Queries • Part 2: DataFrame Queries • Part 3: Additional DataFrame Functions
- 13. DataFrame API • See notebook: http://tinyurl.com/spark-nb2
- 14. Lazy Execution • DataFrame operations are lazy • Work is delayed until the last possible moment • Transformations: DF -> DF • select, groupBy; no computation done • Actions: DF -> console or disk output • show, collect, count, write; computation is done https://www.flickr.com/photos/mtch3l/24491625352
- 15. Lazy Execution Example 1. val df1 = df.select(…) 2. val df2 = df1.groupBy (…) 3. .sum() 4. if (cond) 5. df2.show() • Benefits of laziness • Query optimization across lines 1-3 • If step 5 is not executed, then no unnecessary work was done Transformation: no computation done Transformation: no computation done Action: performs the select, groupBy at this time, then shows the results
- 16. Caching • When querying the same data set over and over, caching it in memory may speed up queries. • Back to notebook … Disk Memory Results Memory Results Without caching: With caching:
- 17. Outline • Part 1: Spark SQL Overview, SQL Queries • Part 2: DataFrame Queries • Part 3: Additional DataFrame Functions
- 18. Use case: Feature Extraction for ML • Example: restaurant finder app • Log data: Timestamp, UserID, Location, RestaurantType • [ 4/24/2014 6:22:51 PM, 1000618, -85.5750, 42.2959, Pizza ] • Machine Learning to train a model of user preferences • Use Spark SQL to extract features for the model • Example features: hour of day, distance to a restaurant, restaurant type Logs ETL Features ML Training Spark SQL Spark SQL See Notebook …
- 19. Functions for DataFrames • See notebook: http://tinyurl.com/spark-nb3
- 20. Dataset (new in 2.0) • DataFrames are untyped • df.select($”col1” + 3) • Useful when exploring new data • Datasets are typed • Dataset[T] • Associates an object of type T with each row • Catches type mismatches at compile time • DataFrame = Dataset[Row] • A DataFrame is one specific type of Dataset[T] case class FarmersMarket(FMID: Int, MarketName: String) val ds : Dataset[FarmersMarket] … Numerical type assumed, but not checked at compile time
- 21. Review • Part 1: Spark SQL Overview, SQL Queries √ • Part 2: DataFrame Queries √ • Part 3: Additional DataFrame Functions √
- 22. References • Spark SQL: http://spark.apache.org/docs/latest/sql- programming-guide.html • Spark Scala API docs: http://spark.apache.org/docs/latest/ api/scala/index.html#org.apache.spark.package • Overview of DataFrames: http:// xinhstechblog.blogspot.com/2016/05/overview-of-spark- dataframe-api.html • Questions, comments: • Spark user list: [email protected] • Xinh’s contact: https://www.linkedin.com/in/xinh-huynh-317608 • Women in Big Data: https://www.womeninbigdata.org/