Using SparkR to Scale Data Science Applications in Production. Lessons from the Field
Download as PPTX, PDF3 likes1,039 views
The document discusses the integration of SparkR in production environments, covering topics such as data science, data engineering, and management aspects of deploying R applications using Spark. It outlines the architecture of SparkR, challenges in parallelization, and techniques for effective data integration, illustrating benefits like improved computation times and streamlined workflows. Key insights include the need for a unified development and production environment and potential future improvements in data engineering and data science collaboration.
![Parallelization with SparkR 2.x
Support for User-Defined Functions
• dapply (dapplyCollect)
• input: DataFrame, func [, Schema]
• output: DataFrame
• gapply (gapplyCollect)
• input: DataFrame¦GroupedData,
groupBy, func [, Schema]
• output: DataFrame
• spark.lapply
• input: parameters, func
• Access to data/HDFS
• output: List](https://image.slidesharecdn.com/april61130wireframekorndorf-170410150418/85/Using-SparkR-to-Scale-Data-Science-Applications-in-Production-Lessons-from-the-Field-16-320.jpg)
Using SparkR to Scale Data Science Applications in Production. Lessons from the Field
- 1. Scaling SparkR in Production. Lessons from the Field. Heiko Korndorf Wireframe, CEO & Founder
- 2. About me Heiko Korndorf • CEO & Founder Wireframe • MS in Computer Science • Application Areas: ERP, CRM, BI, EAI • Serving companies in • Manufacturing • Telecommunications • Financial Services • Utilities • Oil & Gas • Professional Services Rapid Application Development for Hadoop/Spark Test Data Generation/Simulation
- 3. What we’ll talk about Classify this talk …. • Data Science: Scaling your R application with SparkR • Data Engineering: How to bring Data Science applications into your production pipelines, i.e. adding R to your toolset. • Management: Integrating Data Science and Data Engineering with SparkR
- 4. Agenda • SparkR Architecture 1.x/2.x • Reference Projects I + II • Approach with Spark 1.5/1.6 • Parallelization via YARN • Dynamic R Deployment, incl. dependencies/packages • Approach with Spark 2.0 • Parallelization via SparkR • R-Graphics: headless environment, concurrency • Use Spark APIs: SQL, Mllib • On-Prem vs Cloud (Elasticity/decouple storage and compute) • Integrating Data Science and Data Engineering • A Broader Look at the Ecosystem • Outlook and Next Steps
- 5. Data Science with R • Very popular language • Designed by statisticians • Large community • > 10.000 packages • plus: integrated package management • But: Limited as Single-Node platform • Data has to fit in memory • Limited concurrency for processing
- 7. SparkR as seen from R • Import SparkR-package and initialize SparkSession • Convert data frames from local R data frames to Spark DataFrame and back • Read and write data stored in Hadoop HDFS, HBase, Cassandra, and more • Use Spark Libraries, such as SparkSQL and ML • User cluster hardware to distribute data frames and parallelize computation
- 8. SparkR Architecture • Execute R on cluster • Data Integration • Spark DataFrame – R data frame • Access Big Data File Formats • Parallelization with UDFs • Use Spark APIs • SparkSQL • Spark MLlib
- 9. SparkSQL from R • Execute SQL against Spark DataFrame • SELECT • Specify Projection • WHERE • Filter criteria • GROUPBY • Group/Aggregate • JOIN • Join tables
- 10. Native Spark ML
- 11. Time Series Forecasting • ARIMA(p,d,q) • AR: p = order of the autoregressive part • I: d = degree of first differencing involved • MA: q = order of the moving average part • Time Series: a series of data points indexed in time order • Methods: • Exponential Smoothing • Neural Networks • ARIMA:
- 12. “Pedestrian” Challenges • Modify some Spark and R (custom-build) • Submit Spark job with R (incl. packages) as YARN dependency • Challenge: R not installed on cluster • R’s installation location is hard-coded in R • “R Markdown” produces HTML, PDF, and more • Complex objects (.RDS) for metadata, KPIs, etc. • Producing additional output during run • Creating graphics in headless environments Installing R (+Pkg’s) on cluster Managing Non-Tabular Output
- 13. Parallelization with SparkR 1.x • Sequential computation: > 20 hrs. • Single-Server, parallelized: > 4.5 hrs
- 14. Parallelization with SparkR 1.x • Sequential computation: > 20 hrs. • Single-Server, parallelized: > 4.5 hrs • SparkR 1.6.2, 25 nodes, 4 cores: ca. 12 mins.
- 15. Microsoft R Server for Spark • Microsoft R Server for HDInsight integrates Spark and R • Based on Revolution Analytics • UDFs via rxExec() • Data Sources • RxXdfFile • RxTextFile • RxHiveData • RxParquetData
- 16. Parallelization with SparkR 2.x Support for User-Defined Functions • dapply (dapplyCollect) • input: DataFrame, func [, Schema] • output: DataFrame • gapply (gapplyCollect) • input: DataFrame¦GroupedData, groupBy, func [, Schema] • output: DataFrame • spark.lapply • input: parameters, func • Access to data/HDFS • output: List
- 18. The (Data) Science Process Public Perception of Science Source: Birth of a Theorem – with Cedric Villani (https://www.youtube.com/watch?v=yYwydG_aHPE)
- 19. The (Data) Science Process Public Perception of Science Science in Reality Source: Birth of a Theorem – with Cedric Villani (https://www.youtube.com/watch?v=yYwydG_aHPE)
- 20. Integrating Dev and Prod • No Need to Re-Write Applications for Production • Common Environment for Development, Test and Production • “Looks like R to Data Science, looks like Spark to Data Engineers” • Oozie-SparkAction vs ShellAction • Prepare Dev-/Prod-Environment
- 21. 2-Level Parallelization (1) Submit multiple jobs to your cluster: - Cluster Manager (YARN, Spark, Mesos) - Spark Job: Driver and Executors (2) Use GPGPU - Spark Job: Driver and Executor - Let Executor use GPGPU (3) Combine 1 and 2
- 22. Mix Scala and R • Call R from Scala • Add DataScience Module to your Spark Application • Use Spark/Scala for ETL, R for Science code • Call Spark from R • Implement high-performance code in Spark • More granular control over cluster resources
- 23. SparkR: A Dynamic Ecosystem Hadoop, Spark & R: Many interesting projects and options • SparkR (Apache, Databricks) • R Server for Spark (Microsoft) • Sparklyr (RStudio) • Oracle R for Analytics, FastR (Oracle) • SystemML (IBM) • Renjin (BeDataDriven)
- 24. Outlook & Misc • Organizational: Deepen Integration of Data Engineering & Data Science • Source Code Control & Versioning (git …) • Continuous Build • Test Management (RUnit, testthat…?) • Multi-Output (Rmarkdown) • Technical: New Approaches • Simplify/Unify Data Pipelines (SparkSQL) • Performance Improvement: use MLlib • Performance Improvement: move calculation to GPU