Agile data science with scala
7 likes1,824 views
The document discusses the Agile Data Science Toolkit by Data Fellas, aimed at addressing common challenges in data science processes. It outlines the integration of various components like microservices, Spark, and Kafka to create an efficient data pipeline that enhances reliability and accelerates time to market. The toolkit also emphasizes the significance of context and lineage in data handling to improve output quality and reproducibility.
Agile data science with scala
- 1. Agile Data Science with Scala by @DataFellas Xavier Tordoir [email protected] @xtordoir Andy Petrella [email protected] @noootsab
- 2. Data Fellas Andy Petrella Maths Geospatial Distributed Computing Spark Notebook Trainer Spark/Scala Machine Learning Xavier Tordoir Physics Bioinformatics Distributed Computing Scala (& Perl) trainer Spark Machine Learning
- 3. © Data Fellas SPRL 2016 ● Pipeline: productizing Data Science ● Demo of Distributed Pipeline (Spark, Mesos, Akka, Cassandra, Kafka, Spark Notebook) ● Why Micro Services? ● Painful points: ○ Data science is Discontiguous ○ Context Lost in Translation ● Solution: Data Fellas’ Agile Data Science Toolkit Lineup So if you’re not sure you want to stay...
- 4. © Data Fellas SPRL 2016 Pipeline Productizing Data Science Modelling Coding Deploying Finding Data Parsing structures Cleaning (Reducing) Learning Predicting Connect PROD data Tuning training parameters Create Prediction Service Generate Deployable Connect to PROD infrastructure Integration with existing env Allocate (schedule) resources Ensure availability
- 5. © Data Fellas SPRL 2016 Distributed Data Science Demo All-In Spark Notebooks Get data: Source → Kafka Prepare View: Kafka → Cassandra Train Model: Cassandra → ML... Create Server: Cassandra/ML/... → Akka Http Create Client: Json → Html Form, Chart, table, ...
- 6. © Data Fellas SPRL 2016 Bad Pipeline Targeting Dashboard Modelling Coding Deploying Dashboard »»» Data Scientist focusing on the dashboard/report instead of content breaks reusability of data time wasted on learning viz instead of increasing accuracy (or velocity) monolithic instead of service oriented
- 7. © Data Fellas SPRL 2016 Extended Pipeline Micro Services Modelling Coding Deploying Integrating Application Creating Services Abstracts access to prepared views Exposes Prediction capabilities Highly horizontally scalable Scaling micro services cluster → cheaper than computing cluster Customer integration Can be any technologies Can even be another pipeline!
- 8. © Data Fellas SPRL 2016 Painful points Data science is Discontiguous ➔ Highly heterogeneous environment ➔ Too many friction areas ➔ Time to market too long Modelling Coding Deploying Integrating Application Scientist Data Eng. Ops. Eng. Web Eng. Customers ➔ No integration ➔ Error prone ➔ Schedule delays Creating Services Frictions Result: Lack of Agility Collecting Data Eng.
- 9. © Data Fellas SPRL 2016 Painful points Context Lost in Translation Data Lake Processing Machine Learning Model Output Data Input Data No contextual discovery No quality info No lineage (origin of the data) Link to process and input discarded Huge gap in architecture: binary and schema aware serving layer Accuracy depends on concealed quality of inputs No schema! hard and long integration, poor satisfaction Moreover: No backward links → no agility and no context awareness Result: Lack of Reproducibility Application
- 10. Data Fellas… Agile Data Science Toolkit
- 11. © Data Fellas SPRL 2016 Our Approach Agile Data Science Toolkit Automatic Semantics Engine + Autogenerated Microservices Integrated End-to-End Environment Huge gain in Time and Reliability + = Notebook Computing Cluster Access Layer Knowledge Base Consumers Customers Exposes database, learning models, stream sources, notebooks, ... data type process lineage usage Easy to Release Easy to (Re)Use Notebook Version Control (Git) Spark Job Project (SBT) Service Projects (SBT) Metadata (Doc, Logic, Schema, ...) Catalog (ElasticSearch) Deployable (Jar, Docker) Repository (Nexus, Docker Repo, Pypi, Gem Server) Client Projects (Node.Js, Java, Scala, Python, Ruby) Publishable (NPM, Jar, Pip/EasyInstall, Gem) scientist data Engineer ops Engineer
- 12. © Data Fellas SPRL 2016 Agile Data Science Toolkit In a nutshell
- 13. © Data Fellas SPRL 2016 Agile Data Science Toolkit In a nutshell
- 14. © Data Fellas SPRL 2016 Agile Data Science Toolkit In a nutshell
- 15. © Data Fellas SPRL 2016 Agile Data Science Toolkit In a nutshell
- 16. © Data Fellas SPRL 2016 Agile Data Science Toolkit In a nutshell
- 17. © Data Fellas SPRL 2016 Agile Data Science Toolkit In a nutshell
- 19. © Data Fellas SPRL 2016 O’Reilly Online seminar
- 20. © Data Fellas SPRL 2016 Growing We’re Hiring! http://www.data-fellas.guru/#skillsjobs
- 21. Q/A References http://www.data-fellas.guru/ http://spark-notebook.io/ https://github.com/andypetrella/spark-notebook/ https://gitter.im/andypetrella/spark-notebook Come at Strata -- London at least -- We have two talks :-)