ETL Made Easy with Azure Data Factory and Azure Databricks
9 likes2,997 views
This document summarizes Mark Kromer's presentation on using Azure Data Factory and Azure Databricks for ETL. It discusses using ADF for nightly data loads, slowly changing dimensions, and loading star schemas into data warehouses. It also covers using ADF for data science scenarios with data lakes. The presentation describes ADF mapping data flows for code-free data transformations at scale in the cloud without needing expertise in Spark, Scala, Python or Java. It highlights how mapping data flows allow users to focus on business logic and data transformations through an expression language and provides debugging and monitoring of data flows.
ETL Made Easy with Azure Data Factory and Azure Databricks
- 1. WIFI SSID:SparkAISummit | Password: UnifiedAnalytics
- 2. Mark Kromer Sr. Azure Data Program Manager Microsoft ETL Made Easy with Azure Data Factory & Azure Databricks #UnifiedAnalytics #SparkAISummit
- 3. Azure Data Factory Cloud ETL Patterns with ADF 3#UnifiedAnalytics #SparkAISummit
- 4. Nightly ETL Data Loads Code-free
- 5. Slowly Changing Dimension Scenario
- 6. Load Star Schema DW Scenario
- 7. Data Lake Data Science Scenario
- 8. Azure Data Factory Workflow Data Pipelines/Control Flow 9#UnifiedAnalytics #SparkAISummit
- 12. Azure Data Factory Mapping Data Flows 13#UnifiedAnalytics #SparkAISummit
- 13. What is ADF Mapping Data Flow? • Transform Data, At Scale, in the Cloud, Zero-Code – Cloud-first, scale-out ELT – Code-free dataflow pipelines • Serverless scale-out transformation execution engine • Maximum Productivity for Data Engineers – Does NOT require understanding of Spark / Scala / Python / Java • Resilient Data Transformation Flows – Built for big data scenarios with unstructured data requirements – Operationalize with Data Factory scheduling, control flow and monitoring
- 14. Code-free Data Transformation At Scale • Does not require understanding of Spark, Big Data Execution Engines, Clusters, Scala, Python … • Focus on building business logic and data transformation – Data cleansing – Aggregation – Data conversions – Data prep – Data exploration
- 15. Transformation Function Expression Language
- 16. Debug Data Flows with Data Preview and Data Sampling
- 17. Deep Monitoring Introspection of Data Transformations
- 18. DON’T FORGET TO RATE AND REVIEW THE SESSIONS SEARCH SPARK + AI SUMMIT