Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019)
6 likes4,635 views
The document provides an introduction to Azure Data Factory (ADF), focusing on its functionalities like pipelines, activities, datasets, and data transformation capabilities. It discusses the evolution from Azure Data Factory version 1 to version 2, alongside integration with SSIS packages through a process known as lift and shift. Additionally, it covers key concepts such as mapping data flows, schema drift, and practical demonstrations on data transformation.
Pipelines and Packages: Introduction to Azure Data Factory (DATA:Scotland 2019)
- 1. Pipelines and Packages: Introduction to Azure Data Factory Cathrine Wilhelmsen DATA:Scotland · September 13th, 2019
- 2. © 2019 Cathrine Wilhelmsen ([email protected])
- 3. © 2019 Cathrine Wilhelmsen ([email protected]) Pipelines and Packages: Introduction to Azure Data Factory
- 5. © 2019 Cathrine Wilhelmsen ([email protected]) Data Warehousing Business Intelligence Artificial Intelligence Big Data and Analytics Machine Learning Data Science
- 6. © 2019 Cathrine Wilhelmsen ([email protected]) Data Warehousing Business Intelligence Artificial Intelligence Big Data and Analytics Machine Learning Data Science
- 7. © 2019 Cathrine Wilhelmsen ([email protected]) Collect Store Transform Integrate Prepare
- 8. © 2019 Cathrine Wilhelmsen ([email protected])
- 9. © 2019 Cathrine Wilhelmsen ([email protected]) 10 years ago…
- 10. © 2019 Cathrine Wilhelmsen ([email protected]) SSIS SQL Server Integration Services
- 11. © 2019 Cathrine Wilhelmsen ([email protected])
- 12. © 2019 Cathrine Wilhelmsen ([email protected]) Then…
- 13. © 2019 Cathrine Wilhelmsen ([email protected]) ADF v1 Azure Data Factory Version 1
- 14. © 2019 Cathrine Wilhelmsen ([email protected])
- 15. © 2019 Cathrine Wilhelmsen ([email protected])
- 16. © 2019 Cathrine Wilhelmsen ([email protected]) Today…
- 17. © 2019 Cathrine Wilhelmsen ([email protected]) ADF v2 Azure Data Factory Version 2
- 18. © 2019 Cathrine Wilhelmsen ([email protected])
- 19. © 2019 Cathrine Wilhelmsen ([email protected]) Why? How? What?
- 21. © 2019 Cathrine Wilhelmsen ([email protected]) What is Azure Data Factory?
- 22. © 2019 Cathrine Wilhelmsen ([email protected]) What can you do in Azure Data Factory? Copy Data Transform Data
- 23. © 2019 Cathrine Wilhelmsen ([email protected]) What is inside Azure Data Factory? Pipelines Activities Datasets Linked Services Data Flows Templates Triggers
- 24. DEMO Let's look inside Azure Data Factory!
- 25. © 2019 Cathrine Wilhelmsen ([email protected]) Wait… I already have thousands of SSIS packages!
- 26. © 2019 Cathrine Wilhelmsen ([email protected]) And… You told me to use this Biml thing!
- 28. © 2018 Cathrine Wilhelmsen ([email protected]) What does Lift and Shift mean?
- 29. © 2018 Cathrine Wilhelmsen ([email protected]) * Why should you Lift and Shift SSIS?
- 30. © 2018 Cathrine Wilhelmsen ([email protected]) 1. Configure Azure-SSIS Integration Runtime 2. Deploy SSIS Packages to SSISDB in Azure 3. Orchestrate SSIS Packages in Azure Data Factory How do you Lift and Shift SSIS?
- 31. © 2018 Cathrine Wilhelmsen ([email protected]) 1. Configure Azure-SSIS Integration Runtime 2. Deploy SSIS Packages to SSISDB in Azure 3. Orchestrate SSIS Packages in Azure Data Factory How do you Lift and Shift SSIS?
- 32. © 2018 Cathrine Wilhelmsen ([email protected]) Azure-SSIS Integration Runtime like all VMs
- 33. DEMO Let’s lift and shift some SSIS Packages!
- 34. © 2018 Cathrine Wilhelmsen ([email protected]) ≈ ≈ ≈ ≈ ≈ ≈ ADF vs SSIS
- 35. © 2018 Cathrine Wilhelmsen ([email protected]) Pipeline Linked Service Source Sink Activity Data Flow ≈ ≈ ≈ ≈ ≈ ≈ Package Connection Manager Source Destination Control Flow Task Data Flow ADF vs SSIS
- 37. © 2018 Cathrine Wilhelmsen ([email protected]) What are Mapping Data Flows?
- 38. © 2018 Cathrine Wilhelmsen ([email protected]) How do Mapping Data Flows work?
- 39. © 2018 Cathrine Wilhelmsen ([email protected]) Why use Mapping Data Flows?
- 40. © 2018 Cathrine Wilhelmsen ([email protected]) What is Schema Drift?
- 41. © 2018 Cathrine Wilhelmsen ([email protected]) Schema Drift in SSIS
- 42. © 2018 Cathrine Wilhelmsen ([email protected]) Schema Drift in ADF
- 43. © 2019 Cathrine Wilhelmsen ([email protected]) Oh no!
- 45. © 2019 Cathrine Wilhelmsen ([email protected]) Lessons Learned
- 46. © 2019 Cathrine Wilhelmsen ([email protected]) Good luck!
- 47. © 2019 Cathrine Wilhelmsen ([email protected]) @cathrinew cathrinew.net [email protected] thank you!
- 48. © 2019 Cathrine Wilhelmsen ([email protected])