DevOps for Applications in Azure Databricks: Creating Continuous Integration Pipelines on Azure Using Azure Databricks and Azure DevOps
11 likes4,148 views
The document details a presentation by Premal Shah from Microsoft on creating continuous integration pipelines using Azure Databricks and Azure DevOps. It covers the fundamentals of DevOps, the functionality of Azure Databricks, and the integration of CI/CD pipelines for model management and deployment. Key takeaways include utilizing Azure tools for secure collaboration and increased productivity in data processing and machine learning implementation.
DevOps for Applications in Azure Databricks: Creating Continuous Integration Pipelines on Azure Using Azure Databricks and Azure DevOps
- 1. WIFI SSID:SparkAISummit | Password: UnifiedAnalytics
- 2. Premal Shah, Microsoft Creating continuous integration pipelines on Azure using Azure Databricks and Azure DevOps #UnifiedAnalytics #SparkAISummit
- 3. What is DevOps? 3#UnifiedAnalytics #SparkAISummit
- 4. What is DevOps? 4#UnifiedAnalytics #SparkAISummit People. Process. Products. DevOps is the union of people, process, and products to enable continuous delivery of value to your end users. “ ” Build & Test Continuous Delivery Deploy Operate Monitor & Learn Plan & Track Develop Donovan Brown, MSFT PM
- 5. What is Azure Databricks? 5#UnifiedAnalytics #SparkAISummit Increase productivity Build on a secure, trusted cloud Scale without limits Built with your needs in mind Enterprise grade Azure security Native integration with Azure services Live collaboration Enterprise-grade SLAs E2E data pipelines using ADF Integrated billing
- 7. 7#UnifiedAnalytics #SparkAISummit Azure DevOps: Choose what you like Any Language, Any Platform
- 8. 8#UnifiedAnalytics #SparkAISummit Azure Databricks Dev WS Push notebooks to Azure Devops Azure DevOps Repo Build Pipeline Artifact Release Pipeline Deploy Notebook to staging Azure Databricks Staging WS Execute Tests Deploy Notebook to prod Azure Databricks Prod WS Implementation in Azure Databricks Notebooks
- 9. 9#UnifiedAnalytics #SparkAISummit Azure Databricks Dev WS Azure DevOps Repo Build Pipeline Artifact Release Pipeline Run with staging cluster Azure Databricks Staging WS Execute Tests Run with Prod cluster Azure Databricks Prod WS Implementation in IDE (PyCharm, IntelliJ) DB Connect
- 10. Azure Databricks REST API/CLI • Provides an easy-to-use interface to the Azure Databricks platform. CLI (open source project) is built on top of the REST APIs – Workspace API • Deploy notebooks from Azure DevOps to Azure Databricks – DBFS API • Deploy libraries from Azure DevOps to Azure Databricks – Jobs API • Execute notebooks and Spark code once deployed 10#UnifiedAnalytics #SparkAISummit
- 12. DevOps for ML: Goals • Repeatability of model creation & behavior • Evaluation of model predictions • Managing different model versions and files • Operationalization of the model • Monitoring of training and scoring pipelines 12#UnifiedAnalytics #SparkAISummit
- 15. Summary • Two approaches • Implementation in Azure Notebooks • Implementation in IDE • Azure DevOps to build CI/CD pipelines (you can selectively use) • REST/CLI APIs • Model CI/CD on Azure • Azure Databricks: Data preparation and model training • Azure ML: Model deployment and management • Azure DevOps: CI/CD pipeline 15#UnifiedAnalytics #SparkAISummit
- 16. Call to action • Build a CI/CD pipeline using Azure DevOps • Azure Databricks documentation • Azure DevOps pipelines • Incorporate DevOps in your Azure Databricks implementation 16#UnifiedAnalytics #SparkAISummit
- 17. THANK YOU! “It is not the answer that enlightens, but the question” Eugene Ionesco #UnifiedAnalytics #SparkAISummit
- 18. DON’T FORGET TO RATE AND REVIEW THE SESSIONS SEARCH SPARK + AI SUMMIT