DataOps , cbuswaw April '23
1 like485 views
The document provides an overview of DataOps and continuous integration/continuous delivery (CI/CD) practices for data management. It discusses: - DevOps principles like automation, collaboration and agility can be applied to data management through a DataOps approach. - CI/CD practices allow for data products and analytics to be developed, tested and released continuously through an automated pipeline. This includes orchestration of the data pipeline, testing, and monitoring. - Adopting a DataOps approach with CI/CD enables faster delivery of data and analytics, more efficient and compliant data pipelines, improved productivity, and better business outcomes through data-driven decisions.
DataOps , cbuswaw April '23
- 1. APRIL, 2023 DataOps The Future of Data Management - Embracing Agility, Collaboration, and Automation
- 2. Agenda 2 Introductions DevOps to DataOps CI/CD for Data Products Orchestration, Testing and Monitoring Questions
- 3. Jeewan Singh Senior Principal, Data Analytics Tomy Rhymond Principal- Cloud Lead Technology Enablement 3 About Us.
- 4. So…. what is DevOps, really??? DevOps is a cultural movement to: • Improve Collaboration • Automate operations (aka the “plumbing”) • Increase the rate of deployment • Improve quality and security What Source Control CI/CD Infrastructure Automation (IAC) Automated Test and Validation Design for Scalability Use the Cloud How Why Spend more time on valuable work … and have more fun!
- 5. Continuous Deployment Of Databases : Part 1 Data and Analytics professional face unique challenges for automation State Rolling back Other Testing Down Time Application code is stateless Database contains valuable business data Change structure and data without loss Hand crafting release scripts is error-prone Application servers are easy to swap in/out Database servers are very difficult to swap in/out (even in cluster) Can sometimes swap databases or tables in/out Applications easy to roll back from source control Databases must be explicitly backed up and restored Very time-consuming Database unavailable during restore Application code is easy to test with unit tests Unit testing for databases is challenging Unit testing requires test data generation and management which gets complicated quickly Configuration changes deployed via CI/CD Most often only DBAs touch the database (control) Prod databases don’t match source control (drift) Database change management is difficult
- 6. 6 These Roadblocks add friction, prevent automation, and slow adoption of DataOps best practices Fragile Column Mappings Embedded Credentials Hard-coded connections Black-Box SaaS GUI-Only Tools
- 7. 5 Critical Mindset Changes Business Requirements are Static “Our job is to meet the agreed business requirements.” Single-Developer, Individual Ownership “Someone will email me if it breaks.” UAT Testing Approach “We will run some tests before we launch.” Everything Manual “No time to build the automation yet.” Demos at End of Project “Creating demos take time.” Traditional Mindset DevOps Mindset Business Requirements are Fluid “We aren’t doing right if we assume requirements are static.” Multiple Developers, Team Ownership “Someone else may have to fix this if it breaks.” Continuous Testing Approach “We wrote the tests before we started developing.” Mostly Automated “No time to waste on manual stuff.” Demos Daily or Weekly “Continual feedback is critical to success.”
- 8. 8 DataOps is a collaborative and automated approach to managing the entire lifecycle of data, from its creation to its deletion, in a way that ensures that data is trustworthy, accurate, and readily available to the right people at the right time. PEOPLE PROCESS TECHNOL OGY
- 9. DataOps Collaboration Product Owner/Architect Operations/ Administration Chief Data Officer Data Analysts Data Scientist Data Engineer
- 10. 10 DataOps is an approach to data analytics and data-driven decision making that follows the agile methodology of continuous improvement. Source Data Data Ingestions Data Engineering Data Analytics Business Users DataOps CI/CD Orchestration Testing Monitoring
- 11. 11 DataOps practices are an investment whose dividends increase with time and experience Increased speed of delivery from improved processes End-to-end efficient data form automated pipelines with feedback loops Improved productivity and collaboration from empowered developers Better business outcomes from happier customers Secure and compliant data from automated, data quality checks, masking, tokenization and more. Reduced mean time to resolution (MTTR) from shift- left quality approach Increased data reliability and resiliency Developer empowerment with the DevOps culture that promote collaboration and ownership & accountability
- 12. 12 DataOps Principles Analytics is code. Differences can be spotted easily and are all committed to the code repo. Orchestrate. When everything is automated, we never have to choose between delivery new features and performing manual maintenance. Make it reproducible. The code runs the same way every time. There is no state to manage and there are no “two ways” to run it which might produce different results. Disposable environments. There’s no such things as data loss. At any time, the production environment can be recycled, and a new environment can be spun up automatically.
- 15. Taken from Stefana Muller in Dev Leaders Compare Continuous Delivery vs. Continuous Deployment vs. Continuous Integration What do we mean when we say “CI/CD”? CI/CD Definitions Continuous Integration (CI) is a software engineering practice in which developers integrate code into a shared repository several times a day in order to obtain rapid feedback of the feasibility of that code. CI enables automated build and testing so that teams can rapidly work on a single project together. Continuous Deployment (also CD) is the process by which qualified changes in software code or architecture are deployed to production as soon as they are ready and without human intervention. Continuous Delivery (CD) is a software engineering practice in which teams develop, build, test, and release software in short cycles. It depends on automation at every stage so that cycles can be both quick and reliable.
- 16. Developing with CI/CD commit commit commit commit commit main branch dev branch Pull Request ✔ ✔ ✔ ❌ Rebuild a “Beta” Copy of DW Auto-Publish to Production DW ❌ Refreshed daily/hourly 1. Continuous Integration (CI) Testing: Automatic or with every commit! 2. Continuous Delivery (CD): New changes automatically delivered in beta! 3. Continuous Deployment (also CD): New features and fixes delivered to customers automatically! ✔ ❌ 1) Store all your files in source control. 2) Create a full deployment script. 3) Create a text file pointing to your deployment script. CI/CDGettingStartedChecklist
- 18. 18 DataOps Compared to DevOps Develop Build Test Deploy Run CI CD Sandbox Develop Orchestrate Test Deploy Orchestrate Monitor CI CD
- 19. ©4/13/23 Slalom. All Rights Reserved. Proprietary and Confidential. 19 Modern Cloud Data Reference Architecture Data Pipeline Orchestration and Monitoring Security: Authorization & Authentication Continuous Integration, Continuous Deployment (CI/CD) End-User Manufacturer Management Team Internal Analytics Teams External Users Data Source Layer External Unstructured Data Loyalty E-Commerce POS Technology Patient Support Program Wholesale Distribution Vistex JDA MBA Anzio SoloChain MSA Maple CMSV2 PharmaClick POS Reflex POS Tulip MagicBox Guardian Rewards Uniprix Rewards Proxim Rewards Newsletter LMS NPS / Survey IQVIA Nielsen Health Canada Program Participation First Data Bank IQ DataSmart UniBi Website / Facebook Email (Dialogue) Mobile Apps UniSante ProxiSante PTS (db) Proxim POS Cyberlog ICN General Pharmacy Operations Team Data Lake Raw Zone Processed Zone Curated Zone Data Ingestion Batch Ingestion • Cloud based ETL • Event driven f(x) • Rest APIs Streaming Ingestion • Real-time ingestion • IoT Devices Machine Learning (Predictions & Recommendations) Feature Generation Model Development Model Deployment Model Monitoring Central Data Storage Data Warehouse Transformation & Business Rules Data Governance and Access Data Access Layer Governance Layer Management Layer Centralized Policies Data Quality Monitoring Data Lineage & Metadata Data Catalog Consistent Controls Security Policy Enforcement Data Tokenization & Masking Patient Data Hub Facts Dimensions Aggregates Views Merge & Match Deduplication Enrichment Specialty Pharmacy Operations Team Consumption Layer Operational Reports • Warehouse & Specialty • Store Sales & Growth • Kiosk Reports External Data Portal • Neilsen Data • External Kiosk • SharePoint Sandbox Environment • Ad-hoc data analysis • Raw data analysis • Merging / curating data sets Analytical Dashboard • Manufacturer Insights • Patient Insights • Pharmacy Insights API Apps • LifeLabs Apps • Loyalty Program Apps • Etc. VPN Patient / Customer Data Governance SMEs SIR DLD RX Technology Kroll Reflex RX Fillware Compliance Cube AssysteRx PharmaClick RX Applied Robotics Ubik Data Warehouses GCP E- commerce RelayHealth Hub SAP BeWell Diem
- 20. Taken from Stefana Muller in Dev Leaders Compare Continuous Delivery vs. Continuous Deployment vs. Continuous Integration Orchestrated,Test and Monitor Orchestrate • Both Infrastructure as code and data pipeline code with single pipeline • Composer (GCP), Airflow, Azure Data Factory (Azure), DBT, DataOps.live, Informatica, Mattilion, Stitch, AWS Data Pipeline Monitor • Cloud Resources • GCP Monitoring, CloudWatch, Azure Monitor, Datadog • Data pipelines • Respective tools, native cloud monitoring dashboards • Data Quality • ETL tools, manual tools on top of data platforms Test • At the end of the pipeline run • DBT, DataOps.live, Google Dataform, Boomi, Informatica, Matillion, Great Expectations, TSQLT
- 21. 21 From ETL to ELTP Extract Load Transform Publish Extract Transform Load Extract Load Transform Publish Benefits of ELT over ETL: • non-destructive updates • improved stability and recoverability “Publish” step signals that data is available and ready for downstream subscribers, may involve shipping a copy of the data into the data lake, replicating to multiple redshift clusters, populating BI models, or similar actions.
- 22. 22 At the core of DataOps is your organization’s information architecture • How well you know your data? • Do you trust your data? • Are you able to quickly detect errors? • Can you make changes incrementally without “breaking” your entire data pipeline? Critical areas below can transform your data pipeline: • Data Curation services • Metadata Management • Data Governance • Master Data Management • Self-Service interaction