DevOps and Decoys How to Build a Successful Microsoft DevOps Including the Data
1 like452 views
Kellyn Pot’vin-Gorman, a Data Platform Architect at Microsoft, discusses building efficient Microsoft DevOps solutions for data management and automation in a webinar. The document highlights the importance of simplifying deployment processes, leveraging automation tools, and using scripting to enhance interaction with the solutions, ultimately improving customer engagement and resource allocation. It emphasizes the need for continuous enhancement and adaptation to new technologies while providing insights into the structure and execution of deployment strategies.
DevOps and Decoys How to Build a Successful Microsoft DevOps Including the Data
- 1. Kellyn Pot’Vin-Gorman, DevOps Engineer and now Data Platform Architect in Power BI and AI at Microsoft DevOps and Decoys- How to Build a Successful Microsoft DevOps Solution Including Your Data
- 3. Free online webinar events Free 1-day local training events Local user groups around the world Online special interest user groups Business analytics training Get involved Explore everything PASS has to offer Free Online Resources Newsletters PASS.org
- 4. Download the GuideBook App and search: PASS Summit 2018 Follow the QR code link displayed on session signage throughout the conference venue and in the program guide Session evaluations Your feedback is important and valuable. Go to passSummit.com 3 Ways to Access: Submit by 5pm Friday, November 16th to win prizes.
- 5. /kellyngorman @DBAKevlar kellyngorman Kellyn Pot’Vin-Gorman Data Platform Architect at Microsoft, EDU Team Former Technical Intelligence Manager, Delphix • Multi-platform DBA, (Oracle, MSSQL, MySQL, Sybase, PostgreSQL, Informix…) • Oracle ACE Director, (Alumni) • Oak Table Network Member • Idera ACE Alumni 2018 • STEM education with Raspberry Pi and Python, including DevOxx4Kids, Oracle Education Foundation and TechGirls • Former President, Rocky Mtn Oracle User Group • Current President, Denver SQL Server User Group • Linux and DevOps author, instructor and presenter. • Blogger, (http://dbakevlar.com) Twitter: @DBAKevlar
- 6. A Few Facts 1.7Mb of data created per person, per second by 2020 44Zb of data by 2020 80% of data isn’t currently analyzed in any way. Even 10% added access to analyze could result in +$50 million in revenue for every fortune 100 company.
- 7. Automation is a Key to Much of the Tech Acceleration Less intervention from DBAs to maintain, manage and operate database platforms. Advancement in tools to eliminate resource demands. More tool interconnectivity, allowing less steps to do more A resurgence of scripting to enhance automation and graphical interfaces to empower those who have wider demands.
- 9. With Success Comes Culture Change ALTHOUGH CULTURE IS THE BIGGEST HURDLE...
- 10. Our Use Case Large Percentage of University Customers in MHE Searching for same solution EDU team created a Popular Solution that allowed for community involvement Multi-Tier Deployment- VM, ADF, Azure Database, Data and Power BI Ever-evolving
- 11. Drawbacks Teams from universities are made up of varied technical backgrounds ◦ Data Scientists, DBAs, Data Analysts and Educators ◦ Spend More time deploying than working with the solution ◦ Slows down the time the EDU’s limited resources get to work one-on-one with the customers ◦ Discovered some customers lost interest during the deployment phase or didn’t have the time to deploy
- 12. Goal Simplify Simplify the deployment with DevOps practices Remove Remove demands on knowledge required to deploy the solution. Create Create more time for interaction and working with the solution by the education teams.
- 13. Build a Roadmap Document All Pieces of Deployment • Identify any interactive vs. default entries that will benefit the deployment. • Update documentation as you go. Don’t try to do it in the end. 1 Categorize by Physical, Logical and Interactive • Build out Physical Deployment first, as it is the foundation. • Build in easy restart and clean up steps to physical deployments • Test and update any documentation to reflect the automation 2 Begin to automate logical slowly and in phases. • Remove any manual steps or configurations. • Take advantage of any plugins that ease work on end-users side • Continue to accept feedback and enhance. 3
- 14. What is Involved 1. Two SQL Databases- one staging and one data warehouse 2. Azure Data Factory with a SSIS Database 3. Azure Analysis Services 3. Three Power BI Reports 4. CSV files for ongoing data loads, which will need to be configured for ongoing workloads 5. Multiple Solution and project files, some deprecated in VS 2017 * Data loads and configuration via Visual Studio or SSDT solutions already built in. * Sample data files in Excel could be replaced with the customers own data.
- 15. The Solution Was Very Repeatable Most resources and databases could have the same name in every deployment. Outside of the CSV files containing example data, everything else could be deployed without any changes by the customer if a few dynamic parameters were pushed to the deployment. Although official documentation existed, there were numerous versions and the process had evolved with the introduction and ease of Azure deployment.
- 16. Already Using Github Stable code, rarely changed https://github.com/pleblanc72/msedubi
- 17. Visual Studio/SSMS Dev Tools My predecessor and team members built in some automation already using solution files! ◦ Awesome, these can be reused!
- 18. How Do From Point A to Point Automate? Perform the task in the User Interface FIRST. Gather the information with the CLI to build out your scripts Test and retest comparing to the UI deployment Manage and maintain automation. Don’t go back to manual processing or manual intervention. Build out in phases- physical to logical, enhancing and improving as we go along.
- 19. So Many Choices… Multiple options to automate- ◦Azure CLI ◦Terraform ◦PowerShell with Azure Commands ◦Azure DevOps
- 20. Terraform Freemium Product Came to Azure in 2016 Another robust product that allows for automation of processing via script. Supports complex tasks and multi-platform deployments
- 21. Simple CLI
- 23. Azure CLI Allows a command line interface to the Azure cloud Can be installed locally on Windows, Mac and other OS platforms Can be run inside a Docker Container Can be used with the Azure Cloud Shell without installation. Flexible and robust, allows for a CLI solution and automation via scripting in PowerShell/BASH of Azure deployments. https://docs.microsoft.com/en-us/cli/azure/install-azure-cli?view=azure-cli-latest
- 24. Once Installed Test: >az login >az account show <tenant ID> <Subscription ID>
- 25. AZ CLI is Simple to Use and Robust >C:EDU_Docker>az vm create -n LinuxTstVM -g dba_group --image UbuntuLTS --generate-ssh-keys SSH key files 'C:Userskegorman.NORTHAMERICA.sshid_rsa' and 'C:Userskegorman.NORTHAMERICA.sshid_rsa.pub' have been generated under ~/.ssh to allow SSH access to the VM. If using machines without permanent storage, back up your keys to a safe location. - Running .. C:EDU_Docker>az vm list –g dba_group C:EDU_Docker>az vm delete -n LinuxTstVM -g dba_group Are you sure you want to perform this operation? (y/n): Y
- 26. Locating Information on ADF 1. Create one in the GUI 2. Inspect the resource facts via the CLI: az resource list -- location eastus
- 27. Walk Before You Run.. Began to deploy individual resources. Had a final merge script, (aka wrapper) with a test script. Deployed piece by piece until phase I was completed. Received feedback from peers and customers as proceeded.
- 28. Azure CLI Isn’t Enough – Cloud Shell Enhanced Azure CLI commands into BASH script to deploy and automate. A script to enhance automation and set variables to ease customer skill requirements was required. From the Azure Portal: Or Direct: https://shell.azure.com/
- 29. Initial Build in BASH Well, I’m a Linux Person Script is interactive, accepting customer’s requested naming conventions and requirements. Builds out the physical resources in Azure ◦ SQL Server with a data warehouse and staging database ◦ Azure Data Factory ◦ Azure Analysis Server Creates firewall rules for Azure Cloud Shell Creates all user access Creates Database objects
- 31. Two JSON Files, not 64K of JSON
- 32. Living Documentation Made easier transition as redesigned. Kept track of all moving parts. Offered insight to those who knew previous, manual process. Allowed for roadmap to be included. Allowed for troubleshooting section as other sections shrunk with automation.
- 34. PowerShell Love of the SQL Community One the roadmap for future, secondary choice for deployment Still deciding if a PowerShell version is required.
- 35. Benefits and Drawbacks Terraform Azure CLI PowerShell/BASH Freemium, but great user community Newer product, but driven to support Azure PowerShell has a leg up on BASH in the SQL World Scripting is proprietary Scripting can be done in Powershell or BASH Scripting is powerful Can build out anything that has a command line option Is very interactive. If you want to script it, must do more. Can do more than just deploy. Scripting can become more application based. Requires Subscription, tenant, client API and password to be passed Good for check commands and settings Can be as much as needed. Was a bit overkill for what I was doing Offers more support than Terraform, that can’t use some of the defaults Azure CLI was made to work with both.
- 37. Automate the Logical • Migrate any onsite SSIS pkgs and workflows to Azure Data Factory • ADF will build out an SSISDB in Azure SQL database • Store projects in pipelines • Schedule, report and check into a Github repository to automate development cycle. • SLN and PROJ files are recycled.
- 38. Next Steps PowerShell version is in mid-development ◦ More likely know PowerShell over BASH if Microsoft professional. ◦ Will only require two main scripts to be updated with enhancements and additions to the repository. ◦ Automate the SSIS, (Integration Services) and Data Factory steps, (currently an unknown, so figuring out.) ◦ Build out scripts to scale up the current “demo” version to an enterprise version. ◦ Script to automatically pause integration and factory services to save on cost. ◦ Script out data workload inputs for new features/data model and proprietary data loads. ◦ Build out all steps using Azure DevOps to continue the growth of the automation.
- 39. Challenges? No Reference Material on How to Create an Azure Data Factory from the Command Line Azure CLI offered HOPE, but some of it had to be built out in the self-service command of “resource” and use json strings. The motto when working with new tools- “Learn to Fish” •Help menus and examples online are your best friend. •Get a good text editor. •You will be learning new scripting languages •Find support groups, twitter handles and help channels.
- 40. Success Manual Process takes between two full days of onsite meetings, to weeks of webmeetings to deploy. New Automated process deploys in less than 15 minutes after customer answered questions in interactive script. Offers extensively more time for customer to work with solution and Microsoft architects to work with providing value to the customer on how to use Power BI with Azure. In first week, over two dozen customers requested POC deployment with little assistance from heavily limited resource team, allowing for more valuable allocation of resources.
- 41. / You Must Evolve Data WILL come from more than just SQL Server and Azure SQL Databases. 1 You will be expected to do more with less. 2 Embrace automation tools inside the database platform, (backup, recovery, optimization, maintenance tasks.) 3
- 42. Thank You Learn more from Kellyn Pot’Vin-Gorman @DBAKevlar [email protected]