PuppetConf 2017: Unlocking Azure with Puppet Enterprise- Keiran Sweet, Sourced Group
0 likes619 views
The document outlines the integration of Puppet Enterprise with Microsoft Azure, highlighting the role of infrastructure as code and various Azure services. It discusses the benefits of using ARM templates and Puppet for automation, scaling, and deployment workflows in cloud environments. Additionally, it describes the capabilities and strategies for managing infrastructure efficiently through automation and configuration management.
![Applying your Puppet Role to the Instances
What Instance becomes what?
• Leverage trusted facts to assign the instances role
• $trusted[’extensions’][‘pp_role’]
• Prevents reclassification (Security)
• Validate pp_role in CSR against pp_role tag on instance
• Classification opportunities
• Do this in your code
• Use the Puppet node classifier to assign roles
• Use Hiera via hiera_include()](https://image.slidesharecdn.com/sweet-unlockingazurewithpuppetenterpriseoctober112017-final-171025154820/85/PuppetConf-2017-Unlocking-Azure-with-Puppet-Enterprise-Keiran-Sweet-Sourced-Group-21-320.jpg)
PuppetConf 2017: Unlocking Azure with Puppet Enterprise- Keiran Sweet, Sourced Group
- 1. Unlocking Azure with Puppet Enterprise October 11, 2017
- 2. Unlocking Azure with Puppet Enterprise October 11, 2017 v3.0
- 3. Overview • Introduction to Sourced • Introduction to me • Infrastructure as code • Evolving Azure capabilities • Template driven, Puppet delivered services • Scaling Puppet with Azure
- 4. Historically • Sourced Group were founded in 2009 • Significant Financial Services background • Specialize in Configuration Management, Automation, Cloud Computing & Data Management • Achieved a number of industry firsts in these fields • Offices in Australia and Canada • Delivery experience in Amazon Web Services, Microsoft Azure & Traditional infrastructure platforms Major in-flight Projects • 80% data center migration to AWS for a large airline • Includes an Application Delivery Framework • Policy and guidance to underpin this activity • Development of a strategic cloud environment for a global investment bank • Engage with internal stakeholders to define a public cloud environment that is capable of housing material workloads • On-going assistance on the ‘cloud journey’ for large Canadian telco • Full business migration of electronics medical records suite of products to AWS Who are Sourced? Adopting cloud services within an enterprise requires experience
- 5. Our Partnerships Strategic partnerships that align with our customer-centric approach
- 6. Keiran Sweet • Senior Consultant with Sourced Group, Australia • Previously Puppet lead for a large financial organisation • Presentedat multiple Puppet conferences and camps • Background • Linux & UNIX System administration and architecture • Sudden and confusing interest Windows infrastructure as code • Deployment & integration with various cloud providers • Puppet user since ~2008 • Dog Enthusiast Me Who is this guy anyway?
- 8. • Native API’s and SDK’s • AWS / Azure / GCE / vSphere • Write your own scripts and tools to use them • Abstraction Layers • Puppet / Razor / Terraform / Vagrant / Fog • Leverage frameworks that simplify management • Vendor Native Templating Languages • AWS CFN / Azure RM Templates • Express your infrastructure in JSON / YAML Infrastructure As Code What are some of the options today in the cloud?
- 9. • Transparency • Composition of your environment is kept in source control • Greater visibility of changes and history ( git log ! ) • Enhanced scale out, build, test and recovery capabilities • New region expansion, catastrophic simulations • Build an isolated production like environment for testing first • Enhanced automation opportunities • Idempotency and Self healing • CI / CD Further down the stack, unit testing, contestability Infrastructure As Code Benefits
- 11. • Infrastructure Services • Virtual Networks, ExpressRoute, Azure DNS, Load Balancers • Storage services • Blobs / Tables / Queues / Files • Databases & Caching • Azure SQL / CosmosDB/ Azure Redis • Virtual Machines, PaaS and Container Services • Windows / Linux Virtual Machines / Azure App Service/ Docker • Many many many many more…. Microsoft Azure Services overview, it’s more than just compute for Windows!
- 12. Platform Services Infrastructure Services Compute Storage Datacenter Infrastructure Application Platform Web Apps Mobile Apps API Apps Notification Hubs Hybrid Cloud Backup StorSimple Azure Site Recovery Import/Export Networking Data SQL Database DocumentDB Redis Cache Azure Search Storage Tables SQL Data Warehouse Azure AD Health Monitoring Virtual Network Express Route Blob Files DisksVirtual Machines AD Privileged Identity Management Traffic Manager App Gateway Operational Analytics Compute Services Cloud Services Batch RemoteApp Service Fabric Developer Services Visual Studio Application Insights VS Team Services Containers DNS VPN Gateway Load Balancer Domain Services Analytics & IoT HDInsight Machine Learning Stream Analytics Data Factory Event Hubs Data Lake Analytics Service IoT Hub Data Catalog Security & Management Azure Active Directory Multi-Factor Authentication Automation Portal Key Vault Store/ Marketplace VM Image Gallery & VM Depot Azure AD B2C Scheduler Xamarin HockeyApp Power BI Embedded SQL Server Stretch Database Mobile Engagement Functions Intelligence Cognitive Services Bot Framework Cortana Security Center Container Service Queues VM Scale Sets Data Lake Store Dev/Test Lab Integration BizTalk Services Service Bus Logic Apps API Management Media & CDN Content Delivery Network Media Services Media Analytics
- 13. • Azure API • Abstraction Layers • Azure CLI (v1 & v2) • SDK - Ruby / .NET / Python / Node • PowerShell module • Puppet module / Terraform / Vagrant • Azure Resource Manager (ARM) templates Microsoft Azure Infrastructure as Code capabilities “If you are spending significant provisioning time in the Portal You aren’t doing infrastructure as code.”
- 14. Anatomy of the ARM Template • Declare all your Azure resources in JSON • Define parameters to adjust the outcome within boundaries • Define Outputs that are returned to you for consumption • Why ? • Native Templating Language • Templates get the features first – No tracking other projects • The console createsthese templateswhen using the Azure Portal • Store the templateslike any other code • Use Visual Studio Code to help with development Azure’s native templating language
- 15. Template driven, Puppet delivered services
- 16. • Representing our Azure based environment in ARM templates • There isn’t an Azure service for everything we need • Puppet can help here; • We want to also provision instances that run our own services • We don’t want manual intervention to achieve this • We want to ensure that security is still at the forefront • We want to ensure visibility throughout the process I thought this was a Puppet talk ? Where does Puppet sit in all of this?
- 17. Deployment Workflow How do we get there? • Use your CICD Tooling to initiate the deployment • Focus on provisioning consumable services • Abstract away the Operating System – It’s just a commodity run time • Use the Templates Outputs: { } functionalityto return; • Deployment Summary • Service Names & URLs • API Endpoints Deploy Template Provision Azure Services Provision Azure VM’s Puppet Installation Sign Puppet CSR Apply Puppet Catalogue Deployment Complete
- 18. Deployment Workflow Easy, right? Deploy Template Provision Azure Services Provision Azure VM’s Puppet Installation Sign Puppet CSR Apply Puppet Catalogue Deployment Complete • Use your CICD Tooling to initiate the deployment • Focus on provisioning consumable services • Abstract away the Operating System – It’s just a commodity run time • Use the Templates Outputs: { } functionalityto return; • Deployment Summary • Service Names & URLs • API Endpoints
- 19. Bootstrapping the Puppet Agent Azure Custom Script Extensions • User defined code executed on instance launch • Custom Script Extensions also defined in the template • In this case; • Retrieve the script from a URL (ie , Blob storage) • Execute the script • Bootstrap the Puppet agent from the master • Set additional Facts (Optional) • Template parameters can be passed down to the extension if / when required • Failed Puppet runs == Failed Deployment
- 20. Authorising the Puppet Agent Securely signing the CSR Never sign incorrectly configured instances Only sign correctly configured instances • The Puppet CA = Security for the Puppet Service • Policy Based Autosigning • Execute code to validate the incoming CSR • Automatically sign certificates that are validated to have correct: • Name • Subscription • Tags • Role Tag (pp_role)
- 21. Applying your Puppet Role to the Instances What Instance becomes what? • Leverage trusted facts to assign the instances role • $trusted[’extensions’][‘pp_role’] • Prevents reclassification (Security) • Validate pp_role in CSR against pp_role tag on instance • Classification opportunities • Do this in your code • Use the Puppet node classifier to assign roles • Use Hiera via hiera_include()
- 22. Additional Integration Other capabilities to leverage • Azure metadata as structured facts • Azure now has a metadata API ! • Facter feature request ready and waiting.. JIRA FACT-1383 • Available today via keirans/azuremetadata module • Puppet Azure module • Can get you up and running quickly • Keep in mind its current limitations • ARM Template Resource can be used to wrap templates as Puppet resources.
- 23. Scaling Puppet with Azure
- 24. Add load-balanced compile masters to your monolithic installation to increase the number of agents you can manage
- 27. We want to make our compile masters be as disposible as possible, reducing the overhead of their management, while improving reliability, scalability and security
- 28. Automation Challenges • Compile masters • Do not support policy based autosigning for security reasons • Often need additional secrets to function such as hiera eyaml keys • Requires software exposed from the master of masters • Requires a number of executed runs across nodes
- 29. Automation Solutions • Deploy leveraging ARM templates • Leverage Azure Key Vault • Pre-generate our compile master certificates on the Puppet CA • Store the eyaml keys and other certificates in there also • Create an identity that can retrieve them from the keyvault • Bootstrap scripts fetch secrets from the vault on deployment • Orchestrate runs using scripts or Jenkins Puppet Plugin • Result : A Fully Automated Compile Master tier that can be easily reprovisioned
- 30. Solving with Azure capabilities
- 35. Additional Benefits • Rapid scaling out of master capacity ( 2 -20 easily) • Rapid patching of the platform • Tear down • Patch Master of Masters • Redeploy compile masters • Rapid rolling of Compile master certificates • Disaster recovery scenarios and testing • A view to autoscaling long term
- 36. Any questions?
- 37. • Puppet Blog – Policy Based Autosigning • Policy based autosigning in Azure • Puppet Forge - Azure metadatamodule • https://forge.puppet.com/keirans/azuremetadata • ARM Examples - Automated compilemasters • https://github.com/keirans/azure-puppet-compilemasters • Microsoft – Release Pipeline Model • https://msdn.microsoft.com/en- us/powershell/dsc/whitepapers#the-release-pipeline-model • Image Credits • Silicon Valley (HBO) References
- 38. • Using Puppet in Automated Environments • Order in a world of snowflakes Sourced Group, Puppetconf 2015 • Using Puppet in Dynamic Environments • The Evolving Design Patterns of Puppet Enterprise Sourced Group, Puppetconf 2014 • Using Puppet with Multiple Cloud Providers • Using Puppet as heterogeneous cloud glue Sourced Group, Puppetconf 2012 Previous Presentations