Continuous Infrastructure First
2 likes575 views
This document discusses continuous infrastructure and delivery. It describes how automation of build, deployment, and testing processes through improved collaboration between developers, operations, and testers allows delivery teams to release changes in hours or even minutes regardless of project size or code complexity. The document also discusses how treating infrastructure as code through configuration management tools like Puppet, Chef, and Ansible supports continuous delivery.
Continuous Infrastructure First
- 1. Continuous Infrastructure First ! Kris Buytaert @krisbuytaert
- 2. @krisbuytaert Kris Buytaert ● I used to be a Dev, ● Then Became an Op ● 20th year at #fosdem ● CTO and Open Source Consultant @inuits.eu ● Everything is a freaking DNS Problem ● Evangelizing devops ● Organiser of #devopsdays, #cfgmgmtcamp, #loadays, #deliveryconf ? :) ….
- 3. @krisbuytaert What is this is devops thing? ● Culture ● (Lean) ● Automation ● Measurement ● Sharing Damon Edwards and John Willis
- 4. @krisbuytaert devops (<)> continuous delivery
- 5. @krisbuytaert Nirvana An “ecosystem” that supports continuous delivery, from infrastructure,An “ecosystem” that supports continuous delivery, from infrastructure, data and configuration management to business.data and configuration management to business. Through automation of the build, deployment, and testing process, andThrough automation of the build, deployment, and testing process, and improved collaboration between developers, testers, and operations,improved collaboration between developers, testers, and operations, delivery teams can get changes released in a matter of hours —delivery teams can get changes released in a matter of hours — sometimes even minutes–no matter what the size of a project or thesometimes even minutes–no matter what the size of a project or the complexity of its code base.complexity of its code base. Continuous Delivery , Jez HumbleContinuous Delivery , Jez Humble
- 8. @krisbuytaert "Our job as engineers (and ops, dev-ops, QA, support, everyone in the company actually) is to enable the business goals. We strongly feel that in order to do that you must have the ability to deploy code quickly and safely. Even if the business goals are to deploy strongly QA’d code once a month at 3am (it’s not for us, we push all the time), having a reliable and easy deployment should be non-negotiable." Etsy Blog upon releasing Deployinator http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/
- 10. @krisbuytaert For years we've tolerated humans to make structural manual changes to the infrastructure our critical applications are running on. Whilst at the same time demanding those critical applications to go through rigid test scenarios. Who let this happen ?
- 11. @krisbuytaert history of devops ● Europe : – Starting from Operations, – Improved Artifact Quality, – Less pain / stability ● US : – Push from development – Faster Platforms – Faster change
- 12. @krisbuytaert NoOps & YOLO Ops ● Startup ● VC ● Exit Strategy ● 6-9 months ● Actual Business ● Real Customers ● Survival ● 6-9 years
- 13. @krisbuytaert 3 Different Transition Cases: Startups & Multinationals
- 14. @krisbuytaert Case 1: Chaotic Ops ● Complete Chaos ● 10% reproducibility ● CI infra hides under a dev’s desk ● Ops in Debug Mode ● No standardization ● Apollo Moment
- 15. @krisbuytaert Case 1: T0+3months ● Build a Reproducible Jenkins + Slaves ● CI for Puppet by OPS ● Test your code ● Promotion Stage for Infrastructure Code ● Split config out of code ● Keep delivering updates
- 16. @krisbuytaert Case 1: T0+6 months ● Stack Alignment – 1 jdk, 1 jboss , ... ● Project Dolly : – Puppet for everything ● 90% reproducibility ● Standardized Builds ● Increased Test Coverage ● Java Developers contribute to Infra Tests
- 17. @krisbuytaert Case 1: Conclusion ● Started with preparing ops folks to automate ● Learned the same tools developers use ● Developers help the ops folks to improve ● Collaboration + Progress ++
- 18. @krisbuytaert Case 2: CI by Devs ● Some devs have tests ● Some dev teams have “CI” ● Deployments are Chaos ● Ops nags about Artifact Quality
- 19. @krisbuytaert Case 2: T0+18 months ● Found the first ops skills in the org ● Mostly overworked Brent’s ● Move them out of their offices ● Focus team ● Teach Agile ● Adopt IAC (puppet) ● First Successes ● Move people back to teams
- 20. @krisbuytaert Case 2: T0+24 months ● Grey Beard Ops person has converted to Agile Evangelist ● Preaches Kanban (for ops) and Scrum ● Writes Test for his Code ● Coaches developers to achieve CI/CD
- 21. @krisbuytaert Case 2: Conclusions ● Starting with dev delayed the collaboration for 1+ year ● Ops were fire fighting and not involved ● Once ops resources were dedicated collaboration and quality improvement started to happen
- 22. @krisbuytaert Case 3:Ops NOT involved ● Large Transformation ● “devops” team dictates tools they have never used them selves ● Tools they as a team don’t need themselves ● Developers complain about unusable tools ● Developers complain about broken tools ● Tools enforce a manual process
- 23. @krisbuytaert Case 3: 2 years later ● Average “devops” role stays for 2 months , then leaves ● Senior IT management has left (2x) ● Only In house analysts remain ● Mostly contract based developers ● Failing Cloud Strategy ● Legacy Container Ecosystem
- 24. @krisbuytaert Case Conclusions Earlier involvement of the Ops Skills creates much higher success rates , and smoother Continuous Delivery adoption
- 25. @krisbuytaert Why ops first ? ● You can’t support / understand what you don’t do yourself ● Code = Code ● Unblock delivery ● Unblock provisioning ● Metrics & Monitoring Build in
- 27. @krisbuytaert Culture Hack Set up CI/CD for your CI/CD infrastructure first, If the people running your infra don't know how CI/CD works , how do you expect them to support / teach your application teams ?
- 28. @krisbuytaert How do we do this ?
- 29. @krisbuytaert A CI Ecosystem ● Version Control ● Deployment ● Build Tooling ● Artifact Repository ● Code Coverage Tooling ● Testing Tools How many of those tools is your average ops person used to use ?
- 30. @krisbuytaert Arguments against CI ● Setting up the stack costs time ● U don’t have tests anyhow ● Operations and development are different budgets ● One shot projects , fire and forget ● ...
- 31. @krisbuytaert Not Continuous Deployment ● @stahnma @#devopsdays Ohio
- 32. @krisbuytaert Infrastructure as Code ● Treat configuration automation as code ● Development best practices ● Model your infrastructure ● Version your cookbooks / manifests ● Test your cookbooks/ manifests ● Dev/ test /uat / prod for your infra ● Model your infrastructure ● A working service = automated ( Application Code + Infrastructure Code + Security + Monitoring + ... ) PS. Converting Bash to Yaml != IAC
- 33. @krisbuytaert Version all the things No more excuses !– Source code Application – Source code Infrastructure – Builds – Tests – Pipelines – Scripts – Documentation – Monitoring scripts
- 34. @krisbuytaert
- 35. @krisbuytaert Software Release management is not a solved problem
- 36. @krisbuytaert Unless you understand Git Submodules ● Basic git, ● No extra tools required Integrates with other projects too. (No need for *-librarian etc ..)
- 37. @krisbuytaert Continuous Integration Continuous integration (CI) is the practice, in software engineering, of merging all developer working copies with a shared mainline several times a day. It was first named and proposed as part of extreme programming (XP). Its main aim is to prevent integration problems, referred to as "integration hell" (WikiPedia) Does the app you are deploying still work ? Did you break your puppet / chef code ?
- 38. @krisbuytaert What's in your Infra Pipeline ?
- 39. @krisbuytaert A pipeline ● Checkout code ● Syntax ● Style ● Code Coverage ● Tests ● Build ● More Tests ● Package ● Upload to Repo
- 40. @krisbuytaert Artifacts: ● Tested artifacts that go through a pipeline application code, Infra code metadata tests
- 42. @krisbuytaert A pipeline ● Checkout code ● Syntax ● Style ● Code Coverage ● Tests ● Build ● More Tests ● Package ● Upload to Repo ● Deploy on Test ● Check Puppetruns ● Check Monitoring
- 43. @krisbuytaert Testing = Monitoring ● Deploy a host, ● Add it to the monitoring framework ● Add collection tools ● Add check definitions ● Update the monitoring tool config FULLY AUTOMATED
- 44. @krisbuytaert A pipeline ● Checkout code ● Syntax ● Style ● Code Coverage ● Tests ● Build ● More Tests ● Package ● Upload to Repo ● Deploy on Test ● Check Puppetruns ● Check Monitoring ● Promote to next target (e.g UAT) ● Increase Testing ● Promote to next target (e.g Prod)
- 45. @krisbuytaert Jenkins Job DSL ● Groovy ● Git ● Rebuild jobs on commit ● Projects in folders ● 1 seed job to seed them all
- 46. @krisbuytaert Larger CI Stacks ● Generate Pipelines / Jobs based on config files ● Build libraries – CheckoutJob – DeployJob – PackageJob ● Use Groovy / JobDSL to generate PipelineDSL
- 47. @krisbuytaert Testing Multiple Versions ● Initial stage tests code on multiple versions – e.g current puppet version – Next puppet version – Bleeding Edge version ● Only current version breaks build ● Goal = get all versions green
- 48. @krisbuytaert On Prem vs Cloudnative ● Puppet, Chef, Ansible, Terraform ● Kvm/OpenStack/.. ● VPN, Firewalls ● Pipelines & Pipelines ● Standardisation ● Security ● Monitoring ● Puppet, Chef, Ansible, Terraform ● EC2 etc .. ● VPC, SG ● Pipelines & Pipelines ● Standardisation ● Security ● Monitoring
- 49. @krisbuytaert Dev Ooops #container edition ● Put this Code Live, here's a Docker Image ● No machines available ? ● What database ? Where to store the data ? ● Security ? What distro is this even ? Bad Cows ? ● How do we monitor his ? ● Backups ? ● How did you build this ? ● Has DockerHub been hacked again ?
- 51. @krisbuytaert Large Challenge ● How do you automatically spin up a K8s setup in an existing Ecosystem ● No vendor Lockin, Open Source only ? ● How do you deploy applications on K8s , reproducible, not using 30 bash scripts ● Container Ecosystem is no where close to the level of automation / reproducibility which we are used in VM’s
- 52. Contact InuitsInuits Essensteenweg 31Essensteenweg 31 BrasschaatBrasschaat BelgiumBelgium 891.514.231891.514.231 +32 475 961221+32 475 961221 Kris Buytaert [email protected] Buytaert [email protected] Further ReadingFurther Reading @krisbuytaert@krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/ https://inuits.eu/https://inuits.eu/