Continuously serving the developer community with Continuous Integration and Delivery
7 likes2,466 views
The document discusses the processes and challenges of continuous integration and continuous delivery (CI/CD) in software development, focusing on automated deployments and database migrations. It emphasizes the importance of reliability, zero downtime deployments, and feature toggles for managing software changes. Lessons learned highlight that while achieving zero downtime is complex, ongoing improvements and learning are essential to success.
Continuously serving the developer community with Continuous Integration and Delivery
- 1. Continuously serving the developer community with Continuous Integration and Delivery Akshay Karle Fernando Júnior
- 2. “How long would it take your organization to deploy a change that involves just one single line of code?” Mary and Tom Poppendieck
- 3. “(…) build software in such a way that the software can be released to production at any time.” Martin Fowler
- 6. CONTINUOUS DELIVERY ▫︎ Continuous Integration ▫︎ Automated tasks ▫︎ Repeatable/reliable process
- 7. CONTINUOUS DELIVERY frequent, reliable releases of high quality valuable software
- 11. DEPLOYING YOUR APPLICATION ▫︎ Database migrations ▫︎ Infrastructure update ▫︎ Restarting services
- 12. ZERO DOWNTIME DEPLOYMENTS Deployment process of your application has got to be transparent for end users
- 14. WHAT WE DO? ▫︎Continuous Integration and Delivery for repositories on GitHub ▫︎Cloud Hosted
- 15. SNAP INSIDE-OUT
- 16. LIFECYCLE OF A BUILD head repository commit
- 17. LIFECYCLE OF A BUILD head repository commit
- 18. Babysitters LIFECYCLE OF A BUILD head repository commit DATABASE
- 19. LIFE CYCLE ▫︎ Prepare the container ▫︎ Starts database ▫︎ Sets PATH for languages ▫︎ Download artifacts ▫︎Git clone ▫︎Runs each pipeline stage ▫︎Upload artifacts DATABASE
- 20. Babysitters LIFECYCLE OF A BUILD head repository commit DATABASE
- 22. ARCHITECTURE OVERVIEW ▫︎ Rails app fronted by nginx ▫︎ Receives the hooks from GitHub Database web server
- 23. ARCHITECTURE OVERVIEW L B Build Server web server Database
- 24. ARCHITECTURE OVERVIEW ▫︎ Background jobs ▫︎ Babysitters ▫︎ Build Queue ▫︎ Artifacts ▫︎ OpenVZ Containers ▫︎ Virtual machines where the pipeline runs Build Server
- 25. ARCHITECTURE OVERVIEW L B Build Server web server Database
- 26. OUR DEPLOYMENT ▫︎ Do have automated scripts ▫︎ Deployment pipeline ▫︎ 1-click deploy ▫︎ Sort of… Babysitters DATABASE L B Build Server Database web server
- 27. OUR DEPLOYMENT ▫︎ Wait for all builds to finish ▫︎ Put app on maintenance mode ▫︎ No new requests picked up ▫︎ Deploy and wait for migrations Babysitters DATABASE L B Database VZHOSTBuild Server web server
- 29. BLUE-GREEN DEPLOYMENT Web Server App Server Database App ServerApp ServerWeb Server Database
- 30. BLUE-GREEN DEPLOYMENTS FOR SNAP VZHOSTBuild Server web server VZHOSTBuild Server web server L B DATABASE
- 31. CHALLENGES IN SNAP Long running builds Running multiple versions of your app at the same time Database migrations
- 32. LONG RUNNING BUILDS ▫︎ Builds should continue to run ▫︎ Artifacts should continue to be served
- 33. COLOR AWARE BABYSITTERS Build Server Green Stack Build Server Blue Stack web server
- 35. THE SWITCH L B
- 36. THE SWITCH L B
- 37. CHALLENGES IN SNAP Long running builds Running multiple versions of your app at the same time Database migrations
- 38. FEATURE TOGGLES Hide unfinished UI elements Control backend behaviour Test with feature toggles Avoid multi-component feature toggles http://martinfowler.com/bliki/FeatureToggle.html
- 39. FRONTEND <% if feature_enabled?(:parallel_stage) %> <!-- show new html --> <% else %> <!-- show old html --> <% end %>
- 40. BACKEND if feature_enabled?(:parallel_stage) # new logic else # old logic end
- 41. TESTING WITH FEATURE TOGGLES describe "multiple jobs" do describe "feature enabled" do before(:each) do with_feature_enabled(:parallel_stage) end it 'should not show the job tabs when there is only one job' do end end describe "feature disabled" do before(:each) do with_feature_disabled(:parallel_stage) end it 'should not show the job tabs but should show the logs' do end end end
- 42. CHALLENGES IN SNAP Long running builds Running multiple versions of your app at the same time Database migrations
- 43. MIGRATION OF DATA Consider existing schema as well as data Incremental changes Rollbacks
- 46. POPULATE ALL EXISTING STAGES stages id started_at completed_at result … jobs id stage_id …
- 47. COPY ATTRIBUTES TO JOB stages … started_at completed_at result … jobs … started_at completed_at result …
- 48. SWITCH THE APPLICATION TO START USING THE JOB MODEL # After switch class Stage def result results = jobs.collect { |job| job.result } return :failed if results.any?(:failed) :passed end end # Before switch class Stage attr_reader :result end # in transition class Stage def result if feature_enabled?(:parallel_stage) results = jobs.collect { |job| job.result } return :failed if results.any?(:failed) :passed else result end end end
- 50. LESSONS LEARNT
- 51. LESSONS LEARNT Zero downtime is not easy Having reliability, automation, frequent releases helped Watch out your data Things will go wrong, but we keep learning and keep improving