Towards Continuous Deployment with Django
45 likes18,341 views
The document presents a comprehensive guide to implementing continuous deployment using Django, focusing on strategies and tools such as Vagrant, Fabric, and Jenkins. Key concepts include automating the build and deployment process, leveraging virtual environments for consistent development, and utilizing monitoring tools for performance measurement. The talk emphasizes the importance of agile methodologies, testing, and version control in achieving a streamlined continuous delivery pipeline.
![Develop Commit Build Stage Deploy Measure
Using Fabric with Vagrant
def vagrant():
# get vagrant ssh setup
vagrant_config = _get_vagrant_config()
env.key_filename = vagrant_config['IdentityFile']
env.hosts = ['%s:%s' % (vagrant_config['HostName'],
vagrant_config['Port'])]
env.user = vagrant_config['User']
def _get_vagrant_config():
with lcd('../vagrant'):
result = local('vagrant ssh-config', capture=True)
conf = {}
for line in iter(result.splitlines()):
parts = line.split()
conf[parts[0]] = ' '.join(parts[1:])
return conf
Based on https://gist.github.com/1099132](https://image.slidesharecdn.com/towardscontinuousdeploymentwithdjango-120818234852-phpapp02/85/Towards-Continuous-Deployment-with-Django-14-320.jpg)
![Develop Commit Build Stage Deploy Measure
Test separation
# Used to ignore tests
ignore = lambda test_item: None
def pipelineTag(*args):
"""
Let a testcase run if any of the tags are in sys.argv and
none of the tags match not-* in sys.argv.
"""
# Abstain if we're not in a build pipeline (ie using django-jenkins)
# or if no tag(s) specified
if 'jenkins_pipeline' not in sys.argv or not any(['tagged-' in arg for arg in sys.argv]):
return _id
tags = args
assert set(tags) < set(['unit', 'functional', 'flaky', 'slow', 'staging'])
#Silently ignore if no matching tag
if not any(['tagged-'+tag in sys.argv for tag in tags]):
return ignore
# Skip if "not-blah" tagged
if any(['not-'+tag in sys.argv for tag in tags]):
return skip('tagged: ' + ','.join([tag for tag in tags if 'not-'+tag in sys.argv]))
# This test made it through
return _id](https://image.slidesharecdn.com/towardscontinuousdeploymentwithdjango-120818234852-phpapp02/85/Towards-Continuous-Deployment-with-Django-27-320.jpg)
Towards Continuous Deployment with Django
- 1. Towards Continuous Deployment with Django Roger Barnes @mindsocket [email protected] PyCon Australia 2012 1
- 2. Beer Adventure Photography Frisbee Web development Python/Django Co-founder @ Arribaa BTech ICS 1997 1999 2001 2003 2005 2007 2009 2011
- 3. Concepts from other talks ● The why – Lean startups and customer discovery – Do more with less ● The how – EC2 / provisioning – Architecture – Developing for cloud deployment – Monitoring live web applications – IaaS vs PaaS
- 4. What is continuous delivery* "Rapid, incremental, low-risk delivery of high quality, valuable new functionality to users through automation of the build, testing and deployment process" - Jez Humble Deploying every good version of your software... … or at least being able to "Minimising MTTBID (Mean Time to Bad Idea Detection)" – Etsy * delivery == deployment for the purposes of this talk
- 5. More Than Technology ● Technology - For automation ● People - Cross-functional team, organised around product ● Processes – Conventions and some glue that technology can't automate
- 6. Why ● Fail fast, win fast - Build, Measure, Learn ● Competitive advantage ● Avoid YAGNI - do less ● Less manual process, less risk ● Real-time control – Deploy on demand – Decoupled deployment and release – Self service environments
- 7. Continuous Delivery in Practice ● Single path to production ● Optimise for resilience ● Get comfortable with being uncomfortable ● Automate as much as possible ● If it hurts, do it more often ● Becomes natural, return on investment is fast/high ● Lots of ways to do it, here's what I've done...
- 9. Develop Commit Build Stage Deploy Measure Environments ● Make dev/test/prod as similar as possible – Full stack, no "./manage.py runserver" – Some exceptions make sense. eg for development: ● dummy email backend ● dummy message broker ● fewer threads/workers ● less memory ● less analytics instrumentation
- 10. Develop Commit Build Stage Deploy Measure Environments Dev/test virtualised using Vagrant Create and configure lightweight, reproducible, and portable development environments $ vagrant up $ vagrant ssh
- 11. Develop Commit Build Stage Deploy Measure Environments Vagrantfile configures base image, provisioning, shared folders, forwarded ports Vagrant::Config.run do |config| config.vm.box = "precise64" config.vm.box_url = "http://files.vagrantup.com/precise64.box" config.vm.provision :puppet do |puppet| puppet.manifests_path = "puppet/vagrant-manifests" puppet.manifest_file = "dev.pp" puppet.module_path = "puppet/modules" end config.vm.forward_port 80, 8000 config.vm.forward_port 3306, 3306 config.vm.share_folder "arribaa", "/opt/django-projects/arribaa", ".." end
- 12. Develop Commit Build Stage Deploy Measure Environments ● Repeatable, versioned configuration – Snowflake bad, phoenix good – Puppet (masterless) for provisioning OS and services – Fabric for scripted tasks, eg: ● copy database from production ● update requirements ● migrate database ● commit and push to repository ● Anti-pattern – Can't spin up new environment with one command
- 13. Develop Commit Build Stage Deploy Measure Development top level puppet config include uwsgi include statsd include solr include memcached include rabbitmq Test configuration is the same include nginx include testing Production adds: include arribaa backup include arribaa::db_node monitoring include arribaa::nginx include arribaa::celery
- 14. Develop Commit Build Stage Deploy Measure Using Fabric with Vagrant def vagrant(): # get vagrant ssh setup vagrant_config = _get_vagrant_config() env.key_filename = vagrant_config['IdentityFile'] env.hosts = ['%s:%s' % (vagrant_config['HostName'], vagrant_config['Port'])] env.user = vagrant_config['User'] def _get_vagrant_config(): with lcd('../vagrant'): result = local('vagrant ssh-config', capture=True) conf = {} for line in iter(result.splitlines()): parts = line.split() conf[parts[0]] = ' '.join(parts[1:]) return conf Based on https://gist.github.com/1099132
- 15. Develop Commit Build Stage Deploy Measure Dependencies ● virtualenv – separate env for each app ● pip – install dependencies into virtualenv – use requirements file – use explicit versions ● for repository based dependencies, use commit id ● or fork on github
- 16. Develop Commit Build Stage Deploy Measure Dependencies ● Using virtualenv and pip with fabric env.site_dir = '/opt/django-projects/arribaa' env.app_dir = '/opt/django-projects/arribaa/arribaa' env.pip_file = 'requirements.txt' def ve_run(command, func=run, base_dir=env.app_dir, *args, **kwargs): with cd(base_dir): with prefix("source /opt/virtualenvs/%s/bin/activate" % env.virtualenv): return func(command, *args, **kwargs) def update_reqs(): ve_run('pip install -r %s' % env.pip_file, base_dir=env.site_dir)
- 17. Develop Commit Build Stage Deploy Measure Database migration ● South – Schema changes – Data migrations ● Deploy separate expand and contract operations, eg: – Expand ● add new column (update model, manage.py schemamigration …) ● map from old column (manage.py datamigration …) – Deploy – Contract (optional) ● remove old column ( update model, manage.py schemamigration …)
- 18. Develop Commit Build Stage Deploy Measure Database migration ● Using South with fabric def syncdb(): ve_run("python manage.py syncdb --noinput --migrate")
- 19. Develop Commit Build Stage Deploy Measure Tying it all together ● Update pip requirements on vagrant VM $ fab vagrant update_reqs ● Run data migration on vagrant VM $ fab vagrant syncdb
- 20. Develop Commit Build Stage Deploy Measure Source control ● Pick one, learn it ● Use tags to keep track of build status ● Avoid long-lived branches – or integrate them – more overhead ✔ application code ● Tracks all code/documentation ✔ deployment code ✔ provisioning code ✔ configuration code ✔ documentation ✗ not build artifacts
- 21. arribaa Application code ├── apps Django apps │ └── ... ├── static Django static files │ └── ... ├── templates Django templates │ └── ... ├── fabfile.py Fabric scripts ├── requirements.txt Pip └── ... bin └── jenkins.sh Jenkins job docs └── ...
- 22. vagrant ├── Vagrantfile Dev VM config └── puppet ├── modules Puppet modules │ ├── arribaa │ ├── backup │ ├── graphite │ ├── ... │ ├── uwsgi │ └── wget └── vagrant-manifests ├── dev.pp Puppet dev ├── prod.pp Puppet prod* └── test.pp Puppet test * Not used by vagrant, but convenient to store here
- 23. Develop Commit Build Stage Deploy Measure Automated Testing ● Django’s test framework ● Continuous Integration and Testing – Jenkins ● django-jenkins – tests – pylint – coverage – css/jslint ● factory_boy instead of fixtures
- 24. Develop Commit Build Stage Deploy Measure Test separation ● Split up tests by type ● Can parallelise, keeping test run times low – < 20 minutes from commit to production ● Unit tests first, faster feedback ● Run tests on different triggers, keep some out of the main build ● Etsy - Divide and Concur – http://codeascraft.etsy.com/2011/04/20/divide-and-concur/
- 25. Develop Commit Build Stage Deploy Measure Test separation ● Unit ● Functional/UI – Selenium et al ● Staging and production smoke tests – Crawl URLs – Load/performance testing ● Flaky tests, Slow tests – exclude from regular build, run on a separate schedule
- 26. Develop Commit Build Stage Deploy Measure Test separation ● How to do this with Django? – Suites – Custom test runner – Other test systems (Nose has an attrib plugin) ● My current solution – Annotating tests, and adapted django-jenkins command class TestBooking(TestCase): @pipelineTag ("unit") def test_when_lapsed(self): oldbooking = factories.BookingFactory.build(when=some_old_date) self.assertTrue(oldbooking.when_lapsed())
- 27. Develop Commit Build Stage Deploy Measure Test separation # Used to ignore tests ignore = lambda test_item: None def pipelineTag(*args): """ Let a testcase run if any of the tags are in sys.argv and none of the tags match not-* in sys.argv. """ # Abstain if we're not in a build pipeline (ie using django-jenkins) # or if no tag(s) specified if 'jenkins_pipeline' not in sys.argv or not any(['tagged-' in arg for arg in sys.argv]): return _id tags = args assert set(tags) < set(['unit', 'functional', 'flaky', 'slow', 'staging']) #Silently ignore if no matching tag if not any(['tagged-'+tag in sys.argv for tag in tags]): return ignore # Skip if "not-blah" tagged if any(['not-'+tag in sys.argv for tag in tags]): return skip('tagged: ' + ','.join([tag for tag in tags if 'not-'+tag in sys.argv])) # This test made it through return _id
- 28. Develop Commit Build Stage Deploy Measure Build Pipeline ● One pipeline ● All developers feed start of pipeline via VCS (master) – Improve policy of "don't deploy broken code"... – ...with system where it's harder to deploy broken code ● Jenkins Build Pipeline Plugin – Series of dependant jobs, each runs different tests – Feeds into pre-deploy staging job – Successful staging job enables deploy job
- 29. Develop Commit Build Stage Deploy Measure Build Pipeline Currently simple linear flow
- 30. Develop Commit Build Stage Deploy Measure Build Pipeline ● Jenkins job – unit test stage: bash -ex ./bin/jenkins.sh tagged-unit not-flaky not-slow ● jenkins.sh # Setup virtualenv, pip requirements, settings.py etc ...snip... cd $WORKSPACE/arribaa python manage.py clean_pyc python manage.py jenkins_pipeline "$@"
- 31. Develop Commit Build Stage Deploy Measure Staging/QA Job ● fab staging-deploy – very similar process to production deploy ● Test run of production deployment using Fabric – Deploy code and config to staging environment (Vagrant VM) – Apply dependency updates ● puppet config ● pip requirements – Copy database from production and run data migration ● Should now have a deployed QA environment ● Browser smoke tests – currently selenium – Not using Django 1.4 LiveTestCase here, we have a full stack to play with ● Tag successful build as deployable
- 32. Develop Commit Build Stage Deploy Measure Deployment Strategies ● Optimistic – Deploy, reload services – some appservers handle this well ● Blue/Green – Parallel environments, 1 idle – Idle environment gets new version – Smoke tests and monitoring before switching over ● Canary – Take subset of app servers out of pool – Deploy to subset – Smoke tests and monitoring before deploying rest of cluster ● Deploy whole new (app) server instance – then rewire load-balancer
- 33. Develop Commit Build Stage Deploy Measure Deployment with Jenkins & Fabric Using Git tags – Set in Jenkins after successful staging run – Used by deploy in Fabric def pull(): with cd(env.site_dir): run('git fetch') latest_tag = run('git describe --tags --match "ci-passed-staging-*" origin/master', pty=False) run('git reset --hard refs/tags/%s' % latest_tag, pty=False)
- 34. Develop Commit Build Stage Deploy Measure Tags set by build pipeline
- 35. Develop Commit Build Stage Deploy Measure Deployment with Jenkins & Fabric Jenkins script: virtualenv -q /opt/virtualenvs/arribaa-jenkins source /opt/virtualenvs/arribaa-jenkins/bin/activate pip install -r requirements.txt pip install -r requirements-testing.txt cd $WORKSPACE/arribaa fab deploy
- 36. Develop Commit Build Stage Deploy Measure Deployment with Jenkins & Fabric def deploy(): dbbackup() # django-dbbackup – to dropbox changes = pull() # git fetch, git describe, git log, git reset apply_puppet() # puppet apply … prod.pp update_requirements() # pip install -r requirements.txt clean_pyc() # django-extensions command syncdb() # and –migrate (South) collectstatic() reload_uwsgi() # reload app server restart_celeryd() # bounce task server run('git tag -f ci-deployed && git push –tags') # tag deployed version send_email(changes) # send notification
- 37. Develop Commit Build Stage Deploy Measure Decoupling deployment and release ● Feature flags vs feature branches – Real time control with flags – Cleanup/maint needed for flags – Harder/slower to test different combinations in either case ● Good for testing ideas (on/off, A/B, power users, QA etc) ● Gargoyle – includes optional admin interface – selective criteria (%age of users, staff only, ...) ● Alternative: Django Waffle
- 38. Develop Commit Build Stage Deploy Measure Measurement ● Eyes wide open – Track system and user behaviour ● Post-deployment – Error, performance and behaviour heuristics – Trend monitoring – Goal conversion rates
- 39. Develop Commit Build Stage Deploy Measure Measurement
- 40. Develop Commit Build Stage Deploy Measure Measurement - Munin
- 41. Develop Commit Build Stage Deploy Measure
- 42. Develop Commit Build Stage Deploy Measure Measurement ● django-statsd, statsd and graphite – page timing – counters http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/
- 43. Rollback options ● Level of automation – Depends on risk aversion and complexity of environment – 12+ years of deployment, average 5 deployments per week, very very few actual rollbacks ● Engineer around it – lots of small changes → more likely to "roll-forward" – staged deployment → minimise harm ● Engineer for it – avoid backwards incompatible data migrations – tag deployments, keep previous versions handy
- 44. Rollback options "My deployment rollback strategy is like a car with no reverse gear. You can still go backwards, but you have to get out and push. Knowing that makes you drive carefully." - Me
- 45. Where to from here Have an existing app with some gaps? Think about what's slowing you down the most ● Suggested priorities – Automated tests (Django built-in) – Continuous integration (jenkins, django-jenkins) – Measure all the things (sentry, statsd + graphite, new relic) – Scripted database (South) – Scripted deployment (fabric) – Configuration management (puppet) – Virtualised development (Vagrant) – Test separation (test decorator, customise test runner) – Feature flags (gargoyle)
- 46. Looking Forward ● Better version pinning ● Fully automated provisioning ● Better test separation and parallelisation ● Combined coverage results ● Reduce dependence on external services ● Make deployment more atomic ● Staged rollouts to cluster – canary deployment ● Make the "big red deploy button" big and red
- 47. Resources ● Slides – http://slideshare.net/mindsocket/ ● Continuous delivery/deployment – Continuous Delivery Primer - http://www.informit.com/articles/article.aspx?p=1641923 – Anatomy of the Deployment Pipeline - http://www.informit.com/articles/printerfriendly.aspx?p=1621865 – Code as Craft – Etsy http://codeascraft.etsy.com/ – Safe deploying on the cutting edge – Urban Airship http://lanyrd.com/2011/djangocon-us/shbqz/ ● Vagrant – http://vagrantup.com ● Vagrant with Fabric - https://gist.github.com/1099132 ● Jenkins Build Pipeline Plugin - http://www.morethanseven.net/2011/03/20/A-continuous-deployment-example-setup.html ● Git/fabric based rollback options – http://dan.bravender.us/2012/5/11/git-based_fabric_deploys_are_awesome.html – http://www.askthepony.com/blog/2011/07/setup-a-complete-django-server-deploy-rollback-%E2%80%93-all-in-one- powerful-script/ – http://lethain.com/deploying-django-with-fabric/