Experiences from Running Masterless Puppet - PuppetConf 2014
16 likes7,812 views
This document summarizes Spotify's experiences with running Puppet in a masterless configuration. Some key points: - Spotify previously used multiple Puppet masters but switched to a masterless setup to allow for more flexible workflows and continuous delivery of applications and configurations. - In the masterless setup, each node runs Puppet apply directly using Hiera data to determine which modules to use for that run. - Benefits of the masterless approach include easier debugging and ability to control modules on a per-node basis. Drawbacks require more manual configuration. - Spotify uses a custom Ruby wrapper, PuppetDB for facts/catalog storage, and a secret management service to support the masterless infrastructure.
Experiences from Running Masterless Puppet - PuppetConf 2014
- 1. September 24, 2014 Experiences from running masterless Puppet Erik Dalén
- 2. whoami • System engineer at Spotify since almost 3 years. • Community contributor to Puppet. • Author of Puppet Explorer web UI. • Author of the puppetdbquery, dnsquery and puppet ls modules. 2
- 3. Agenda • Background about our puppet setup • How we implemented masterless • Why we did the switch to masterless • New workflows and future improvements 3
- 4. Background • Puppet users since 4 years • Two large Puppet installations, 20+ Puppet masters. • A single repository with all modules • Git branches mapped to Puppet environments • Code review in Gerrit to merge into production branch • Tests using AWS virtual machines on every code review 4
- 5. Section name Implementation Started with just switching to the same workflow but masterless. Using a custom Ruby “wrapper” around puppet apply. 5 Gerrit SITE X SITE X git mirror git mirror SITE Y git mirror git mirror PuppetDB Node Node Node ● ServerDB ● LDAP Gerrit replication Gerrit replication git pull git pull failover if pod mirrors are unavailable Queries * Queries * Store facts * Store catalog * Store report hiera secrets hiera secrets HTTPS GET auth by cert Puppet CA request certificate
- 6. Benefits of this setup • Debug logging of compilation and hiera lookups • Ability to get traceback of custom functions directly on the node • Deprecation warnings from compilation visible on the node $ sppuppet hiera resolvconf::servers 8.8.8.8 8.8.4.4 6
- 7. Drawbacks • Requires more configuration, and can’t use puppet to manage it. • Quite specialised. 7
- 8. Secret management • Small web service where nodes authenticates using their SSL cert • Calculates the facts for the node from the certificate • Sends the secret hiera data that applies to that particular node 8
- 9. PuppetDB access control • Very binary per default, allow everyone or a whitelist 9 Node Apache PuppetDB mod_ext_filter jq
- 10. Why switch to masterless? • Scaling the Puppet masters not an issue • But scaling the workflow is an issue 10
- 11. Growing amount of committers 11 Number of monthly committers 220 165 110 55 0 2 years ago 1 year ago now
- 12. Growing amount of modules 12 Number of modules 600 450 300 150 0 2 years ago 1 year ago now
- 13. More complex dependencies • 126 modules using the apache module • 91 modules using the nginx module • 92 modules using the postgresl module Backwards incompatible changes to shared modules almost impossible to do. 13
- 14. Flexible workflow needed • Allow one set of modules per Puppet run determined by hiera • Easier ways to do continuous delivery of both application and configuration • Gradual rollout of new module versions 14
- 15. Why not r10k or librarian-puppet? Both great utilities that allow building a environment dynamically. But it is still a fixed environment for many nodes. production_new_apache production_new_apache_new_postgres production_old_apache_new_postgres production_new_apache_old_postgres 15
- 16. Our solution to the problem At each puppet run we look in hiera data which modules to install. 16 hiera/common.yaml: modules: spotify-hostbase: latest hiera/role/spotify_web.yaml modules: spotify-apache: latest hiera/role/puppetexplorer.yaml modules: puppetlabs-apache: latest
- 17. Internal forge • A simple forge implementation using the new V3 forge API • Mirrors the upstream forge • Will be used for distributing internal modules as well github.com/jhaals/puppet-anvil 17
- 18. Questions?