Iteratively introducing Puppet technologies in the brownfield; Jeffrey Miller

ORNL is managed by UT-Battelle, LLC
for the US Department of Energy
Iteratively introducing Puppet
technologies in the brownfield
Jeffrey Miller
HPC Linux Systems Engineer
HPC Core Ops Group
National Center for Computational Sciences

22 Open slide master to edit
Citation and Disclaimer
This work was supported by the Oak Ridge Leadership
Computing Facility (OLCF) and the Computer and Data
Environment for Science (CADES) at Oak Ridge National
Laboratory (ORNL) for the Department of Energy (DOE)
under Prime Contract Number DE-AC05-00OR-22725
This presentation does not contain any proprietary or
confidential information.

Acknowledgements
Greg Shutt, CADES Task Lead
Cory Stargel, HPC Infrastructure Task Lead
Larry Orcutt, HPC Linux Systems Engineer
Michael Shute, HPC Linux Systems Engineer
James “Jake” Wynne, III, HPC Linux Systems Engineer

Contact Information
Jeffrey Miller
Email: millerjl@ornl.gov
LinkedIn: https://www.linkedin.com/in/millerjl1701/
GitHub: https://github.com/millerjl1701
On Slack:
– https://puppetcommunity.slack.com
– https://simp-project.slack.com/
– https://hangops.slack.com/
ID: millerjl1701

What We Do:
Infrastructure Team – HPC Core Ops Group
As part of the National Center for Computational
Sciences (NCCS), the HPC Core Ops group provides all
the necessary infrastructure services, networking
support, security oversight, and monitoring analytics
required to keep the OLCF leadership supercomputing
systems healthy.
The Infrastructure Team provides necessary external
services for use by the OLCF HPC resources as well as
other programs and projects supported by NCCS.
CADES provides a compute and data infrastructure
environment to enable the scientific discovery process
for researchers at ORNL and their collaborators.

Wouldn’t this be great?
From Deer Standing, by Petr Kratochvil. Retrieved from
https://www.publicdomainpictures.net/pictures/20000/velka/deer-standing.jpg

Brownfield Infrastructure
• Preexisting environment
providing production services
• Inventory? What inventory?
• Documentation?
• Compliance?
• Conglomeration of
configuration methods?
• Disaster recovery? Backups?
• etc.
From Dry Agricultural Brown Soil, by George Hodan.
Retrieved from
https://www.publicdomainpictures.net/pictures/220
000/velka/dry-agricultural-brown-soil.jpg

Don’t Touch Anything
From Fire in Dumpster, by Ben Watts, 2009. Retrieved from
https://commons.wikimedia.org/wiki/File:Dumpster_Fire_(4088047046).jpg

Where to start???
Retrieved from https://www.reddit.com/r/factorio/comments/982cw3/spaghetti_spaghetti/

Bolt – What is this?
“An open source orchestration
tool that automates the manual
work it takes to maintain your
infrastructure”
- Works against local or remote
- Run scripts or commands
- Organized in tasks and plans
- Plans can be written in
Puppet or yaml
- Connects to remote targets
over SSH or WinRM
From https://puppet.com/docs/bolt/latest/bolt.html

Bolt – What can you do?
You probably have a desktop system and/or a
management server…
• Install bolt and start writing a “laptop_config” plan
– Install git and other tools
– Run .dotfiles setup script
• Keep your code in git and commit often
• https://forge.puppet.com/encore/patching
• Start a habit of automate first
• Read Ben Ford’s April 2, 2020 blog post :
https://puppet.com/blog/automating-from-zero-to-
something/

Automate All the Repos
Consider setting up a GitLab instance using bolt and the
Vox Pupuli GitLab puppet module on a system if you
don’t have an instance already.
GitLab and GitLab runners can enable:
- Code review process
- Infrastructure code deployment to a management
server
- Puppet code repositories validation testing and
deployment to Puppet Servers

Provisioning
Razor
”Advanced provisioning
application that can deploy
both bare metal and virtual
systems” by Puppet
- PXE boot management
- Hypervisor deployment
- Automate the hand off to
configuration management
https://github.com/puppetla
bs/razor-server/wiki
Terraform
“a tool for for building,
changing, and versioning
infrastructure safely and
efficiently” by HashiCorp
- Infrastructure as Code
- Terraform creates the VM and
razor provisions
https://www.terraform.io/intro/i
ndex.html

Puppet Agent and Facter
• Facter: Puppet’s system profiling library that is included
with the Puppet Agent package
– Bolt leverages facter to retrieve node facts
– But, for Bolt to use facter, the Puppet agent needs to be
installed
• Enter apply_prep
– Built in Bolt function like run_command, run_script, etc.
– Installs the Puppet agent package if it isn’t already installed
– Collects facts from the node into the running inventory
– This can be an expensive operation…

Puppet Infrastructure
• Using Bolt:
– Install the Puppet Agent on several new VMs
– Install and configure a PuppetCA (and optionally catalog
compile servers)
– Deploy PuppetDB and backend PostgreSQL database
– Reconfigure Puppet Server systems to use PuppetDB
• Using Puppet or Bolt
– Deploy Puppetboard (or alternative) dashboard to PuppetDB
Then…

Puppet Agent Rollout
• Using Bolt:
– Install the Puppet agent on each system (hardware or VM)
– Configure the Puppet agent to register with the Puppet Server
infrastructure
– Then, (this is key) have the Puppet agent configure absolutely
nothing
Yes… Absolutely nothing. Null. Zero. Zilch.
What you have now is a growing inventory that furthers
systems and services discovery.

--noop or noop()
When Puppet is run locally on
the system, a “--noop” flag may
be passed to report what
would change but not actually
change anything.
Similar result with the “noop”
parameter set puppet.conf.
noop() function
Function in the trlinkin-noop
module that sets a scope to
noop
https://forge.puppet.com/trlinki
n/noop
For examples of this working see: Puppet noop, no-noop, and the path
to safe Puppet Deployments, Alessandro Franceschi, 2017.
https://blog.example42.com/2017/12/27/noop-no-noop-and-the-path-to-safe-
puppet-deployments/

Systems Configuration – Iterative Style
• If you haven’t already, `git init`
• What do you want to configure today?
• Leverage a git workflow for puppet code use
– Create branch
– Add puppet code
– Deploy branch to Puppet Servers
• Use bolt to test canary systems using noop
• Find edge cases on the other nodes using noop
• Merge and enforce (i.e. no-noop)

Even better…
Test environment…
- Dedicate Puppet Servers to serve out test code
- Test VMs for types of nodes you support in production
- Consider using a unique environment for testing
instead of a branch against production
- Code dev -> squash -> merge -> cherry pick

Even better better…
Watch:
Multi-node acceptance tests for fun and profit. Trevor
Vaughan, 2019.
https://www.youtube.com/watch?v=4iBEIMQkBCk

Final thoughts…
Implementation takes a willing team.
Learn to trust the process and the tooling.
Trust, but verify.
Be flexible. There will be landmines in the brownfield.
Focus on what can be done rather than shortcomings.
Patience – the iterative process may take a long time…
Notice I didn’t mention Ruby skillz… oops. #facepalm
The Puppet Community.

Questions?

Iteratively introducing Puppet technologies in the brownfield; Jeffrey Miller

More Related Content

What's hot (20)

Similar to Iteratively introducing Puppet technologies in the brownfield; Jeffrey Miller (20)

More from Puppet (20)

Recently uploaded (20)

Iteratively introducing Puppet technologies in the brownfield; Jeffrey Miller