DevOps and Build Automation

Editor's Notes

• #14: Infrastructure as code - Allows to automate deployments in different environments (dev, test, staging, prod)
• #15: DevOps practitioners rely on automation. An automation platform gives us the ability to describe our entire technology stack as executable code.
• #21: Some people might question whether gains in efficiency are typical, noting that they have worked on projects where, in their view, trying to automate tasks actually cost the project more time than it saved. Depending on the situation, this may be a real concern. In addition, automation might have been implemented poorly or carried too far on some projects – but keep reading for on what to automate and when.
• #32: Because DevOps is a cultural shift and collaboration (between dev, ops and testing), there is no single “DevOps tool” – it is rather a set, consisting of multiple tools (DevOps toolchain). Generally, DevOps tools fit into one or more of these categories, which is reflective of key aspects of the software dev and delivery process.
• #37: The more components you add to your software, the more lines of code you maintain, and the more tests and routines you run as part of your build process- the longer the build will take to run. Also, some development environments and technologies, like C/C++, tend to be correlated with longer builds. In Agile work environments, builds are expected to be run frequently and the organization depends on build output to guide development work. Long builds can be a big problem, and one that can “creep up” on a development team. Why it matters? Some builds are so long they can only be ran nightly. But even if your “normal” build takes just 20 or 30 minutes more than it should – there are significant costs to your organization: Developer wait time – as we shown in a survey we conducted last year, most developers spend between 2-10 hours per week waiting for builds to complete – more often than not, this is lost time. Context switch – while the build runs, developers switch away from the current task to other ones. If the build is long, developers lose context related to the problem they were working on, which reduces productivity. In other words, if builds are supremely fast, developers never have to stop thinking about the problem they are working on right now. Build bottleneck – during intensive development phases such as “integration storms”, developers sync changes into the main code line and frequent builds are needed to get everything working. The less frequent the build, the less problems dev can solve every workday. Product quality – when a developer commits changes that break something, it will only be discovered after the build runs. When the bug is fixed, again there is a lag waiting for the next build before QA can verify the fix. The less frequent the build, the less issues can be fixed and verified before the release – hurting product quality.
• #38: Sometimes, individual builds run relatively quickly. But in some organizations, there could be dozens or hundreds of builds run in each dev/test cycle. This could be because numerous teams (sometimes thousands of developers) developing different software components, and each running its own build. Or, the dev team might need to deliver numerous versions of the software for different platforms, customized builds for different customers, etc. Some organizations have relatively short builds– but find themselves needing to support dozens, if not hundreds, of these builds at any given time. Why it matters: If you’re a build engineer responsible for running 500 builds, you’ll feel the pain even if each of them is 10 minutes long (=~83 hours build time without parallelization). While builds may be short, cumulatively they take very long to run. But if you’re a developer or QA engineer using such as build system, and your build takes only 10 minutes, why should you care? Limited build resources – because the organization is running large numbers of builds, you’ll find you have limited access to build servers during specific time windows, or the servers are often overloaded and builds will take much longer.When you rely on running builds often to get fast feedback and fix bugs, you’ll notice that having to “wait in line” for your builds to run hurts your productivity (particularly if you’re practicing Agile), and that you can’t move development fast enough because your waiting for a build server.
• #39: Software projects use a large number of modular components: different frameworks, components developed by different teams or by 3rd-party partners, open source libraries, and so on. As your product evolves, there are multiple versions of your own code, and also multiple versions of these many components, creating a many-dimensional matrix of dependencies and supported behaviors. That’s where things get complex to build. Why it matters: Complex builds reduce the flexibility of the build process and makes it much more difficult to manage and run the build: Complex builds are brittle – interactions between many different components often lead to manual error, broken builds and worse – builds that run correctly but introduce bugs due to partial or incorrect sources. Extensive manual efforts – executing a complex build and delivering its results requires a substantial manual effort. Even if the build is automated, it is typically automated in small pieces/components, and there is no orchestration of the entire process. Incremental builds are difficult– often you’ll want to run a partial build and re-purpose items that haven’t changed and were previously compiled. With complex builds – due to partially specified dependencies – an incremental run could break the build and teams are forced to run the entire build in all scenarios. Legacy components and fear of change – complex builds tend to have legacy components written years ago by staff members who are no longer at the organization. This impedes changes or optimizations to the build, for fear of possibly breaking legacy components that are not well understood. Complex builds are long – there is a correlation between the complexity of the build and the time it takes to run, which introduces additional issues as described above.