Building an Automated Database Deployment Pipeline

BUILDING AN AUTOMATED DATABASE
DEPLOYMENT PIPELINE
Grant Fritchey
Red Gate Software
Continuous delivery for databases

Goals
 Understand the technology and process
requirements to work towards automation step-
by-step in your release pipeline.
 Learn about the organizational changes
necessary to support process modifications.
 Appreciate why these changes are necessary in
support of modern development and
deployment methodologies.

grant@scarydba.com www.scarydba.com
@gfritchey www.linkedin.com/in/scarydba
Grant Fritchey
Product Evangelist, Red Gate Software

ALM – and where the database fits in
Three core processes in Application Lifecycle Management:
Governance Development Operations

Natural friction across pipeline
Development → Operations

Building an Automated Database Deployment Pipeline

Natural friction across pipeline
Development → Operations
Operations → Development
Why?

 Agile
 Scrum
 Lean
 Feature-driven Development
 Iterative

Operations focus is on production protectionprotection

 Monitoring
 Deployment
 Integrity
 Data Management

Databases as a bottleneck – historically
Odd languages
 SQL
 Cubes
 X-Query
3 reasons why databases have traditionally slowed down
deployments:
Data persistence
 Data outlives
applications
 Data can’t be
replaced
DBA paranoia
 Frankly…
1 2 3

A quick primer on continuous delivery
Common misconception: it’s about automating to Production

A quick primer on continuous delivery
Common misconception: it’s about automating to Production
In fact –
there’s a difference between three continuous approaches:
Development
Source
control CI server Test Production
Continuous integration
Continuous deployment
Continuous delivery

The goals of continuous delivery for
databases
 Faster feedback on changes made
o Continuously integrate team changes
o Automated testing
o Releases rehearsed in testing environments before
deployed to Production
= repeatability, repeatability and
a strong team process for changes

Focus on the pipeline
Think of the Toyota production system…
 Start with Lean
o Focus on the customer, eliminate waste
o Continuously Improve
o Empower the team
o Optimize the whole
o Plan for change
o Automate processes
o Build quality in

SOURCE
CONTROL
CONTINUOUS
INTEGRATION:
FUNDAMENTALS
CONTINUOUS
INTEGRATION:
ADVANCED
AUTOMATED
DEPLOYMENT
Four key stages of the deployment pipeline

SOURCE
CONTROL
 Greenfields
 Existing systems
 Test data

 Automate builds as first step
o Incremental
o Complete
 Build library of manual tests
 Automate testsCONTINUOUS
INTEGRATION:
FUNDAMENTALS

 Select CI tooling
 Team discipline
 Customization of build steps
 Automate the testingCONTINUOUS
INTEGRATION:
ADVANCED

 Testing
 Automation
 Safety
o Backups
o Rollback strategyAUTOMATED
DEPLOYMENT

Create development environment
for automation
Empower Development
 Environment should let them work at their speed
Take part in Development
 Instead of stopping bad deployments, stop bad development
Automate your build process
 Supply clean production data or supply good sample data
Get started on writing tests
 Build your library

Different types of testing for different stages of the pipeline:
Development
Integration
Testing QA
Pre-Production/
Staging Production
Automate testing
Unit tests Integration tests
Automated tests
Deployment validation
Behaviour validation
Other validations

B C D
A
Always Be Continuously Delivering
Deliver

Fail faster
 But less frequently
 Smooth the process

Testing, testing, testing
 Select the right tests for each stage
 Review and act on the results

DBAs must work with Devs…
Yes, you must protect the data for the business, but
that must be tempered with helping deliver functionality.
Changes to philosophy

Devs must work with DBAs…
Yes, you may know more about business needs,
so educate rather than isolate.

Project Management must think of operations as part
of development…
Yes:
 Deployment is part of development
 Release 1.1 and on are part of development
o And planning for 1.1 is not premature optimization
 Data retention is part of development

Changes to workplace
SOURCE
CONTROL
CONTINUOUS
INTEGRATION:
FUNDAMENTALS
CONTINUOUS
INTEGRATION:
ADVANCED
AUTOMATED
DEPLOYMENT

Changes to workplace for continuous
database delivery
 Empower developers to do more
 Bring DBAs into development teams
 Management must buy-in
 Management must get out of the way

How are other companies getting started?

Case Study: Boxon
 Global packaging and labelling company
 Based in Sweden; sub-division in China
 Three business areas: profitable packing
solutions; intelligent marking; customized
big-bags solutions for bulk handling
 One developer responsible for ASP.NET
application with SQL Server backend,
enabling customers to control consistent
printing of labels wherever the print shop
is located in the world

Boxon – Initial Process
 Deploying to production was scary…
 Database changes not version-
controlled
 2-hour window - at night - to
complete deployments to production
– difficult to find quiet period with a
global customer base
PAIN
POINTS

Boxon – Improved Process
 Deployments are less scary now…
 Tickets are logged and prioritized in Unfuddle
(unfuddle.com)
 Ticket numbers are logged in SVN and
TeamCity to track items
 Now use SQL Source Control and Subversion
to version DB changes
o Source of authority on database build
 TeamCity (CI build server) is triggered on
check-in of changes
o Fast response – changes checked in frequently
 Notification if build fails
 Build packaged into a Nuget file for deployment
BUILDING A
PIPELINE

Case Study: Move with Us
 Risk management solution and
sales and marketing channel
provider for residential property
businesses in the UK
 Located near Cambridge, UK
 Customers put a heavy load on
databases, so updates need to
be efficient

Move with Us – Initial Process
 Database code often not checked in
to VCS due to risk of conflicts –
changes were communicated by
email instead
 Hot fixes sometimes done directly
on integration server
 1 day needed to release updates –
a lot of work in building and testing
 Problems with merging code led to
a lot of firefighting
PAIN
POINTS

Move with Us – Improved Process
 Each developer has a local copy of database
 SQL Source Control and Subversion used to
check in database changes to version control
 TeamCity (CI build server) automatically
compiles changes on check-in
 Successful builds packaged into a Nuget file
automatically
o Fast getting here – multiple database changes
checked in per day
 Nuget package deployed to Test on demand
 If problems detected, return to Dev to fix
BUILDING A
PIPELINE

Case Study: Practice Fusion
 Web-based electronic health record
(EHR) company
 Founded in 2005
 Based in San Francisco
 Hosts over 50 million patient records
 Used in all 50 states and by 150k+
physicians
 3 DBA engineers; 1 data architect;
10+ developers

Practice Fusion – Initial Process
 Hand-crafted SQL scripts - inconsistent
 Database changes not always version
controlled
 Changes released by opening dozens
of files in SSMS
 Long-winded team comms if further
changes needed
 Smoke-testing deployments showed
some objects had been deployed
straight to Prod and weren’t in
deployment script
PAIN
POINTS

Practice Fusion – Improved Process
 SQL Source Control and Subversion to
version DB changes
o Source of authority on database build
 Jenkins (CI build server) is triggered on
check-in of changes
o More time to develop; less time managing scripts
 Build failed by Jenkins if problems detected
o Fewer issues deploying to production
 Jenkins, SQL Compare and SQL Data
Compare used to deploy changes to
environments
 Custom scripts for replication
BUILDING A
PIPELINE

Summary
 Eliminate or mitigate friction
o Slow is smooth, smooth is
fast
 Adopt lean methodologies
and focus
 Four key stages
o Source control
o Build automation
o Continuous Integration
o Continous Deployment
 A-B-C-D
 Change your philosophy
 Change your workplace

Documentation and resources
 Continuous Delivery by Jez Humble and David Farley (Addison Wesley)
 The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win by
Gene Kim, Kevin Behr and George Spafford (IT Revolution Press)
 The Goal: A Process of Ongoing Improvement by Eliyahu M. Goldratt and Jeff Cox
(Gower Publishing Ltd.)
 Agile Organization by the agile admin (theagileadmin.com)
Further resources:
 Database Delivery Learning program: www.red-gate.com/delivery
o Patterns and practices on Simple-Talk
o Tutorials in Red Gate training academy
 www.youtube.com/user/RedGateVideos - for recorded seminars

Image sources
Author Source Information
Chiltepinster Wikimedia Commons Mocking Bird Argument.jpg – Wikimedia Commons. This file is licensed under the Creative Commons Attribution-Share Alike
3.0 Unported license. Source on Wikimedia Commons: “Own work”
Tableatny Wikimedia Commons Athlete at Starting block.jpg – Wikimedia Commons. This file is licensed under the Creative Commons Attribution 2.0 Generic
license. Source on Wikimedia Commons: “BXP135671”
n.raveender Wikimedia Commons Stop No Entry.jpg – Wikimedia Commons. This file is in the public domain. Source on Wikimedia Commons: “Own work.”
Original uploader was Liface
at en.wikipedia
Wikimedia Commons Cross Country US.jpg – Wikimedia Commons. This file is licensed under the Creative Commons Attribution-Share Alike 3.0
Unported license. Source on Wikimedia Commons: “Transferred from en.wikipedia; transfer was stated to be made by
User:TFCforever.” Notes on license: Liface at the English language Wikipedia, the copyright holder of this work, hereby
publishes it under the following license:GNU head Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software
Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
Subject to disclaimers.
Horia Varlan Flickr Graduated cylinders and beaker filled with chemical compounds – Flickr. This file is licensed under the Creative Commons
Attribution 2.0 Generic license.
Henry Mühlpfordt Flickr CERN Atlas Control Room 2010-07-01 – Flickr. This file is licensed under the Creative Commons Attribution-ShareAlike 2.0
Generic license.
Department for Business,
Innovation and Skills
Flickr Toyota’s new Auris – Flickr. This file is licensed under the Creative Commons Attribution-NoDerivs 2.0 Generic license.
Qrodo Photos Flickr Sport Action – Fencing – Flickr. This file is licensed under the Creative Commons Attribution 2.0 Generic license.
William Warby Flickr Gears – Flickr. This file is licensed under the Creative Commons Attribution 2.0 Generic license.

Building an Automated Database Deployment Pipeline

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Building an Automated Database Deployment Pipeline (20)

More from Grant Fritchey (20)

Recently uploaded (20)

Building an Automated Database Deployment Pipeline