SQL Server Source Control Basics
0 likes1,075 views
This document is an excerpt from a book about SQL Server source control basics. It discusses the history and concepts of source control, including versioning, branching, merging and database deployment. The book uses examples with the Subversion source control system and the Bookstore database to demonstrate key source control processes for database development projects.
![Chapter 3: Getting Started with Source Control for Databases
94
The database creation script
Let's start with the script to create the Bookstore database. The database needs to
be only a basic one that uses the default configuration settings. Therefore, in its very
simplest form, we can open a SSMS query window, or even a text editor such as Notepad,
and create a script such as that shown in Listing 3-1.
USE [master]
GO
CREATE DATABASE [Bookstore] ;
GO
Listing 3-1: A basic script to create the Bookstore database.
Of course, we can customize the CREATE DATABASE statement as required, to specify
values for configuration settings such as data and log file locations, file sizes, file growth
increments and so on. We then simply save the file as Bookstore.sql to the appropriate
location in our working folder, in this case to the Database directory in the working
folder (i.e. SVNFredBookstoreProjectBookstoreDatabasesBookstoreDatabase).
Alternatively, we can create the database through SSMS (right-click the Databases node
in Object Explorer, and select New Database), and then right-click the newly created
Bookstore database and script it out (Script Database As | CREATE TO | File), choosing
the same file name and path.](https://image.slidesharecdn.com/sql-server-source-control-basics-04-06-14-v2-140904022337-phpapp02/85/SQL-Server-Source-Control-Basics-94-320.jpg)
![Chapter 4: Editing and Versioning
127
Create a new view
So far, we've only created three tables, plus some test data, so our first task is going to be
to create a code module, namely a view. Listing 4-1 creates the AuthorsBooks view. As
discussed earlier, the script is such that any modification further down the line, when
this view is "live" and we need to change its structure, incurs minimal interruption to
end-users.
USE Bookstore;
GO
IF OBJECT_ID('dbo.AuthorsBooks') IS NOT NULL
SET NOEXEC ON
-- if the object exists, SQL Server will compile, but not run, the
-- subsequent CREATE VIEW batch
GO
PRINT 'dbo.AuthorsBooks: creating initial stub';
GO
CREATE VIEW dbo.AuthorsBooks
AS
SELECT 'A code stub, which will be replaced by an ALTER Statement'
AS [stub]
GO
SET NOEXEC OFF
-- the ALTER VIEW batch will always run
GO
PRINT 'dbo.AuthorsBooks: view code is updated';
GO
ALTER VIEW AuthorsBooks
AS
SELECT ( a.FirstName + ' ' + a.MiddleName + ' ' + a.LastName )
AS FullName ,
b.Title
FROM Author a
INNER JOIN AuthorBook ab ON a.AuthorID = ab.AuthorID
INNER JOIN Book b ON ab.BookID = b.BookID;
GO
PRINT 'dbo.AuthorsBooks: view creation script finished';
Listing 4-1: Creating the AuthorsBooks view in the Bookstore database.](https://image.slidesharecdn.com/sql-server-source-control-basics-04-06-14-v2-140904022337-phpapp02/85/SQL-Server-Source-Control-Basics-127-320.jpg)
![Chapter 4: Editing and Versioning
As a result of the modification, Fred's working folder now contains changes that we need
to commit to the repository, as shown in Figure 4-2.
Figure 4-2: Edited file waiting to be committed to the repository.
Finally, right-click on BookstoreDatabases and commit the change (be sure to add a
comment to the Commit dialog box).
129
Update the test data
We may also need to generate some test data for the new YearOfPublication column.
UPDATE [dbo].[Book]
SET [YearOfPublication] = 1884
WHERE BookID = 1;
GO
UPDATE [dbo].[Book]
SET [YearOfPublication] = 1969
WHERE BookID = 2;
GO](https://image.slidesharecdn.com/sql-server-source-control-basics-04-06-14-v2-140904022337-phpapp02/85/SQL-Server-Source-Control-Basics-129-320.jpg)
![Chapter 4: Editing and Versioning
130
UPDATE [dbo].[Book]
SET [YearOfPublication] = 1987
WHERE BookID = 3;
GO
UPDATE [dbo].[Book]
SET [YearOfPublication] = 1813
WHERE BookID = 4;
GO
UPDATE [dbo].[Book]
SET [YearOfPublication] = 1963
WHERE BookID = 5;
GO
UPDATE [dbo].[Book]
SET [YearOfPublication] = 1818
WHERE BookID = 6;
GO
UPDATE [dbo].[Book]
SET [YearOfPublication] = 2001
WHERE BookID = 7;
GO
UPDATE [dbo].[Book]
SET [YearOfPublication] = 1944
WHERE BookID = 8;
GO
UPDATE [dbo].[Book]
SET [YearOfPublication] = 1920
WHERE BookID = 9;
GO
UPDATE [dbo].[Book]
SET [YearOfPublication] = 1977
WHERE BookID = 10;
GO
Listing 4-3: Adding new test data to the Book table.
Again, we have the option to save Listing 4-3 as a new upgrade script, or we can modify
the existing TestData.sql file, which is the option we'll choose here. In order to do this,
first run the code in Listing 4-3 to add the new data to your development database, then
follow the steps below.](https://image.slidesharecdn.com/sql-server-source-control-basics-04-06-14-v2-140904022337-phpapp02/85/SQL-Server-Source-Control-Basics-130-320.jpg)
![Chapter 6: Merging
If Barb clicks Update, SVN will bring her working folder up to Revision 16 and attempt to
automatically merge the Revision 16 version of the file in the repository with her working
copy of the file. If it can't do so safely, SVN will request manual help with the merge. If
the change was complex, Barb might choose to cancel at this stage so as not to risk losing
those changes, save them off somewhere else first, as described earlier, and then perform
an update.
Here, however, Barb simply clicks Update. In this case, the changes she made do not
conflict with those of Fred, so SVN performs an auto-merge and invites Barb to commit
the new auto-merged version of the file, which contains both her new Title column and
the nullable MiddleName column. The resulting commit is Revision 17 in the repository.
Remember that Fred will still have Revision 16 of the Author.sql file until he performs
an update of his working folder.
Things get more interesting, however, when our developers' changes overlap.
Manual merge (dealing with conflicts)
Once again, both Fred and Barb update their respective working folders so that they both
have the latest revision (now Revision 17). Fred adds a new Type column to the Book
table and commits the change. Listing 6-1 shows the change that Fred makes to the Book.
sql file, with his added code highlighted.
187
CREATE TABLE dbo.Book
(
BookID INT IDENTITY
PRIMARY KEY ,
Title NVARCHAR(1000) NOT NULL,
YearOfPublication INT NULL ,
[Type] CHAR(12) NULL
);
GO
Listing 6-1: Adding the Type column to the Book table.](https://image.slidesharecdn.com/sql-server-source-control-basics-04-06-14-v2-140904022337-phpapp02/85/SQL-Server-Source-Control-Basics-187-320.jpg)
![Chapter 6: Merging
In addition, SVN marked Barb's working copy of the Book.sql file with a yellow
exclamation point, indicating that the file is in a "conflicted" state.
If Barb opens the file in a text editor, she'll find the content displayed in a "unified diff"
format (see Chapter 5), similar to what is shown in Listing 6-2.
191
CREATE TABLE dbo.Book
(
BookID INT IDENTITY
PRIMARY KEY ,
Title NVARCHAR(1000) NOT NULL,
YearOfPublication INT NULL,
<<<<<<< .mine
[Type] VARCHAR(15) NULL
=======
BookCategory CHAR(12) NULL
>>>>>>> .r19
);
GO
Listing 6-2: A unified diff of the Book.sql file.
Barb does not work with any of these files directly. Instead, she closes the Book.sql file
and launches a merge tool, in this case TortoiseMerge, to help her perform the merge
operation. Her goal is to come up with a new "merged" version of the Book.sql file in
her working folder that reflects her change or Fred's, or both, and which she can commit
to the repository, as shown in Figure 6-8.](https://image.slidesharecdn.com/sql-server-source-control-basics-04-06-14-v2-140904022337-phpapp02/85/SQL-Server-Source-Control-Basics-191-320.jpg)
![Chapter 7: Manually Deploying Databases from Source Control
Create a new tag for the v2.0.1 build
Barb's first step is to create a new tag that identifies all the files that comprise
the v2.0.1 build:
1. Right-click the trunk folder, navigate TortoiseSVN | Branch/Tag.
2. Name the new tag v2.0.1 (in To path: enter /tags/v2.0.1).
3. Create the tag from the HEAD revision in the repository.
Again, Barb must update the working folder to retrieve the new tag from the repository.
Creating a Master upgrade script
We'll again create a master PowerShell script for upgrading the database. The script will
target the original build (2.0.0) with updates for the new build (2.0.1). In other words, the
script is developed specifically to upgrade 2.0.0 to 2.0.1.
Listing 7-4 shows the master PowerShell upgrade script.
# define variables
# modify server instance and folder paths as necessary
$instance = "localhostSqlSrv2012"
$db = "Bookstore"
$date = Get-Date -Format yyyyMMddHHmmss
$file = "C:DataFilesBackups$($db)_$($date).bak"
$folder = "C:SvnBarbBookstoreProjecttagsv2.0.1DatabasesBookstore
ChangeScripts
252
ChangeScripts_2.0.1"
[string]$LastBuild = 200
# variable to verify existence of database
$id = Invoke-SqlCmd `
-ServerInstance $instance `
-Database "master" `
-Query "SELECT database_id FROM sys.databases WHERE name = '$db'"](https://image.slidesharecdn.com/sql-server-source-control-basics-04-06-14-v2-140904022337-phpapp02/85/SQL-Server-Source-Control-Basics-252-320.jpg)
![Chapter 8: Automating Database Versioning and Deployment from Source Control
Therefore, we're going to generate the migrations script that SQL Compare will incor-porate
into the transition script at deployment time. Unfortunately, at time of writing,
SQL Source Control's migrations tooling only supports "up" (i.e. "apply this") scripts, and
not "down" (i.e. "undo this") scripts.
In SSMS, select Fred's FredBookstore database and then hop to the Migrations tab in
SQL Source Control, choose Add Migration Script, and paste in the script in Listing 8-3.
291
DECLARE @ShouldRunMigrationScript BIT
SET @ShouldRunMigrationScript = 1
IF NOT EXISTS ( SELECT 1
FROM Information_Schema.Columns
WHERE table_schema = 'dbo'
AND TABLE_NAME = 'Author'
AND COLUMN_NAME = 'Country' )
BEGIN
SET @ShouldRunMigrationScript = 0;
PRINT 'Column [Country] in [dbo].[Author] could not be found -
skipping migration.';
END
IF @ShouldRunMigrationScript = 1
BEGIN
UPDATE [dbo].[Author]
SET [Country] = 'United States'
WHERE [Country] IS NULL
ALTER TABLE [dbo].[Author]
ALTER COLUMN [Country] NVARCHAR(100) NOT NULL
END
Listing 8-3: The migration script.
Give it a descriptive name, such as Set Author.Country Not Null, along with a
description, and click Save and Close. Run the migration script on the FredBookstore
database and then switch over to the Commit Changes tab.](https://image.slidesharecdn.com/sql-server-source-control-basics-04-06-14-v2-140904022337-phpapp02/85/SQL-Server-Source-Control-Basics-291-320.jpg)
SQL Server Source Control Basics
- 1. SQL Server Source Control Basics Robert Sheldon, Rob Richardson & Tony Davis ISBN: 978-1-906434-79-3
- 2. SQL Server Source Control Basics Robert Sheldon, Rob Richardson & Tony Davis First published by Simple Talk Publishing, 2014
- 3. Copyright Robert Sheldon, Rob Richardson & Tony Davis 2014 ISBN: 978-1-910035-00-9 The right of Robert Sheldon, Rob Richardson and Tony Davis to be identified as the authors of this book has been asserted by them in accordance with the Copyright, Designs and Patents Act 1988. All rights reserved. No part of this publication may be reproduced, stored or introduced into a retrieval system, or transmitted, in any form, or by any means (electronic, mechanical, photocopying, recording or otherwise) without the prior written consent of the publisher. Any person who does any unauthor-ized act in relation to this publication may be liable to criminal prosecution and civil claims for damages. This book is sold subject to the condition that it shall not, by way of trade or otherwise, be lent, re-sold, hired out, or otherwise circulated without the publisher's prior consent in any form other than which it is published and without a similar condition including this condition being imposed on the subsequent publisher. Technical Review: Rob Richardson and Tony Davis Editor: Tony Davis Cover Image: Andy Martin Typeset: Peter Woodhouse and Gower Associates
- 4. Table of Contents Foreword: A Brief History of Source Control______________________________viii Introduction_ _________________________________________________________ 13 The Hands-on Examples_ _______________________________________________15 Chapter 1: Core Source Control Concepts_______________________17 Why Source Control?___________________________________________________17 The Source Control Repository__________________________________________ 20 The Working Folder____________________________________________________22 Versioning and Collaborative Editing____________________________________ 24 How versioning works________________________________________________25 Collaborative editing_________________________________________________28 Branching_ ___________________________________________________________36 Merging______________________________________________________________39 Summary_____________________________________________________________43 Chapter 2: The Source Control Repository_ ____________________ 44 Centralized Versus Distributed Repositories________________________________45 Centralized repositories_ ____________________________________________ 46 Distributed repositories_______________________________________________48 Comparing centralized and distributed repositories________________________52 Choosing a VCS for Database Development_ _______________________________56 Versioning a Project in the Repository_____________________________________58 Getting Started with Source Control______________________________________59 Setting up Subversion_______________________________________________ 60 Setting up the repository______________________________________________62 Setting up the working folders_________________________________________65 We've Only Just Begun__________________________________________________77
- 5. Chapter 3: Getting Started with Source Control for Databases____ 78 Planning for Database Source Control____________________________________ 79 Integrated Development Environment (IDE) integration____________________79 Disconnected versus online database development_______________________ 80 Shared versus dedicated database development_ __________________________81 Source control and Application Lifecycle Management_ ____________________82 Database build and deploy considerations________________________________82 Database versioning challenges_ _______________________________________84 Database source control architecture___________________________________ 86 Getting the Bookstore Database into Source Control_ ______________________ 90 Script database objects for source control_ _______________________________92 Create, Add and Commit database scripts________________________________93 Updating the working folder__________________________________________104 Summary____________________________________________________________108 Chapter 4: Editing and Versioning_ __________________________ 109 Database Versioning Strategies__________________________________________110 Versioning code modules_ ___________________________________________ 110 Versioning tables___________________________________________________ 111 Source Control Versioning Mechanisms_ _________________________________119 Version numbering and history_ ______________________________________ 119 Deltas and diffs_ ___________________________________________________ 121 Modifying the Bookstore Database_______________________________________126 Investigating Bookstore Revision History_________________________________ 132 Diffing files in Bookstore_____________________________________________ 136 What to Do When Things Go Wrong: Revert_ _____________________________144 The World of Source Control_ __________________________________________148 Chapter 5: Branches and Tags_______________________________ 149 How Branching Works_________________________________________________150
- 6. Branching Strategies___________________________________________________ 152 No branching______________________________________________________ 153 Branch when needed (release branches)_________________________________ 153 Feature branches___________________________________________________ 157 Branching strategy for database development____________________________ 158 Tags_________________________________________________________________160 Branching in SVN_____________________________________________________161 Create the branch in the repository____________________________________ 163 Update the working folders___________________________________________ 165 Switching the working folder_________________________________________166 Modifying the branch_ ______________________________________________167 Diffing between branches and the trunk_ _______________________________168 Creating Tags in SVN__________________________________________________174 Summary____________________________________________________________176 Chapter 6: Merging_________________________________________177 How to Avoid Painful Merges_ __________________________________________178 How Merging Works___________________________________________________179 Intra-branch merges: conflicting changes on the same branch_______________179 Inter-branch merges: moving changes between branches___________________182 How to Merge in SVN__________________________________________________184 Intra-branch merges in SVN__________________________________________184 Inter-branch merging in SVN_________________________________________ 195 Summary____________________________________________________________219 Chapter 7: Manually Deploying Databases from Source Control_ _ 220 Database Deployment Pitfalls___________________________________________221 Version drift_______________________________________________________222 Lack of upgrade script testing_________________________________________223 No rollback plan____________________________________________________226
- 7. Building and Deploying Databases_______________________________________227 Database build numbers_____________________________________________230 Deploying a new database build_______________________________________ 233 Upgrading an existing database_ ______________________________________236 Deploying a New Build of the Bookstore Database__________________________241 Saving database backup files__________________________________________241 Create the build table_ ______________________________________________241 Create a tag for the latest Bookstore revision_____________________________242 Create the build script_______________________________________________243 Run the new build__________________________________________________246 Upgrading the Bookstore Database_ ____________________________________ 248 Preparing the environment___________________________________________248 Creating the upgrade scripts__________________________________________249 Preparing to deploy_________________________________________________ 251 Deploy the upgrade_________________________________________________ 255 Automating Deployments for Upgrade Script Versioning____________________256 Source Control and Database Deployment________________________________256 Chapter 8: Automating Database Versioning and Deployment from Source Control_ __________________________________________ 258 Versioning the Database________________________________________________259 CREATE scripts versus upgrade scripts_ ________________________________259 Deployment challenges______________________________________________261 Choosing a versioning approach_______________________________________262 Automation for CREATE Script Versioning and Deployment_________________265 Automated Versioning and Deployment for Bookstore_____________________ 269 Versioning databases with SQL Source Control___________________________270 Versioning the existing Bookstore database______________________________274 Modifying Bookstore________________________________________________282 Deploying databases with SQL Compare________________________________285 Database Versioning and Deployment with Continuous Integration___________293 Summary____________________________________________________________295
- 8. Foreword: A Brief History of Source Control viii By Phil Factor The fledgling computer industry did not invent source control. It merely adopted practices that were already well established in the industrial manufacturing and design industries. The design and manufacture of large, complex machines such as aircraft, tanks or motor cars required a great deal of discipline, particularly in the technical drawing of components. The engineering process involved identifying the assemblies and components in a design, and representing them in a hierarchical map. Each component was defined by its technical drawing, and the "spec" which specified the materials and manufacturing techniques. Having defined the context of a component within the overall design, for example the dimensions within which the component had to fit, and the tolerances within which it had to work, a team of engineers could work on each one relatively independently. Over the course of the engineering process, they would construct and test the various assemblies, and change the drawings and specifications as required. Each version of every component had a version number. At this stage, or perhaps retrospectively, variants would be designed. For an estate version of an automobile, for example, a stronger back axle and suspension might be required. However, any drastic changes to the design, such as a change in width, which affected all variants, would require a new version of all the affected components. At the stage of testing the prototypes, some changes would have a knock-on effect, requiring changes to other components. The drawing office and project office evolved a robust system for dealing with all these processes; thus were born the fledgling change control and releasing systems, and much of the software engineering process and terminology still shows signs of these roots. Source control for software mimicked the established techniques of the drawing office, but only the parts that were possible within the crude technology of the time. Back in the days of punch cards, once the programmers had finished with their routine, they would
- 9. take the punched cards, change cards, or coding forms to the source-control or config-uration- management team who would duly update the library, and build the customer-ready product from the official source. At the same time, there was an office process that recorded the work and managed the "versioning" task. The work was treated as if it were a technical drawing. To alter source, you checked the code in and out as though it were a technical drawing that had to be altered. Evolution and refinement of the process proved to be slow. As soon as programs became text files, there was an automation of the manual process, using programs such as Source Code Control System (SCCS). At this stage, there wasn't the concept of storing just the difference between file versions. The technology was there but we weren't culturally ready. This practice of delta-storage became mainstream in the 1980s with Revision Control System (RCS), but the whole process still used the analogy of the file being the unit of work, as if it were a technical drawing. The idea of being able to deal with a whole lot of files at once took a long time to catch on. Also absent for a long time was the idea of having more than one person working on a file at the same time. Although Concurrent Versions Systems (CVS) was the first to break free from the idea of locking out the file to work on it, it was unreliable until the introduction of Subversion. Even now, it is only too easy for the source control system to fail to sort out the merging of two simultaneous alterations of a source file, and "merge failure" is still a phrase to be dreaded by most developers. The original source control software was mainframe-based, and individual programmers accessed the system via a terminal. UNIX systems were the first to introduce server-based, or "centralized" source control systems that relied on a single, shared repository, and these eventually became available on MSDOS and Windows. Whilst these worked well when a team was co-located on the same file-sharing server, they were useless where some of the team were outside the domain. In the mid-nineties, source control became network-based, but still ultimately hosted on a server. From this point, the idea of a hosted service on the Internet was just a short step. ix
- 10. More fundamental was the change to a distributed model for source control, in which no single computer held the master copy. The distributed model became mainstream with BitKeeper, followed by Git and Mercurial, and it made the process of forking, branching and merging far easier and more reliable. Source control has always been an intrinsic part of software development. The recent advances in source control have led to a fundamental change to the way we develop software. It has made possible the current surge of collaborative open-source software and made the rapid delivery of software achievable. It has revolutionized team-working in the past decade. If we've seen rapid improvements in source control over the past decade, I suspect we'll see even more, since the recent advances have been fuelled by frustration with the existing system and that frustration still remains, in large part manifest in the pain inherent in many merge operations. The problem with merging boils down to the fact that source control systems still stick to the idea of the text file as the unit of work, just as if it were the generic technical drawing or engineering specification. Comparing the textual differences between source files isn't always a good way of merging, or of detecting the repercussions of a change. Without the ability to perform any a semantic analysis of the source file, it cannot, for example, judge whether a change is a code logic change, or merely the addition of comment blocks. I long for the day of a generic version-control system that understands any language once I've added the appropriate parser as a plug-in. In the meantime, in order to achieve rapid delivery of software, some teams avoid the pain of the merge by avoiding branches altogether, and keeping all the development work in the trunk, using software "flags" to disable the functionality of the components that are not ready for release. However complex a software system may be, there is no "opt-out" alternative to source control. A database, for example, can be of startling complexity but everything can be scripted, even if built step-wise, using a GUI. Whatever the system, the ultimate source of that system, and the script to build it, is what lies in the script repository. Despite the occasional frustrations, the benefits far outweigh the effort. Any team development, even a one-person team, needs source control. x
- 11. About the Authors and Technical Reviewers Robert Sheldon After being dropped 35 feet from a helicopter and spending the next year recovering, Robert Sheldon left the Colorado Rockies and emergency rescue work to pursue safer and less painful interests – thus his entry into the world of technology. He has worked as a technical consultant and written numerous books, articles, and training material related to Microsoft Windows, various relational database management systems, and business intelligence design and implementation. He has also written news stories, feature articles, restaurant reviews, the novel, Dancing the River Lightly, the publishing guide Ebook Now, and six books in the 5-Spot ebook travel series. You can find more information at http://rhsheldon.com. Bob contributed Chapters 1–7 to this book. xi Rob Richardson Rob Richardson is a software craftsman from Phoenix, Arizona building web properties in ASP.NET and Node. Rob specializes in translating business requirements into technical solutions for small- to medium-sized businesses. He has created software applications ranging from enterprise-scale applications to PDA-based systems, web applications to embedded database synchronization and postscript processing systems. He's a frequent speaker at conferences, user groups, and community events, and a diligent teacher and student of high quality software development. You can find recent talks on his blog at http://robrich.org/ and follow him on Twitter at @rob_rich. Rob contributed Chapter 8 to this book, plus additional material elsewhere, and was the technical reviewer for Chapters 1–7.
- 12. xii Tony Davis Tony is an editor with Red Gate Software, based in Cambridge (UK), specializing in databases, and especially SQL Server. He edits articles and writes editorials for both the Simple-talk.com and SQLServerCentral.com websites and newsletters, with a combined audience of over 1.5 million subscribers. You can sample his short-form writing at either his Simple-Talk.com blog at http://tinyurl.com/nt2wnab his author pages at http://tinyurl.com/otq9rwf, or at his SQLServerCentral.com author page at http://tinyurl.com/ms3d6dd. As the editor behind most of the SQL Server books published by Red Gate, he spends much of his time helping others express what they know about SQL Server. He is also the lead author of the book, SQL Server Transaction Log Management for which, see http://tinyurl.com/o89ocxl. In his spare time, he enjoys running, football, contemporary fiction and real ale. Tony contributed additional material throughout the book, and was the technical reviewer for Chapter 8.
- 13. 13 Introduction For efficient team-based database development, and reliable and repeatable database deployments, source control is not optional. The job of a source control system is to maintain a change history of all the files in a project. As soon as we enter a new file into source control, the system assigns it a version. Each time we commit a change to that file, the version increments, and we have access to the current version and all previous versions of the file. For good reason, we often refer to a source control system as a version control system, or VCS. Few software developers would consider building an application without the benefits that a VCS offers, but its adoption for databases has been slower, partly because of the added complication of the need to preserve data during database upgrades. However, unless we have in the VCS the correct versions of all the scripts necessary to create our database objects, load lookup data, add security accounts, and take any other necessary actions, we have no hope of a reliable and repeatable database deployment process, let alone of coordinating database upgrades with changes to the associated application. We also run a very real risk that our "ad hoc" database patching will cause inconsistencies and data loss. Version control can and should play a key role in the database development and deployment process, and this book will show you exactly how to get started. It provides "just enough" detail about the core components of a source control system, and the operations that allow us to commit our database changes and retrieve others' changes. It then focuses on getting a database into source control and working with it, covering: • Database source control architecture – what needs to be in there, how to structure it all. • Collaborative editing – working as a team on a database project, while minimizing the potential for conflicting changes and data loss.
- 14. • Change auditing – what changed between one version and another, and who changed it? • Branching – allows teams to work independently on separate features, and control what they deploy and when. • Merging – what happens if one user changes the name of a column while another updates its data type? A merge operation lets the team decide the correct outcome. • Building and deploying databases – building new databases and upgrading existing databases from source control, both manually and using automation tools. Every chapter follows the same "half theory, half practice" template, so you learn the concepts and then immediately see how they work in a particular source control system, in this case, Subversion (SVN). It's important to state up front that this is not a book about Subversion. It will explain conceptually the purpose and function of some of the primary source control opera-tions, and familiarize you with their implementation in SVN, but it is not a reference for SVN commands. We focus on the advantages and challenges of working with databases in source control, and on what's required for efficient, repeatable database deployments, from source control. 14 Intended Audience The primary audience for the book is the database developer, though it is relevant to any developer, data architect, DBA, or anyone else who needs to work with or deploy databases from a version control system. This is an entry-level book, suitable for anyone not currently using a version control system (VCS), or using it for the application only, and not the database. It does not assume any prior knowledge of source control operations, although some familiarity would be an asset. It assumes basic technical competency with SQL Server.
- 15. 15 Required Tools For the majority of this book (Chapters 1–7), you will need only access to: • A SQL Server installation – even if it is the evaluation edition available free from Microsoft. We tested the code examples on SQL Server 2008 and SQL Server 2012, though most should work on later editions, as well as on SQL Server 2005. • A source control management system – we used VisualSVN Server standard edition, which is free (see Chapter 2 for details). • A native source control client – we used TortoiseSVN, again freely available (see Chapter 2 for details). If you wish to work through Chapter 8, you will also need access to three Red Gate Software tools (or suitable alternatives): SQL Source Control, SQL Compare and SQL Data Compare. You can download free trial versions of each of these tools (see Chapter 8 for details). The Hands-on Examples The practical sections of each chapter work progressively through an example of getting a database for a fictional Bookstore application into source control and working with it. The book provides various scripts, shown as listings, to create the Bookstore database, and various database objects, and then progressively modify them while tracking changes through the VCS. You can download all of these scripts from: http://tinyurl.com/lum8ryl.
- 16. The example builds sequentially through each chapter, but to make it easier to jump in at any point, we also provide a SVNBackups folder, containing a backup repository as it existed at the end of a given chapter, which you can restore to recreate the repository. To do this: 1. In VisualSVN Server Manager, create a new empty repository (not with the trunk/ branch/tag structure) e.g. called MyBookstoreProject. 2. At the SVN Server level, select Action | All Tasks | Start Command Prompt. 3. Restore the appropriate repository backup. E.g. to restore the repository as it existed at the end of Chapter 2: svnadmin load D:RepositoriesMyBookstoreProject < D:SVNBackups BookstoreProject_Ch02.bak. 4. Check out the BookstoreProject to a new working folder, using the SVN Checkout command (described in detail in Chapter 2). 16
- 17. Chapter 1: Core Source Control Concepts This opening chapter takes a conceptual, "big picture" approach to explaining core source control concepts, using diagrams and anecdotes as necessary to describe how all the pieces fit together. • Why source control – the basic purpose of source control and the benefits it offers, including: • maintaining an "audit trail" of what changed, who made the change, and when. • allowing team collaboration during development projects. • Basic source control components and concepts, including: • the repository, which stores the files and maintains their histories. • the working folder, which provides an isolated environment (sandbox) for creating, editing, and deleting files without affecting those in the repository. • workflow concepts, such as versioning, branching and merging files. 17 Why Source Control? A source control solution provides users with the tools they need to manage, share, and preserve their electronic files. It does so in a manner that helps minimize the potential for conflicting changes and data loss, in other words one user inadvertently overwriting another user's changes when multiple users work on the same files. If one user changes the name of a column while another one updates its data type, the source control system will alert us to the conflict and allow the team to decide the correct outcome, via a merge operation.
- 18. Chapter 1: Core Source Control Concepts Critically, a source control solution maintains a version history of the changes made to every file, over time, and provides a means for users to explore those changes and compare different file versions. This is why we often refer to a source control system as a version control system, or VCS for short. These days, few software developers would consider building an application without the benefits that a VCS offers, but its adoption for databases has been slower. This is mainly because of the nature of a database, which must preserve its "state" (i.e. business data) between database versions. It means that having, in source control, the files that define the schema objects and other code is not the whole story. When upgrading a "live" database to a newer version, as it exists in the VCS, we can't just tear down objects that store data and re-create them each time. Nevertheless, despite this added complication, there is no reason why we should exclude databases from our source control practices. In fact, a VCS can be one of a database devel-oper's most valuable tools and the foundation stone for an effective and comprehensive 18 change management strategy. A Brief History of Source Control If you skipped straight to Chapter 1, you missed Phil Factor's trawl through the recent history of source control, and its roots in the mechanisms used to control the process of designing and manufacturing large, complex machines such as aircraft and tanks. I recommend you go back and read his Foreword at some point. From here in, we focus on the use of source control to store and manage electronic files, and database files in particular. At its heart, the purpose of a VCS is to maintain a change history of our files. As soon as we enter a new file into source control, the system assigns it a version. Each time we commit a change to that file, the version increments, and we have access to the current version and all previous versions of the file.
- 19. Chapter 1: Core Source Control Concepts This versioning mechanism, which we discuss in more detail in Chapter 3, serves two core purposes: • Change auditing • Compare versions, find out exactly what changed between one version and another. • Find out who made the change, and when; for example, find out when someone 19 introduced a bug, and fix it quickly. • Team collaboration • Inspect the source control repository to find out what other team members have recently changed. • Share recent changes. • Coordinate work to minimize the potential for conflicting changes being made to the same file. • Resolve such conflicts when they occur (a process called merging). By maintaining every version of a file, we can access the file as it existed at any revision in the repository, or we can roll back to a previous version. Source control systems also allow us to copy a particular file version (or set of files) and use and amend them for a different purpose, without affecting the original files. This is referred to as creating a branch, or fork. I hope this gives you a sense of the benefits a source control system offers. We'll look at more as we progress. The following sections paint an initial picture of the source control components and workflow that enable this functionality. We review the most important concepts in terms of the content creation, the storage, and the tracking strategies they enable, but we won't go into detail, because we'll cover these topics in depth from Chapter 2 onwards.
- 20. Chapter 1: Core Source Control Concepts The Source Control Repository At the heart of a VCS is the repository, which stores and manages the files, and preserves file change histories. Centralized source control systems support a single, central repository that sits on a server, and all approved users access it over a network. In distributed source control systems, each user has a private, local repository, as well as (optionally) a "master" repos-itory, accessed by all users. We'll discuss both centralized and distributed repositories in more detail in Chapter 2, but assume a centralized model for the conceptual examples in this chapter. Regardless of the model used, when we add files to the repository, those files become part of a system that tracks who has worked on the file, what changes they made, and any other metadata necessary to identify and manage the file. Repository storage mechanisms The exact storage mechanism varies by source control system. Some products store both the repository's content and its metadata in a database; some store all content and metadata in files (with the metadata often stored in hidden files); other products take a hybrid approach and store the metadata in a database, and the content within files. The repository organizes files and the metadata associated with each file, in a way that mirrors the operating system's folder hierarchies. In essence, the structure of files and folders in a source control system is the same as in a typical file management system such as Windows Explorer or Mac OS Finder. In fact, some source control systems leverage the local file management system in order to present the data in the repository. Figure 1-1 shows the server repository structure for a BookstoreProject repository, typical of the one we'll see throughout this book, with the Databases folder expanded to reveal a Bookstore database. Don't worry about the details of this structure yet, as we'll get to them later. 20
- 21. Chapter 1: Core Source Control Concepts Figure 1-1: Typical hierarchical folder structure in the repository. What sets a source control repository apart from other file storage systems is its ability to maintain file histories. Everything we save to a repository is there for ever, at least in theory. From the point that we first add files to the repository, the system maintains every version of every file, recording every change to those files, as well as to the folders that form the repository structure. Source control of non-text files Ideally, a VCS manages and tracks the changes made by all contributors to every type of file in the system, whether that file is a Word document, Excel spreadsheet, C# source code, or database script file. In reality, however, traditional source control solutions usually track changes only on text files, such as those used in application and database development, and tend to treat binary files, such as Word or Photoshop files, as second-class citizens. Even so, most solutions maintain the integrity of all files and help manage processes such as access control and file backup. 21
- 22. Chapter 1: Core Source Control Concepts 22 The Working Folder Most users care less about how their source control system stores the file content and metadata, and more about being able to access and work on those files. Each repository includes a mechanism for maintaining the integrity of the files within their assigned folder structure and for making those files accessible to authorized users. However, to be able to edit those files, each user needs a "private workspace," a place on his or her local system, separate from the repository, to add, modify, or delete files, without affecting the integrity of the files preserved in the repository. Figure 1-2: Typical working folder structure.
- 23. Chapter 1: Core Source Control Concepts Most VCSs implement this private workspace through the working folder. The working folder is simply a folder, and set of subfolders, on the client computer's file system, registered to the source control repository and structured identically to the folders in the repository. Figure 1-2 shows the working folder structure for a user of the BookstoreProject repository. This user has copied the entire repository to a working folder called BookstoreProject. Each user stores in their working folder copies of some or all of the files in the repository, along with the metadata necessary for the files to participate in the source control system. As noted above, that metadata is often stored in hidden files. We can update our working folder with the latest version of the files stored in the repository, as illustrated in Figure 1-3. Figure 1-3: Copying files from the repository to the working folder. 23
- 24. Chapter 1: Core Source Control Concepts We can edit the files in the working folder as necessary and then, eventually, commit the edited versions back to the repository. This process of "synchronizing" the working folder with the repository, i.e. updating the local working folder with any changes in the repos-itory and committing any local changes to the repository, works differently from product to product and depends on whether the repository is centralized or distributed. Regardless of repository type or product, the source control system always keeps the files in the repository separate from changes made to files in the working folder, until the user chooses to commit those changes to the repository. Versioning and Collaborative Editing The exact architecture and mechanisms that underpin versioning and collaborative editing vary by VCS, but the basic principles are constant. A user can obtain a local "working copy" of any file in the repository, make additions and amendments to that file, and then commit those changes back to the main repository. At that point, other users can request from the repository the amended version of the file or any of the previous versions. The VCS maintains a full change history for each file, so we can work out exactly what changed from one version to another. In this section we'll discuss, at a high level, how the repository maintains these file versions, as users make progressive changes to those files. Chapter 4 will discuss versioning in much more detail, along with the specifics of how it works in various source control systems. Notionally, this versioning process is easy in a single-user system. A user updates his or her working folder with the latest versions of a set of files, and then edits those files as appropriate in a suitable client, such as Notepad for a plain text file, Visual Studio for an application file, or SQL Server Management Studio for a database file. The user then commits the changes to the repository, creating new versions of the files. 24
- 25. Chapter 1: Core Source Control Concepts However, another key function of source control is to enable a group of people to work collaboratively on the set of files that comprise a development project. In other words, the VCS must allow multiple users to modify a file concurrently, while minimizing the potential for conflicts and data loss. Here, we'll take a first look at how source control systems allow for and manage these concurrent changes, but we'll discuss this topic in more depth, and with worked examples, from Chapter 3 onwards. 25 How versioning works Let's assume we've created a project directory in source control for an Animal-Vegetable- Mineral (AVM) application and that we've established a working folder for this project. Figure 1-4 depicts the progressive changes to the application over three revisions. Notice that the repository preserves all the changes to the files, with each version assigned a revision number. Note that Figure 1-4 is not in any way a depiction of how a VCS maintains different file versions internally. It is merely to help visualize the process of how it can allow us to access different file versions, and provide a history of changes to our files over time. In Revision 1, we committed to the repository (from our working folder) two new files, Animals.txt and Vegetables.txt. Revision 1 represents the first and latest file versions in the AVM repository. We edited Animals.txt to replace skunk with elk, and created a third file called Minerals.txt, and committed the changes to the repository. Collectively, these changes form Revision 2. Vegetables.txt remains unchanged from Revision 1. Next, we edited Vegetables.txt, changing sprouts to carrots, and edited Minerals. txt, changing potash to pyrite and adding silica. These changes form Revision 3, with Animals.txt unaltered from Revision 2.
- 26. Chapter 1: Core Source Control Concepts Mostly, users are interested in working with the latest folder and file versions in the repository, but we can also request to see the repository as it existed at any earlier revision, with each file exactly as it existed at that point in time. For example, if we were to pull Revision 2, we would get the Revision 2 copies of the Animals.txt and Minerals.txt files, as well as the Revision 1 version of the Vegetables.txt file. In this way, we can build and deploy a specific "version" of the application or database. Figure 1-4: Working with files from the AVM project in source control. 26
- 27. Chapter 1: Core Source Control Concepts Conceptual versus actual source control implementation At this stage in the book, the descriptions of the versioning process are conceptual in nature. An actual source control implementation will vary according to product. It might not store this many different file versions, or it might build a given version by storing a record of the differences (the delta) between a current version and the previous version. We'll discuss this fully in Chapter 4. Since developers usually want to ensure they are working with all the most recent versions of their project files, they update their working folders regularly to get the latest versions. When they view their working folders, they are viewing the latest version of the repository and all its folders and files, at the point in time they did their last update. However, we can also request to view the revision history for the repository (for example, by accessing the repository's log – more on this in Chapter 3). Exactly what we see when viewing the revision history depends on the VCS, but it's likely to include information such as the revision number, the action, who made the changes, and when, and the author's comment, i.e. a description of the change. Figure 1-5 on the next page shows what the log might look like for the AVM project folder, after the sequence of changes (and assuming two users, Fred and Barb). A VCS, as noted earlier, often stores the differences between each file version, rather than the full file for every version. We refer to each set of differences as the delta. If we request to view a file as it existed at a particular revision, the VCS might, for example, retrieve the last stored complete file and then apply the deltas in the correct order going forward. Likewise, a VCS will usually provide an easy visual way for users to see a list of what changed between any two revisions in the repository. We call this performing a diff, short for "difference between revisions." We'll cover deltas and diffs in more detail in Chapter 4. 27
- 28. Chapter 1: Core Source Control Concepts Figure 1-5: Storing files in the repository of a source control system. 28 Collaborative editing Any source control solution must provide the structure necessary to permit collaboration on different file versions in the repository, while preserving every version of each file. The potential difficulty arises when more than one user works on the same version of a file at the same time. Let's say both Fred and Barb have in their respective working folders Revision 2 of our AVM app. Fred edits Vegetables.txt, changing sprouts to carrots and commits the change. At roughly the same time, Barb edits the same file, changing sprouts to peas, and commits the change. What should be the result in the source control repos-itory? If the "last commit wins," we'd simply lose Fred's changes from the current version of the file.
- 29. Chapter 1: Core Source Control Concepts Older source control systems (often referred to as "first generation") get around such difficulties by imposing an isolated editing (or locking) model, whereby only one user at a time can work on a particular version of a file. Most modern source control systems enable a group of users to work collaboratively on the same version of a file. Referred to as a concurrent editing model, this process allows them to reconcile, or merge, the changes made by more than one user to the same file and, in the process, resolve any conflicting changes. 29 Isolated editing A traditional "first-generation" source control solution, such as Source Code Control System (SCCS), developed at Bell Labs in 1972, uses a central repository and a locking model that permits only one person at a time to work on a file. To use a database metaphor, we can liken the isolated editing model to SQL Server's pessimistic concurrency model. It assumes, pessimistically, that a conflict is likely if multiple users are "competing" to modify the same file, so it takes locks to prevent it happening. A typical workflow in source control might looks as follows: 1. Fred performs a check-out of the latest version of the Animals.txt file, in this case, Revision 1. a. If the file does not exist in Fred's working folder, the source control system will copy it over. b. The source control system "locks" the file in the repository. (The exact "locking" mechanism varies by system.) 2. Fred edits the file in his editor of choice. He deletes skunk and adds elk. 3. Fred saves the changes to his local working folder.
- 30. Chapter 1: Core Source Control Concepts 4. Barb attempts to check out Animals.txt, but cannot because Fred has it locked. Although Fred saved his changes locally, he has yet to perform a check-in to the repository and so the file remains locked and no one else can check it out. 5. However, Barb can download Animals.txt to her working folder as a read-only copy, so she can at least see the latest version as it exists in the repository. Figure 1-6 provides a pictorial overview of this process, to this point. Figure 1-6: Allowing only one user at a time to modify a file. From this point, the workflow might proceed as follows: 6. Fred performs a check-in, which copies the updated file from his working folder into the repository. a. The repository stores Revision 2 of Animals.txt, while retaining Revision 1. b. The source control system releases any locks, so both versions of the file are 30 available for check-out. 7. Barb can now work freely on Animals.txt. She can: a. View Revision 2, with an elk instead of a raccoon, by re-syncing her working folder.
- 31. Chapter 1: Core Source Control Concepts b. Compare the two versions to determine what changed. c. Check out the file for editing. The source control system will automatically copy the latest version, Revision 2, to her working folder and lock the file in the repository. Remember that source control terminology can vary a lot depending on the system. For example, some systems refer to the check-out operation as a "get," and the check-in operation as an "add," a "delta," or a "commit." An important point in all this is that the repository only increments the revision number in response to the check-in operation. A user can check out a file, edit it and save it to his working folder but then decide to "revert" to the original version of the file. The source control system will release the file, and the revision number will remain unchanged. The user's local changes are lost, unless he or she saved them elsewhere. 31 Concurrent editing If working on the files in isolation is the safest route to controlling changes and avoiding conflicts, it's also the slowest. For a more efficient workflow, modern source control systems allow for the possibility that two or more users will edit the same file at the same time. The notion of a check-out operation, with its attendant locking, disappears and instead each user performs an "update" to refresh his or her working folder with the most recent copies of the repository files. Users can then work on these files concurrently, then commit their changes to the repository. If isolated editing is akin to SQL Server's pessimistic concurrency model, then the concurrent editing model is more like SQL Server's optimistic concurrency model. It hopes, optimistically, that no other user will "interfere" with a file on which another user is working, but it has to deal with the consequences if it happens.
- 32. Chapter 1: Core Source Control Concepts Let's see how this might alter our typical workflow. Our description of the various processes uses the most common terms associated with centralized version control systems, with the usual proviso that you will see differences even among centralized systems, and certainly for distributed systems where these processes work slightly differently. 1. Fred performs an update to retrieve the latest files. In this case, he now has in his working folder Revision 2 of Animals.txt. 2. Barb does likewise. 3. Fred edits his working copy of the file by adding wolf. 4. Barb edits her working copy of the file by adding fox. 5. Fred saves the changed file to his working folder. 6. Fred commits the changed file to the repository. The updated file becomes Revision 3 32 in the repository. 7. Barb saves her edited copy to her working folder. 8. Barb tries to commit to the repository, but the repository detects that the version of the file has changed since Barb's last update. The commit fails. 9. Barb performs an update, retrieving Revision 3 into her working folder, and must now "merge" the changes in her working copy of the file with those in the Revision 3 copy of the file. This merge process might be automated, manual, or a combination of both. 10. Barb commits the merged file to the repository. The updated file is designated as Revision 4. Figure 1-7 provides an overview of this process.
- 33. Chapter 1: Core Source Control Concepts Figure 1-7: Concurrent editing of the Animals.txt file. In this simple example, Fred and Barb each make changes that do not really conflict; the wolf and the fox can easily co-exist, at least within the confines of a text file. In such cases, the source control system will probably perform this sort of merge automatically, but in this case even merging the documents manually is a relatively painless process whereby Revision 4 simply contains both the users' changes, as shown in Figure 1-8 overleaf. Again, this may not be exactly how a VCS implements a merge, but it gives a good idea of how it works conceptually. Some users don't even consider this a merge operation, since it can occur automatically as part of the update operation. Some merges, however, are not quite so straightforward. 33
- 34. Chapter 1: Core Source Control Concepts Figure 1-8: Merging two versions of a file to create a third version. Dealing with conflicts during concurrent edits A VCS can sometimes auto-merge changes made by different users to the same file. Sometimes, however, concurrent changes to the same version of a file will cause a real conflict, and to resolve it one of the users will need to perform a manual merge operation, within his or her working folder. Let's rewind to the stage of our Animals.txt example, where each user was working with Revision 2. Suppose that, in addition to adding wolf to his file, Fred changed bear to black bear, and committed the changes (creating Revision 3). At the same time, in addition to adding fox to her file, let's assume Barb changed bear to brown bear. Now when Barb tries to commit her file, an actual conflict emerges, one that Barb must resolve, as shown in Figure 1-9. 34
- 35. Chapter 1: Core Source Control Concepts The source control system can't merge the two file versions until Barb resolves the conflict between black bear and brown bear (the additions of wolf and fox still cause no problem). When conflicts of this nature arise, someone must examine the comparison and determine which version of bear should win out. In this case, Barb decides to go with black bear. Figure 1-9: Addressing conflicts when trying to merge files. 35
- 36. Chapter 1: Core Source Control Concepts We're going to discuss the practicalities of merging in much more detail later in the book, but it's worth considering the risk associated with this merge process. Barb's commit fails, so she can't save her changes to the repository until she can successfully perform a merge. If something goes wrong with the merge operation, she risks losing her changes entirely. This might be a minor problem for small textual changes like these, but a big problem if she's trying to merge in substantial and complex changes to application logic. This is why the source control mantra is: commit small changes often. Merging in distributed source control systems In distributed source control systems, each user's client hosts a local repository as well as a working folder. In the distributed model, Barb commits to her local repository, saving her changes, and then pushes the updated file to the central repository, which would fail because Fred got there first. We won't go into detail on what comes next, but it means we'd see different file versions and version numbers in the central repository from what we saw in our "centralized" example. The main point at this stage is simply that merging is "safer" on distributed source control systems because users can always commit their local changes to their local repository. Another wise practice is: update often. The earlier and more often we update our working folders and commit our changes, the better the processes work for everyone and the easier it is to merge files and resolve conflicts. 36 Branching The way in which we organize files within a repository depends on the team's preferences, the project, and the source control system itself. One common approach is to organize files by project, with each set of project files, whether related to database development, application development or something else, stored within the main project folder.
- 37. Chapter 1: Core Source Control Concepts We organize the project folder itself according to the standards set for the organization. In many cases, we will store the files related to the main development effort in a common root subfolder, often named trunk, but sometimes referred to by other names, such as master. We then create subfolders in the trunk folder as required; for example, as we saw earlier in Figure 1-1, a subfolder for database development and another for application development, and so on. However, relying exclusively on the trunk folder assumes that everyone on the team is working simultaneously on the project files associated with the main development effort. In reality, it is common that certain team members will wish to "spin off" from the main effort to work on separate, but related projects, often referred to as branches. For this reason, you'll find that the main project folder will also contain a branches folder (or a folder with a comparable name), in addition to trunk. Let's assume that we've created a project folder in our repository for our AVM project, added the trunk and branches folders, and built our data-driven application. At this point, all our files are stored in the trunk folder. At Revision 100, the team releases the latest version of the application to customers as v1.0. They're now ready to begin developing v2.0. At the same time, customers are submitting feedback and bug reports for v1.0. This is a classic example in software development of where it will be useful to create a branch (we'll discuss various other branching strategies in Chapter 5). We create a branch at a particular revision in the repository. When we do this, the repository creates a new path location, identified by whatever we call the branch. In this case, we'll create a branch at Revision 100 and call it 1.0_bug fixes. From the user's perspective, this process creates a separate 1.0_bug fixes subfolder in the branches folder, at Revision 101, and populates it with project files that point back to the Revision 100 files in the trunk folder. Developers can work on the branch files as they would in the trunk, but their efforts remain independent of the trunk development efforts, while preserving the fact that each set of files shares the same roots. 37
- 38. Chapter 1: Core Source Control Concepts Tags and build numbers Closely related to the concept of a branch is a tag. In fact, a tag is virtually identical to a branch in that it is a named location for a set of files in the repository, created at a particular revision. We can think of creating a tag as a way to name a set or subset of files associated with a particular revision. The big difference is that we don't ever modify tags. They represent a stable release of a product, with the tag usually representing some meaningful build number. In our example, we might create a tag at Revision 100, called "v1.0." We won't discuss tags again until Chapter 5. While part of the team works on the 1.0 bug fixes in the branch, the rest of the team continues the 2.0 development in the trunk. Those assigned to the bug fixes can work with the trunk folders and files as they would in the trunk. They can edit the files in their working folders and commit changes to the repository, which will maintain revision histories for the life of each branch file, starting from the point of branch creation. For example, let's say Barb creates a branch of the AVM project, containing all the project files, called 1.0_bug fixes, in preparation for the first maintenance release (v1.1). This means that the branch folder is at Revision 101, and contains, among other files, the latest revision (let's say Revision 100) of Animals.txt, as it existed in the trunk at the time she created the branch. Meanwhile, Fred is working in the trunk in preparation for the release of v2.0. Fred updates his working folder with the latest file versions in the trunk. He retrieves Revision 100 of the Animals.txt file, adds chipmunk and walrus and deletes fox. He saves the file and commits his changes to the repository. This creates Revision 102 in the repository. Barb updates her working folder with the latest file versions in the branch. She retrieves the latest version of the Animals.txt file in the branch, which is still Revision 100; Fred's commit has no effect on the file versions in the branch. Barb changes black bear to bear and commits the change, creating Revision 103 in the repository, as shown in Figure 1-10. 38
- 39. Chapter 1: Core Source Control Concepts Figure 1-10: Creating a branch for the Animals.txt file. As you can see, at this point, we have two simultaneous but separate development efforts. Of course, at some point the team may want to merge changes made in the 1.0_bug fixes branch back into the trunk so the next full product release can benefit from the maintenance fixes. 39 Merging Previously, we discussed the need to perform a merge operation to resolve conflicting changes to the same version of a file in the repository. Similarly, when we create a branch, we'll at some point want to merge the changes made to the branch into the trunk. Conversely, we will also need to merge changes from the trunk into a branch. For example, this might be necessary so that a long-running "feature branch" can catch up with the changes in the trunk. As you can imagine, a merge in either direction could be a complex process, especially if conflicts arise across many project files.
- 40. Chapter 1: Core Source Control Concepts The type of comparison made when merging between branches is similar to merging files during normal concurrent editing. In fact, within a particular source control system, the two processes might seem nearly identical. Conceptually, however, they are somewhat different in intent. When merging during concurrent editing of the same file, we're trying to resolve all conflicts to produce a copy of the file that represents a single source of truth. When merging a branch to the trunk, or vice versa, we're retaining two sources of the truth and we're interested only in incorporating the changes from one into the other. For example, suppose Barb wishes to merge into the trunk the changes she made in the branch. Barb's latest commit to the repository was Revision 103 (+bear, −black bear), affecting the Animals.txt file in the branch. Meanwhile, Fred's latest commit to the repository was Revision 102 (+chipmunk, +walrus, −fox), affecting the Animals.txt file in trunk. The resulting merge operation will produce a new version of the Animals.txt file in the trunk (Revision 104), while the branch version will remain untouched at Revision 103. Let's consider each of these changes in turn, in the context of the merge operation. Fred's additions of chipmunk and walrus present no problem because they affect only the file slated for v2.0 (the trunk) and have nothing to do with version v1.1 (the branch). In other words, Revision 104 will still contain the chipmunk and walrus entries. The next potential conflict is with the fox listing. Fred removed fox from the file in trunk, but Barb did not remove it from the version in her branch. You might think that in merging from branch to trunk, we'd add fox back in to the trunk version. However, in effect, this would simply overwrite Fred's change and any effective source control solution will recognize the potential for this sort of problem and avoid it. It may help to think of the merge in terms of the branch change set, in this case all the changes in the current branch revision compared to the revision on creating the branch. Barb wishes to apply this change set (−black bear, +bear) to the trunk file. This means, in SVN at least, that it will retain all other entries as they currently exist in trunk, and so fox will still not exist in the merged version in the trunk. Another VCS might mark this as a possible conflict, and the fact that the branch version assumes that the fox entry exists, but it doesn't exist 40
- 41. Chapter 1: Core Source Control Concepts in the trunk, is a potential problem. Ideally, before performing the merge, Barb would have updated her working copy of trunk and merged from trunk to branch, or at least inspected the differences and raised any possible issues with the team before performing her merge in the opposite direction. Figure 1-11: Resolving conflicts in the Animals.txt file. 41
- 42. Chapter 1: Core Source Control Concepts Merging works differently in different source control systems The methods used to perform the branch comparisons and merges vary from product to product, as does the amount of manual input involved to ensure that the merge does not do more damage than good. Each product is different and each configuration within a product can vary. Finally, we have the potential conflict between bear and black bear. Just because Barb changed it in the trunk does not automatically mean it should be changed in the new release. Even so, SVN does not treat this as a conflict, though another VCS might. Only if Fred had also changed the black bear listing would SVN treat it as a conflict. The result is that the merged file in the trunk will contain the entry bear. Again, the onus is on Barb to ensure that this change to the maintenance release will cause no problems when applied to v2.0. Branching usually comes off without a hitch; we point to the files and folders in the trunk that we want to branch and within seconds, off we go, with a new branch. Merging is the tricky part. The risks and overhead associated with merging can sometimes keep organizations from taking advantage of these capabilities. They might branch, but they don't merge, often resulting in duplicated development efforts and manual copy-and-paste 42 operations. On top of this, it's arguable that some source control systems are better at merging than others. Git, for example, was designed to merge, whereas SVN has a reputation for inflicting its fair share of agony on those trying to merge files. Ultimately, though, the ability to branch and merge is crucial to any organization that needs to expedite their projects and work on those projects in parallel. The mantra is to merge early and often. If you do so, then it's not usually too painful.
- 43. Chapter 1: Core Source Control Concepts 43 Summary In this chapter, we've discussed many of the primary benefits of a source control system, and taken an initial, largely conceptual, look at the components (repository and working folder) and processes (branching and merging) of such a system, which help us realize these benefits. Application developers have been reaping these benefits for many years, but database developers are only just starting to catch up. It's true that database development is different from application development and we'll discuss these differences, and the extra challenges they pose for version control, in Chapter 3. Often database developers have to remind the rest of the development team of those differences, but that does not mean they cannot benefit from a proven system to store and manage files, track changes, work with different versions, and maintain a historical record of the file's evolution. In the next chapter, we're going to take a deeper look at the role of the repository, for both centralized and distributed source control systems. We'll discuss briefly the factors to consider when choosing a source control system for database development work, and then install our chosen system and prepare it so we're ready, in Chapter 3, to start source controlling our databases.
- 44. Chapter 2: The Source Control Repository All source control systems have a repository, a mechanism for storing and managing the files and preserving their histories. The repository organizes into a hierarchical tree structure every version of every file that comprises each branch of work. It also maintains comprehensive metadata that describes details about each file and folder, including who made the changes, when those changes occurred, comments submitted with those changes, and any other information the system needs in order to track each file and folder and maintain its version history. All source control systems also provide a means to coordinate the efforts of a group of users working with those files. Via a working folder, users can add, modify, or delete files, without affecting the integrity of the files preserved in the repository. When changes are then committed back to the repository, the new versions are available to other users, and the repository retains the file version history. Together the repository and private workspace provide the structure necessary to support all other source control components and actions. In this chapter, we examine the two primary types of repositories, centralized and distributed, looking specifically at how the two compare and how they differ. We also consider briefly the many factors that can influence the choice of a source control system for database development work. In order to offer practical examples of the concepts, we had to pick a VCS for use in this book, and opted for Subversion (SVN), a centralized repository. It is just one of a number of systems we could have chosen as a vehicle to demonstrate universal source control concepts; this is not a book about the SVN product, but SVN is free, easy to set up, and implemented widely, which made it a good candidate. The concluding sections of this chapter describe how to install Subversion and configure the repository and working folder, in preparation for putting a database under source control. 44
- 45. Chapter 2: The Source Control Repository Centralized Versus Distributed Repositories Source Control systems with a centralized repository are client-server applications that use a central server to host the repository, which stores the files and their version histories, along with any metadata necessary to manage those files and versions. All client computers communicate with the repository over a network, updating their working folders with the latest files in the repository, and committing changes in the opposite direction. All user interaction is between a local working folder and the remote repository, on the central server, which in turn oversees all operations related to the repository. These centralized solutions emerged into the market in the 1990s and dominate it still. In the last decade, a new generation of source control tools has emerged that use a distributed model (also referred to as decentralized). In a distributed version control system (a DVCS), each client has a local repository, which is a clone and a peer of the repository on every other client. Most user interaction is between a local working folder and the local repository. Distributed systems are still as common as centralized ones, but they're quickly gaining popularity among the development community. Many of the concepts fundamental to source control, such as updating working folders, committing changes to a repository, and so on, remain more or less the same regardless of the repository type, although individual features are implemented in different ways and the presence of local repositories in the distributed model alters the workflow. More notable differences come to the fore mainly when performing more complex operations such as merging, which we won't cover in detail until Chapter 6. 45
- 46. Chapter 2: The Source Control Repository 46 Centralized repositories A VCS with a centralized repository, often referred to as "second generation," remains at the heart of the industry. When most people think of a VCS, they're usually thinking in terms of a centralized repository. Major document management systems, such as SharePoint Server, clearly rely on a central management server. Figure 2-1 illustrates how a centralized system works. At the heart of the system is a central server that manages the repository, controls versioning, and orchestrates the update and commit processes. It maintains all the files, and tracks who made what changes, and when, via a series of deltas, which describe the modifications to the files over time, from one version to the next. In such a system, only the central repository stores the metadata necessary to describe these changes and maintain version history. Since there is a single central repository, it is an easy task to back up all the files stored in the repository. Deltas and diffs Sometimes the term delta is used interchangeably with diff, a dynamically generated comparison between two versions of a file. However, the two are not the same. Chapter 4 covers deltas and diffs in detail. In short, the central repository serves as the single source of truth. The users are, for all practical purposes, isolated from each other and all interactions are with this one repos-itory, via a working folder on the client machine. Each user can update the files in the working folder, edit those files, and then commit the changes to the central server.
- 47. Chapter 2: The Source Control Repository Figure 2-1: Working with a centralized repository. Following is a summary of a few of the core source control operations depicted in Figure 2-1. Of course, there are many more, some of which we'll encounter as we progress through the book. • Add – Register a new file or folder in the working folder. When a user creates a file or folder for the first time (a file or folder that is currently no part of the source control system) they save it to their working directory. However, the user must then perform an Add operation to "register" it as part of that structure. Thereafter, the user can simply open the file in his or her editor of choice to edit it and then save the changes. 47
- 48. Chapter 2: The Source Control Repository • Commit – Save to the repository changes made to the files and folders in a working folder. • Update – Retrieve files and folders from the repository into the working folder so it has the correct version. As noted in Chapter 1, you will see and hear different terms used to refer to the same operation, or even the same term used to refer to different operations! For example, in many second- and third-generation source control systems, developers tend to use the term update interchangeably with checkout. You might also hear the term get used in the same sense as checkout or update. In SVN, however, checkout has a specific meaning and refers to the initial synchronization of the working folder to the repository; essentially it's the first-ever update of a working folder. An obvious drawback of a centralized system is that the users need the central server to be accessible in order to commit changes or update their working folders, as well as to carry out administrative tasks such as performing a diff or viewing file histories in the log. Users can, and do, work offline when editing the files in their working folders, but there is no history saved for any of these changes and, of course, no other user can access those changes until the user commits them to the repository. In general, a down server, as well as slow network connections or unavailable networks, can wreak havoc on the workflow. 48 Distributed repositories A VCS based on the distributed model, often referred to as "third generation," is becoming an increasingly popular choice, because these address some of the limitations of centralized systems. In a DVCS, users each have a local clone of the repository on their client computer. Each local instance of a distributed repository maintains the entire project history of all changes to the files, as well as other metadata. In a DVCS, users can work offline and independently of each other, while still having a fully functioning source control system available at all times, thanks to their local repository.
- 49. Chapter 2: The Source Control Repository In a centralized system, we saw that most of a user's interactions were across a network, between a local working folder on a client and a remote repository. In a distributed system, most of the interactions are between a local working folder and a local repository. When users wish to update their local repositories with other users' changes, or make their changes available to other users, they synchronize their local repository with any other remote repository, by pushing their changes to a remote repository and pulling others' changes into their local repository. Each repository can push changes to, and pull changes from, any other repository. Figure 2-2 shows an example of a DVCS that includes four users (Fred, Barb, Jane and Joe), each running a client machine with a local working folder and repository. Users of a DVCS are likely to adhere to the following workflow, or some variation of it. 1. Pull the latest files and folders from a remote repository into their local repositories. 2. Update their working folders with the latest versions in their local repository. 3. Edit those files. 4. Commit changes to their local repositories. 5. Push files and folders to a remote repository. A peer-to-peer relationship exists between each of the repositories. No one repository has any special significance over any other; each is a clone of the others. Of course, if users do not regularly push and pull changes, the repositories will quickly get out of sync, but eventually all will be consistent. Users can choose to pull only the branches, folders, or files that are relevant to their current work. However, this is not the default behavior and, culturally, most teams frown on this way of working. Such a process can become unwieldy, particular for large projects with a large number of users, and it becomes much harder to obtain a "master" view of the project as a whole. 49
- 50. Chapter 2: The Source Control Repository Figure 2-2: Working without a central server in a distributed system. In some DVCS architectures, a team often designates one of the peer repositories as a "central server" to which all users push their changes and from which they can pull recent file versions. In this sense, the central server is no different from any other peer, but it does represent the "one version of the truth" for the whole project, to which any local repository can refer. 50
- 51. Chapter 2: The Source Control Repository Figure 2-3 shows an example of a distributed source control system in which we have designated one of the peers, Jane's computer, as the central server. Notice, however, that all peers can still push and pull from all other peers. Figure 2-3: Working with a central server in a distributed system. It is also possible to designate a separate computer to use specifically as a central server for the repository in order to support all users as well as provide a mechanism for backing up the repository at regular intervals. In this case, there would be no specific user associated with the machine in the sense of adding, editing and committing changes, although the server would still be considered a peer. 51
- 52. Chapter 2: The Source Control Repository Comparing centralized and distributed repositories Having seen the basic architecture for a centralized versus a distributed VCS, let's compare and contrast the two in terms of basic source control functionality, as well as administration. 52 Versioning In the centralized model, users commit changes to, and update their local working folders from, a single central repository, which maintains the full version history for every file. The central server assigns a global sequential revision number to each commit. As a result, each file or set of files associated with a particular commit receives a revision number that represents a specific file version. The revision numbers increment sequen-tially making it easy to identify file versions at a specific point in time. In a distributed model, users commit changes to their local repositories and then push and pull changes with any other remote repository. A distributed system will assign a commit hash to every change (for example, Git uses 40-character SHA1 hashes), but global sequential versioning is more difficult to achieve. Chapter 4 provides a more detailed discussion of versioning. Concurrent editing and merging In the centralized model, all commits and updates require a connection to the central server, as does performing a diff or viewing revision history, and so on. Users can work on files offline, but what they do on their clients within their working folders is neither tracked nor versioned by the repository. A user can edit and save a file multiple times within a working folder, but it is only upon commit that those changes register in the repository as a new revision. At this point, other users can update their working folder to retrieve those changes into their working set.
- 53. Chapter 2: The Source Control Repository If a conflict arises during a commit or an update, a user must perform a merge operation on their client and then commit the new, merged file. It's a potentially tricky operation and, in a centralized system, merges tend to be something to avoid, although modern centralized systems have become better at handling them. Although most centralized systems no longer exclusively lock a file upon checkout, it is possible to do so and, used sparingly, this is a possible solution in cases where you want to maintain absolute control over the versioning process and avoid issues of contention. As discussed in Chapter 1, merging is much easier if the team adopt the "update often, commit often" philosophy. Counter to this, however, is the problem that with a single, central repository, every commit means inflicting the change on the rest of the team. It means users have a tendency to delay committing until they are sure the change won't cause disruption, which can sometimes result in a "big bang" merge operation. In the distributed model, users commit changes locally and then push them to a remote repository as necessary, and pull the changes of others. A user can work offline, safe in the knowledge that their local repository maintains a full revision history. Users can also perform locally operations such as comparing versions (diffs) or viewing the log. In short, each user has a local, fully operational version control system. Merge operations are safer, in that the user commits changes locally before performing a merge. After committing their changes locally, they can pull others' changes, update their working folder, perform a merge if necessary, commit again, and then push their changes. A user can make many small commits to their local repository, and then "publish" them all to a peer repository in one go, once debugging and testing is complete. This avoids potential disruption to the work of others while still offering each user the full benefit of a VCS. A DVCS is designed from the ground up to support branching and merging, so it is optimized for that specific use case and encourages a culture of frequent merging. As a result, branches don't get stale enough to require a "big bang merge," as is often the case with second-generation systems. 53
- 54. Chapter 2: The Source Control Repository Generally, third-generation systems have more sophisticated diffing algorithms, which often makes user intervention during merges less necessary. Centralized systems tend to track changes on a file by file basis. For example, let's say that during concurrent editing, one user renames a file, while another modifies the contents of the same file (with its original name). A system such as SVN will recognize this internally as a rename, rather than a "delete old file create new one" (as was the case for first-generation source control systems like CVS and SourceSafe). However, such situations will still raise a conflict and require a manual merge operation. An important feature of distributed systems is that they track changes across all files collectively, rather than on a file-by-file basis. The text in a file has associated metadata that stores the filename. In our previous example, where one user renames a file, while another modifies its contents, a system such as Git records the former as a modification of the metadata attached to the text, and the latter as a modification of the text. Generally, it will be able to merge such changes without any manual user intervention. Even so, if the possibility of conflict exists, the source control system will call for the user to step in to resolve those conflicts. The degree to which the user must get involved depends, in part, on the source control system itself as well as the files being compared. If distributed systems offer much stronger support for merging (and therefore branching), the flipside is that you have much less control over what other users are doing. Although a distributed system can prevent changes from being lost or overwritten, it cannot prevent time being wasted on duplicate efforts. 54 Ease of backups In a centralized system, the repository represents a single, consistent source of truth for all the files and folders, and all version histories. As such, it also provides the mechanism necessary for making reliable backups.
- 55. Chapter 2: The Source Control Repository Even if a distributed system uses a central server, and we take backups off that repository, it can be difficult to guarantee that the backup represents a single, consistent view of the system. Of course, through repeated push/pull operations, each repository becomes a clone of any other repositories. In other words, we can achieve eventual consistency in a distributed system, as opposed to immediate consistency in a centralized system, and in that sense, each distributed peer could be considered a backup. 55 Repository size Most modern source control systems are very efficient at storing file version histories while taking up minimal extra storage space. Using clever packing and compression techniques, a modern repository containing mainly text-based files can store decades of history with a total repository size of little more than that for a single working folder, and still maintain the current version of every file. However, the type of file stored can and will affect the overall repository size. Binary files can take up lots of space. Each commit, after a change to one of these files, is likely to result in a whole new copy of the file. Under such conditions, the repository can rapidly grow in size. The repository in a centralized system can, at least from a user's perspective, grow as large as it likes, as long as we can scale up the central server to accommodate the entire repository. Users can easily retrieve only a subset of files into their working folders in order to conserve space on their client machines. However, size is more of an issue in a distributed system, with each user's client hosting a repository. Even if users quickly delete those large files, the repository maintains a permanent history of all changes, so the repository size is never reduced. This is true for first-, second-, and third-generation systems. For database developers, this might not be much of an issue, but it should be a consideration.
- 56. Chapter 2: The Source Control Repository 56 Security A centralized system can restrict access to certain specific files and folders. In a distributed system, every peer gets all files and all history. This can become a concern, for example, if a contractor should have access to only a portion of the files. However, many teams get around this limitation by creating multiple smaller repositories, rather than maintaining one large one, as is more often the case in a centralized system. Choosing a VCS for Database Development There are many factors to consider when choosing both the repository type (centralized or distributed) and the specific source control product that is most appropriate for the needs of the organization, team, and project. It is necessary to consider the needs, not only of application developers, but also of the database developers, the administrators who will access the VCS to manage operations, the QA testers who must be able to test applications, the project managers who must be able track progress, and so on. That said, perhaps the most important factor is continuity and consistency among teams. If the application development team uses a specific source control system then there will be a strong pull toward using the same one for the databases, and for good reason. Having the database content in the same repository as the application content provides a single, consistent versioning scheme across all project assets. Most source control solutions support different configurations within the same system, in order to maximize user flexibility. The geographical distribution of your development team is also an important factor. A centralized source control solution can become a hindrance if your team is distributed to the extent that network bandwidth and connectivity become an issue. Under such circumstances, a distributed system might offer the necessary flexibility that a centralized one cannot.
- 57. Chapter 2: The Source Control Repository Many application developers now work in Agile team environments, where code changes are rapid and frequent. They appreciate the flexibility of a distributed source control system, the strong support for concurrent editing of files via branching and merging, and the freedom to work locally as much as possible. When we want to replace the existing version of a client application with the version in source control, it doesn't really matter how we apply the changes as long as the modified version still builds and runs correctly. For database developers, accustomed to a more sedate rate of change, this can seem an alien environment. In contrast to upgrading a live application, when we want to upgrade an existing database, for instance to add or remove a column, change a data type, split a table, and so on, we must be certain that what we have stored in the VCS describes, not only the required changes, but the exact order and manner in which to perform these changes in order to preserve data integrity. Historically, most teams haven't had in place the processes, tools, checks, and controls that allowed frequent modification of live databases without significant risk of data loss, database downtime and subsequent disruption to users and to the business. Therefore, historically, database development work, in contrast to application development, has tended to be more conservatively paced, with longer upfront design and less frequent subsequent changes. In this sense, database development seems to be a better fit with the higher degree of control afforded by a single, centralized "master" repository, rather than the speed and high flexibility of a multi-repository distributed system. However, with better tools, processes, and automated testing procedures in place, most teams now embrace a unified approach to the development and deployment of both application and database, and therefore a unified strategy for versioning a project, with all project resources organized side by side in the same repository. When all these are versioned together, everything can, in theory at least, change as often as necessary. Ultimately, in such teams, it's highly likely that the database will be absorbed into the existing VCS strategy for the application. 57
- 58. Chapter 2: The Source Control Repository Versioning a Project in the Repository As noted in the previous section, the secret of successful source control is to have all project assets versioned together, in the same repository. This will, of course, include the database, which is our primary interest in this book, but also the application code, QA resources including automated and functional tests, certain configuration details for each environment, scripts and tools for accomplishing various tasks, deployment scripts, and so on. In many development projects the trunk, often referred to as master, is where the mainstream development effort takes place, and so this is where all the project folders live. The team create branches off of trunk as it becomes necessary to separate out certain strands of the development effort. However, different teams take different approaches, and we'll discuss this topic in more detail in Chapters 3 and 5. The important point is that, with all the assets together, it's possible to use a single version number to represent a point in time for the entire application. This is immensely useful for fulfilling one of the main purposes of source control, namely being able to reproduce the entire project as it existed at a particular revision. For example, let's say a customer reports a bug with a release version of our software. We can re-create the project, from the revision in the repository that corresponds to this release, reproduce the reported defect, debug it, create a fix, and redeploy the new version to customers. Each user checks out to their working folder at different levels of the repository structure, according to their role. In Figure 2-4, Jane, a data architect, checks out the entire repository. She creates a BookstoreProject folder on her client, linked to the root of the BookstoreProject repository on the server. Fred, a database developer, only needs the database assets so he creates a subfolder of BookstoreProject on his client, called BookstoreDatabases, and during checkout links it to the Databases sub-directory of the BookstoreProject repository. 58
- 59. Chapter 2: The Source Control Repository 59 Figure 2-4: Project versioning. Getting Started with Source Control In order to turn concepts into practical examples, we needed to choose a source control system for this book. We decided to use a centralized system over a distributed one, and out of all the available centralized solutions, we chose Subversion (SVN), a system created by Collabnet and provided as an open source Apache project. In addition to the above considerations, with regard the specific nature of database devel-opment work, we also took into account the following factors: • SVN is an open source solution, so it's free. • It is relatively easy to set up and runs on multiple platforms. • It is one of the most widely implemented source control solutions, so you can find plenty of information about how it works, and people who are currently using it.
- 60. Chapter 2: The Source Control Repository Ultimately, though, it is just one of any number of systems we could have chosen, and our choice of SVN in no way implies that we endorse this product above all others. For demonstrating the concepts in this book, SVN is a good fit, but you must decide what will work best for you. 60 Setting up Subversion The best way to get started with source control is to jump right in; fortunately, SVN makes that very easy. All we need to do is create the repository and configure the working folder and we're good to go. To help us do this, we use the tools below. • VisualSVN Server standard edition – A free SVN management program that lets us create and manage an SVN repository on a Windows computer. For information about downloading, installing, and running VisualSVN Server, see http://www.visualsvn.com/server/. • TortoiseSVN – A free, user-friendly interface that integrates with the Windows Shell, making most SVN-related commands available through Windows Explorer. We'll use this tool for all interactions between a user's working folder and the repository. For information about downloading, installing, and running TortoiseSVN, see http://tortoisesvn.net. In any real development environment, we'd have a separate server for the central SVN repository, and each user would work on their own machine, interacting with the central server through their working folder. If a proper client-server architecture is available to you, perhaps via virtual machines, then that's the setup to use. However, for convenience, we create both the SVN repository and the working folders for all users on a single machine, specifically a Windows 7 virtual machine running an instance of SQL Server 2012.
- 61. Chapter 2: The Source Control Repository Avoid using TortoiseSVN for both working folder and local repository Built into TortoiseSVN is a feature that lets us create a local repository. This configuration permits one user to connect to the repository on the same machine as the working folder. If you plan to try out the examples in this book, this approach works fine except when we need two users to demonstrate such concepts as branching and merging. However, if you plan to use the same computer to participate in a server-based SVN system, you're better off not using this feature because it can complicate any subsequent participation in the server-based implementation. For the VisualSVN Server installation, we used the following default settings: • Location – Default location (C:Program FilesVisualSVN Server). • Repositories – Default folder (C:Repositories), where VisualSVN Server stores the repository data and metadata. • Server Port – Default port (443) – if another service, such as Internet Information Services (IIS), uses port 443, you will need to select a different port, such as 8443. • Use secure connection – Option selected. • Authentication – Subversion authentication (we cannot use Integration Windows Authentication with the VisualSVN Server Standard edition, but it is available in the Enterprise Edition). You can change the default settings in the server properties. To access the properties, open VisualSVN Server Manager, right-click the VisualSVN Server node and then click Properties. In the VisualSVN Server Properties dialog box, select the appropriate tab and configure the settings as necessary. For example, to change the authentication mode, go to the Authentication tab. 61
- 62. Chapter 2: The Source Control Repository For the TortoiseSVN installation, simply follow the onscreen prompts and accept all the default settings. With both tools installed, we're ready to go. The rest of this chapter is devoted to setting up a repository and working folder. 62 Setting up the repository Not surprisingly, to set up a source control environment, one of the first tasks is to create the repository. Our repository will host a fictitious development project called BookstoreProject. As its name implies, it relates to a bookstore and its inventory. Before we actually create the repository, we will add a few users to VisualSVN Server so we can control access to the repository and its files. We open up VisualSVN Server Manager. Create repository users We want to create three authorized repository users: Jane, the data architect, and two database developers, Fred and Barb. We'll make sure that no other user accounts can access our repositories. 1. In the left window, right-click the Users node, and then click Create User. 2. In the Create New User dialog box, create the first user, Jane, assign a password, and then click OK. 3. Repeat Steps 1 and 2 to create the users, Fred and Barb. The Users node should now look similar to Figure 2-5.
- 63. Chapter 2: The Source Control Repository Figure 2-5: Viewing users in VisualSVN Server Manager. 63 Create the repository Our next task is to create the actual repository, which we'll name BookstoreProject. 1. Right-click the Repositories node, and then click Create New Repository to launch the wizard. 2. Enter BookstoreProject in the Repository Name text box, and then click Next. 3. On the Repository Structure screen, select the option Single-project repository (more on this below) and click Next. 4. In some cases, we would select the default option, Empty repository, but for now we want to use the predefined trunk, branches, and tags folders, a structure common to many development projects. 5. On the Repository Access Permissions screen, select the Customize Permissions option and then click the Custom button. 6. In the Customize Permissions dialog box, click the Add button.
- 64. Chapter 2: The Source Control Repository 7. In the Choose User or Group dialog box, select the users Barb, Fred, and Jane, and 64 then click OK. 8. The three users should now appear in the Customize Permissions dialog box. Ensure that they all have Read/Write permission, and then click OK to close the dialog box. 9. On the Repository Access Permissions screen, click Create. 10. The Repository Created Successfully screen should appear and provide details about the main repository. Click Finish to complete the wizard. Beneath the Repositories node in VisualSVN Server Manager, you should now see the BookstoreProject repository, with three folders: branches, tags, and trunk, as shown in Figure 2-6. Figure 2-6: Creating the main repository with the default structure. 11. Notice that the URL for the BookstoreProject repository appears above the right window. We will need it later, so right-click the BookstoreProject node and click Copy URL to Clipboard to get the URL.
- 65. Chapter 2: The Source Control Repository As we progress, we can create the subfolders needed to store our project files, but we'll do that through the users' working folders. Interaction with the repository and its structure via Visual SVN Manager should be kept to a minimum, and should be the preserve of the project administrator only. That's all there is to setting up the repository. If you want to test access to the repository, you can use your browser by going to https://localhost:443/svn/ (or whichever port you selected). If prompted, enter one of the usernames and passwords. You should then be able to see the repository content. Note, however, that the URL and port number may vary depending how you initially configured the repository. Setting up the working folders Now that we've set up the repository on the server, it's time to set up a working folder on each client so that our users can access the repository and interact with it. In this chapter, we'll define the high-level structure for the project as a whole. The focus of the book is purely on the database development but, as discussed, it's important to have all project assets stored side by side in a single repository. Subsequently, the database developers can update their respective working folders with this structure or, more specifically, the parts of it that are relevant to them, and start development work. Each user's client, in this example, is TortoiseSVN, which integrates into Windows Explorer. As discussed earlier, you will ideally set up each user on a separate client. If you're working on a single machine, you'll need to remember to clear TortoiseSVN's authentication data each time you want to switch between users. Multiple users on the same client: clearing authentication data To clear out the previous authentication details, and connect as a different user, right-click the working folder, or one of its subfolders, navigate TortoiseSVN | Settings. Go to the Saved Data node, and click Clear All for the Authentication data listing. 65
- 66. Chapter 2: The Source Control Repository Create a working folder: checkout In order to set up a working folder for each of our three users (Jane, Fred, Barb), we simply need to create a regular folder in Windows Explorer (or use an existing empty folder), and then perform what SVN calls a checkout. As we discussed earlier, a checkout has a very specific meaning in SVN and what it does is register the working folder with the repos-itory and perform the initial synchronization necessary to copy over the repository folder structure (trunk, branches and tags) and files. To start, we'll create a working folder in the VCS for Jane, the data architect, who is responsible for defining how to structure the whole project. Jane will connect to the repository to register her working folder and perform an initial checkout of the whole BookstoreProject repository. 66 To create Jane's working folder: 1. In the root of the C: drive on your client machine (or use whatever drive and name you wish), create a folder called SVNJane. This allows us to distinguish each user when working on the same client. If you have each user on a separate client this folder is unnecessary. 2. Within the SVNJane folder, create a subfolder named BookstoreProject. This directory will form the root of Jane's working folder and will correspond to the root folder of the BookstoreProject repository (see Figure 2-4). 3. Right-click the BookstoreProject directory, and then click SVN Checkout. 4. In the Checkout dialog box, you should find a URL that points to the repository and the path to your working folder, similar to what is shown in Figure 2-7.
- 67. Chapter 2: The Source Control Repository Figure 2-7: The repository URL and working folder. 5. If you've already connected to a different SVN repository from your computer, another URL might be listed in the URL of repository text box. In such cases, you can retrieve the correct URL from within VisualSVN Server Manager by right-clicking the main repository and then clicking Copy URL to Clipboard. 6. Having verified that the information in the Checkout dialog box is correct, click OK. 7. When the Authentication dialog box appears, enter Jane's credentials, select the Save authentication check box and click OK. 8. When the Checkout Finished dialog box appears, it should confirm the update of Jane's working folder to reflect the initial trunk/branches/tags structure that we established when we created the repository. Creation of this initial repository structure formed Revision 1 in the repository, so Jane's folder is now "synchronized" with Revision 1 in the repository. 67
- 68. Chapter 2: The Source Control Repository That's all we need to do to create a working folder! Jane's folder, as it appears in Windows Explorer, should now look similar to the one shown in Figure 2-8. Figure 2-8: Jane's working folder in Windows Explorer. The white-on-green checkmark on each folder except the .svn folder indicates that those folders are in sync with the latest revision in the repository. The .svn folder contains the metadata necessary to manage the files and folders in the working folder and to keep them in sync with the repository. When viewing the working folder, be aware that TortoiseSVN can sometimes be slow in applying the appropriate icons to the various folders and files, especially with old hardware, very large repositories, or when the system is under load. Sometimes, refreshing Windows Explorer can help hurry things along. 68
- 69. Chapter 2: The Source Control Repository Create the project structure in the repository: Add and Commit Jane can now add the subfolders to her working directory that will establish the high-level project structure within the VCS. For this project, the major development effort will take place in trunk, so this is where we need to locate all the project assets. To establish the basic project structure in the repository we'll use two very common source control operations: 1. Add the file or folder to the working folder – Add is a formal source control process that "registers" the file or folder within the working folder. 2. Commit the file or folder (or both) to the repository – Commit is a formal source control process to move data from the working folder to the repository. Remember, that these terms might have subtly or even very different meanings in a different VCS. In Git, for example, Add means to add to the staging area. However, for our purposes here, we referring specifically to the registering of the file or folder with the source control process. We can commit empty folders to source control, we can commit individual files to source control, or we can commit a folder and all its subfolders and files in one go. In each case, we follow essentially the same process. We create the folder in the working folder, or save the file to it, and then perform an Add followed by a Commit. Updating the working folder When we commit the changes in the working folder to the repository, we only move data from the working folder to the repository, not in the other direction. Any changes committed to the repository by other users in the meantime are not copied to our working folder until we explicitly do an Update. The working folder is only as current as the initial checkout or the last update. Chapter 3 covers the Update operation in more detail. 69
- 70. Chapter 2: The Source Control Repository 70 Add the folders In Windows Explorer, navigate to Jane's working folder (SVNJane), and then to the trunk folder, and create several new subfolders for AppCode, Configuration, Databases, DeployScripts, and Design. Since we are interested primarily in the database, let's also create a folder for our Bookstore database, as a subfolder of Databases. All of the other asset folders are empty, and are included simply to emphasize the importance of versioning all project assets together. We won't discuss them further. Figure 2-9: Creating new folders in the working folder. The "?" icon overlay for each folder indicates that it is currently "untracked" by the repository. At this stage we've created the folders but not registered them as part of the working folder structure. In order to do this, we need to Add them. Right-click the Bookstore- Project folder, point to TortoiseSVN, and then click Add. The Add dialog box lists the new folders, all selected by default, as shown in Figure 2-10.
- 71. Chapter 2: The Source Control Repository Figure 2-10: Adding new folders to the working folder. Jane can deselect any folders she doesn't want to add to her working folder at this time, but in this case she wants to add all of them, so click OK. We'll see confirmation of the folders that TortoiseSVN added to the working folder. With the Add complete, Jane's working folder should look as shown in Figure 2-11. We see a blue plus sign next to each of our new folders, indicating that they have been newly added to the working folder, and white-on-red exclamation points on all the trunk and BookstoreProject folders, higher in the hierarchy, indicating one or more changes or additions to these folders that we are yet to commit to the repository. 71
- 72. Chapter 2: The Source Control Repository 72 Figure 2-11: Folders added, but not committed. We can verify the current mismatch between the working folder and the repository using the Repository Browser, a built-in tool for TortoiseSVN. Right-click the BookstoreProject folder, point to TortoiseSVN, and then click Repo-Browser. There you'll see that the trunk folder in the repository contains none of the new subfolders. For the subfolders to appear, we must commit our changes to the repository. Commit the folders During general development, we'd probably add files to a new folder and commit them together, but here we're only interested in establishing our initial project structure. Back in the working folder in Windows Explorer, right-click the BookstoreProject folder and then click SVN Commit. In the Commit to dialog box (shown in Figure 2-12) we can and should add a comment to the Message text box to describe what we're committing to the repository. Some organizations have specific guidelines for commit messages but even if
- 73. Chapter 2: The Source Control Repository not, it's a very good practice to provide a description, as it can make auditing tasks much easier down the road. 73 Figure 2-12: Committing new folders. Click OK to close the Commit to dialog box. The subsequent Commit Finished dialog box reports which folders we committed and a revision number, in this case Revision 2. We'll discuss revision numbers in much more detail in Chapter 3, but for now simply note how our source control system increments the revision numbers with each commit operation.
- 74. Chapter 2: The Source Control Repository Figure 2-13: A successful commit creates a new revision in the repository. Click OK to close the Commit Finished dialog box. White-on-green checkmarks should now overlay each of the folders we added to the working folder and then committed to the repository. In fact, this icon now precedes all folders (except .svn), indicating that the working folder contains no additions or changes that need to be committed to the repository, as shown in Figure 2-14. Figure 2-14: Jane's working folder with no changes to commit. If desired, we can reopen the repo-browser to confirm that the subfolders now also exist in the repository. 74
- 75. Chapter 2: The Source Control Repository Setting up working folders for the database developers It's now time to set up the working folders for our developers, Barb and Fred. Barb is the lead developer and will need to check out the entire project. Fred is a database specialist and will be working only with the files in the Databases folder. For Barb, we perform a checkout to a BookstoreProject folder, exactly as we did for Jane (except within a parent folder called SVNBarb). If you are working from a single client, remember to right-click BookstoreProject, choose Settings, and under Saved Data clear the authentication data, as described previously, before right-clicking again and performing the checkout. This will update Barb's working folder with the project structure Jane created. Fred, instead of checking out the entire repository structure, will check out only the Databases folder, as follows: 1. In Windows Explorer, create a working folder named C:SVNFred and then a 75 subfolder called BookstoreProject. 2. Create a subfolder in BookstoreProject called BookstoreDatabases. This is the folder to which we will check out the Databases folder in the repository. 3. Right-click the BookstoreDatabases folder. a. If you are working from a single client, reset the authentication data. If not, go straight to b. b. Select SVN Checkout. 4. Provide a URL to the repository's Databases folder. Simply append the repository's root URL with /trunk/Databases, or click the ellipsis button and navigate to the folder in repository browser.
- 76. Chapter 2: The Source Control Repository Figure 2-15: Mapping server to client URLs during checkout. 5. Click OK to close the Checkout dialog box. 6. If you receive a message that your certification failed, click Accept the 76 certificate permanently. 7. When the Authentication text box appears, enter Fred's credentials and then click OK. The Checkout Finished dialog box will appear and provide an overview of the folder you checked out. Fred's working folder is now at Revision 2 and he has in his working folder only the contents of the Databases folder. Figure 2-16: Fred's working folder. Remember that Fred checked out the trunk/Databases folder, so he doesn't actually have the trunk folder in his working directory, but implicitly all the work he does in his working folder will be committed to trunk.
- 77. Chapter 2: The Source Control Repository 77 We've Only Just Begun At the end of this chapter, you should understand the fundamental differences between centralized and distributed repositories and have a better appreciation of the factors that will affect your choice. Given the number of source control solutions now available to database developers, there's no reason that anyone who creates, modifies, or manages database components should not be taking advantage of source control both personally and across their organization. Of course, this is just the start of our journey. We've set up a source control repository and a working folder with which to interact with it, and now it's time to put a database under source control!
- 78. Chapter 3: Getting Started with Source Control for Databases Over the course of the previous chapters, we've discussed many of the primary benefits of source control systems in coordinating the efforts of the development team, ensuring a complete audit trail of all changes to the code files, and allowing the team to reproduce any specific revision or build. I hope by this stage you are convinced that a source control solution can and must play a key role in pulling together the database development process in a way that makes it more productive and efficient, while maintaining a history of database changes. Database developers can and should benefit from source control's audit history and change-tracking capabilities, but there's more to it than simply placing a few database build scripts into a subfolder of the application development team's project folder in source control. This chapter discusses many of the factors that will influence how a development team works with a database via the VCS, and how the team creates, organizes and updates the relevant database files in the VCS. Unlike application developers, database developers are not assembling files into a neat little application package, but are instead running scripts that feed off each other and off existing database objects, while negotiating the close interdependency between the code and the data. However, with the appropriate planning, our VCS will not only protect our files and provide a single authoritative source for our schema, so we can incorporate changes into our database design in a controlled and consistent manner; it will also allow us to build any version of the database stored in the VCS and use it to upgrade any existing database, without fear of causing data loss or data integrity issues. 78
- 79. Chapter 3: Getting Started with Source Control for Databases Planning for Database Source Control The majority of this book concerns itself directly with the practicalities of using a version control system in a development environment to improve the efficiency of team-based database development. Regardless of the type of source control repository they choose, team members need to give thought up front to their development model, while coordi-nating their efforts through that VCS. They must also consider how they wish to build and deploy databases from source control. Both factors, and others, will affect how the team uses the VCS, how they organize the source files in the VCS, and how they handle modification of those files. Integrated Development Environment (IDE) integration Application developers work on files within their preferred IDE, usually Visual Studio for Windows platform developers. The IDE provides tools, either built in or via a plug-in, for syntax checking, debugging, compiling, and other development-related tasks. To give one obvious example, VisualSVN, a plug-in for the Visual Studio IDE, makes it very easy to work with a VCS, allowing developers to add and edit files directly from within the Visual Studio IDE. When developers modify an application, they retrieve the necessary code files from the VCS, build the application in their IDE, edit the underlying files directly, and commit the changed files back to the VCS. This is a natural fit for working with a VCS, and a very natural development model. In recent years, good VCS plug-ins have also emerged for database development IDEs such as SQL Server Management Studio (SSMS) and SQL Server Data Tools (SSDT), offering some degree of source control for database development. The majority of this book does not rely on any particular plug-ins, but in Chapter 8 we demonstrate a database build and deployment scheme using the Red Gate SQL Source Control SSMS plug-in, which allows developers to work on a live database while ensuring every change gets checked into source control and becomes immediately available to other team members. 79
- 80. Chapter 3: Getting Started with Source Control for Databases Disconnected versus online database development The model whereby developers work in their preferred IDE, directly modifying the files stored in source control, is great for application development but a less natural fit for database development. It means that database developers need to work on a source-code "representation" or "model" of the database, in the form of the SQL scripts that create or alter the database objects, rather than working directly with a live database (the term "live" does not necessarily refer to a production database, but rather any running database, usually one used for development). We call this a disconnected development model. For example, for each modification to a database object, a developer might generate a separate change script (we'll discuss other options shortly) and commit it to the VCS, and then run the script against the live database. Some database developers, however, update the live database directly. In this online development model, we construct retrospectively the scripts necessary to describe the change (i.e. the source code), once the change is complete. Many teams use a schema comparison tool to generate the change scripts by comparing the updated live database with the latest revision in the source control repository. Some teams favor the disconnected model because it is a highly controlled process, but the online model is equally viable, as long as the team has in place the tools, workflow, and business processes to ensure that all changes get from the live database into source control. These requirements are really no different from those used to control how application code gets from a developer's workstation to the repository. As the tools and processes to support the online database development model improve, it will likely become more common because it represents a much faster, more intuitive way to build databases. You can work through the majority of this book using either development model, but Chapter 8 follows the online model. 80
- 81. Chapter 3: Getting Started with Source Control for Databases Shared versus dedicated database development It is a very common practice for every application developer to have their own "sandbox" environment, rather than have everyone work in a shared server environment. It means every developer can work in relative isolation, unaffected by the changes made by others. As soon as a change is completed and tested, the developer commits the change to the VCS, so that it is available to other developers the next time they update their working folders. Most developers would prefer to use the same model for database development, working on their own dedicated sandbox databases, and committing changes to a shared database directory in source control, from where they are available to other team members. However, for various reasons, many teams develop against a single shared database. It can mean lower licensing costs, the need to generate only one copy of test data, and the ability for the DBA to manage the environment much like a production instance, applying the correct security configuration, as well as instance and database properties. The biggest impact on the development team, however, is the need to ensure non-inter-fering changes in the shared database environment. There are ways to achieve this. For example, if the database design makes intelligent use of schemas, such that each schema represents a logical area of the overall application then, notionally, we could assign each developer to a logical area, and there would be few overlapping changes. However, most developers relying on a shared database have no doubt found that changes made by others can and do break things and affect their work, causing delays and frustration. Given a choice, most developers would likely choose the dedicated model. 81
- 82. Chapter 3: Getting Started with Source Control for Databases Source control and Application Lifecycle Management No VCS is an island, and in most development teams the VCS will be integrated into a larger suite of tools and policies that govern how the team design, build, test, and deploy applications and databases. The umbrella term for this suite of tools and policies is Application Lifecycle Management (ALM). Not surprisingly, the ways in which these tools integrate varies from solution to solution, and tools can, and often do, cross multiple categories. For example, Microsoft offers the Visual Studio ALM solution, which includes Team Foundation Server for version control, bug tracking, build automation, reporting, and other functionality. The solution also includes the Visual Studio IDE for coding, debugging, and testing. SourceGear Fortress, on the other hand, is a cross-platform ALM solution that integrates version control, work item tracking, and other tasks into a single environment that can be integrated with IDEs such as Eclipse and Visual Studio. The choice of ALM strategy will largely define the team's approach to source control issues such as support for concurrent editing, when and how to create branches and tags, how to name and comment on files, and how to organize the folder structure. We'll discuss good practices for each of these issues as we progress through the book, but it's essential to establish that there are many possible approaches, and that the one you choose will depend on many other factors, including the overall ALM strategy. Database build and deploy considerations Source control protects our files and provides a single authoritative source for our schema so that we can incorporate changes into our database design in a controlled and consistent manner during development. Of course, at some point the development effort will be complete, and the team will need to build and deploy the database to a target environment, such as test, staging or production, either to create a new database or to update an existing one in that environment. 82
- 83. Chapter 3: Getting Started with Source Control for Databases At the deployment stage, our goal is to have in source control a single source of "truth" that defines precisely the database we wish to deploy. In other words, source control provides the foundation for reliable and repeatable database deployments to a range of environments. Always deploy from source control As tempting as it might be, never "sneak in" a direct modification to a test, staging, or production data-base, whether to add a feature or to quickly solve a customer concern. Always make the effort to make the change in the VCS, retest, and redeploy through the official deployment process. If someone modifies a target database directly, then it renders invalid the scripts we have in source control to roll that data-base from one state to another. This is a problem termed version drift and it can and will cause failed 83 deployments (see Chapter 7 for further details). We won't discuss the intricacies of database build and deploy until Chapters 7 and 8, but it's a potentially complex process with many moving parts, all of which rely on the fact that the correct versions of all the required files are in the VCS and accessible by the build engine. The developers must ensure, for example, that the VCS contains the correct versions of the scripts necessary to create objects, load reference (lookup) data, migrate data, add security accounts, set configuration settings, provide parameter values, or take any other necessary actions. If batch files, PowerShell files, or other types of script files are necessary, those too must be ready to go. One of the major issues to address is how to apply those scripts and, particularly, in what order to apply them. The most basic solution is to document each step and ask the DBA to execute the scripts in the prescribed order. A slightly better option is to create a batch file that executes them in the preferred order. We can maintain (in source control, of course) a "change log" that supplies the batch file with an ordered list of scripts. However, considering that building or upgrading a database might require hundreds of script files, this is clearly a task crying out for automation. Many organizations turn to tools that automate most or all of the steps necessary to build and deploy their applications and databases.
- 84. Chapter 3: Getting Started with Source Control for Databases 84 We have numerous options. For example: • Red Gate's Deployment Manager is a package-based management tool for deploying databases. • Microsoft's Visual Studio Team Foundation Server includes Team Foundation Build, which automatically compiles and tests applications. • BuildMaster automates build management and supports workflow-driven approvals, database change scripts, and production deployments. • CruiseControl provides a framework for a continuous build process that includes plug-ins for email notification, Ant build software, and various source control tools. Each tool helps automate the process of getting from Point A (source control) to Point B (the deployed application or database) and has its own methods for getting there. Chapter 7 will discuss build-and-deploy issues and others in a lot more detail, as well as demonstrating a very basic but reliable build scheme based solely on a build table and a PowerShell script. Chapter 8 will discuss and demonstrate a more automated, tool-based approach to building and upgrading databases. Database versioning challenges When developers modify an existing code object during application development, they simply change the code file as appropriate and save the modified file over the top of the old version. In other words, a developer saves an object's code file to source control and then, during the development cycle, continually modifies that same file, as required. This is a natural fit with the standard source control model; the repository stores one file per object, regardless of how often we change it, and it can reproduce the exact state of that file at any given revision number. Whenever developers update their working folder they get the latest version of each object.
- 85. Chapter 3: Getting Started with Source Control for Databases Working with database objects is sometimes different. We can work with the code modules that form our database interface, such as stored procedures and views, in much the same way as any other code object. We simply write the script that creates the object if it doesn't exist, or alters it if it does, and then we continually refine that same script in source control. However, when working with tables, we need a different approach, one that accounts for the complex dependencies between these objects and, critically, the need to preserve the table's state, in the form of its business data, from one version to the next. This need to preserve state when upgrading an existing database is the main factor that distinguishes working with databases in a VCS from working with standard application files. Let's say a developer adds a field to a C# class in source control, and then later someone else changes the type of that field. It doesn't usually matter exactly how we apply those changes to the application as long as it builds correctly in its final state. For the equivalent database changes (adding a column then changing the data type), the exact manner in which we apply each change matters a great deal, as each one has implications for the business data. We have two basic approaches to versioning a database: 1. Save the current state of the schema – we store one CREATE script per object and edit each file over time, to reflect each schema modification. 2. Save the migration scripts – for each schema object, we store the string of migration scripts (ALTER <ObjectName>…), usually one change per script, required to advance the object from one defined state to the next. Most teams would prefer to use Scheme #1 during development, since it's analogous to the way they work with all other application files. However, when it comes time to deploy an upgrade, Scheme #1 implies that we can simply tear down any database object that already exists in the target database and re-create it with the version from source control. As we've just discussed, this is not possible when that target database stores business data. 85
- 86. Chapter 3: Getting Started with Source Control for Databases For this reason, many teams adopt Scheme #1 during development, and then switch to Scheme #2 after deployment of v1, stringing all of the migration scripts together in the right order, to arrive at a build script to go from v1 to v2. Ideally, however, we'd have a unified process in place that could take us through initial development and subsequent upgrade, at least as far as possible. We can use Scheme #1 for most cases, as long as we have tools we trust to automate the synchronization of the latest schema as it exists in the VCS, with the database schema of the target database. This may include, for example, use of a schema comparison tool, such as Red Gate's SQL Compare, among others. However, sometimes modifying a CREATE script is not sufficiently descriptive to allow our automation tools to handle the data correctly during the state change, and we must create a migration script to nudge the tool in the right direction. In other words, the team optimizes its general VCS and deployment processes around Scheme #1, but incorporates migration scripts for the "hard" cases. We cover migration scripts in great depth in Chapter 8. Chapter 4 will discuss each of these schemes in more detail. Throughout Chapters 3 to 6, we adopt Scheme #1 when working on a database in the development environment. Chapter 7 demonstrates a build-and-deploy scheme that assumes Scheme #1 during development, but switches to Scheme #2 after deployment. Chapter 8 demonstrates a build-and-deploy mechanism that follows Scheme #1, but incorporates migration scripts as necessary. Database source control architecture As we've discussed, our chosen source control architecture will reflect the needs of the organization, the needs of those who will use source control, and all of our ALM processes, including how we design and develop applications and databases, the manner in which we deploy them, how we assign version and build numbers, and the general strategies for working with source control on a day-to-day basis. 86
- 87. Chapter 3: Getting Started with Source Control for Databases There are many possible architectures and it's impossible to be prescriptive, but it's worth discussing general considerations, in terms of what needs to be in the VCS and how we organize those folders and files. Our goal should be to define our source control structure and the policies that govern that structure, before we actually start the first project, rather than waiting till we're in the middle of it. What needs to be in source control? The first step to getting a database into source control is to create a script file for each database object and then add each one to the VCS, in the appropriate directory. As we'll demonstrate later in the chapter, we can simply write the script to create each database object and save it as a separate file, or we can generate script files from existing database objects or diagrams. Primarily, the development team needs to ensure that the scripts in source control represent the correct and most recent versions of all database schema objects, including the base tables and indexes, as well as the code modules (views, stored procedures, triggers, and so on) that define the interface to the base tables and indexes. The devel-opers must also ensure the VCS contains the latest versions of the scripts to load any necessary static data, including both reference data and data from other sources. While, in this book, we'll focus only on database schema objects and reference data, a lot more goes into a database. To get a feel for what else is involved, it's instructive to view the permission hierarchy of SQL Server instance and database objects, as shown in Figure 3-1. As you can see, building and deploying a database is about more than just implementing the schema. Separately from the schema objects, but still a part of the VCS, we'll need scripts to create all other necessary database objects, notably the users, roles, and permis-sions that govern who can access which objects. Again, these scripts will form a separate but integral part of the overall database build-and-deploy process. 87
- 88. Chapter 3: Getting Started with Source Control for Databases Figure 3-1: The SQL Server permission hierarchy. Other script files that go into a database build Even Figure 3-1 doesn't tell the whole story. A database build might also include scripts to perform admin-istrative tasks, such as define database configurations, schedule tasks, define file groups, perform Extract- Transform-Load (ETL) operations, or any number of interrelated and interdependent tasks. We don't cover this in the book, but http://tinyurl.com/ko99fnu provides a good overview of what's required. 88
- 89. Chapter 3: Getting Started with Source Control for Databases On top of all this are the scripts necessary to define the deployment itself, such as batch files that list the order in which SQL scripts should be executed. The developer will have to work hand in hand with the DBA to provide all necessary scripts and to ensure that everyone is implementing the scripts through the change control processes that have been put into place. Of course, in addition to the database, we have the applications that rely on it. As discussed in Chapter 2, right alongside the database in the VCS will be the files that comprise the relevant application, plus other related files, such as QA resources, test and design resources, and so on. In short, only when the repository contains everything associated with the project, both database and application, are we truly gaining the benefits of the VCS audit trail. Organizing the database source files After what, the next question becomes how to store all of this in the VCS, i.e. how to determine the most appropriate folder and file structure for the database. One simple but common organizational structure for database source control is to have a project folder (named after the database) and within it a series of subfolders for each relevant type of database schema object (a subfolder for tables, one for views, and so on). For simplicity, it's a model we use in this book, though some developers prefer an even simpler structure, with just a dedicated folder for tables (which need to maintain state) and a dedicated folder for all code modules that define the database interface. Another approach to organizing files is to group schema-related objects together. For example, in a book-related application, we might subdivide the database objects and functionality into schemas such as BookOrders, Accounts, Dispatch, and so on. As a result, in our source control repository, we might create a subfolder named SchemaObjects under the database project folder. Within SchemaObjects, we would then create a subfolder for each schema (e.g. BookOrders, Accounts, Dispatch, etc.). 89
- 90. Chapter 3: Getting Started with Source Control for Databases Finally, within each schema folder, we would create subfolders such as Tables, Views, Types, and so on. All files not related to schemas, such as database-level objects, batch files, or test data scripts, would be stored in folders outside of the schema hierarchy. Simplifying the permission architecture Although it's out of scope to go into detail here, such a schema-driven design also simplifies permissions, since we simply create a database login and role that has access only to the relevant schema and the objects therein. For more information, see: http://technet.microsoft.com/en-us/library/ms191465.aspx. Again, for simplicity and because we need to focus on source control concepts, we don't make use of schemas in this book. We use only the default (dbo) schema in our Bookstore database. As a result, the way we organize the files in repository might be quite different from what you've seen elsewhere. Getting the Bookstore Database into Source Control Now that we’ve covered the basics, it's finally time to get our sample Bookstore database into source control and start developing it! In light of all the previous "planning" discussion, the following summarizes our basic approach to working with our example database and storing it in SVN: • We'll develop our database in SSMS and from there we'll script it into the VCS. • We assume online database development with a dedicated database for each developer (though the alternative model in each case will work equally well). • We don't, at this stage, assume reliance on any particular ALM toolset. 90
- 91. Chapter 3: Getting Started with Source Control for Databases • We'll subdivide our database project folder in source control according to database object type. As noted in Chapter 2, this folder forms just one part of a larger folder structure that accommodates all the project files. • We'll version the database by saving the latest CREATE script for each object (i.e. saving the current state of the schema). When we left off in Chapter 2, we'd used the VisualSVN Server Management tool to create a single-project BookstoreProject repository and three users (Jane, Fred, Barb) for our fictitious bookstore development project. As part of this process, VisualSVN Server automatically created our initial project folders (branches, tags, and trunk) at the root level. We could have just as easily taken a different approach, such as to create a single repository (often named after the organization, for example) to hold multiple projects, with each individual project containing its own branches/tags/trunk structure, or any number of other approaches. Within the trunk folder, our data architect, Jane, via her working folder, created and committed all the required high-level project folders, such as AppCode, Databases, DeployScripts, and so on. If you followed along precisely, launching the Repository Browser (TortoiseSVN | Repo-browser) from the BookstoreProject folder, in one of the users working folders, should reveal a structure that looks as shown in Figure 3-2. Figure 3-2: The current structure of the BookstoreProject repository. 91
- 92. Chapter 3: Getting Started with Source Control for Databases Don't worry if your revision numbers don't match exactly. Every commit creates a new revision in the repository. For example, if you committed the Bookstore folder as an additional commit, separate from the other folders, then the Databases folder would be at Revision 3. For our lead developer, Barb, we created a BookstoreProject folder and checked out the entire project structure, as created by Jane. For our database specialist, Fred, we created a working folder called BookstoreDatabases, and linked it to the Databases folder in the repository (https://<server>/svn/BookstoreProject/trunk/databases). Right now, though, all our developers have in their working folders is an empty subfolder called Bookstore, in which to house our Bookstore database and all its required objects and data. We're now ready to start creating all of the necessary database script files in our working folder, and committing them to the repository. Script database objects for source control When it comes to using source control for database development, one of the most important points that will emerge from our example is the need to create a script file for each database object and then add each one to source control. We can simply write the script to create each database object and save it as a separate file, or we can generate script files from existing database objects or diagrams. For a large, existing database, scripting out every object can seem a daunting process. However, most tools, including SSMS, support methods for scripting out all database objects at one time, and putting each object in its own file. Simply right-click the database in Object Explorer, point to Tasks, and then click Generate Scripts. The interface walks through the process of scripting selected objects, each to a separate file in the chosen destination, in this case the working folder. Of course, as discussed earlier, when we need to build the schema from those files, we'll have to run them in the correct order, but we'll save the details on that for Chapter 7. 92
- 93. Chapter 3: Getting Started with Source Control for Databases To get started with source controlling the Bookstore database, we'll create four T-SQL DDL scripts: a databases creation script and a script for each of the three tables. We'll add each file to our working folder, in the appropriate subfolder, and then commit these folders and files to our BookstoreProject repository. All interactions with the repository will be through the working folders of our developers, starting with Fred. Create, Add and Commit database scripts When working on database projects, we need to organize the necessary files into subfolders of our working directory, and then commit both to the repository, at which point they are available to the other source control users. As discussed earlier, we're going to subdivide our database directory according to database object type. Each time we save a new file, folder, or both, to our working folder, we then add it to the working folder and then commit to the repository, as described in Chapter 2. To get us started, within Fred's Bookstore folder, create two new subfolders, as follows: • Database, which will hold script files related to the Bookstore database as a whole, including our database creation script. • Tables, which will hold script files specific to tables. Don't worry about whether to add or commit them yet, as we'll get to that shortly, after we've created our first database script. 93
- 94. Chapter 3: Getting Started with Source Control for Databases 94 The database creation script Let's start with the script to create the Bookstore database. The database needs to be only a basic one that uses the default configuration settings. Therefore, in its very simplest form, we can open a SSMS query window, or even a text editor such as Notepad, and create a script such as that shown in Listing 3-1. USE [master] GO CREATE DATABASE [Bookstore] ; GO Listing 3-1: A basic script to create the Bookstore database. Of course, we can customize the CREATE DATABASE statement as required, to specify values for configuration settings such as data and log file locations, file sizes, file growth increments and so on. We then simply save the file as Bookstore.sql to the appropriate location in our working folder, in this case to the Database directory in the working folder (i.e. SVNFredBookstoreProjectBookstoreDatabasesBookstoreDatabase). Alternatively, we can create the database through SSMS (right-click the Databases node in Object Explorer, and select New Database), and then right-click the newly created Bookstore database and script it out (Script Database As | CREATE TO | File), choosing the same file name and path.
- 95. Chapter 3: Getting Started with Source Control for Databases Figure 3-3: Saving Bookstore.sql to the Database folder. Having created our database creation script file, we need to add this file to the working folder and then commit it to the repository. In fact, we'll commit our two new folders, and the file, in one step. We'll keep this brief, as Chapter 2 already described the Add and Commit steps; refer back there if you need more detail. Right-click on the BookstoreDatabases folder and perform the SVN Add operation (TortoiseSVN | Add), ensuring that it selects the two folders and the Bookstore.sql file. After the add, TortoiseSVN displays the blue "add" overlays for the folders and the script and the white-on-red exclamation points for the Bookstore folder and all folders higher in the hierarchy. Next, right-click again on the BookstoreDatabases folder and perform the SVN commit (remember to clear the current authentication data first, if you have all three users on the same client, as mentioned in Chapter 2). We've chosen to commit from our root folder, in order to include in the commit every-thing we've added to this folder and subfolders. 95
- 96. Chapter 3: Getting Started with Source Control for Databases Committing from the root folder We could, instead, have committed from the Database folder, or just committed the Bookstore.sql script, in which case we'd include that folder and its file in the commit, but not the Tables folder or any of its files. It's a good practice, as far as possible, to commit from the root folder to keep everything in sync. Enter a descriptive commit message along the way and, ultimately, we should see the Commit Finished dialog box, informing us of what we committed and the new revision number. Figure 3-4: Commit the Database and Tables folders and Bookstore.sql. Having committed to the repository all additions to our working folder, we should see plenty of those white-on-green checkmarks next to our folders. We can also open the Repository Browser and confirm that this structure and the database creation script are now versioned in the repository. 96
- 97. Chapter 3: Getting Started with Source Control for Databases 97 The table scripts Our next step is to create three script files for three tables, Author, Book, and AuthorBook, in the Bookstore database, one file per table. The database diagram in Figure 3-5 shows the three tables and the one-to-many relationship that exists between the Author and AuthorBook tables and between the Book and AuthorBook tables. Figure 3-5: The Author, Book, and AuthorBook tables in the Bookstore database. Listing 3-2 provides a single T-SQL script to create all three tables. To implement the one-to-many relationships, we create two FOREIGN KEY relationships on the AuthorBook table, one to Book and one to Author.
- 98. Chapter 3: Getting Started with Source Control for Databases 98 /*Author.sql*/ USE Bookstore; GO CREATE TABLE dbo.Author ( AuthorID INT IDENTITY PRIMARY KEY , FirstName NVARCHAR(30) NOT NULL , MiddleName NVARCHAR(30) NOT NULL , LastName NVARCHAR(30) NOT NULL ); GO /*Book.sql*/ USE Bookstore; GO CREATE TABLE dbo.Book ( BookID INT IDENTITY PRIMARY KEY , Title NVARCHAR(1000) NOT NULL ); GO /*AuthorBook.sql*/ USE Bookstore; GO CREATE TABLE dbo.AuthorBook ( AuthorID INT NOT NULL , BookID INT NOT NULL , CONSTRAINT pk_authorid_bookid PRIMARY KEY CLUSTERED ( AuthorID, BookID ) , CONSTRAINT fk_authorid FOREIGN KEY ( AuthorID ) REFERENCES Author ( AuthorID ) , CONSTRAINT fk_bookid FOREIGN KEY ( BookID ) REFERENCES Book ( BookID ) ); GO Listing 3-2: Creating the tables.
- 99. Chapter 3: Getting Started with Source Control for Databases To save these scripts to our working folder, we can place the code for each table in a separate query window in SSMS and save each as a separate script file (Author.sql, Book.sql, and AuthorBook.sql) to the BookstoreTables folder. Alternatively, simply run Listing 3-2 to create all three tables, and then script out each table to an individual file, as described earlier (or we could even use Tasks | Generate Scripts for the database to generate them in one go). Whichever way we choose to save the script files to the folder, we then need to add them to the working folder (right-click the BookstoreDatabases folder and select TortoiseSVN | Add) and then commit them to the repository (right-click BookstoreDatabases, select SVN Commit), exactly as described previously. The Commit Finished dialog box will verify the files committed and report the new revision number (in this case, Revision 4). Windows Explorer should now show the reassuring green ticks next to the files we just added and committed to the repository. On the server, we can confirm that everything has been properly synced with the repository by opening the Repository Browser and verifying the contents of our BookstoreProject folder, as shown in Figure 3-6. Notice that we see the same revision number (in this case, 4) for each one of the files we committed. SVN assigns the same revision number to each object that is part of the same commit operation. It increments the revision numbers for each commit, across the entire repository, without regard to folder structures or file types or to how many different projects we might have within the repository, which is why the numbers are referred to as global sequential revision numbers. 99
- 100. Chapter 3: Getting Started with Source Control for Databases Figure 3-6: Viewing our files in the Repository Browser. In other words, the BookstoreProject repository, as a whole, is at Revision 4. Within the root folder, each subfolder reflects the last revision to affect it or its contents. So, if you explore the repository, you'll see that the last revision to affect the Databases subfolder was Revision 4, whereas for the Database subfolder it is Revision 3, and for the other project folders that we created in Chapter 2, it is Revision 2. We can review in more detail what SVN committed at each revision by using the SVN log. We're not going to discuss the log in detail until the next chapter, but to take a sneak preview, right-click on the BookstoreDatabases folder, point to TortoiseSVN, and then click Show Log. 100
- 101. Chapter 3: Getting Started with Source Control for Databases 101 Figure 3-7: Viewing the log. When we first set up the repository, VisualSVN Server added the branches, tags, and trunk folder, to which it assigned 1 as the revision number. However, that's not shown here because Fred did not sync to the entire repository, only to the Databases folder and its subfolders, so he sees only those revisions. When Jane added and committed the project folders to the trunk, those were all grouped together as Revision 2. Fred then added and committed the Database and Tables folders and the Bookstore.sql file, which was marked as Revision 3. Finally, when he added the table creation scripts, SVN assigned Revision 4 to those files. Essentially, each line repre-sents a commit, i.e. a particular version in the repository. Of course, this is just how revision numbering works in SVN, but it gives a reasonable feel for how a source control system tracks revision history. Other source control systems might take a different approach to tracking revision numbers, just as they would any other feature.
- 102. Chapter 3: Getting Started with Source Control for Databases Adding test or reference data to source control Often, our database schema will include certain special tables called lookup tables, which store static or reference data, usually for the purpose of displaying data in human-readable form in the application. A typical example would be a lookup table that translates US state codes into state names. Another type of data we might add to source control is the test data necessary to facilitate integration testing, that is, if we don't instead create scripts that retrieve the data. In any case, if we choose to hard-code any test data, we can add the script files to source control, just like any database code or schema object. To illustrate the point, we'll just add some simple test data to bookstore tables, as shown in Listing 3-3. 102 USE Bookstore; GO -- test data SET IDENTITY_INSERT dbo.Book ON GO INSERT INTO Book ( BookID, Title ) VALUES ( 1, 'Huckleberry Finn' ), ( 2, 'Slaughterhouse-Five' ), ( 3, 'Beloved' ), ( 4, 'Pride and Prejudice' ), ( 5, 'Cat''s Cradle' ), ( 6, 'Frankenstein' ), ( 7, 'Atonement' ), ( 8, 'The Razor''s Edge' ), ( 9, 'The Age of Innocence' ), ( 10, 'Song of Solomon' ); SET IDENTITY_INSERT dbo.Book OFF GO SET IDENTITY_INSERT dbo.Author ON GO
- 103. Chapter 3: Getting Started with Source Control for Databases 103 INSERT INTO Author ( AuthorID, FirstName, MiddleName, LastName ) VALUES ( 1, 'Kurt', '', 'Vonnegut' ), ( 2, 'Edith', '', 'Wharton' ), ( 3, 'Mary', '', 'Shelley' ), ( 4, 'W', 'Somerset', 'Maugham' ), ( 5, 'Ian', '', 'McEwan' ), ( 6, 'Toni', '', 'Morrison' ), ( 7, 'Jane', '', 'Austen' ), ( 8, 'Mark', '', 'Twain' ); SET IDENTITY_INSERT dbo.Author OFF GO INSERT INTO AuthorBook VALUES ( 8, 1 ), ( 1, 2 ), ( 6, 3 ), ( 7, 4 ), ( 1, 5 ), ( 3, 6 ), ( 5, 7 ), ( 4, 8 ), ( 2, 9 ), ( 6, 10 ); GO Listing 3-3: Some test data. Generally speaking, every script we store in the VCS will describe a change to a single object. Of course, here we have a script that modifies the data of several objects at once. We could, certainly, split this into three separate data loading scripts, but because of the FOREIGN KEY dependencies, and because it's such a small amount of test data, it seems an unnecessary complication in this case.
- 104. Chapter 3: Getting Started with Source Control for Databases We add this script to source control in exactly the same manner as described previously. In brief: • In TortoiseSVN, create a new folder called TestData, as a subfolder of Bookstore. • Save Listing 3-3 to the TestData folder as TestData.sql. • Add the TestData folder to the working directory – this will add both the folder and the file. • Right-click BookstoreDatabases and perform a commit to the repository – again, this will commit both the TestData folder and the file (remember to include a meaningful commit message). Once again, if we right-click the BookstoreDatabases folder, and navigate TortoiseSVN | Repo-Browser, we will see that the Databases folder in the repository contains the new TestData subfolder and file. Updating the working folder Up to this point, we've demonstrated two users (Jane and Fred) accessing the repository through their working folders. We performed an add operation to register new folders and files in our working directory, and then a commit to copy those files over to the central repository. When we perform a commit, we move data in one direction, from the working folder to the repository. If another user adds new files to the repository, there is no mechanism that will alert us automatically and then auto-update our working folder. In Chapter 2, our second database developer, Barb, also performed the initial repository check-out at the same time Fred did, but she has done no further work on the bookstore project. When Barb next opens her working folder, it will look exactly as she left it, at Revision 2, reflecting none of the work that Fred committed to the repository, adding files to define the Bookstore database, plus three tables. 104
- 105. Chapter 3: Getting Started with Source Control for Databases Figure 3-8: Barb's working folder is still at Revision 2. Barb can rectify this situation easily by performing an update of her working folder. From TortoiseSVN, she simply right-clicks the BookstoreProject folder and selects SVN Update. TortoiseSVN displays an Update Finished dialog box that shows the files and folders being updated in the target folder. 105 Figure 3-9: Barb's update of her working folder.
- 106. Chapter 3: Getting Started with Source Control for Databases Although this dialog box is specific to TortoiseSVN, most source control systems will provide details about what's being updated in our working folder. This will include all updates made by other users and committed to the repository. Barb's working folder is now up to date, as shown in Figure 3-10. Figure 3-10: Barb's working folder now has the latest revision in the repository. A word of caution, though: systems such as SVN let us specify which folder to update. It will update the specified folder with any files or subfolders that have changed within that folder, but not any changes that fall outside the folder, even if related. For example, if Fred subsequently performs an update on the Database folder, then this will get any changes to Bookstore.sql, but not to any of the table files. This can sometimes cause issues, such as another user or an administrator moving a file. For example, we have a TestData.sql file, originally committed to the BookstoreTestData folder, and reflected as such in our current working folder. 106
- 107. Chapter 3: Getting Started with Source Control for Databases However, let's say another user subsequently moves the file into the Database folder. If we then update only the Database folder, the file will show up there but will also be retained in the TestData folder because we did not update that folder. If we then edit the file in the TestData folder and try to commit our changes, we'll receive an error saying that the commit operation failed and we'll have to try to perform a merge operation to resolve the problem (more on this in Chapter 6). To avoid any potential issues, it is best to update the folder at the highest level of the hierarchy. In Fred's case, that would be the BookstoreDatabases folder. Whenever we edit a file in our working folder, and commit the change to the repository in a multi-user system, we "optimistically" assume that no one else modified the file in the repository since we last performed an update. If a user forgets to perform an update for a substantial period and then blindly edits a file, the risk is high that the user is editing an invalid version of the file, because there is a more recent version of the file in the repos-itory. When the user tries to commit the change, it will reject the commit and again, the user will need to attempt to perform a merge to resolve the conflict. Whenever we're about to edit a file, we can perform an update first to make sure we have the most recent version. In general, we should update our working folder as often as necessary to ensure we're working with up-to-date files, and try to hit the right balance between too few and too many updates. The exact frequency will depend on the project and the existing state of the development effort. In Chapters 5 and 6, when we get into the topics of merging and branching, it will become clearer why striking the right balance is important. In addition, most source control systems provide a way to view details about the history of revisions made to the files and folders. TortoiseSVN provides the Repository Browser, as we saw briefly in the previous chapter. In addition, we can use the SVN log to review in detail what SVN committed at each revision. We can access the log whenever we update our working folder (there is a Show Log button at the bottom right of the Update Finished dialog), or at any other time by right-clicking a file or folder, and navigating TortoiseSVN | Show Log. 107
- 108. Chapter 3: Getting Started with Source Control for Databases 108 Summary We now have our database files and a small amount of test data in source control! Granted, this is an extremely small database, but from these files we can assemble our complete schema at any time, without worrying that the files will be lost or overwritten. Using SQL script files for database development is nothing new. Most database devel-opers use them at least to some extent, even if their organization's approach to database development is, at best, a lackadaisical one. The challenge for many database developers, in terms of embracing source control as part of their development efforts, is not so much the capabilities of the source control systems themselves, but rather a willingness on the part of the developers to embrace a different development methodology and change-management strategy. A source control solution represents an important component of effective database development but, more importantly, it is a mindset that understands the importance of scripting all database development, implementing versioned builds, and committing 100 percent to the source control system. The key to all of this, in practical terms, is to make certain that all developers script every database object to an individual script file, and commit it to source control. If someone needs to change that object, then they should update their working folder with the appro-priate script, modify it as necessary, and commit it. No one should be making changes directly to the database schema. In Chapter 4, we move on to editing, and versioning our source-controlled files.
- 109. Chapter 4: Editing and Versioning By now, you should have a good sense of what source control is and how it works, at least at a high level. We've discussed how the repository, whether distributed or centralized, is the heart of any source control solution. It manages our files and maintains a history of every change to those files since they entered the source control system. Most of our interaction with the repository is through a working folder on our local computer. From there, we can access and edit the files stored in the repository and view the history of their changes. The previous chapter discussed many of the factors that can influence the way in which a team stores database and application files within the repository; how we work with these files during team-based development; and how we assemble a particular revision in the repository into a build for deployment to a target database. In this chapter, we delve into the mechanisms and mechanics of modifying database files in the VCS during development. We'll discuss the main strategies for versioning database scripts, distinguishing between code modules that we can simply re-create and later deploy to the target database, and tables where we will need to preserve the business data during any modifications. We'll also see how the repository keeps an "audit trail" of our modifications, and how we can access and compare the various file revisions to identify the differences between them. Once again, we'll be using SVN, a centralized repository, to demonstrate the various source control concepts; see Chapters 2 and 3 to ensure you're using the same setup that we use here. However, the tasks we'll perform are common to all source control solutions, including distributed systems, although each product has its own methods for delivering specific functionality. In this book we're more concerned with possibilities than we are with absolutes. 109
- 110. Chapter 4: Editing and Versioning Database Versioning Strategies Before we begin modifying existing database files, we need to consider exactly how we want to track those changes, as this will affect what we store in the repository and how the repository can track the change history for the database as a whole. During the development phase, it's often permissible to simply tear down the existing version of a development database, and replace it with the latest revision in the VCS. However, when it comes time to deploy what's in the VCS to a target environment, upgrading the existing database, we have two major challenges to consider: • Maintaining state – How to deploy the necessary table changes to a live database in a way that preserves existing data. We can't simply recompile the database and roll it out. We often have to provide scripts to modify the schema, migrate and update data, or take other steps to protect and archive data. • Avoiding downtime – Many databases are subject to rigorous Service Level Agree-ments (SLAs) regarding their availability. We can't simply take the database down each time we want to make a change. We often have to find a way to modify the live database with minimal user disruption. Over the following sections, we'll consider how we modify both code modules and tables in our database schema, in light of these challenges. 110 Versioning code modules A common scheme when working with modules in source control is simply to modify the object code and replace the existing file with the new one. Usually, the CREATE script will start with an "if exists" clause, such that when we run it the script will drop the object, if it exists, and replace it with the new one.
- 111. Chapter 4: Editing and Versioning However, this approach can be problematic down the line, when it comes time to update an object that users may be accessing in a live system. With this approach, there is always a short period, between dropping and re-creating the object, when it will be unavailable. Also, after re-creating the object, we must reapply the appropriate permissions so that users can access it. Instead (see http://tinyurl.com/ma7u6a3) we adopt the pattern used by Alexander Karmanov, whereby our script turns the NOEXEC option ON if the object already exists, then creates a "stub" for the object (which SQL Server will compile and not run, if NOEXEC is ON), and then turns NOEXEC OFF and defines the object logic in an ALTER statement. When working with this script, we simply modify the logic in the ALTER statement, as required, and resave the file. This approach minimizes user interruption when deploying to a live database, and leaves any permissions unchanged. We'll see an example a little later in the chapter. 111 Versioning tables So far in this book, we've created only new tables, saving each one to its own .sql file in the repository. What happens, however, if we wish to modify one of those tables, such as adding a new column to the Author table, or changing a data type? We have to choose between several possible approaches to versioning our database. 1. Save the modified CREATE script – We store one file per object and edit each file as appropriate to reflect each schema modification. In this approach, we are effectively versioning the current state of the schema. 2. Save a new upgrade script – For each schema object, we store the string of upgrade scripts (one change per script) required to advance the object from one defined state to the next.
- 112. Chapter 4: Editing and Versioning 3. Save both the CREATE and the upgrade scripts – Store both the modified CREATE script and the new ALTER file to different folders, so we can either create the current schema directly or upgrade the existing schema. There are advantages and disadvantages to any of these approaches, and the choice is not necessarily straightforward. In the coming sections, we'll consider each in turn. Database versioning via an ORM Of course, you'll likely encounter other approaches too. For example, modern Object Relational Mapping (ORM) tools, such as Entity Framework, will automatically generate database changes from code changes. However, we won't discuss this sort of approach further in this book. Whichever approach we choose, we have to factor in the need to preserve data when deploying these changes to an existing database. For example, suppose we move a column from one table to another and delete a column from a third table. We need to provide the scripts necessary to alter the three table definitions. We also need a script to migrate the data from the first table to the second table, a process that might require that we store the data temporarily during the migration and update processes or even beyond, until we can verify the changes. In addition, we might need a script to archive the data in the column we're deleting from the third table. And this is just a simple example! Imagine the complexities involved in a large database with lots of data and lots of changes. On top of that, we have to consider how to make all these changes with minimal downtime and user disruption. This usually involves progressing in small, well-defined steps, each step verified with a suite of automated tests, and only switching users over to the new version once we're certain of the correctness and their impact on overall application performance (see for example http://tinyurl.com/kl8a3dd). With all these considerations in mind, we must decide which of our three approaches to take when versioning our database. So let's take a closer look at our options. 112
- 113. Chapter 4: Editing and Versioning Approach #1: Version the current state of the table Each time we modify a table, we commit the latest table creation script to source control. If we need to add a new column to a table, for example, we might update our working folder with the latest version of the table file, create the table, add the new column, regenerate the CREATE script, save it back to the Tables folder, and then commit it to the repository. In other words, we save each CREATE script to the repository, one table per file, and change each file as necessary to reflect each schema modification. Any user performing an update to their working folder will simply get the latest version of the table's creation script. There are obvious advantages to this scheme for the development team. It makes it easy for each developer to create a new environment, with the current state of the database schema, by simply running each CREATE script in dependency order. It's an approach that works naturally with a VCS, making it very easy to perform a diff (see later) to find out exactly how each object changed between revisions; and this, in turn, makes it relatively easy to perform a merge into a separate branch of the development project. Another advantage with this approach is that it makes it easy to audit the change history and quickly identify who added or removed a line of code, and when. SVN, for example, offers this capability via the blame command, often used to blame and shame the "developer who broke the build." Of course, there is an apparent underlying assumption to this approach that we can simply drop any existing table and re-create it, in order to get the latest version. While this may be possible in the development environment, we cannot extend this approach to any environment where we must retain the data. This is why it's reasonably common for a development team to adopt this "version the current state of the table" model during development, but then adopt the "upgrade script" approach once a database has been built and deployed (we use this approach in Chapter 7). 113
- 114. Chapter 4: Editing and Versioning Approach #2: Version the upgrade scripts In this approach, we store in the VCS an upgrade script to describe each schema modifi-cation, with one change (i.e. one ALTER command) per script file. In this case, every script is an upgrade script (also called a migration script). The team create a build script to CREATE, ALTER and DROP objects as required, in the right order, in order to advance a database from one defined database state, such as an empty database or a specific database build, to the next defined state or build. In other words, when developers update their working folders, they don't retrieve the latest versions of each object, but instead a series of scripts that they must run, in the right order, to progress the database from one state to the next. One change per script versus all changes in one script Creating one script file per ALTER command is generally considered best practice because it provides for flexible deployment and management strategies and more granular control over your operations. If we need to change the order in which we apply the modifications, we simply tweak the build script. However, another method sometimes used is to create one script file that includes all the ALTER commands. Certain automation tools, such as Red Gate's SQL Source Control Migrations, take this approach. Of course, if we need to change the order, we have to edit the T-SQL script manually. We adopt the one-script-per-upgrade approach in this book. This approach works across all environments, from development through to production. To build a new database, we generate a creation script that includes all the migration scripts, applied in the correct order (and we must be certain to include all the scripts, which is not always an easy task). To upgrade an existing database, we run all the upgrade scripts not previously applied. This technique relies on the team knowing exactly which scripts they previously applied to an environment, and then applying those upgrade scripts in source control that they've yet to run. It can also mean keeping track of any scripts we don't want to run. For example, let's say in one revision someone deleted a table, but in a later revision we spot it as a mistake and restore the table. We would not want to run the script that deleted the table as part of a deployment, or we'd risk losing data. 114
- 115. Chapter 4: Editing and Versioning The problem of version drift It is vital that nobody modifies a database directly, outside of the source control process. If a target database has drifted from the expected starting state, then running the upgrade scripts may not work as expected. Chapter 7 discusses version drift in more detail. In a basic implementation of this approach, we might save our upgrade scripts to a specific folder associated with the build to which they apply, with each file named according to the action, the date, and an "index number" indicating the sequence. In our bookstore project, it could look something like Figure 4-1, where Fred has created an UpdateScripts subfolder in his working folder, and then within that several version-specific folders containing the scripts for each version. Figure 4-1: Adding update scripts to a Windows Explorer folder. 115
- 116. Chapter 4: Editing and Versioning Of course, this is only an illustration of the types of update files we might create. Our build mechanism would also have to track the individual update operations in some way, through version numbers, timestamps, or some other mechanism (we get to this in Chapters 7 and 8). There are several advantages to the upgrade scripts approach. As noted, it works for development as well as deployment environments. It provides an audit trail of every modification. It's repeatable; as long as we apply every upgrade script to every environment, in the right order, we will create the same database each time. Since every script is a migration script, we can exert precise granular control over exactly how we wish an upgrade to proceed. If we need to perform a data load or other manipulation in order to support a schema change, we simply add it to the indexed list of files. There are some obvious downsides too. Since every change is a separate script, we can't simply do a diff to compare the current version of an object with a previous version, to work out what changed. If someone's change introduces a problem, we may need to search through all recent upgrade scripts to find the offending change. Likewise, it's much harder to work out what the schema as a whole looked like at any point in history. It's a method best suited to "one developer at a time," to avoid the likelihood that two devel-opers submit a script with the same index and then need to resolve the conflict. Since every change is a new script, it's also much harder to know if one change might conflict with another. Again, we often resort to searching through all recent upgrade scripts to reassure ourselves that a proposed change is compatible. For similar reasons, branching and merging are hard in this scheme, and developers often avoid them as far as possible. To make this approach work, we need some automated way of storing and applying the upgrade scripts in a consistent fashion. Chapter 7 of this book will demonstrate a simple approach that relies on build tables and a PowerShell script to run the upgrade scripts in the correct order. Alexander Karmanov demonstrates a more rigorous and automated approach in his incremental database development and deployment framework (http://tinyurl.com/ pta9bcs). He associates every set of upgrade scripts with a database build number. For 116
- 117. Chapter 4: Editing and Versioning example, when we first create an empty database, we associate it with a build number (say, "0.0.0.1"), saved in the deployed database in an extended property (or a table valued function). Thereafter every set of changes increments the build number, so the upgrade scripts folder for build 0.0.0.2 contains the scripts necessary to progress the database from build 0.0.0.1 to 0.0.0.2. A configuration file, called a change log, stored in source control, associates each build number with the correct set of upgrade scripts, and establishes the order in which we must apply them. If we always know the build of the database that currently exists in an environment, and we have a change log, then we use a deployment tool (in this case a VB script, but equally well a PowerShell script) that will always apply the correct set of upgrade scripts to advance from one build to any other. Finally, several tools help support the upgrade script approach, such as dbdeploy (http://dbdeploy.com/) or migratordotnet (http://tinyurl.com/pc8hqjh). Approach #3: Save both the modified CREATE script and the upgrade script There are many advantages to having the latest state of the database schema in source control, and it's a natural fit for the core source control processes. However, it relies on schema synchronization scripts to apply a set of changes to an existing database, and there will always be certain types of changes with which these tools will be unable to cope. With the upgrade scripts approach, we exert granular control over our database upgrades, but it can be painstaking and slow, and works against some of the natural source control processes. In essence, this approach is optimized to deal with the edge cases, i.e. those changes that required very strict control, rather than the common cases, those changes we could apply by simply comparing the source and target schemas and having a tool generate a script to synchronize them. 117
- 118. Chapter 4: Editing and Versioning Some teams opt for a "bit of both" approach, storing both the latest creation scripts and, in a separate source control directory, the upgrade scripts. It sounds tempting, but without the right tools and a strict control process, there be dragons here. What if someone updates one but not the other? Which one is right? What if creating the object according to the latest CREATE script doesn't give the same result as running the specified set of ALTER scripts on the existing build? However, tools exist that allows us to adopt this "both ways" approach but remove some of the associated danger, and move us more towards the best of both worlds. Most development teams would like to optimize their source control processes around Approach #1. However, because we can't drop and re-create the tables when upgrading an existing database, we use schema comparison tools instead to compare the source database as described by the script in the working folder, to the target database as it exists in the environment, and generate automatically the upgrade script necessary to bring them into alignment. We can even extend this approach to upgrading existing databases in environments such as Test, Staging and even Production. Of course, such an approach requires extreme care. While we can rely on a synchronization tool for 95% of database modifications, there will always be the other 5% that require manual intervention. For example, there is no way for a tool to know that we renamed a table, rather than dropped one table and created a new one. In such cases, we still need to write a migration script to specify exactly how we wish the synchronization to proceed. Likewise, we'll also need migration scripts to perform any data load and manipulation operations that might be necessary to incorporate our schema changes. An example might be trying to deploy a change that adds a NOT NULL constraint to a column that contains existing data, including NULL values. In order to deploy such a change successfully, we'd need to write a migration script that loaded a default value to any NULL columns in the table, stored that script in source control, and incorporated it in the deployment process. 118
- 119. Chapter 4: Editing and Versioning Migration scripts in SQL Source Control Chapter 8 will review a technique that allows us to adopt this "version the current state" method but also store migration scripts in source control and incorporate them in a database deployment, as required. Source Control Versioning Mechanisms Regardless of how we choose to version the objects in our database, we can rest assured that our VCS will store all of these files, and preserve the history of any changes we make to them over time. There is a good reason why we've referred interchangeably to a source control system as a version control system or VCS. As soon as we commit a file or folder, the source control repository maintains a full history of revisions to that object. It means, for example, that we have access to every file as it existed at every version in the repos-itory (although, as discussed, if we're versioning only upgrade scripts, there is notionally 119 only one version of every file). To track all these changes and provide access to their histories, source control systems assign some form of revision number to each set of changes that comprise a single commit. Some systems refer to these revision numbers as changeset numbers. Most source control systems will also provide a way to view revision history, revealing what changed, who made the change, and when, for each version in the repository. Version numbering and history Different source control systems approach versioning in different ways. SVN's versioning takes the form of global sequential revision numbers, an incremental numbering system. It assigns a number to each changeset (committed revision), incrementing the number for each new revision. The more recent the revision, the higher the revision number.
- 120. Chapter 4: Editing and Versioning As discussed in Chapter 2, SVN tracks changes within every file, on a file-by-file basis. In contrast, a distributed system such as Git uses Git hashes to ensure that it can identify changesets uniquely across all files, across multiple peer repositories. Among VCSs that use an incremental numbering system, each will likely differ in implementation. Some will apply sequential numbering at the folder or file level, so the sequence of revision numbers for a specific folder, for example, will be "unbroken" by changes made to other folders. Other repositories will increment revision numbers across the entire repository, so any change to any structure in the repository will increment the revision number. Regardless of implementation details, every VCS can identify every changeset, such that it can present a full change history for each file and structure, and can reproduce the repository as it existed at any particular revision. Each time we commit a set of changes, the VCS will increment the revision number and we'll subsequently have access to both versions of the changed file and be able to compare different versions (most source control systems don't store both versions, but rather the "delta" between the two versions, as we'll discuss shortly). Older source control systems maintained histories for files only, which could result in a fair amount of confusion if someone changed the directory structure itself. These days many source control systems, including SVN, store the directory structure as well as the files, thus maintaining a history of changes to the directory structure resulting from moving, renaming, or deleting folders. Git and Mercurial work slightly differently in that they don't store the directory, only the file path, so we can't commit an empty folder. However, the net effect to users is the same, and the working directory's folder structure is as they expect. 120
- 121. Chapter 4: Editing and Versioning 121 Deltas and diffs Although, from a user's perspective, the repository maintains each version of every file, that's not how it works internally. Given all the data that a repository stores, particularly if we consider that every version of every file is accessible through the source control system, a repository has the potential to grow enormous, up to many terabytes if it includes lots of binary files. Even if it stores mostly text files, we still have the potential for extremely large repositories when we take into the account the number of files and the number of revisions for each file. Add to that the metadata that tracks information about each version, and you could end up with a repository so large that backup and maintenance operations become enormous tasks and performance starts to degrade. A modern VCS doesn't actually save each version, but instead saves the differences (deltas) between versions so that it can reassemble a particular file version. A delta is a record of the differences between two files or trees. The VCS compares one or more of these records to a baseline version to construct the requested version. The evolution of deltas When source control systems first began using deltas to maintain change histories, the deltas applied only to the files themselves, or more precisely, to the text files. But as source control systems have evolved, delta calculations have incorporated binary files, as well as tree structures. In other words, many modern source control systems can now record every change that occurs within the repository. How deltas work Once again, different source control systems calculate deltas in different ways. For instance, solutions that can calculate deltas only on text files tend to do line-by-line comparisons between those files to arrive at the delta. On the other hand, systems that support binary deltas use a byte-oriented algorithm to calculate the differences between files.
- 122. Chapter 4: Editing and Versioning Regardless of the exact approach, the end result is that the repository stores each file version as either the entire file or a delta that specifies what changed in the file since the previous revision. For example, let's say that in an idle moment a Hunter S. Thomson fan among our developers wants to add the book "Hell's Angels" to the test data. He modifies the TestData.sql file appropriately, and commits the change (this will create a new revision in the repository if you go ahead and do it). We can represent the associated delta that the repository stores for the TestData.sql file as follows (assuming the original data load was a multi-row INSERT INTO…VALUES rather than separate INSERTs for each row): 122 + ,(9, 'Hunter', 'S', 'Thomson') + ,(11, 'Hell''s Angels') + ,(9, 11) The "+" signs represent the fact that we've added three rows to the file. Of course, this is only a rough representation of the delta, as it must also take into account where to insert the data, along with other information, but it should give you a sense of how deltas work and how they can save space in the repository. Deltas work in much the same way for changes to the directory tree. For example, if we were to rename the TestData folder to Data, we would represent the delta as follows: - /trunk/BookstoreProject/Databases/BookstoreDB/TestData + /trunk/BookstoreProject/Databases/BookstoreDB/Data This looks like two actions, deleting the TestData folder and adding the Data folder. However, internally, if we rename or move a file or folder, the SVN repository will not store a new copy, but rather store the delta, in this case the file path.
- 123. Chapter 4: Editing and Versioning "Move" versus "delete then add" Some systems treat a move or rename, whether to a file or folder, as a deletion from one location and an addition to another location, and will therefore actually store two copies. Even so, the final result is the same from the user perspective. When a repository stores deltas for each revision rather than the actual files, the source control system must often "build" a file when we request a particular version, starting with the last stored file and applying the deltas in the order they were committed. For example, let's say our TestData.sql file entered the repository as Revision 7, and the latest version is at Revision 15, when we added the Hunter S. Thomson book. The VCS may retrieve the original stored file (Revision 7) and then apply any subsequent deltas to reconstruct the latest version. Not surprisingly, as the number of revisions grows, so too does the number of deltas. In fact, sometimes that number gets to be so great that performance degrades significantly when we try to retrieve a version. For example, Revision 30 might require the repository to apply 21 deltas to the first version! For this reason, a source control system might periodically store the full version of a file. For example, it might, depending on how it is configured, store every tenth file as its full version. As a result, it need never apply more than nine deltas to a single version. Some VCSs, such as Microsoft's Team Foundation Server (TFS), use reverse deltas to store different versions of a file. In that case, the repository stores the most recent version in its entirety and applies reverse deltas to rebuild earlier versions. This makes sense when we consider that most users are interested in the most recent version. 123
- 124. Chapter 4: Editing and Versioning The exact method that the VCS uses to store and maintain versions in a repository is not nearly as important as its ability to retrieve specific versions as we request them. In addition, the VCS must provide some mechanism to allow users to easily view infor-mation about different versions and compare them in order to see their differences. And 124 that's where the diff comes in. Performing diffs A delta is a recording of the differences between file versions, and the repository uses it to construct a file when we request a particular version. In other words, a delta is systemic and integral to the repository storage process. Related in concept, but distinct, is a diff, which is a function available to allow users to compare files visually. Although some of the mechanisms used to calculate deltas and calculate diffs may be the same, the functions themselves are different. In essence, a diff is a visual tool, separate from the repository, which provides a line-by- line comparison between two files. By performing a diff, we can see exactly what changed between one version of a file and another (or even compare two different files). For example, we can perform a diff between Revision 20 of a file and Revision 30, and see exactly what is different. This process can be particularly useful when trying to trouble-shoot a database issue and needing to find out, for example, when someone changed a column from nullable to non-nullable. File encoding If creating script files in SQL Server Management Studio and saving them to SVN, you might run into file encoding issues when trying to perform a diff between file versions. In some cases, SVN incorrectly infers that such a file is not text but is binary. If this occurs, right-click the file, point to TortoiseSVN, and then click Properties. In the Properties dialog box, check whether a mime-type property is listed. If so, delete the property and click OK. You'll then have to commit your change to source control. Check the TortoiseSVN documentation for more information.
- 125. Chapter 4: Editing and Versioning Source control systems let us run a variety of diff operations in order to compare specific file and folder revisions. The types of diffs we can perform vary from one system to the next, so refer to the product documentation for specifics. That said, some common types of diff allow us to compare: • the working folder to the repository • any two revisions of the same file, in the repository • any two revisions of the same folder, in the repository to produce a unified diff that shows, in a single view, the changes to all files in the folder • two file revisions to determine not only their differences but also to identify "blame," that is, to include details about who revised which lines of code, and when • specified revisions of two different files • the trunk to a branch or two branches • folder hierarchies • revisions • the working folder or the repository to the index (the staging area between the working folder and repository in products like Git). When we use a GUI to perform the diff, we see a graphical side-by-side comparison of the two file versions, as you'll see later in this chapter. If a source control system doesn't support a specific diff operation, you'll often find a third-party tool that you can use to augment the built-in features. For example, TortoiseSVN doesn't include built-in tools to view the differences between folder hierarchies, but tools such as WinMerge or Beyond Compare will. 125
- 126. Chapter 4: Editing and Versioning Modifying the Bookstore Database It's time once again to convert theory into practice, and see how database versioning and investigating that version history in the repository, work in SVN. We'll walk through some examples of modifying some code modules and tables in our Bookstore database, as follows: • create a new view, AuthorsBooks, to view a list of books and their authors • alter the Book table to add a new column, YearOfPublication • update the test data to populate this new column • alter the AuthorsBooks view to return the new YearOfPublication column. In each case, we'll adopt Approach #1, versioning the latest version of the creation script for the object. We'll edit these files from within SSMS "optimistically," updating our working folder just before the change to ensure we have the latest versions, but then making the change and committing it to the repository, hoping that no other user changed that file in the meantime. If they did, we'll need to resolve any potential conflict with a merge operation (see Chapter 6). Some centralized source control systems, including SVN, still allow us to lock a file explicitly, to prevent someone else trying to edit it at the same time as us. In SVN, to place a lock on a file, right-click the file in the working folder, and navigate TortoiseSVN | Get lock. When we commit the change, SVN will release the lock automatically (or we can release the lock manually at any point). This pessimistic approach can occasionally be useful for very high-priority changes, but the use of locks should be the exception not the rule in most teams, and we won't cover this approach in detail. Third-generation, distributed source control systems do not even support such a locking mechanism. Once we've made all our changes, we'll see how the repository allows us to view them, who made them, and when, and how the current file versions differ from the previous ones. 126
- 127. Chapter 4: Editing and Versioning 127 Create a new view So far, we've only created three tables, plus some test data, so our first task is going to be to create a code module, namely a view. Listing 4-1 creates the AuthorsBooks view. As discussed earlier, the script is such that any modification further down the line, when this view is "live" and we need to change its structure, incurs minimal interruption to end-users. USE Bookstore; GO IF OBJECT_ID('dbo.AuthorsBooks') IS NOT NULL SET NOEXEC ON -- if the object exists, SQL Server will compile, but not run, the -- subsequent CREATE VIEW batch GO PRINT 'dbo.AuthorsBooks: creating initial stub'; GO CREATE VIEW dbo.AuthorsBooks AS SELECT 'A code stub, which will be replaced by an ALTER Statement' AS [stub] GO SET NOEXEC OFF -- the ALTER VIEW batch will always run GO PRINT 'dbo.AuthorsBooks: view code is updated'; GO ALTER VIEW AuthorsBooks AS SELECT ( a.FirstName + ' ' + a.MiddleName + ' ' + a.LastName ) AS FullName , b.Title FROM Author a INNER JOIN AuthorBook ab ON a.AuthorID = ab.AuthorID INNER JOIN Book b ON ab.BookID = b.BookID; GO PRINT 'dbo.AuthorsBooks: view creation script finished'; Listing 4-1: Creating the AuthorsBooks view in the Bookstore database.
- 128. Chapter 4: Editing and Versioning In Fred's working folder, create a new folder called Views, as a child of Bookstore, and then save Listing 4-1 to this folder, as AuthorsBooks.sql. Right-click on the BookstoreDatabases folder and Add the new folder and file to the working folder, then right-click to reset the authentication data if necessary, then right-click again and commit the changes to the repository. 128 Alter the Book table Let's now assume that a little later we receive a request to store the year of publication for each book. We need to add a new column to our Book table. Listing 4-2 shows the ALTER TABLE statement. ALTER TABLE dbo.Book ADD YearOfPublication INT; Listing 4-2: Altering the Book table in the Bookstore database. For the sake of simplicity, we've assumed we'll only ever want to store the year, and we've used an INT data type for the YearOfPublication column. In reality, we'd be more likely to use a DATETIME data type, and store dates that are more accurate. As discussed in the previous sections, we have the choice to save this to source control as a new upgrade script, or we can simply regenerate the new CREATE statement after adding the column, and save to the repository a modified version of the Book.sql creation script. In this example, we'll modify the existing Book.sql file. In my case, I altered the Book table in my sandbox database, as per Listing 4-2 and then used Script Table As | CREATE To | File to replace the existing Book.sql file in my working folder (having first performed an Update of my working folder, to ensure no one had altered the file in the meantime).
- 129. Chapter 4: Editing and Versioning As a result of the modification, Fred's working folder now contains changes that we need to commit to the repository, as shown in Figure 4-2. Figure 4-2: Edited file waiting to be committed to the repository. Finally, right-click on BookstoreDatabases and commit the change (be sure to add a comment to the Commit dialog box). 129 Update the test data We may also need to generate some test data for the new YearOfPublication column. UPDATE [dbo].[Book] SET [YearOfPublication] = 1884 WHERE BookID = 1; GO UPDATE [dbo].[Book] SET [YearOfPublication] = 1969 WHERE BookID = 2; GO
- 130. Chapter 4: Editing and Versioning 130 UPDATE [dbo].[Book] SET [YearOfPublication] = 1987 WHERE BookID = 3; GO UPDATE [dbo].[Book] SET [YearOfPublication] = 1813 WHERE BookID = 4; GO UPDATE [dbo].[Book] SET [YearOfPublication] = 1963 WHERE BookID = 5; GO UPDATE [dbo].[Book] SET [YearOfPublication] = 1818 WHERE BookID = 6; GO UPDATE [dbo].[Book] SET [YearOfPublication] = 2001 WHERE BookID = 7; GO UPDATE [dbo].[Book] SET [YearOfPublication] = 1944 WHERE BookID = 8; GO UPDATE [dbo].[Book] SET [YearOfPublication] = 1920 WHERE BookID = 9; GO UPDATE [dbo].[Book] SET [YearOfPublication] = 1977 WHERE BookID = 10; GO Listing 4-3: Adding new test data to the Book table. Again, we have the option to save Listing 4-3 as a new upgrade script, or we can modify the existing TestData.sql file, which is the option we'll choose here. In order to do this, first run the code in Listing 4-3 to add the new data to your development database, then follow the steps below.
- 131. Chapter 4: Editing and Versioning 1. In SSMS, right-click the Bookstore in Object Explorer, point to Tasks, and then click 131 Generate Scripts. 2. In the Generate and Publish Scripts wizard, click Next if the Introduction page appears. 3. On the Choose Objects page, select the option Select specific database objects, select the Tables check box so that all tables are selected, and then click Next. 4. On the Set scripting options page, click the Advanced button and in the Advanced Scripting Options dialog box, navigate to the Type of data to script option in the General category and select Data only. Click OK to close the dialog box. 5. Back on the Set scripting options page, save the script to the TestData subfolder in the working folder, replacing the existing file (or save it to a query window to review it first). Then click Next. 6. On the Summary page, review your chosen options and then click Next. The Save or Publish Scripts page will appear and display the status of the script generation process. When that has completed, click Finish. Once again, commit the change to the repository, as for the Book table. Alter the view Finally, we want to modify our view definition to return the new YearOfPublication column. This time, we can simply update the AuthorsBooks.sql file in our working folder by opening it in SSMS and modifying the ALTER VIEW statement, as highlighted in Listing 4-4. After you've made the change, save the file back to Fred's working folder.
- 132. Chapter 4: Editing and Versioning 132 ALTER VIEW AuthorsBooks AS SELECT ( a.FirstName + ' ' + a.MiddleName + ' ' + a.LastName ) AS FullName , b.Title, b.YearOfPublication FROM Author a INNER JOIN AuthorBook ab ON a.AuthorID = ab.AuthorID INNER JOIN Book b ON ab.BookID = b.BookID; Listing 4-4: Modifying the AuthorsBooks view to return YearOfPublication. Finally, commit the change to the repository, as for all the other changes. Investigating Bookstore Revision History Now that we've made changes to our Bookstore database, we can view the history of these revisions and compare what changed between one revision and the next. Although SVN approaches versioning differently from other source control systems, the concepts behind versioning are consistent among the various products: the repository can identify each changeset uniquely, allowing us access to the histories associated with every file and folder. SVN maintains revision numbers at the repository level. Whenever we commit a set of changes, which may involve, for example, moving folders, editing files, adding files or deleting files or folders, SVN assigns a revision number to the changeset. It increments the number by one with each commit, based on the last assigned revision number, which is the last changeset we committed to the repository. Figure 4-4 depicts the revision history as it might look for a typical set of database revisions. In fact, it's the revision history for our Bookstore database, specifically the view from the SVN log. To access the log, right-click the BookstoreDatabases folder,
- 133. Chapter 4: Editing and Versioning point to TortoiseSVN, and click Show Log. This launches the Log Messages dialog box, which lists the most recent revisions associated with the file or folder from which we launched the log. If we launch the log from a folder, the information includes revisions to any of the files or subfolders within that folder. Of course, a different VCS will display this sort of information differently, but it will always be available in some form. For example, in Git and Mercurial, we would see all changesets regardless of the folder we used to launch the log, rather than only the changesets "from here down." Figure 4-3: Viewing basic revision history in the log. 133
- 134. Chapter 4: Editing and Versioning For every commit to the repository, SVN increments the revision number. For each revision, we see: • The revision number – in this case, we see Revisions 2 through 9. • We do not see Revision 1, since this change, which established the initial repository structure, is out of the scope of the repository's Databases folder, which is the folder to which we linked Fred's BookstoreDatabases folder. • If we launch the log from the TestData folder, we see only Revisions 8 and 5. • Other source control systems may show all changesets, regardless of from where we launch the log. • A symbol that indicates the action associated with the commit. • For Revisions 2 through 6, each action adds either a new file or folder. • For Revisions 7 through 9, each action modifies an existing file. • The author of the change – here, we see Jane associated with Revision 2 and Fred with all other revisions. • A timestamp indicating when the author made the change. • The associated Commit message. If we select a particular revision, the lower pane reveals more details about the actions associated with that revision. The revision number not only identifies a specific set of changes, but we can also use it to access a snapshot of the repository at a specific point in time. When we open the Repos-itory Browser (shown in Figure 4-4), we're viewing the most recent revisions of all the files and folders stored in the repository. Collectively, these files form the project's current version, which SVN refers to as the head (notice the HEAD button in the upper-right corner). You'll find that other source control solutions use the term head in different ways. In Git, for example, the term refers to the location of the working directory. 134
- 135. Chapter 4: Editing and Versioning When we select the Databases folder, it will show the latest revision for the project as a whole (Revision 9), since that was the last revision to affect the contents of this folder. In Figure 4-4, we've drilled down to the Bookstore subfolder, and we can see the latest revisions to affect each folder and its contents. Figure 4-4: Viewing the revision number associated with the Bookstore folder. SVN will also let us view the repository as it existed at a previous version. Click the HEAD button, and in the Select Revision dialog box we can specify the revision we would like the repository to display. In this case, we'll go back in time to see the repository as it existed when we committed Revision 5, as shown in Figure 4-5. Figure 4-5: Viewing the repository at Revision 5. 135
- 136. Chapter 4: Editing and Versioning The Bookstore folder no longer contains the Views subfolder, which we did not commit till Revision 6. It does contain the TestData subfolder, and the TestData.sql file, since these formed the Commit for Revision 5. However, the TestData.sql file will not have data for the YearOfPublication column, nor will the Book.sql file include this column. Although we're viewing the repository as it appeared at Revision 5, the repository still retains all versions. Viewing by revision number acts simply as a filter that excludes any changes that occurred after the specified revision number; it does not delete any data. We can return to the head version at any time. 136 Diffing files in Bookstore In the following sections we'll look at some of the types of diff operations we can perform using TortoiseSVN, but most source control systems support similar operations. Diffing non-text files In many source control systems, diffs are limited to text files. Even systems that support binary deltas do not necessarily support binary diffs. In such cases, users will need external tools to perform certain types of diffs. SVN provides the built-in TortoiseIDiff to compare image files, though we won't cover its use here. As discussed earlier, if you need more extensive features than are available in TortoiseSVN, you can turn to third-party tools such as WinMerge and Beyond Compare. The important point to take out of all this is not how SVN works, but rather how a source control system provides a number of ways to track information about a file throughout its history. That is what source control is all about.
- 137. Chapter 4: Editing and Versioning 137 Diffs from TortoiseSVN The most basic diff operation in TortoiseSVN, and the easiest to perform, is to compare a current version of a file in the working folder with the version that preceded it. Note that, with TortoiseSVN, we can diff only files, not folders, meaning that we can't, for example, perform a diff between two folder hierarchies. Diff with previous version To compare the last two versions of the AuthorsBooks.sql file, right-click the file in Fred's working folder, point to TortoiseSVN, and then click Diff with previous version. This launches TortoiseMerge, with the two versions of the file opened side by side, as shown in Figure 4-6. Figure 4-6: Viewing a diff between file versions. The version on the right is the current version (the working copy, which is Revision 9) and the version on the left is the file as it existed at the previous revision (Revision 8). TortoiseMerge performs a line-by-line comparison, highlighting the differences and using plus and minus signs to indicate what changed between the two versions.
- 138. Chapter 4: Editing and Versioning In this case, we are comparing two commits. However, our current working copy may contain uncommitted changes, in which case we'll get to see the differences between the file version we want to commit and the file version with which we started. Remember that this is only a local diff; we'll only see the changes we made locally. If, in the meantime, someone else changed the revision we started with, we won't know about that until we update the working folder. To demonstrate this, let's make just a small change to the AuthorsBooks.sql in the working folder. Move the comments on Lines 5 and 6 down two lines, to after the GO command, and modify the text. Save this change to the working folder but don't commit the change. In Fred's working folder, right-click the file in TortoiseSVN (it will be marked with a red exclamation mark), and navigate to TortoiseSVN. We see two options, Diff and Diff with previous version. Figure 4-9 shows the result of Diff. This time, we're comparing our working copy (uncommitted) with the last committed version. Figure 4-7: Comparing the working copy of AuthorsBooks.sql with the committed version. 138
- 139. Chapter 4: Editing and Versioning If we were to pick the Diff with previous version, we'd see Revision 8 on the left-hand side, and the diff would also highlight the addition of the YearOfPublication column. If we're happy with our changes, we can commit and create a new revision (Revision 10). If we spot something wrong, we can right-click the file and navigate to TortoiseSVN | Revert. The subsequent Revert dialog box asks us to confirm that we want to roll back our changes, which will return our working folder to its original state, and our working copy of the file will be back in sync with the most recent version in the repository (Revision 9). Again, however, this does not necessarily mean the working folder is still coordinated with the repository; other users might have committed their changes since our last update. 139 Diff with URL TortoiseSVN also offers a Diff with URL function. We use it primarily to compare files and folders in a branch with the same file or folder in trunk, which we'll cover in the next chapter. However, we can also use it to compare a file in the working folder with a version of a file in the repository. To try it out, hold down the Shift key, right-click the AuthorsBooks.sql file and navigate TortoiseSVN | Diff with URL. When the Diff with URL dialog box appears (shown in Figure 4-8), choose Revision 6 (the revision where we created the view initially) and click OK. Figure 4-8: Selecting a revision for viewing differences.
- 140. Chapter 4: Editing and Versioning You'll then be presented with a side-by-side comparison of the working copy of the file, as shown in Figure 4-9. In the left window is our working copy and on the right is Revision 6, as it's stored in the repository. Figure 4-9: Diff with URL, comparing a working copy of a file to the repository version. The repository represents the one source of truth, so the plus signs on the repository copy (Revision 6) show what lines we must add to the working copy to bring it into sync with the repository copy, and the minus signs on the working copy indicate what lines must be removed from that copy to bring it into sync with the repository copy. In this case, we don't want to make any changes, but it does provide us with a way to view what has changed over time and perhaps better understand why those changes were made. We can also launch Diff with URL from a folder. For example, launch it from Fred's Bookstore folder and compare to Revision 6. We see a File patches box containing all the files that changed within that folder between the two revisions, and we can click on each one to compare the individual file versions. Figure 4-10 compares the working copy of TestData.sql (on the right) with Revision 6 in the repository (on the left). 140
- 141. Chapter 4: Editing and Versioning Figure 4-10: A diff with URL from the Bookstore folder. 141 Diff with blame Many source control systems support a feature called blame, which can identify who changed a line of code, and when. In SVN, the blame feature works in conjunction with a type of diff that shows you how a file has changed, and who made those changes. To perform a blame, right-click a file in Fred's working folder, such as Book.sql, select TortoiseSVN | Blame. This launches the blame dialog box, shown in Figure 4-11. In this case, the dialog box refers specifically to the file in Fred's working folder. Notice that, in the From revision box, I've added the 4, which is the revision number assigned to the file when it was first committed to the repository. The current revision is 7, which was populated automatically.
- 142. Chapter 4: Editing and Versioning 142 Figure 4-11: Performing a blame and a diff. Make sure the Use text viewer to view blames option is deselected so we get the GUI in TortoiseBlame, as shown in Figure 4-12. Figure 4-12: Performing a diff and blame on the Book.sql file.
- 143. Chapter 4: Editing and Versioning Notice that Fred's name is listed next to each line of code, along with the applicable revision number for each line he added or modified. Hovering over the name will also give us the timestamp for that commit and the associated commit message. If we right-click the user's name, we can select further options. For example, we can open the log, blame the previous revision, or launch TortoiseMerge. 143 Diffs in the SVN log We can also perform some diffs via the SVN log. If we launch the log from a file, such as the AuthorsBooks.sql file, we can then right-click a revision within the log, and select the appropriate option from the context menu, as shown in Figure 4-13. Figure 4-13: Performing a diff on a revision in the log.
- 144. Chapter 4: Editing and Versioning For example, to find out simply how a file changed between any two revisions in the repository, click on the first revision and then Ctrl-click on the revision to which you wish to compare. With both selected, right-click and select Compare revisions from the context menu. Alternatively, we can also choose to see the blame revisions, as well as a "unified diff," which is like a summary of all the changes in a single file. If we launch the log from a folder then we can perform similar diff operations, but in this case we get to see a Changed files list of all the files that changed between the two selected revisions, and we can click on each to compare them side by side, or perform a unified diff to see all changes to all files in a single view. We'll see further examples in the next chapter, where we'll perform a diff between a project branch or a specific tag, and the trunk. What to Do When Things Go Wrong: Revert Occasionally, when working with files and editing them, we will err and make an unintended change, or a change we regret. If we spot our mistake early, after saving the change to our working folder but before committing it the repository, it is fairly easy to roll back our changes with the Revert command, as we discussed earlier in the chapter. However, if we commit the change to the repository and then realize we just stuck something out there that our coworkers will be laughing about for the next five years, what do we do? Not to worry. Source control once again comes to our rescue. Suppose, for example, Fred is editing the AuthorsBooks.sql view and adds the ISBN column to the SELECT list, in the mistaken assumption that the column exists in the Book table. It is such a quick and easy edit, that he uses Notepad to add the column to the script, without verifying his change. 144
- 145. Chapter 4: Editing and Versioning A source control system is, for all practical purposes, indifferent to such mistakes, and lets Fred commit the file to the repository. The database is then deployed to QA, where testers quickly realize that all those failing queries are the result of Fred's flawed view. After receiving an Inbox full of rebukes, Fred immediately returns to his working folder to roll back his changes. The easiest way to roll back a file is to use the file's log. Right-click the file in the working folder, and navigate TortoiseSVN | Show log. The log will reflect all of the file's commits, including the most recent one, which contains the error Fred wants to revert, which is Revision 10 in Figure 4-14. Figure 4-14: Viewing the log for the AuthorsBooks.sql file. 145
- 146. Chapter 4: Editing and Versioning We can roll back a file to any previous revision, as well as roll back sets of files or folders, if necessary. In this case, Fred just wants to roll AuthorsBooks back to Revision 9, so right-click the Revision 9 entry, and select Revert to this revision. Simply select Revert on the Reverting changes dialog box, and then wait as SVN reverses our changes and displays the Merge Finished dialog box (shown in Figure 4-15), indicating that SVN has essentially performed a reverse merge by comparing Revisions 9 and 10. Again, each source control system will have its own methods for reverting to a previous revision. Figure 4-15: Displaying the results of the reverted changes. Fred can verify that the code has reverted to its previous state, without the reference to the ISBN column, and he can then commit the file to the repository. If we reopen the file's log, we can see results that some might not expect: a new revision has been added to the repository, as shown in Figure 4-16. 146
- 147. Chapter 4: Editing and Versioning Figure 4-16: A new revision number resulting from a file rollback. The goal of any source control system is to maintain a record of all changes to a file throughout its history. Although SVN rolled back Fred's changes, the file itself keeps moving forward. Revision 10 still exists, but we now also have Revision 11, which happens to look exactly like Revision 9. What all this means is that Fred's error has not disappeared after all, at least not from the file's history, but at least the most current version of the file is now correct and the QA folks can quit badgering him, at least until he does something like this again. 147
- 148. Chapter 4: Editing and Versioning The World of Source Control I hope this chapter demonstrates that there's a lot more to source control than simply storing files. A source control system's ability to manage files and folders as well as preserve a history of their changes makes it possible to perform a variety of tasks, including those we've covered in this chapter: managing files and folders in the repos-itory; editing files in the working folder; accessing specific file and folder revisions; and performing diffs that tell exactly what has changed between those revisions. Yet the tasks we've performed so far have been relatively straightforward, in no small part because only one user has been making all the changes. But that's not how it works in the real world. One of the greatest values that a source control system brings to a development team is its ability to handle multiple users, some of whom are working on the same files at the same time. From here in, we enter the real world with a discussion of how to branch and merge files in order to support concurrent development efforts. As you'll soon see, that's where source control really starts to get interesting. 148
- 149. Chapter 5: Branches and Tags All our work so far on our BookstoreProject and all files relating to that project, has been in a single directory, i.e. the trunk directory (often called "master") of our working folder. Our development work has proceeded in a "straight line" from Revision 1 to the current revision. At some point in the development process, however, the team will find a need to separate out development efforts, so that two groups can work simultaneously on the same version of the repository, but for different purposes. An example often quoted is one team working on new development (e.g. version 2.0) of a project, while another works on bug fixes for version 1.0. Another example is the need to separate off and stabilize a certain version of the repository for QA testing or customer assessment or similar, while the development effort continues. To accommodate either case, we need to create a branch (or fork) in our repository to allow the growth of our development tree to diverge from the straight and narrow. From the version in the repository at which we create the branch, the growth continues in the original location (trunk or branch), but also sprouts off into one or more new branches. From the users' point of view, each branch represents a distinct copy of the project files on which a team member can work independently, while preserving the fact that each set of files shares the same roots. Of course, at some point we may need to merge the changes from one branch back into the master, or into another branch, or both. We won't cover merging until Chapter 6, but it's worth remembering that branching and merging are flip sides of the same coin. A team's confidence and ability to perform merges, with conflict resolution as necessary, will dictate their policy towards branching. In this chapter, we'll discuss how branching works conceptually, review briefly some popular branching strategies, and then demonstrate how to create branches in SVN. We'll also cover the related topic of tags, where a tag is essentially a branch that the team agree not to modify, and often denotes a fully tested and releasable version of a project. 149
- 150. Chapter 5: Branches and Tags 150 How Branching Works The methods used to implement branching vary from one VCS to the next, but the basic branching concepts apply universally. From the point of view of a user accessing the repository through their working folder, branching is similar to copying a folder and its contents from one location in the file system to the other. At that point, we have two distinct versions of the folder, and we can modify the contents of each independently. Figure 5-1 shows our working folder, with the trunk folder, as we've become accustomed, but also a second "copy" of the project, v1.0BugFixes, in the branches directory. Figure 5-1: Creating a branch based on a version in the trunk. From the VCS's viewpoint, however, the branch is usually not a full copy of the files and their histories, but rather pointers to the files' origins. There are several advantages to creating a branch in a VCS, as opposed to just making a second copy of a set of files in the file system.
- 151. Chapter 5: Branches and Tags 1. The VCS tracks all changes made to the branch files – it stores the file deltas within the branch, rather than in the trunk where the files originated. 2. The VCS knows the exact revision in the trunk from which the branch originated – since the VCS knows the exact point in time when the code paths separated, it knows how far back to look on each path to gather the information it needs to carry out any subsequent merge operation. Figure 4-1 provides an overview of how the branching process might work. 151 Figure 5-2: Creating a branch at Revision 100. In this example, let's assume Revision 100 of our database project occurs when we're nearing a release. As is often the case when working with a second-generation VCS like SVN, we create a branch that we'll use for working with QA and refining our release (v1.0). Within the branch, developers can fix bugs and address problems related to such issues as the need to change lookup data or to modify a stored procedure. At the same time, other developers will move forward with the next new version of the database (v2.0), which will incorporate elements of the v1.0 design, but with significant additions and changes.
- 152. Chapter 5: Branches and Tags Someone on the team creates a branch from the project folder in the trunk, at Revision 100. The set of files and folders in the branch becomes Revision 101 in the repository. From this point, any changes to files in the branch are independent of the trunk. The VCS stores in the branch all changes made to the branch files, with earlier revision histories available only in the trunk, at Revision 100. When we modify a file within the branch and commit the change, that change becomes Revision 102, with the changeset stored as part of the branch. The corresponding file within the trunk will still show as Revision 100 since, at this point, we've made no changes in the trunk. However, if we now modify the file in the trunk and commit the change, the new file version becomes Revision 103, but that revision has nothing to do with Revision 102, or anything in the branch. At some stage, the team will produce a fully tested v1.0 release and deploy it to production. However, they will also want to merge some or all of the v1.0 fixes to the v2.0 code base in the trunk. To do this, they'll need to apply to the trunk the changes made to the branch since creation, while taking into account any potentially conflicting changes made to the trunk in the meantime. We'll see how to do this in the next chapter. 152 Branching Strategies How a team uses branching is specific to that team and how its source control system ties into its deployment and product lifecycle strategies. Some teams simply don't create branches, some create them only when strictly necessary, and others create them as soon as a development effort starts, to separate out different strands of the development effort. Understand that it's not the process of branching that is problematic, but the process of merging changes from one branch into the master or another branch, the latter of which has also changed in the meantime. The more branches diverge, the more difficult merging becomes, which is why you should merge early and often. Again, we'll cover merging in Chapter 6.
- 153. Chapter 5: Branches and Tags All modern VCSs include the mechanisms necessary to merge files and resolve any conflicts that should arise, though the process isn't always straightforward. Third-generation distributed VCSs, such as Git and Mercurial in particular, treat branching and merging as a very natural part of the source control process. Let's review briefly this "spectrum" of branching strategies. 153 No branching Some development teams rarely stray from the straight and narrow. Users on a first-generation VCS, or even some on second generation, don't support concurrent editing, let alone branching and subsequent merging. Users check out files, placing locks on them to prevent simultaneous update attempts, and check the files back in when they've finished editing them. At no time do multiple developers work on the same file at the same time. They don't really need to create branches to support concurrent development efforts. They do not ever need to merge files and resolve conflicts. For some development teams, this sort of strategy might work just fine. However, such an approach can result in lower productivity, as team members end up waiting for others to complete their work before they can proceed. It can also result in irregular and delayed deployments. Teams that support concurrent development efforts, whether through branching or concurrent editing, can often achieve more in a shorter space of time. The flip side to this is that they often have to merge files and, in that process, sometimes have to resolve conflicts as a result of that merging. Branch when needed (release branches) A common strategy is to create a branch only as required, typically as the team approach a release. Release branches provide a mechanism for more smoothly incorporating source control into the team's quality assurance and deployment processes.
- 154. Chapter 5: Branches and Tags For example, let's say a team uses a very simple approach where they use master for releasable code, and create a branch for development work. During the project devel-opment phase, all members of the team work on a single "development" branch. At a defined point, they merge the changes into master, tagged appropriately, and these are then deployed from master to their QA or staging environment. Note that, in our worked examples to this point, we've effectively used trunk as the development branch, so we would create the tag to a different branch, but the principles are exactly the same. Figure 5-3: Working with a master and development branch. Team members might create additional branches under duress, but they essentially adopt a two-prong strategy. They set up the initial project structure as the master and create a branch for development, or they might start with the trunk for development and create a branch for the master (for our discussion here, we'll assume that the trunk is the master). This approach usually requires complex folder structures to support multiple releases and ongoing maintenance within a single branch. To overcome this difficulty, many teams move to a branching strategy that also includes one or more release branches. A release branch lets a team fix bugs and refine the application in preparation for its final release. Figure 5-4 shows a release branch created for the 1.0 release. 154
- 155. Chapter 5: Branches and Tags Figure 5-4: Using a release branch to test and refine an application. The application and database developers work in the development branch until the code is ready for QA. At that point, someone creates a branch for the 1.0 release. The team deploy to QA from the release branch, fix bugs that QA reports and address any other outstanding issues, and then merge into the master, tagging the release and deploying from there to production. After deployment the team might continue to use the release branch for ongoing maintenance, or create another branch specifically for that purpose. In the meantime, development continues in the development branch in preparation for the 2.0 release. During the QA process, the team might wish to merge into the development branch some or all of the bug fixes on the release branch, either at regular intervals or when the product is ready for release. It will also be necessary to merge regularly from development to release, to ensure the two branches don't drift too far apart. Release branches solve certain issues towards the end of a development cycle but, ultimately, having a single branch for development, although it makes the VCS processes rather straightforward, can lead to delays in deployment. Some features take longer to develop than others and the team can notionally deploy only when all planned features for that release are complete. This starts to become a problem if, for example, we fix a serious bug while building a new feature, and would really like to deploy the bug fix immediately to the released version, but have to wait till the feature is complete. 155
- 156. Chapter 5: Branches and Tags We can get around this problem by "cherry picking" that one commit that fixed the bug, merge it into the release branch and deploy it. We'll discuss this in more detail in Chapter 6, but it's a far from ideal solution. Another problem that can arise is that all features are ready to go except one which has thrown up unexpected difficulties, and management decides to release the product without the errant feature. One option, although again it's not ideal, is to create a temporary branch that contains all features except the errant one, and deploy from the temporary branch to QA/Staging/Production. Once v2.0 is complete, the team can merge to master and remove the temporary branch. A better option, and the norm in a third-generation VCS, is to have created a specific feature branch for the errant feature (see the next section) and not merge it into master for deployment until it is ready. Feature flagging instead of branching One way that some teams remove the concern of delayed deployments, without the need to introduce branching, is to use "feature flags" within the application code, whereby they wrap every feature code that prevents it from running until enabled via configuration. It means the team can deploy all features, even incomplete ones, only enabling each feature for the user as it becomes ready. We won't discuss this approach in detail, as we implement it in application code not in the VCS, but this article and the links in it offer a good overview: http://tinyurl.com/p8eulma. Typically, when the team start "cherry picking" commits or creating temporary branches, it's an indicator of a broader problem in their development process. The team may need to consider refining their branching strategy to separate more cleanly the various strands of development For example, they could create a separate branch for release patching, and a separate branch for major feature. This is a strategy we call "feature and release branching." 156
- 157. Chapter 5: Branches and Tags 157 Feature branches A popular approach within development teams who work to shorter, more frequent release cycles, is to create feature braches in addition to release branches. The team creates temporary branches to develop specific product features, rolling out one branch per feature, or "user story," so that they can develop each one in isolation, without causing instability in the development branch, or allowing any single development effort to disrupt the nightly build process. When that feature is complete and fully tested, the team can merge the feature into the development branch, from where they can incorporate it into the development and QA processes. Figure 5-5 illustrates one example of feature branching with multiple active features, none of which will affect the main line of development. Figure 5-5: Implementing feature branches to isolate development.
- 158. Chapter 5: Branches and Tags Use of feature branches, with a dedicated purpose for each branch, allows for a consistent deployment strategy for every release, and it allows "maximum concurrency" in the team's development efforts. Ultimately, one may arrive at a rather "freewheeling" branching scheme, such as the one described in Vincent Driessen's excellent article, A successful Git branching model (http://tinyurl.com/39hd87l). As you can imagine, however, such a strategy also requires very strict discipline with regard to the subsequent merging processes. The last thing you want is for a stored procedure to get updated in the release branch, but never have the update make it back to the development branch. There is also a good reason why frequent branching is a model best suited to a third-generation VCS such as Git, since they offer very strong support for branching and merging. Branching strategy for database development The way a team approaches branching is tied to its development and product lifecycle strategies. Every team is different, and no single approach is the absolute right way to branch. What complicates matters for database developers is that teams often plan their product lifecycle strategies around application development, and expect the database team to tag along (excuse the pun). Often the database team has little or no input on when and how to create branches. However, product groups are trying more than ever to incorporate database development into their development processes, particularly those groups that embrace Agile development methodologies. In such situations, branching often plays a significant role. As a result, it's more likely than ever that at some point, as a database developer, you'll find yourself having to work within one or more branches during a database's development cycle. 158
- 159. Chapter 5: Branches and Tags For database developers, release branching with a single development branch is relatively painless because the team simply place the database in the VCS in the development branch, alongside the application, and the team are, most of the time, not trying to develop different versions of the same database in two places at once. When the team create a release branch, of course, there will be two different versions of the database but, as long as they merge frequently in both directions, and adhere strictly to the established development and source control procedures, it should not make life too difficult. In this book, in order to focus on basic database source control concepts, we don't stray too far beyond simple branching of this nature. However, as discussed, some teams will find this limiting, and see the need to use feature branching alongside release branching, to provide a more flexible structure for maximizing development efforts and improving efficiency. For database developers, however, the more branching that occurs, the greater the number of database versions out there. Without careful planning and good database and interface design, this can get dangerously complicated. At this stage, it becomes important that your database design makes intelligent use of schemas (http://tinyurl.com/5nd8jn), to group together database objects that support each logical area of the application's functionality. That way, if a developer creates a branch to work on a specific feature, any related database changes will be on one specific schema, representing one logical area of the database. This is likely to make the subsequent merging processes less painful. Regardless of the exact branching strategy, a team that creates an application that makes direct access to base tables in a database will have to put a lot of energy into keeping database and application in sync (to say nothing of having to tackle issues such as security and audit). This issue will only worsen, the more branches we maintain and the more versions of the database we have in the VCS. A good database development team will, firstly, make considerable efforts to ensure that they understand the application domain as fully as possible, and so get the database design right up-front and minimize the need for substantial refactoring later. Secondly, they will never expose the inner workings of the database to the application. 159
- 160. Chapter 5: Branches and Tags Each application should have a stable, well-defined interface with the database (one for each application, usually, if more than one application uses a single database). The appli-cation development team owns the interface definition, which should be stored in the VCS. The database developers implement the interface, also stored in the VCS. Database developers and application developers must carefully track and negotiate any changes to the interface. Changes in it have to be subject to change-control procedures, as they will require a chain of tests. The database developers will, of course, maintain their own source control for the database "internals," and will be likely to maintain versions for all major releases. However, this will not need to be shared with the associated applications. An application version should be coupled to an application-database interface version, rather than to a database version. In many respects, a branching strategy for database development is no different from any other application development. Indeed, it's likely that the team will adopt the same strategy for applications as for databases. However, I think it fair to say that, from the database developer's perspective, they would like the branching strategy to be as simple as possible, perhaps adopting a simple "branch-as-needed" approach, at least until the whole team become highly competent at managing database source control processes. Feature branching should be used judiciously and is best suited to cases where developers can work on discrete parts of the database, and where there exists a well-defined interface between application and database. 160 Tags The revision numbers that a source control system uses to track version histories are separate from the types of numbering systems used to identify product builds, versions or releases. They can be specifically associated with one another, but they are separate systems. That's where tags come in. Tags provide an easy way to denote a snapshot of your repository at a given point of time, usually based on a particular revision number.
- 161. Chapter 5: Branches and Tags A tag is essentially just a branch, but one that we should not modify, once created. We use tags, generally, to associate a build number with a stable, releasable revision in the repository. As we saw earlier, in Figure 5-3, some teams only ever have tagged versions in master, since they always want master to contain only code that can be, or has been, released. For instance, suppose we had deployed the Bookstore database based on the files in our repository, as they existed at Revision 4. The database deployment is part of the product release 1.0.0.0. We can create a tag named release 1.0.0.0 and associate it with Revision 4. Thereafter, we'll always know exactly what was in release 1.0.0.0, just by referencing the tag, as long as no one subsequently changes the files and folders associated with the tag. A deployment strategy might include a process that extracts the most recent revision number from the source control system and incorporates that number into the build number. However, the number structures themselves remain separate. In other words, if our most recent database build is 1.2.5.35, then that build number might or might not somehow reflect the revision number, but the build number has no bearing on how the VCS assigns revision numbers. Tags are usually on the most recent revision, of course, but we can tag any revision in the repository. 161 Branching in SVN It's time to see branching in action in SVN. The team have completed v1.0 of the Bookstore application. At that point, they created a tag for the release in their Tags folder, which effectively forms their "deployment" branch. In other words, they will deploy from this folder to production. We'll see how to create tags shortly, in the Creating Tags in SVN section later in this chapter. Our development duo now want to create a branch off of trunk, containing the whole project for the v1.0 release, where they can perform some bug fixing on the released version, while mainline development continues in the trunk, on the next major release (v2.0).
- 162. Chapter 5: Branches and Tags Since Barb is the only one who checked out the entire repository, she will be the one to create the branch in the repository. Fred can just check out, to his working folder, the bits of the branch he needs, specifically the Databases folder. When we create a branch, it must still represent a "single version for the entire project." If Fred were to create a branch off of the Databases directory (since he doesn't have trunk checked out), then we defeat this objective. For example, when it comes time to build and deploy the branch, a CI server might check out the branch directory and run the build; if all it had was the Databases folder then it wouldn't have all the project files it needed to run the full build. One way for Fred to create a branch on his own would be to create a second, separate working folder, check out trunk to that folder, and then create the branch from there. With the branch created in the repository, Fred can once again just check out what he needs to his original working folder, and probably delete the second working folder. However, for simplicity, in these exercises we're simply going to have Barb create a branch called v1.0BugFixes, based on the current revision of the trunk folder in the trunk. Before creating the branch, Barb needs to update her working folder with the latest revision in the repository, so right-click BookstoreProject and click SVN Update. If you've been following along with the exercises up to this point exactly as described, then this should bring Barb's working folder to Revision 11. However, if you've been experimenting with other options or trying side-experiments, then your revision numbers won't match. It doesn't matter; as long as you provided a descriptive commit message, you'll still know which changes you committed at each revision. Clearing the authentication As you'll recall from Chapters 2 and 3, if you've set up working folders for Jane, Barb, and Fred on the same computer, you will need to clear the authentication history before signing in as a different user. To clear the settings, right-click a folder in the working directory, point to TortoiseSVN, and then click Settings. On the Saved Data node, click Clear All next to Authentication data. 162
- 163. Chapter 5: Branches and Tags Create the branch in the repository To start creating the branch, right-click the trunk folder in Barb's working folder, point to TortoiseSVN, and then click Branch/tag. This launches the Copy (Branch/Tag) dialog box, shown in Figure 5-4. 163 Figure 5-6: Creating a branch in TortoiseSVN. The dialog box will automatically display the folder from which it was launched as the source of the branch. Instead, we must specify a folder for our branch, so in the To path text box, type in /branches/v1.0BugFixes. In the Create copy in the repository from section, select the option HEAD revision in the repository to ensure that we're creating the branch based on the most recent revision in the repository.
- 164. Chapter 5: Branches and Tags Click OK, and a Copy Finished dialog box confirms that the new revision is Revision 12. In other words, creating the branch creates a new revision in the repository. You may also see a message regarding your working copy: "Your working copy remains on the previous path. If you want your next changes to be committed to the copy or branch you have just created, then you need to switch your working copy over to the new URL. Use the switch command to do that." SVN uses a path (URL) to serve as the pointer from a working folder to the repository. SVN uses that path to keep the working folder and repository in sync. This is simply a reminder to Barb that, despite creating a branch, the path remains unchanged (pointing to <server>/svn/BookstoreProject, since Barb checked out the entire project). Therefore, when Barb updates her working folder after creating the branch, she'll get all changes to the branch as well as to the trunk. In other cases, a user that checks out only specific portions of the repository (such as Fred) may need to switch their working folder to tell SVN to use a different path, pointing to different, but related, content in the repository. We'll see how to do this shortly, when we discuss how Fred updates to retrieve the new branch. That's all we need to do to set up a branch in SVN but, returning to Barb's working folder, you may be surprised not to see the new branch in the branches folder. When we create a branch, we add it directly to the repository. To verify this, launch Repository Browser by right-clicking the BookstoreProject folder, pointing to TortoiseSVN, and then clicking Repo-browser. Figure 5-7 shows the repository with the new v1.0BugFixes branch. We've created a separate copy of the project, in the branch, from Revision 11 in trunk. However, as noted, on the server the repository has merely created a new branch folder with pointers back to the original content in trunk. 164
- 165. Chapter 5: Branches and Tags Figure 5-7: The new branch as it appears in the repository. Update the working folders For the branch to appear in Barb's working folder, she needs to update her folder, so right-click her BookstoreProject folder, perform an SVN Update, and verify that the branches subfolder now contains the new branch (as shown previously in Figure 5-1). If Fred needs to work on the new branch, he has two ways to update his working folder. The first, and the one we use here, is to now create a new subfolder of BookstoreProject called something like BookstoreDatabases_v1.0Bugfixes and from there perform an SVN checkout of the /branches/v1.0BugFixes/Databases folder. In effect, Fred is creating a second working folder, one that points to the Databases folder in the branch rather than in the trunk. 165
- 166. Chapter 5: Branches and Tags Figure 5-8: Updating Fred's working folder with the new branch. Switching the working folder Fred's other option, when he needs to work on the branch, is to switch his current working folder (BookstoreDatabases) to point to the branch, instead of the trunk. To do this, right-click the current root folder in the working folder (in Fred's case, the BookstoreDatabases folder), point to TortoiseSVN, and then click Switch. In the Switch To Branch/Tag dialog box, switch the path from /trunk/Databases to /branches/ v1.0BugFixes/. This will update the working folder with the files from the new target. Any commits Fred makes from that point forward will occur on the branch, until he switches the working folder back to trunk. Keep in mind, however, that the way we're dealing with working copies here is specific to TortoiseSVN and, by extension, second-generation source control systems. In third-generation systems, our only choice is to check out the entire repository, and switching back and forth between branches is generally much more common. 166
- 167. Chapter 5: Branches and Tags 167 Modifying the branch In order to prove that anyone can now work on this branch independently of the BookstoreProject in the trunk, let's have Fred make a small modification to the AuthorsBooks view in the branch, by adding a comment and a calculated column. Within Fred's BookstoreDatabases_v1.0Bugfixes branch, open the AuthorsBooks.sql file, add a comment, and then add the NewBookID calculated column, which creates an ID based on the BookID value and part of the LastName value. Listing 5-1 shows the new code for the ALTER VIEW statement, highlighted to make it easier to identify. (Don't dwell on the logic of this column too much; it's only for demonstration purposes.) ALTER VIEW AuthorsBooks -- retrieves all authors and all their books AS SELECT ( a.FirstName + ' ' + a.MiddleName + ' ' + a.LastName ) AS FullName , (CAST(b.BookID AS VARCHAR) + '_' + LOWER(LEFT(a.LastName, 2))) AS NewBookID, b.Title, b.YearOfPublication FROM Author a INNER JOIN AuthorBook ab ON a.AuthorID = ab.AuthorID INNER JOIN Book b ON ab.BookID = b.BookID; GO Listing 5-1: Modifying the AuthorsBooks view definition. Save and close the modified AuthorsBooks.sql file and then, back in TortoiseSVN, right-click BookstoreDatabases_v1.0Bugfixes and perform an SVN Commit, with a suitable commit message (again, make sure you're connecting to the repository as Fred).
- 168. Chapter 5: Branches and Tags Figure 5-9: Commit the modified AuthorsBooks view to the v1.0BugFixes branch. The Commit Finished dialog box will appear, showing that we're now at Revision 13 (or whatever the revision number is on your system). Remember, we started with Revision 11, created our branch (Revision 12), and altered the AuthorsBooks.sql file (Revision 13). The BookstoreDatabases_v1.0Bugfixes branch in Fred's working folder should now show all green, indicating that there are no current changes that need to be committed to the repository. At this point, the trunk remains unchanged at Revision 11. The changes made in this exercise affect only the branch. Diffing between branches and the trunk In Chapter 4, we worked through some examples of performing a diff between different file versions. However, we can also perform diffs between files or folders in the trunk (or a branch) and the corresponding files or folders in another branch (or associated with a tag). Let's look at a few examples. 168
- 169. Chapter 5: Branches and Tags 169 Comparing folders We can perform a high-level diff between a folder in the trunk and a corresponding folder in our branch. From Fred's working folder, launch repo-browser, right-click the trunk folder, and click Mark for comparison. This will turn the folder bold. Next, navigate to the v1.0Bugfixes folder, right-click it and select Compare URLs. This will launch the Changed Files dialog box, which provides a list of differences between the trunk folder and the branch folder, as shown in Figure 5-10. Figure 5-10: Viewing differences between the branch and trunk project folders. The dialog box lists all the objects that have been added or have changed. As expected, it shows only the AuthorsBooks.sql files, which we modified in the branch. Double-click the file to launch TortoiseMerge and see a side-by-side comparison of the file as it exists in our branch (on the left in Figure 5-11) and as it exists in trunk in the repository (on the right).
- 170. Chapter 5: Branches and Tags Figure 5-11: Performing a diff between a file in a branch and file in the trunk. Not surprisingly, the diff shows the column and comment that Fred added. 170 Unified diffs Another useful type of diff to perform is a unified diff, which shows all the file changes that have occurred between a specific revision, such as the first revision in the new branch, or a tagged revision, and the current revision. To demonstrate this, let's make another simple change to a different database object in the branch. In Barb's branch (for a change!), add a comment of your choice to the AuthorBook.sql file, and commit it. Then, launch the SVN log from BookstoreProject, and highlight the two revisions to compare, in this case, Revisions 14 and 12, as shown in Figure 5-12.
- 171. Chapter 5: Branches and Tags Figure 5-12: Selecting revisions in the repository log. Having highlighted the two revisions, right-click one of the files and click Show differences as unified diff. This launches the TortoiseUDiff utility, which displays all the differences (as additions and deletions) that have occurred between the two versions, as shown in Figure 5-13. 171
- 172. Chapter 5: Branches and Tags Figure 5-13: Viewing a unified diff of revisions in the branch. As the figure shows, the diff reflects Barb's changes to files, with the changes highlighted in green. Notice too, highlighted in yellow, the path and file names, along with the revision numbers. 172 Revision graphs TortoiseSVN also lets us generate revision graphs, which provide a fast and simple way to view where our branches and tags came from in the trunk. In other words, a revision graph provides a visual aid for seeing the source revision on which we based our branch or tag.
- 173. Chapter 5: Branches and Tags In Barb's working folder, right-click the v1.0BugFixes branch folder, point to TortoiseSVN, and click Revision Graph. This launches the Revision Graph dialog box, shown in Figure 5-14. Figure 5-14: Reviewing a revision graph of the v1.0BugFixes branch. The revision graph shows that we created the branch based on Revision 11 in the trunk and that the newly created branch became Revision 12. It also shows that the branch is now at Revision 14. We can hover over any of the revision icons to reveal a pop-up message providing additional details about that revision, such as the URL, the author, and the date and time that user committed to the repository. The Revision Graph dialog box also provides options for displaying more information and taking additional actions. For example, we can display all the revisions in which changes were made or mark a revision as the working copy. 173
- 174. Chapter 5: Branches and Tags 174 Creating Tags in SVN To demonstrate how tags work, we'll create one for our Bookstore project. Conceptually, tags function the same way in other VCSs, but there may be implementation differences. Creating a tag in TortoiseSVN is nearly identical to creating a branch, except that we specify a different location (in this case, the tags folder in BookstoreProject). For this example, we'll create a "retrospective" tag based on Revision 4 and imagine that this related to release 1.0 of our Bookstore database. Revision 4 also represents our very first schema, with just our three initial tables, so if we ever want to see everything that changed since, we can compare our current release to this tag. In Barb's (or Jane's) working folder, right-click the trunk folder and navigate TortoiseSVN | Branch/tag to launch the Copy (Branch/Tag) dialog box that we saw previously, when creating a branch. To name the tag, we simply type the correct path into the To path text box so it reads /tags/v1.0. SVN will apply the tag to all content in the trunk as it existed at Revision 4. Next, add a comment, and then specify the revision number in the text box associated with the Specific revision in repository option. Figure 5-15 shows the Copy (Branch/Tag) dialog box with the configured information. Once satisfied that the tag is exactly as you want it, click OK to close the Copy (Branch/Tag) dialog box.
- 175. Chapter 5: Branches and Tags 175 Figure 5-15: Adding the v1.0 tag. As was the case when creating a branch, SVN creates the new tag in the repository, but it does not appear in Barb's working folder until we right-click her working folder and select SVN Update. Having done this, it should show the new v1.0 tag, in the tags folder. Notice that the tag contains the folders (and files) as they existed at Revision 4. Figure 5-16: Viewing the v1.0 tag in Barb's working folder.
- 176. Chapter 5: Branches and Tags If we want to see exactly what has changed since that revision, we can open Repository Browser and compare URLs between the trunk and the v1.0 tag or between the v1.0BugFixes branch and the v1.0 tag. In either case, the Changed Files dialog box will show all added and modified files and folders. We can then click on any one to see exactly how it changed. Alternatively, we can perform a unified diff to see all the changes in a single, shortened view. Finally, remember that, just as a branch in the repository is really just a new pointer to the same content, so is a tag. That means our new tag is pointing back to the original content, as it existed at Revision 4. Another way to consider it is that the repository stores deltas, as described in the previous chapter, not copies. Here, the delta is just a new path to the same content. 176 Summary Branching is an integral feature of any modern source control system. Branching lets teams work on distinct sets of files that share the same roots, without one team overwriting the work of another team. We can create a branch just about as easily as we can copy a folder. Our approach to branching will depend, at least in part, on our broader management strategies, which determine how files are stored, deployed, implemented, and in other ways managed throughout the product lifecycle process. Of course, having created a branch and made changes as required, we'll often want to merge those changes back into trunk, or into another branch. That's the challenge for the next chapter.
- 177. Chapter 6: Merging If branching is the process of "segregating" the development effort into different streams, then merging is the process of bringing these streams back together. Branches allow team members to work on a particular feature or logical area of an application, or a set of bug fixes, without their changes affecting users working on trunk or on other branches. Merging allows team members to incorporate changes made in one location (branch or trunk) into another. In this chapter, we refer to this as "inter-branch" merging. For example, when we create a branch for bug fixes of the current database release or for developing a particular feature, then team members working on the branch will, at some point, want to merge those changes into the "ongoing development" branch. To make this process as painless as possible, it's also important that the team perform regular merges in the opposite direction, incorporating into their branch the changes made by the ongoing development team. In addition to inter-branch merging, we may also need to perform "intra-branch" merging, when more than one user makes changes to the same file in the trunk, or in the same branch. For example, let's say Barb and Fred both start work on the same revision of one of our database files. Barb commits her change first. When Fred commits, the operation will fail and Fred will need to perform an update. If Fred's proposed commit does not conflict with Barb's commit, most VCSs will perform an auto-merge, as part of the update, producing a third file version that reconciles Barb's change and Fred's proposed change, and which Fred can then commit. If the changes do conflict, Fred will need to assist manually in the merge operation. While inter-branch and intra-branch merge operations serve different purposes from the team's perspective, conceptually they are similar in nature. If the team adopts good practices, committing small changes often and updating often, then intra-branch merges should be painless, since they will entail merging a single (small) proposed commit with a single existing commit. Inter-branch merge operations can be trickier, since the merge operations may need to reconcile a range of commits in the branch with a range of commits in another branch, manually assisting with any conflicts that arise. The longer users wait between merges, the more painful is the process. 177
- 178. Chapter 6: Merging In this chapter, we'll discuss a few best practices to help avoid painful merge operations, then explain conceptually how various varieties of intra-branch and inter-branch merges work, pointing out significant differences between merging in a centralized VCS versus a distributed one. Finally, we'll walk through some simple merge examples in SVN, so you get a feel for how it works in practice. How to Avoid Painful Merges The more work you do in your working folder without committing, and without checking what other users have been up to via an update, the more painful you'll find the resulting "big bang" merge. To minimize the potential pain, follow these simple rules: 1. Update your working copy often – Before starting work on a file, retrieve the latest version into the working folder to make sure you've got the very latest revision from the repository. 2. Commit small changes often – Having made and tested the smallest possible changeset that will leave the database in a consistent working state, commit it. 3. Always start a merge from a clean working folder – No uncommitted changes. 4. Merge often – By following the first two rules we will, by definition, perform frequent small intra-branch merges. However, we must also apply the spirit of these rules to our inter-branch merging. For example, if we're working on a feature branch that we will need to merge into the mainline development branch, then during development we should: a. frequently merge from mainline to branch to "update" the branch with any changes made in mainline development so the two strands don't drift too far apart b. frequently merge small, tested changes from branch to mainline. 178
- 179. Chapter 6: Merging Even if you follow all of these guidelines, merges can sometime still be tricky and, without due care, can leave your working folder or repository, or both, in a mess. If a failed merge leaves the working folder in an unstable state, almost every VCS will provide a way to undo changes and get back to the starting revision. In the case of SVN, for example, we can use the revert command (see http://tinyurl.com/86s4sc8). We can also simply take a copy of the current working folder prior to commencing the merge. For particularly knotty merges, talk to whoever is responsible for maintaining the repository, and consider taking a full backup (assuming the repository is relatively small) or an incremental backup, before starting the merge. 179 How Merging Works The basic concept behind merging is universal: we need to merge changes in one location with changes made in another location. However, perhaps more so than for any source control concept discussed previously in this book, different VCSs have different ways to perform the various merge operations, and merging in a centralized VCS is quite different from merging in a distributed one. In this section, we'll try to focus on conceptual descriptions of each variety of merge operation and along the way point out significant differences in how the type of merge works in centralized versus distributed systems. Intra-branch merges: conflicting changes on the same branch This type of merging occurs when a potentially conflicting change occurs during concurrent modification of the same file in the same branch of a repository. It results in the need to merge the repository's version of a branch or the trunk, with the local version of the same branch or trunk.
- 180. Chapter 6: Merging A typical workflow leading up to the need to perform an intra-branch merge in a centralized VCS might look something like this: 1. Barb updates her local working copy – Barb has the latest revision, 29 say 180 (R29 for short). 2. Fred updates his local working copy – Fred also has R29. 3. Fred commits successfully a change to file x – the repository generates a delta to describe the change and registers R30. 4. Barb finishes modifying file x and either: a. attempts to commit – the repo can't generate a delta because the "base" for Barb's proposed commit is R29 whereas the repository version is now at R30, so the repository notifies her of the need to update before committing b. performs another update before committing – and gets notified of Fred's change. 5. Barb needs to perform a merge. As we discussed, briefly in Chapter 1, in a centralized VCS this intra-branch merging process can become somewhat "dangerous," especially if we forget to employ good practices that help avoid large, complicated merges. The essential problem is that a centralized VCS will "block" a user's attempt to commit a change from their working folder to the central repository if another user has changed that file in the meantime. Instead the VCS invites the user to perform an update, which incorporates a merge operation (auto or manual), before committing. Having established a "merged" version of the file, in the working folder, the user will typically have no history of the state of the file when attempting the initial commit, so if the merge goes wrong, the user risks losing work. To avoid this possibility, users have to resort to "pre-VCS versioning" of their working folder, by simply making a copy of the working directory to a folder, named by date, or appended with a letter or number.
- 181. Chapter 6: Merging Welcome to the Stone Age It seems ironic that, when using a second-generation system VCS, users often resort to the sort of "copy the folder" version control that would make even a first-generation VCS blush with shame. Fortunately, third-generation systems have figured how to avoid having to make a precautionary backup. By contrast, intra-branch merging in a distributed VCS is safer, since each user has a local repository. In a DVCS, we can commit locally and then "pull" the latest changes from a remote repo and merge as necessary. The merge operation is safer in that it involves a local commit and a remote commit, and we need to create a third commit that combines the two. We commit the merged version locally and finally push the merged version to the remote repository. If the merge fails, we can simply try it again. Continuing our example, if Barb's proposed commit does not conflict with Fred's commit, the repository will perform an auto-merge, creating a merged file that contains both sets of changes, which Barb can commit to the repository (as R31). Barb will have the option to review how the VCS achieved the merge, but it does the work automatically. Users of a centralized VCS, such as SVN, typically don't even think of this as a "merge" because they clicked on Update to begin the process and the merge was automatic. Distributed VCSs make these two distinct steps, allowing us to separate them, and making it clear that users are merging as they pull. If Barb's proposed commit does conflict with Fred's commit, Barb must perform a manual merge, and then commit the merged version. The VCS will store some temporary files in Barb's working folder to allow Barb to view various versions of the "conflicted" file, examine each conflict and take steps to resolve them, usually with the help of a merge tool (you'll see this in action shortly). In most cases, regardless of whether we use a centralized or a distributed VCS, the more complex the merge, the more likelihood there is for conflicts and the more work we will have to do to resolve them. 181
- 182. Chapter 6: Merging However, as noted in Chapter 2, a DVCS, which will track changes across all files collec-tively rather than on a file-by-file basis, does generally offer much stronger support for merging and can more often merge changes without any manual user intervention. Inter-branch merges: moving changes between branches For teams that segregate their development efforts into branches, with each branch serving a defined purpose, such as to create a specific feature, there will inevitably be the need to merge changes from the trunk to the branch, from the branch to the trunk, or from one branch to another branch. A user might also decide to merge only specific changes, or all changes since the branch was created. The team will potentially need to merge in a lot of files that have changed both in the origin (the trunk, say) and in their branch. If the changes don't conflict, the VCS will auto-merge, as described previously. However, if they conflict then the user faces a lot of extra work when trying to merge the changes manually. Fortunately, the previously described danger of intra-branch merging does not apply to inter-branch merges, since the merge operation will always involve two commits, as opposed to a commit and a proposed commit. Nevertheless, the best way to avoid unnecessary pain is to perform frequent merges. It will make it easier to iron out conflicts before the files in the source and the destination diverge too far. Don't be afraid of branching, but have a healthy respect for the subsequent merging process, and use branches only when they'll augment the development process. A superfluity of branches can result in a lot of extra work. 182
- 183. Chapter 6: Merging Merging from mainline to branch Let's say that a small team of developers, led by Fred, need to work on a set of complex analytical database functions. They will take some time and testing and, in order not to disrupt other developers, they decide to create a feature branch off of "mainline." Meanwhile, Barb and the rest of the team continue the main development effort, modifying the database schema as required in mainline (which could be in trunk, or a dedicated branch). The onus is on Fred and his team to periodically merge changes from mainline into the feature branch in order to ensure the two strands of development do not drift too far apart. Every developer should regularly update their working folders with the latest changes by other team members working on the same branch. In addition, the team should periodically pause their feature development to merge the mainline development changes into their feature branch. Typically, one team member will merge in all changes since branch creation, resolving any conflicts, then commit the merged files to the branch project directory in the repository, from where the other team members can update their working folders. Merging from branch to mainline At a certain point, Fred's team will finish work on each analytical function and will want to merge the feature into mainline. Fortunately, their task is made less onerous by their periodic merges in the other direction, since now all they need to do is merge into mainline the specific changes relating to their new function. At this point, it may be that the feature branch is no longer required and the team can delete it. Alternatively, they may want to keep it open to develop the features further. 183
- 184. Chapter 6: Merging 184 How to Merge in SVN Up to this point, we've lived in a conflict-free SVN world, but that is about to change. We're going to walk through both intra- and inter-branch merges for our Bookstore project, resolving conflicts as we encounter them. Intra-branch merges in SVN The best way to demo how SVN handles "intra-branch" merging, in other words merging that arises from two users committing changes to a single branch, is to walk through a couple of examples, one where the VCS is able to perform an auto-merge, and another that requires user intervention to resolve a merge conflict. Auto-merge Fred and Barb are working in trunk (https://<server>/svn/BookstoreProject/trunk) and both update their working folders, so each has the latest revision in the repository (Revision 15, in this example). Both users decide to modify the definition of the Author table: 1. Barb changes Author.sql to add a Title column to go along with the name columns. 2. Fred changes Author.sql to make the MiddleName column nullable. Fred successfully commits his change, creating Revision 14 in the repository, as shown in Figure 6-1.
- 185. Chapter 6: Merging Figure 6-1: Committing Fred's changes to the repository. Subsequently, Barb tries to commit her change and receives a Commit failed message (shown in Figure 6-2), telling her that Author.sql is out of date. If Barb had committed first, then Fred would see this message instead; another reason to commit early and often! 185
- 186. Chapter 6: Merging Figure 6-2: Receiving a Commit Failed message when trying to commit. When Barb clicks OK on the Commit Failed dialog box, SVN automatically suggests that she update her working copy, as shown in Figure 6-3. Figure 6-3: TortoiseSVN prompts Barb to update or cancel the operation. 186
- 187. Chapter 6: Merging If Barb clicks Update, SVN will bring her working folder up to Revision 16 and attempt to automatically merge the Revision 16 version of the file in the repository with her working copy of the file. If it can't do so safely, SVN will request manual help with the merge. If the change was complex, Barb might choose to cancel at this stage so as not to risk losing those changes, save them off somewhere else first, as described earlier, and then perform an update. Here, however, Barb simply clicks Update. In this case, the changes she made do not conflict with those of Fred, so SVN performs an auto-merge and invites Barb to commit the new auto-merged version of the file, which contains both her new Title column and the nullable MiddleName column. The resulting commit is Revision 17 in the repository. Remember that Fred will still have Revision 16 of the Author.sql file until he performs an update of his working folder. Things get more interesting, however, when our developers' changes overlap. Manual merge (dealing with conflicts) Once again, both Fred and Barb update their respective working folders so that they both have the latest revision (now Revision 17). Fred adds a new Type column to the Book table and commits the change. Listing 6-1 shows the change that Fred makes to the Book. sql file, with his added code highlighted. 187 CREATE TABLE dbo.Book ( BookID INT IDENTITY PRIMARY KEY , Title NVARCHAR(1000) NOT NULL, YearOfPublication INT NULL , [Type] CHAR(12) NULL ); GO Listing 6-1: Adding the Type column to the Book table.
- 188. Chapter 6: Merging This takes the repository to Revision 18. Barb subsequently updates her working folder and retrieves the latest version of the Book.sql file, with the new column. Then Fred decides that Type was a poor choice of column name. He changes it to BookCategory. In the meantime, Barb, who has a strong preference for using varchar data types, as opposed to char, changes the Type column to a varchar(15). Fred commits his changes first, creating Revision 19 in the repository. Figure 6-4 provides an overview of the changes both Fred and Barb have made and reflects Fred's commit. Figure 6-4: Updating the Type column in the Book table. This time Barb does the right thing and checks the log (shown in Figure 6-5) before attempting her commit. 188
- 189. Chapter 6: Merging Figure 6-5: Viewing the log to verify the revision history. She sees that her working folder is out of date and, thanks to Fred's descriptive commit message, understands quickly what has happened. Again, she might at this stage consider saving off her current working folder to a separate location, but in this case she simply updates her working folder and, as she expects, the Update Finished dialog contains a "conflict" warning, as shown in Figure 6-6. Figure 6-6: Conflicting versions of the Book.sql file. 189
- 190. Chapter 6: Merging Barb clicks OK to close the dialog box. Her Tables sub-directory now looks as shown in Figure 6-7, with multiple copies of the Book.sql file. Figure 6-7: Conflicting versions of the Book.sql file in Barb's working folder. When SVN updated Barb's working folder and detected the conflict, it added three temporary files to the Tables folder, all of them related to the original Book.sql file: • Book.sql.mine – Barb's current working copy of the file before she updated her working folder. The file includes no conflict markers. It contains only her latest changes, and nothing else. • Book.sql.r18 – This is a copy of the file as it existed at Revision 18. In other words, it is the "base" revision from which both users started, before making their conflicting changes. • Book.sql.r19 – This is the most recent revision of the file, committed to the repository by Fred. Some VCSs refer to this simply as "theirs." 190
- 191. Chapter 6: Merging In addition, SVN marked Barb's working copy of the Book.sql file with a yellow exclamation point, indicating that the file is in a "conflicted" state. If Barb opens the file in a text editor, she'll find the content displayed in a "unified diff" format (see Chapter 5), similar to what is shown in Listing 6-2. 191 CREATE TABLE dbo.Book ( BookID INT IDENTITY PRIMARY KEY , Title NVARCHAR(1000) NOT NULL, YearOfPublication INT NULL, <<<<<<< .mine [Type] VARCHAR(15) NULL ======= BookCategory CHAR(12) NULL >>>>>>> .r19 ); GO Listing 6-2: A unified diff of the Book.sql file. Barb does not work with any of these files directly. Instead, she closes the Book.sql file and launches a merge tool, in this case TortoiseMerge, to help her perform the merge operation. Her goal is to come up with a new "merged" version of the Book.sql file in her working folder that reflects her change or Fred's, or both, and which she can commit to the repository, as shown in Figure 6-8.
- 192. Chapter 6: Merging Figure 6-8: Merging file changes in TortoiseMerge. To open TortoiseMerge, Barb right-clicks the Book.sql file, points to TortoiseSVN, and then clicks Edit Conflicts. This launches TortoiseMerge, as shown in Figure 6-9. 192
- 193. Chapter 6: Merging Figure 6-9: Viewing the file conflicts in TortoiseMerge. What Barb sees on the top screen in TortoiseMerge is the change "they" (Fred, in this case) made. In other words, it shows a comparison of Revision 19 to its base (Revision 18). In the right-hand screen she sees a depiction of how her working copy of Book.sql changed, compared to its base (Revision 18). The bottom half of the screen shows the current state of the "merged" Book.sql file, with the exclamation point and row of question marks denoting where we need to resolve the conflict. Barb can right-click the exclamation point in the left margin to see a range of options for fixing the conflict. For example, she can choose the Use text block from 'theirs' option, Use text block from 'mine', Use text block from 'mine' before 'theirs', or Use text block from 'theirs' before 'mine'. She can also type directly into the bottom pane to tweak the resulting output. 193
- 194. Chapter 6: Merging In this case, Barb decides that the best option is to combine the two changes, accepting the new column name of BookCategory, but changing its data type to varchar(15), as shown in Figure 6-10. Figure 6-10: Resolving file conflicts in TortoiseMerge. On the top toolbar of TortoiseMerge, click the Mark as resolved button (represented by a green tick), and close the TortoiseMerge window. Barb can now return to her Tables directory of her working folder. Book.sql has the normal white-on-red exclamation mark to denote an uncommitted change. She can commit her change as per usual, creating Revision 20, and Fred, when he updates his working folder, will retrieve Barb's changes. 194
- 195. Chapter 6: Merging Inter-branch merging in SVN Let's now move on to the inter-branch varieties of the merge process, moving changes between branches. The merge operation is similar but the "business case" for performing such a merge is rather different. In our previous intra-branch merge examples, a user encountered the merge operation via an update or commit command. With inter-branch merges, the user will instigate the merge using the TortoiseSVN | Merge command, in order to move changes from the current branch into the originating branch (in other words, the branch or trunk from which the current branch grew) or vice versa. If the operation encounter merge conflicts, due to incompatible changes to the equivalent file in the current branch and originating branch, then SVN may be able to perform an auto-merge, or the user may need to intervene manually, as described for the intra-branch merge examples. When we define a merge operation, we need to define three key elements: FROM (the source), TO (the destination) and WHAT (the specific revisions we wish to merge): WHAT <revisions> FROM <Source URL or path> TO <Destination URL or path> In TortoiseSVN, we start the merge by right-clicking the TO directory in our working folder and selecting TortoiseSVN | Merge, to bring up the Merge type page of the Merge wizard appears, as shown in Figure 6-11. 195
- 196. Chapter 6: Merging Figure 6-11: Choosing the type of merge you want to perform. The current version of TortoiseSVN supports two possible merge options: • Merge a range of revisions – Copy one or more changes from a branch or the trunk to another branch (or trunk). • Merge two different trees – Reintegrate a branch (source) into the trunk or into another branch (destination). You can merge a specific revision from the source into a specific revision of the target. By default, the HEAD revisions are merged. After the merge, the target will look exactly like the source. This option is commonly used to reintegrate a feature branch into the trunk. Note that previous versions of TortoiseSVN offered a third option, Reintegrate a branch/ automate merge, used for the specific case of doing a final merge from a branch back to its source (either the trunk or another branch). However, we now simply use the Merge two different trees option, with the assumption then being that we will be doing no more development on the branch and can remove it. 196
- 197. Chapter 6: Merging 197 SVN merging precautions When SVN encounters a formal conflict during inter-branch merging, where SVN cannot reconcile one user's changes to a file in the source branch with another user's changes to the same file in the target branch, it will ask the user to intervene. Earlier in the chapter, we've seen how conflict resolution works for intra-branch merging, and the same principles apply to dealing with such conflicts in inter-branch merging. However, in the absence of these formal conflicts during inter-branch merging, the extent to which SVN will simply "do exactly as it's told" can surprise new users. It will happily make quite drastic changes during a merge, such as overwriting one column with another, without a murmur. After all, it performs what is simply a textual comparison; the VCS has no "context" to understand whether or not a change is potentially dangerous, or what the user really intends. Let's say Fred is working on our AuthorsBooks view in a branch (and has not yet merged the changes to mainline development in the trunk). In the meantime, the team working in trunk have deleted the view and implemented the same functionality in a stored procedure. If Fred were to blindly merge all revisions from trunk to branch, one might expect these changes to raise a conflict, but in fact there is a good chance they won't; Fred will lose his view, and any work he did on it, and gain a stored procedure. All this is a longhand way of saying that we must proceed with caution when approaching a potentially troublesome merge. Follow good merging practices and take necessary precautions, as described earlier in the How to Avoid Painful Merges section. In SVN, it is relatively easy to use the log to revert back to a specific revision, That is, we can roll back to the previous revision before we commit, and after we commit. In addition, SVN offers a variety of repository backup mechanisms, including "hot" backups (svnadmin hotcopy) and full and incremental backups (svnadmin dump).
- 198. Chapter 6: Merging We won't discuss repository maintenance in any detail (see http://tinyurl.com/o5jr7xg for full details), but if you do need to back up our small Bookstore repository, while working through the examples in this book, or create a repository "clone," perhaps to perform some of your own experiments, then the following simple steps will do the trick: 1. In VisualSVN Server Manager, at the SVN Server level, select Action | All Tasks | Start 198 Command Prompt. 2. Back up the repository: a. Either a full backup of all revisions, e.g.: svnadmin dump D:RepositoriesBookstoreProject > D: SVNBackupsBookstoreProject.bak b. Or a full backup of a revision range, e.g.: svnadmin dump D:RepositoriesBookstoreProject > D: SVNBackupsBookstoreProject_r26.bak -r 1:26 3. Create a new empty repository (not with the trunk/branch/tag structure) e.g. called BookstoreProject_2. 4. Restore the backup to the new repository: svnadmin load D:RepositoriesBookstoreProject_2 < D: SVNBackupsBookstoreProject_r26.bak 5. Check out the cloned repository to a new working folder, as described previously in the book.
- 199. Chapter 6: Merging 199 Merging a range of revisions Within this type of merge, we can specify either an: 1. All revisions merge – for example: a. Calculate all changes made to the trunk (FROM), since we created Branch x, and apply them TO the working folder of Branch x, or b. Calculate all changes made to Branch x (FROM), since branch creation, and apply them TO the working folder of trunk. 2. Specific range or cherry pick merge – specify one or more specific revisions, a range of revisions, or a combination of both. Try to use the cherry pick merge sparingly, and only when strictly necessary. It can quickly get confusing keeping track of what specific changes you've merged where. Generally, it is better to make small regular merges of all revisions. If you find yourself reaching for a cherry pick merge regularly, examine your development workflow, work out why and try to fix the problem. In order to see each type of merge in action, we'll return to our Bookstore example. Back in Chapter 5, our team created a new branch of the Bookstore project, called v1.0BugFixes. In this branch, Fred made some improvements to the AuthorsBooks view, adding a calculated NewBookID column. Let's continue that development effort and see how to move changes from branch to mainline, and vice versa. Let's say that Fred is continuing development work on the v1.0BugFixes branch, but is aware of the continuing work on mainline development, in the trunk, and doesn't want his bug-fixing branch to drift too far apart from the trunk. From clean working folders (no uncommitted changes), he updates both his trunk and branch directories and then performs a diff between the two to find out the extent of the changes.
- 200. Chapter 6: Merging 200 In Fred's working folder: • Launch the repository browser. • Right-click the Bookstore folder in the v1.0BugFixes branch, select Mark for comparison. • Right-click the Bookstore folder in the trunk and select Compare URLs. The resulting Changed Files dialog box, shown in Figure 6-12, summarizes the differences between the two locations. Figure 6-12: A diff comparing the Bookstore directory in branch and trunk. Double-click each change file to review exactly what changed in each case. The Author and Book files represent changes made in the trunk, not yet applied to the bug fixes branch, and vice versa for the other two files. Fred doesn't feel any of the changes to the trunk will conflict with his changes in branch so, after discussion with Barb, they decide that Barb will merge into the bug fixes branch the changes made in trunk, since branch creation.
- 201. Chapter 6: Merging 201 All revisions merge In this merge operation, we're merging mainline to branch. We want SVN to calculate the changes necessary to advance the trunk from the revision at which we created the branch to the current revision (the HEAD), and apply that change set to the working folder in the v1.0BugFixes branch. As discussed previously, we begin the merge operation from the TO directory (the desti-nation for the merge), so in Barb's working folder, right-click the branchesv1.0BugFixes folder and select TortoiseSVN | Merge, to bring up the Merge type page (refer back to Figure 6-11). Select Merge a range of revisions and click Next. In the URL to Merge from text-box, use the ellipsis button to navigate to the trunk folder. By default, we'll see the specific range radio button activated, but the range left blank, meaning we'll merge all revisions (alternatively we can explicitly select all revisions). Figure 6-13: Determining what revisions to merge.
- 202. Chapter 6: Merging Reverse Merge We can also specify a reverse merge, which is essentially a way to undo a commit. Triple-check that the merge from URL is correct, and then click Next. The Merge options page appears, as shown in Figure 6-14, providing several options for how to perform the merge. A full discussion of these options is beyond the scope of this chapter, but you can find more information in the TortoiseSVN documentation, as well as the SVN "red book" (http://svnbook.red-bean.com/). Figure 6-14: Reviewing merging option before testing or performing the merge. Accept all the default selections in the screen, and click Test merge to verify that the merge will work exactly as expected. The Merge Test Only dialog box, shown in Figure 6-15, reports that it will merge in all changes for Revision 12 (the revision that created the branch) through to the latest revision (20), and that it will entail changes to the Author and Book tables in the branch. 202
- 203. Chapter 6: Merging Figure 6-15: Verifying that the merge will perform as expected. This is exactly what we expect, so click OK and return to the Merge wizard. There, we can click Merge to begin the actual merge operation. We should see a Merge Finished screen, indicating success. Since there were no conflicting changes, TortoiseMerge has auto-merged into branch the changes made to the Author and Book tables in trunk, since 203 branch creation. Back in Barb's working folder, we should see red exclamation marks indicating that we need to commit to the repository these changes to the v1.0BugFixes branch project folder. Figure 6-16: The working folder showing changes to be committed.
- 204. Chapter 6: Merging In Barb's working folder, right-click BookstoreProject and commit the changes in the normal way. Figure 6-17 shows the Commit dialog box, listing the details of the merge. In this case, the merge operation creates Revision 21 in the repository. Figure 6-17: Committing a merge to the repository. Notice in Figure 6-17 that the folder listing shows modified (property change only). This refers to the fact that SVN notes in the folder's metadata that a merge occurred (the folder structure itself has not changed). Fred can now update his BookstoreDatabase_v1.0BugFixes working folder to capture the changes. Work would continue on branch, with the team regularly merging the latest changes to trunk into their v1.0BugFixes branch. At a certain point, however, the team may want to transfer the bug fixes into mainline development. For the sake of this example, we can stick with the branch changes already applied in Chapter 5 (which added a comment to the AuthorBook table and adding the NewBookID calculated column to the AuthorsBooks views). These changes do not conflict with 204
- 205. Chapter 6: Merging any changes in the trunk, so we can simply perform an all-revisions auto-merge in the opposite direction (branch to trunk). In this case, we want SVN to calculate the changes necessary to advance the branch from the revision at which we created the branch (R12) to the current revision in the branch (R21), and apply that change set to the working folder in the trunk. After ensuring that working folders are up to date and contain no uncommitted changes, we start the merge operation from Barb's trunk folder (C:SVNBarbBookstoreProject trunk) since we want to merge the changes TO this folder. The URL to merge FROM is //<server>/svn/BookstoreProject/branches/v1.0BugFixes. The test merge confirms that we're merging into the trunk the effects of Revisions 12 through 21 in the branch, and that it involved modification of the AuthorBook and AuthorsBooks files, in the trunk. Figure 6-18: Verifying our merge operation into the trunk. Once the merge completes, Barb commits the changes to create Revision 22. Fred can then perform an update of his BookstoreDatabases working folder to capture the changes. 205
- 206. Chapter 6: Merging 206 Cherry pick merge The second type of "range of revisions" merge is what we call the "cherry pick," where we want only to move a specific change, or a specific range of changes, from one location to another. The cherry pick works very similarly to the all-revisions merge we just saw, so we'll work through this example very quickly. In one of our developers' v1.0BugFixes branch folder (for example, C:SVNBarbBookstoreProjectbranchesv1.0BugFixesDatabases Bookstore), we'll make three separate changes to our Bookstore database (three separate commits), as follows: • R23 – Create a new BookCategory table to define the categories in the Book table, rather than supporting arbitrary category names: • Save the new BookCategory.sql to Tables and commit. • R24 – Populate the BookCategory table with lookup data: • Create a new LookupData folder, save the new INSERT script, BookCategoryData.sql, to this folder, and commit. • R25 – Update the Book table to reference the BookCategory table • Edit the existing Book.sql script and commit. Listing 6-3 shows all three changes in a single script (purely to save space). Save BookCategory.sql and BookCategoryData.sql to separate files, as described above, and then edit the existing Book.sql file, as described in the listing.
- 207. Chapter 6: Merging 207 /*BookCategory.sql*/ USE Bookstore; GO CREATE TABLE dbo.BookCategory ( BookCategoryID INT PRIMARY KEY , CategoryName NVARCHAR(50) NOT NULL ); GO /*BookCategoryData.sql*/ USE Bookstore; GO INSERT INTO dbo.BookCategory VALUES (101, 'Art & Architecture'); INSERT INTO dbo.BookCategory VALUES (102, 'Children'); INSERT INTO dbo.BookCategory VALUES (104, 'Fiction'); INSERT INTO dbo.BookCategory VALUES (103, 'Food & Beverage'); INSERT INTO dbo.BookCategory VALUES (105, 'History'); INSERT INTO dbo.BookCategory VALUES (106, 'Mind & Body'); INSERT INTO dbo.BookCategory VALUES (107, 'Religion & Spirituality'); INSERT INTO dbo.BookCategory VALUES (108, 'Sports & Recreation'); INSERT INTO dbo.BookCategory VALUES (109, 'Technology'); INSERT INTO dbo.BookCategory VALUES (110, 'Travel'); GO /*Book.sql*/ USE Bookstore; GO CREATE TABLE dbo.Book ( BookID INT IDENTITY PRIMARY KEY , Title NVARCHAR(1000) NOT NULL , YearOfPublication INT , BookCategoryID INT , -- replaces BookCategory VARCHAR(15) CONSTRAINT fk_bookcategory FOREIGN KEY ( BookCategoryID ) REFERENCES BookCategory ( BookCategoryID ) ); GO Listing 6-3: Changes relating to the new BookCategory table and lookup data.
- 208. Chapter 6: Merging For the Book table, notice that we're changing the existing BookCategory VARCHAR(15) column to BookCategoryID INT and adding a FOREIGN KEY constraint that references the new BookCategoryID column in the BookCategory table. In our development environment, the existing BookCategory column contains no data, so we can get away with applying this upgrade to our "live" Bookstore database by simply altering the Book table to remove that column and adding the new column definition, and constraint (the code download for this chapter provides an ALTER TABLE script that does this). However, this is a classic example of an upgrade that would require a lot of care in any environment where the BookCategory column contained data. We'd need to be sure to make the upgrade in a way that preserved the integrity of any existing data. We'll discuss this sort of issue in more detail in Chapter 8. Now comes the fun part. The powers that be decide that we should add only the BookCategory table to the trunk for version 2.0, as well as update the Book table. As a result, Barb plans to merge to the trunk Revisions 23 and 25, but not Revision 24, because it is as yet undecided how to handle the lookup data in v2.0. For this situation, we need the cherry pick merge. We start the merge operation in Barb's trunk folder (C:SVNBarbBookstoreProject trunk) since we want to merge the changes TO this folder. Again, we choose the Merge a range of revisions option, and the URL to merge FROM is //<server>/svn/ BookstoreProject/branches/v1.0BugFixes. This time however, instead of leaving the specific range blank (indicating all revisions), we specify just two specific revisions by entering 23,25 (or use Show log to locate the correct revision numbers on your system). Figure 6-19 shows how the Merge revision range page of the Merge wizard should look. 208
- 209. Chapter 6: Merging Figure 6-19: Merging specific revisions from the branch to the trunk. A Test merge will verify that the merge will successfully perform the changes you expect. If everything looks good, we can execute the merge and return to Barb's folder where we'll see in the trunk the new BookCategory.sql file (R23), and the modified Book.sql file (R25), but not the LookupData folder or file. We can have Barb commit the merge at this point, and it should go fairly smoothly. We know no one has been working in the trunk since our last merge, so SVN should perform an auto-merge with no conflicts, and designate the commit as Revision 26. 209
- 210. Chapter 6: Merging 210 Merge two different trees In the "range of revisions" merge, SVN calculates the change set to describe a string of revisions to one branch and applies them to another branch. The "merge two trees" merge is somewhat different. In this type of merge, SVN compares two different trees with the goal of making them identical. Let's say we've made our final bug fix to our v1.0BugFixes branch and want to merge it into trunk and close off the branch (delete it). If we perform a "merge two trees" merge, from branch to mainline, SVN will compare the two trees (trunk and branch) and calculate the changes necessary to get from the current revision in the trunk to the current revision in the branch, and apply those changes to the trunk of the user's working folder. This type of merge requires a lot of care. We need to be certain that no one has committed further changes to the trunk since we last merged all revisions from trunk to branch, otherwise we risk losing those changes to the trunk (we'll discuss this in a little more detail in the Inter-branch merges with conflicts section). With this note of caution in mind, let's walk through an example. Step 1: Modify AuthorsBooks view in the branch Let's make some final changes to our v1.0BugFixes branch in Barb's working folder. Listing 6-4 shows an updated definition for our AuthorsBooks view, with the changes highlighted. We remove the NewBookID computed column, add an alias to the Title column, add another inner join to the BookCategory table, include the CategoryName column in the SELECT list, and define a CASE expression to handle the author names.
- 211. Chapter 6: Merging ALTER VIEW AuthorsBooks -- retrieves all authors and all their books AS SELECT ( CASE WHEN MiddleName IS NULL OR MiddleName = '' THEN a.FirstName + ' ' + a.LastName ELSE a.FirstName + ' ' + a.MiddleName + ' ' + a.LastName END ) AS FullName , b.Title AS BookTitle , bc.CategoryName AS BookCategory , b.YearOfPublication AS YearPublished 211 FROM Author a INNER JOIN AuthorBook ab ON a.AuthorID = ab.AuthorID INNER JOIN Book b ON ab.BookID = b.BookID INNER JOIN BookCategory bc ON b.BookCategoryID = bc.BookCategoryID; GO PRINT 'dbo.AuthorsBooks: view creation script finished'; Listing 6-4: Updating the AuthorsBooks view in the v1.0BugFixes branch. Update the copy of AuthorsBooks.sql, in C:SVNBarbBookstoreProjectbranches v1.0BugFixesDatabasesBookstoreViews, as described in Listing 6-4, and then commit the change, which creates Revision 27 in the repository. In order to test the view, we can populate the BookCategoryID column of all rows in the Book table with the value 104, which points to the Fiction category in the BookCategory table. The relevant portion of the new TestData.sql file will look as shown in Listing 6-5. INSERT INTO Book ( BookID, Title, YearOfPublication, BookCategoryID ) VALUES ( 1, 'Huckleberry Finn', 1884, 104 ), ( 2, 'Slaughterhouse-Five', 1969, 104 ), (...etc...) Listing 6-5: Loading test data for the BookCategoryID column. Commit the changes to the repository to create Revision 28.
- 212. Chapter 6: Merging Step 2: Merge two trees (branch to trunk) We're going to merge the branch into the trunk, but this time around we're going to do a tree merge, which merges all the files. The assumption is that the team have decided to use the lookup data in 2.0, so everything can now be merged. In other words, all changes in the branch will be incorporated into the trunk. Barb should be sure to perform a final update of her working folder before going ahead with the merge. To perform a tree merge, navigate to the trunk folder in Barb's working folder and launch the Merge wizard. When the Merge type page appears, select the Merge two different trees option. Next comes the confusing part. On the Tree merge page, make sure the URL displayed in the From text-box points to the trunk, and the URL in the To text-box points to the branch, as shown in Figure 6-20. This might seem rather odd at first, and it is, but TortoiseSVN thinks in terms of where the roots of the branch come from (the From text-box) and where they have ended (the To text-box). Figure 6-20: Performing a tree merge from the branch to the trunk. 212
- 213. Chapter 6: Merging Those are the only options we need to set to perform a tree merge. To recap, we're expecting to merge the new LookupData folder and BookCategoryData.sql file into trunk, since we skipped those in the previous cherry pick merge, plus our new changes to the AuthorsBooks view, and to TestData.sql. We can confirm this with a test merge. Figure 6-21: Viewing the results of the tree merge. If it all looks good, execute the merge operation. Step 3: Commit the effect of the merge to the repository With the merge complete, the trunk in Barb's working folder should show that it has changes ready to commit to the repository. Go ahead and commit those changes. This creates Revision 29 in the repository. Given all the changes that Barb made to the AuthorsBooks view, you might have expected SVN to baulk at the merge, but it automatically incorporates all Barb's changes, overwriting anything that gets in the way. We can confirm this with a diff between the most current revision and the preceding revision. Right-click the AuthorsBooks.sql file in the trunk, point to TortoiseSVN, and then click Diff with previous version. 213
- 214. Chapter 6: Merging This launches TortoiseMerge and compares the two versions of the file, as shown in Figure 6-22. Figure 6-22: Comparing revisions of the merged AuthorsBooks.sql file. When we did the merge, SVN treated the revised copy of the file in the branch as the correct version of the file and incorporated all changes. The assumption in this case is that Barb knew what she was doing and, by merging the files, she was essentially giving her blessing to these changes, in both the branch and the trunk. Inter-branch merges with conflicts: down the SVN merge rabbit hole As we've tried to stress throughout this chapter, merge operations of all types require extreme care, or you can very quickly end up in a mess. So far, we haven't run into any formal "conflicts" from SVN's point of view; as we've seen, it's pretty ruthless about the sort of changes it will make without complaint during inter-branch merges. However, it's very easy to merge in changes that SVN won't complain about, but your team members will. It's also possible to run into what SVN regards as a formal conflict. We're not going to offer fully worked examples here, as it's a deep rabbit hole, but we will offer some idea of what can go wrong so that you can practice and experiment on your own systems. 214
- 215. Chapter 6: Merging In our previous merge to trees example, we stressed the importance of no one else making changes to the trunk, after merging trunk to branch and before performing the tree merge (branch to trunk).However, what if someone did make a change? Let's say that, after our cherry pick merge to trunk (of the BookCategory.sql file and modified Book.sql file), but before we merged the two trees, Fred made a further change to the Book.sql file in trunk. Cloning the repository If you want to follow along from here in, you may want to create a clone of the BookstoreProject repository. I created a backup of the repository as it existed after the cherry pick merge (Revision 26, in my case) and restored it to a new copy of the repository, exactly as described in Steps 1-5 in the previous SVN merging precautions section. So, let's rewind in time and pick up on our previous example straight after the cherry pick merge. The new sequence of events from this point might be as follows: 1. FRED: Edits the Book table in trunk. For example, changes the Title column to BookTitle, adds a BookPageCount NVARCHAR(5) column, and commits the changes. 2. BARB: In the v1.0BugFixes branch, she modifies AuthorsBooks view as described in 215 Listing 6-4 and commits the change. 3. BARB: Performs an update of her working folder. 4. BARB: Performs the "merge two trees" as described in the Step 2: Merge two trees section of the previous example. When we have Barb execute Step 4, SVN will see no problem. It will simply report that, as part of the "merge two trees" merge, it plans to update Book.sql in trunk, as well as add Barb's new LookupData folder and file, and her update to the AuthorsBooks view, as shown in Figure 6-23.
- 216. Chapter 6: Merging Figure 6-23: A potentially problematic "merge two trees" operation from the branch into the trunk. If Barb doesn't have her wits about her, and simply performs the merge and commits the changes to trunk, the net result is that Fred's changes are simply lost. Fred might consider this a conflict, even though SVN doesn't! Alternatively, let's imagine that Barb is eagle-eyed enough to spot this problem before committing, and so reverts the effects of the merge. How does she resolve this problem? Let's say she tries an "all revisions" range merge from trunk to branch, to capture Fred's changes into the branch. Figure 6-24 shows the error messages Barb receives when she tries this type of merge. The problem she has here is that she can now only merge all revisions from trunk to branch, if the branch hasn't changed since the last time she merged all revisions from branch to trunk. Figure 6-24: A failed "all revisions" range merge from the branch to the trunk. 216
- 217. Chapter 6: Merging Instead, she'll need to make sure no one is working on trunk, and cherry pick from trunk to branch just the revision relating to Fred's Book.sql edit, and commit it. Now that her working folder in branch has Fred's latest change to the trunk, she can perform a "merge two trees" merge safely. The net result should be that Fred's change is preserved and the branch and trunk trees are now identical. As you can tell, the situation can get complicated quickly, and it can get even worse. Let's rewind in time again, back to just after the original cherry pick merge (Revision 26, if your revision numbers exactly match those in the books), and consider this new workflow, containing an extra change by Fred: 1. FRED: Edits the Book table in trunk. For example, changes the Title column to BookTitle, adds a BookPageCount NVARCHAR(5) column, and commits the changes. 2. BARB: In the v1.0BugFixes branch, she modifies AuthorsBooks view as described 217 in Listing 6-4 and commits the change. 3. BARB: Performs an update of her working folder. 4. FRED: edits the AuthorsBooks view in trunk to reflect his changes to the Book table. 5. BARB: Performs the "merge two trees" as described in the Step 2: Merge Two trees section of the previous example. Now, when Barb attempts the "merge two trees", not only does she have the problem of potentially overwriting Fred's changes to the Book table, in trunk, but also an out-of-date copy of the AuthorsBooks view. Figure 6-25 shows the formal conflict, although SVN lists even this as a "maybe."
- 218. Chapter 6: Merging Figure 6-25: A very problematic "merge two trees" operation with conflict. If Barb decides to press ahead with the merge anyway (we hope she wouldn't do this!) she would be invited to try to resolve the conflict, as shown in Figure 6-26. Figure 6-26: Taking steps to resolve conflicts in the AuthorsBooks.sql file. Normally, of course, Barb will want to view each conflict and attempt to resolve it with the help of TortoiseMerge, as described earlier in this chapter, in which case she would choose Edit conflict. However, she also has the options to Prefer local or Prefer repos-itory. These options are not self-explanatory. In short, Prefer local means "in the case of a conflict prefer what is there already over the incoming change" and Prefer repository means the opposite. SVN will auto-merge all other changes 218
- 219. Chapter 6: Merging So, for example, if Barb selects Prefer local, SVN will prefer Fred's local changes to the working folder in the destination (trunk) over the changes Barb is attempting to merge into trunk. In general, it's worth avoiding these options unless you're absolutely positive it's what you want. Likewise, it will be rare to simply opt to let SVN auto-merge what it can and leave everything else in a conflicted state for you to resolve later." Barb's situation is getter rather dire here. Her merge is still going to update Book.sql and so will remove the changes Fred made to that file. So, she's trying to edit a conflict on a view that she changed in the branch and that Fred changed in trunk, while another part of the merge will remove the reason for Fred's change to the view! At this stage, the best choice is probably to abort the whole sorry mess of a merge, have a word with Fred and try to sort the mess out manually. Merge with caution! 219 Summary Merging can be the scariest of all source control functions. Working with revisions, updating your working folder, committing changes to the repository, even creating a branch, are all easy operations in comparison. But merging calls for a different awareness altogether. You're trying to join together two different, but related, lines of development into a single structure. Depending on your circumstances, you might need to perform inter-branch merges or intra-branch, and the merge might occur automatically or you might have to step in to decide how the merge is carried out. In most cases, however, as long as you update your working copy often, commit small changes often, and merge often, you should be okay. Certainly, merging (and the branching that goes with it) should be used carefully and judiciously, but they can both be valuable tools in helping to streamline and maximize your development efforts. In Chapter 7, we'll move on to take a look at the bigger picture, in terms of deploying a database from what we have in source control. There, you'll get a better sense of how source control is just one piece in a much larger puzzle.
- 220. Chapter 7: Manually Deploying Databases from Source Control We've covered a lot of ground since Chapter 3, when we first discussed all of the planning considerations that help determine how we create, organize and modify database files in source control. Over the course of Chapters 4 through 6, we discussed how to manage concurrent editing of our database files, and how the VCS maintains every version of every file, allowing us to see an audit history of every item that comprises our database. We also looked at how to use branching and merging to streamline the development efforts of the team. It's time now to pull all the pieces together and build and deploy a database. Our stated goal, right from the start, has been that, once the development effort was finished, we would have in source control a single source of "truth" for defining precisely the database we wish to deploy to our users, along with the ability to deploy that database to any given environment (such as the test, staging or production environments), either to create the database from scratch or to upgrade an existing database. This chapter will start by taking a closer look at some of the potential issues surrounding database deployment, specifically as they relate to deploying new databases, and deploying upgrades to our existing databases. We'll then walk through an example of each type of deployment for our Bookstore database. • Building a new Bookstore database from scratch – from the latest version of all the CREATE scripts in the VCS. • Upgrading an existing Bookstore database – we'll store in the VCS the migration scripts necessary to describe a series of changes, and then string them together into an upgrade script. 220
- 221. Chapter 7: Manually Deploying Databases from Source Control In terms of tools, we'll rely on nothing more than a build table to track which build we've deployed to which environment, and a PowerShell script to string together in the correct order the scripts that comprise the build. As such, this chapter provides only a solid but basic and rather manual way to deploy our databases. It's a good starting point but we'll soon want to introduce further automation. Chapter 8 will describe a more automated and tool-based approach, and will explore how to work with migration scripts, which will allow us to deploy more complex changes, ones that we need to map out carefully and precisely, in order to ensure we preserve the integrity of existing data. Database Deployment Pitfalls Regardless of the approach we take to deploy a database, the main obstacles to database deployment are often the same. Back in Chapters 3 and 4, we discussed a few of the major challenges, including: • The vast array of object and configuration files that comprise a database and its environment – all of which, potentially, we need to script and maintain in the VCS, and include in the build. • The need to maintain a table's "state" when we upgrade an existing database – a database is all about data persistence. We can't simply recompile a database and roll it out over the top of the existing one. We often have to provide scripts to modify the schema, migrate and update data, or take other steps to protect and archive data. • The need to minimize downtime and user disruption during database upgrades – if the database is down, a large part of the business is down. In this section, we'll consider three more potential obstacles in the path to smooth database deployments, in the form of version drift, improperly tested upgrade scripts, and failure to consider how to roll back problematic deployments. Each of these can, and frequently does, cause failed deployments, or deployments that introduce problems. 221
- 222. Chapter 7: Manually Deploying Databases from Source Control 222 Version drift Some organizations, even those that perform due diligence when it comes to their application files, make most or all their production database changes directly to the database, on an ad hoc basis. Such changes are often at the expense of lost data or broken applications, and can even threaten the integrity of the database itself. When this happens, it is a very difficult task to work out what change caused the problems, and when. The original script files, if they still exist, might be spread across multiple file servers or developers' desktops – duplicated, overwritten, out of sync. It's a mess. Even with database source control practices in place, it can be tempting for users to make small, "quick fixes" directly to a live database, rather than going to the trouble of making the change in source control, retesting everything, and requesting a new deployment. It's particularly hard to resist when users and managers are clamoring for an urgent fix to go live "yesterday." Suppose the latest database build (2.4.6) in production reflects exactly the most recent version of your schema in source control. However, QA discovers a bug, in the form of a malfunctioning stored procedure, that's affecting online users. A manager calls the DBA in the middle of the night and demands that she fix the bug immediately. She tracks the problem to a data type mismatch. Either she alters the stored procedure directly in the production database now, or she follows the change control procedures and alters it in source control and requests a new deployment. In the latter case, the bug won't get fixed till sometime the next day. She opts for keeping her job and so modifies the appropriate stored procedure parameter, in the production database, goes back to bed, and promptly forgets all about it. At this point, although the DBA has fixed the immediate problem, she's introduced version drift, and likely stored up bigger problems for down the line. The production database is out of sync with the build 2.4.6 files in source control. Any further changes in the VCS that assume the original data type may no longer work correctly when deployed to production.
- 223. Chapter 7: Manually Deploying Databases from Source Control At that moment, the fun really begins, as everyone tries to figure out what went wrong and how everything got out sync. Let's hope the DBA isn't still catching up on her sleep. A VCS is effective only if all database developers and DBAs, and anyone else with their hands on the database, make all changes through the VCS, and use official deployment processes. If even one person fails to follow protocol, it will inevitably lead to problems. Unfortunately, it's nearly impossible to prevent direct changes to a database, but we can take steps to "encourage" our developers to follow the straight and narrow. For example, we can implement DDL triggers to catch database modifications, or include in our deployment process a verification step that compares schemas in order to ensure that the target database is in the state we expect. Lack of upgrade script testing Many development teams will automate the testing of their application code, but not apply the same rigor to testing database changes. Ideally, each time we make a database change we should, for both database and dependent application: 1. Perform some basic tests locally to ensure the new functionality works as expected. 2. Commit the change (and any new tests) to the VCS. 3. Automatically deploy the change from the VCS to the integration environment, to which all other developers' changes are also deployed, and run the team's full suite of tests. 4. Periodically deploy to a production-like environment (such as QA) and perform a further range of tests, including performance testing under a full workload. 223
- 224. Chapter 7: Manually Deploying Databases from Source Control 224 Development testing When a developer makes a database change, he or she should verify the change locally, using some basic unit tests, and then commit the change and any new tests to the VCS. Unit testing databases It is out of scope to cover this topic in any detail, but we recommend Alex Kuznetsov's series of articles as a good starting point for what's required. See http://tinyurl.com/oxu6tnz. Each and every commit will, ideally, trigger an automated build process, which "contin-ually integrates" each change with the changes of others, and tests it. In other words, a process that automatically deploys every commit to an integration (or "CI") server, loads test data and runs the team's growing suite of tests. Loading test data into development databases in order to perform testing is not neces-sarily a trivial issue. It's often unrealistic for developers, or even the QA team, to work with the full set of production data in their sandbox or test environments, simply because the database is too big, contains sensitive data, or both This is one reason why some teams work on a shared database server during development (see Chapter 3). One way to do it, assuming data sensitivity is not an issue, is to pull a subset of data from the production database. However, the new database might have enough schema changes that such a migration becomes a significant task. An alternative is to script out a subset of data or create a staging database from which to pull test data that is safe to use. In either case, preparing the test data for a large database can turn into a time-consuming process. Even so, it's better to spend the time up front preparing test data that everyone can use, rather than everyone trying to create their own data sets on the fly.
- 225. Chapter 7: Manually Deploying Databases from Source Control Generating test data Hugo Kornelis offers a detailed description of how to generate test data in T-SQL: http://tinyurl. com/qyn8q3b. If time is in shorter supply, as is almost always the case during a development effort, you might consider a tool that automatically generates test data, such as SQL Data Generator from Red Gate Software. See Phil Factor's article for some tips: http://tinyurl.com/oktjh8d. By carrying out this form of basic integration testing, we verify that all dependent appli-cations communicate correctly with the database and behave as expected. In other words, integration tests ensure that a change doesn't introduce any new problems (regressions), and that the application code and database can properly exchange data. Production-readiness testing At some point in the development process, ideally as early as possible, the team must verify that the database (and application) builds and works in a production environment just like it does in development. Often, QA testing in an environment that mimics that of the production server is missing, or happens in a rush just prior to the deployment deadline. It is often at this late stage, when problems are hard to fix without causing substantial delays, that DBAs discover problems, for example with security and configu-ration settings, or find that the updates have badly degraded database performance. DevOps and early deployments One of the drives behind the "DevOps" movement is to promote the ideas, tools and automation that will allow developers to refine their deployment processes to the point where they can deliver early and often to a "production-like" environment. From here, DBAs and operations staff can review deployments and advise on possible issues much earlier in the project lifecycle. 225
- 226. Chapter 7: Manually Deploying Databases from Source Control Briefly, the team need to plan in time and establish the appropriate environments to perform several different types of database testing. For example: • QA/regression testing – Verifies whether the latest changes introduced any new problems into the production environment. Regression testing a database can include testing data load and extract operations, as well as the application's access and transactional operations. • Load testing – Verifies that the database can perform as expected when handling a realistic data load and production-level workload. 226 No rollback plan Many deployments to production, application or database, lack a "get-out clause." What if we're halfway through a database upgrade and our deployment fails? What if we upgrade the database and we or worse, our customers, discover that we've introduced a serious bug? How do we back out, or roll back, our changes and return the database to its previous consistent state? Rolling back database changes is a tricky business. Again, we're up against the issue of data. Schema changes are easy to roll back, but not changes to data. Still, we have to start somewhere. At the very least, before starting any sort of deployment, we should back up the target database. This provides an immediate restore point to the moment before the deployment process started. Unfortunately, this approach doesn't account for data that might have changed during deployment, such as a new sales order or an update to a medical record. Important information could be lost.
- 227. Chapter 7: Manually Deploying Databases from Source Control Some deployment processes, in addition to creating a backup, also create database scripts that reverse schema and data changes. This requires the deployment plan to include a specific rollback component that defines how to handle each operation that we poten-tially need to roll back. The component also records the order of the operations to ensure that rollback can occur in the correct order. Database Migration Scripts: Getting from place A to place B Phil Factor's post illustrates the sort of thing that's required. See: http://tinyurl.com/oe63cta. Even with rollback scripts, however, rolling back a database is no small task, and the possibility of lost data still exists. A deployment strategy that includes a rollback component requires carefully considered procedures and scripts to carry out the opera-tions. If we could lock the database against write operations during the deployment process, we could save a lot of trouble, but such an approach is often not possible because there's usually only one database and when it is offline the business is offline. Of course, we'll want to deploy a production database when traffic is at its lowest, but we might still have to keep our database online and available to users during the deployment process. In such cases, our rollback solution may depend not only on our tools and scripts but on other SQL Server tools and components, such as replication or the transaction log. Building and Deploying Databases A VCS should be integral to any development effort, but it remains only one piece of a much larger puzzle. As we discussed in Chapter 3, most well run product teams have in place Application Lifecycle Management (ALM) strategies to handle all aspects of appli-cation and database development, from inception and planning to maintenance and support. All these strategies and tools need to fit together smoothly in order to ensure successful releases. One of these pieces is the structure that defines how we build and deploy our applications and databases to their target environments. 227
- 228. Chapter 7: Manually Deploying Databases from Source Control Having all the correct files checked into source control is only the first step in a build and deployment process. After that, we need in place a build-and-deploy system, usually a set of policies, processes, and automated tools, which know the files to retrieve from source control, how to use those files to build the applications and databases, what steps to take and in what order to take them. Overall, this system must put in place all the necessary processes to ensure that we build the correct versions of the correct components in the correct order on the correct systems. The build and deployment processes vary greatly from one organization to the next and, to add confusion to the mix, an organization likely deploys applications very differ-ently from how it deploys databases, using different tools and procedures for each. For example, the degree of automation can vary, despite that fact that the more manual the process, the more time it consumes and the more it is prone to error. All that said, Figure 7-1 attempts to provide a high-level, conceptual overview of how we might deploy a database. It represents only one of many possible ways to approach database deployment but it should, at least, provide you with a good sense of the issues our deployment system needs to address. Not surprisingly, the build process starts with source control. The build engine then performs the following steps: 1. Retrieves the files from the repository and copies them to the build share, a network share created specifically to support build operations. 2. Connects to the target database and, optionally, creates a backup; whether it takes a backup depends on the size and purpose of the particular database, but we do need some way to ensure that we can restore the database to its original state should something go wrong. 228
- 229. Chapter 7: Manually Deploying Databases from Source Control 3. Prepares for the build – this might entail creating build packages or taking some other action to prepare the source files and any supporting files for the build process. a. If replacing the target database with a clean build, the engine deletes that database and creates the new one by using the files it retrieved from source control. b. If updating the target database, the engine instead applies the update 229 scripts to the existing database. 4. Verifies the database creation or update. 5. Inserts the current build number. The build is then complete and the database deployed to its target environment. Again, this is a very simple overview of the deployment process as a whole, and in particular of the area enclosed in the dotted lines in Figure 7-1. Preparing the source files, coordinating the various tasks, handling errors, rolling back changes, verifying each step, and the assortment of other tasks, represent a series of sophisticated operations, different for each deployment solution, with no one way being definitively the right way to approach any of it.
- 230. Chapter 7: Manually Deploying Databases from Source Control 230 Figure 7-1: Building or updating a database. Database build numbers Build numbers provide a way to easily identify a particular application or database build. Build numbers are especially important in database development. They make it possible to distinguish the structure of one database version from another. If the build numbers are different, we should assume that the schemas are different. If the build numbers are the same, we should be able to assume that the databases share the same schema, unless
- 231. Chapter 7: Manually Deploying Databases from Source Control the team permit changes directly to the database. However, if you do then, as discussed in the previous section on Version drift, it's difficult to trust anything you see because direct changes essentially undermine the advantages of assigning build numbers, as well as many of the advantages of source control. For example, if a developer and tester are working against different SQL Server instances, but the same database build is installed on both instances, the assumption would be that the databases are the same. However, if someone were to directly modify the developer's database, the tester could run into unexpected results when running a stored procedure because of unknown differences in the schema. The bottom line? Don't permit direct changes to a database unless you can confirm with absolute certainly that every one of those changes will be properly scripted and added to source control. Even then, we'd recommend against it. Direct changes make it difficult to know which version of the database is the correct one. Build numbers also help to ensure that when we update a database, we're updating the correct build. For example, a set of upgrade scripts developed for build 2.0.33 will fail, or lead to unexpected results, if we run them against build 1.3.7. The build number serves as the universal identifier that all users, applications, and processes can reference to verify procedures and carry out tasks. Our database deployment process must include a mechanism to assign a new build number to a database build. Most tools that automate the database deployment process include this capability. Even if they don't, it's easy to add the script file necessary to assign that number (we'll see an example of how to do this later in the chapter). The question then becomes how to store the build number. Many teams, and some third-party deployment tools, use the database's extended properties for the build number. This approach is fine if all your database systems support extended properties, but many don't, such as Windows Azure SQL Database (formerly SQL Azure). To get around this, we can, for example, store the build number in a table-valued function instead. 231
- 232. Chapter 7: Manually Deploying Databases from Source Control Rather than support multiple systems for assigning build numbers, it's better to devise a single solution that works for any environment. One safe, cross-platform solution is to add a table to your database that tracks build numbers. Its structure is completely customizable, as long as we can query the table to retrieve the current build. Unfortu-nately, some organizations have restrictions against adding tables, which is why teams and tools often turn to extended properties or table-valued functions. Of course, whenever we rebuild the database, we'll also rebuild this table so it will only ever contain one row, the current build. However, whenever we update the database, it will track each new build. Later in this section, we provide an example of a table that tracks build numbers. We can assign build numbers based on whatever numbering system works best for the team. Many teams opt to mirror the application's build number for the database build number. Another common approach is to leverage the build number assigned to a revision in source control, via a tag. Often the application and VCS number are one and the same. Either of these approaches is usually fairly easy to implement and maintain. Build numbers are often sequential for ease of tracking (though they don't have to be) but other than that, a build number can be a single number, have multiple parts, or be a mix of alphanumeric characters. We can, if desired, tie all or part of the number to our product version number. For example, we might have a three-part build number in which the first part reflects the product version. The exact build numbering scheme is flexible, as long as each build number can identify uniquely a single database build, and any two databases that share that number also share the exact schema. 232
- 233. Chapter 7: Manually Deploying Databases from Source Control Deploying a new database build In Figure 7-1, only the left portion of the area enclosed in the dotted lines refers to building a database from scratch, and yet many steps can go into building a database, and the larger and more complex that database, the more steps involved. Above all, we must create the database objects in the correct order. We cannot, for example, create a foreign key that references a column that does not yet exist, and we cannot create views that reference tables that don't exist. Our build process must account for all dependencies, not only among the database objects, but also those outside the database, such as login accounts and server configurations, with an assurance that no objects, configuration settings, variable values, security principals, or any other components are missed. Although the methods used to build a database will vary from one team to the next, we can separate them into three broad categories: 1. Uncontrolled, ad hoc deployments. 2. Manual deployments from source control, via scripting. 3. Automated deployments from source control. Ad hoc database deployments This is database deployment at its most basic. We use a tool such as SSMS to build and test our database. When we're done, we script it out and run those scripts, in the right order, to create the database on the target environment, and then we add the necessary lookup data. Alternatively, we might use a schema comparison tool to compare our devel-opment database to the database as it exists in the target environment and let the tool generate and run the necessary deployment script to synchronize the two environments (and use a data comparison tool to synchronize the reference data). 233
- 234. Chapter 7: Manually Deploying Databases from Source Control At the very least, we should save to source control the individual script files for each object, and the reference data, along with the deployment script. However, even if we do this, the essential problem remains that we are deploying from a moving target – a database running on a SQL Server instance – rather than from a known version in source control. It quickly becomes very difficult to track which changes are deployed where, and very easy for what's in source control to become out of sync with what's in development or in each of the deployment environments. Its obvious shortcomings make it an approach suited only to very small databases and usually only for testing or development purposes. Sadly, it is surprisingly common, not only for development and test databases, but also for ones in production. Hopefully, this far through the book, it's not an approach you'll consider for long. Scripting database deployments from source control This is still a manual deployment method, but its cornerstone is that we save to our VCS all scripts and all changes to those scripts, and we only ever deploy to any environment from a known revision in the VCS. Rather than define manually the order in which we must run the script to deploy the new database, we create a deployment script file, such as a batch or PowerShell file, which serves as a master file that runs the SQL script files for us, in the prescribed order. Whenever we want to create or recreate the database, we run the master script, which must spell out each and every task that's required in order to build each database. Whenever we make a change to the database that affects how the object creation and data load scripts should be run, we must carefully (and manually) update the master file. Of course, the more tasks we can include in the master file, the better we can automate the process. We can use the master file, not only to call SQL script files, but also for other types of files, such as additional batch or PowerShell files. 234
- 235. Chapter 7: Manually Deploying Databases from Source Control We can even set up the master file to accept arguments so that, when we call it, we can specify the instance on which to create the database and create it on as many instances as we like. It's a much better approach, and one we demonstrate in this chapter, but still subject to the failing of any manual approach. It's easy to make a mistake. For example, if we forget to include an object creation script in the master file then it won't be deployed. If it's a code object, or a table without dependencies, the database will build correctly, but the object will be missing and so will cause application errors. Automating database deployments Not surprisingly, the most effective strategy for ensuring reliable, repeatable builds is to automate every step of the build and deployment process, from accessing the correct files in source control to assigning a build number, to deploying it to specified target environments. The next step for a team that already works from VCS, and has scripting in place to semi-automate builds, is to start using available tools to automate as many aspects of the build process as possible. An organization might develop its own tools for automating deployment, but most will use tools available with the database management system or their VCS, or commercially available tools that plug smoothly into their broader ALM strategy (see the Planning for Database Source Control section of Chapter 3). As noted earlier, in many organizations there is a disparity between deployment mecha-nisms for applications and those for the databases. For applications, it's becoming more common to see fully automated build processes. The application team add their files to source control, and the build mechanism takes over from there. At the end of each day, or even every time a team member commits a change, there is an automated build from source control to the test environment, followed by an automated suite of integration tests, so that the team deal with all problems as soon as they happen. 235
- 236. Chapter 7: Manually Deploying Databases from Source Control It's common, however, that the database will deploy by a much less frequent, much less automated process. In Chapter 2 we discussed the technical and cultural reasons for this, such as the difficulties of refactoring a live database while preserving the data but, nevertheless, the drive in many teams is towards a unified deployment process for both the database and the application. The database team add their files to source control right alongside the application files, and the framework incorporates the tools necessary to automate building the database alongside the application. In this chapter, we only go as far as scripting the database build process, but Chapter 8 demonstrates an automated approach, based on use of the Red Gate's SQL Source Control SSMS plug-in for managing the database development process via a VCS, with integration into a build server for Continuous Integration. Upgrading an existing database Once we've deployed a database to a production environment and it's populated with real data, we're much more likely to upgrade the existing database rather than re-create it from scratch. The scripts we run during an upgrade deployment must account for the need to preserve the table's data from one version of the database to the next. As discussed in Chapter 4, as the team modify a database during the development process, they can elect to save to the VCS either the latest CREATE script for each object (versioning the current state of the database), or the string of upgrade scripts required to advance the object from one defined state to the next. Some teams adopt the former method during development, when it's permissible to tear down and rebuild databases, but switch to storing upgrade scripts after the first deployment to production. In any event, the choice of how to version the database will, in turn, affect the tools and processes the team use when it comes time to apply these scripts during the database upgrade process. In some teams, the approaches used to upgrade a database can be a bit haphazard, sometimes applying changes directly to the database and sometimes through 236
- 237. Chapter 7: Manually Deploying Databases from Source Control upgrade scripts, leaving everyone uncertain as to which database includes the correct schema. Again, automating this upgrade process is the best way to ensure consistency and repeatability. However, whether we implement a semi-automated or fully automated solution, we're likely to take one of three approaches to upgrading an existing database: 1. Rebuild the database from scratch. 2. Apply a set of upgrade scripts. 3. Use a tool to auto-generate a transition script. 237 Rebuild the database Once past the development stage, we're much more likely to perform an upgrade, rather than rebuild a database. That said, some teams choose to rebuild, at least on occasion, partly because it's a much simpler process. They will often rebuild, for example, when implementing a significant number of database changes and they want to tie the database to a new product version, or even a new product. Of course, this means that the team must contend with migrating the existing data from the old database to the new, a task that grows in time and complexity as the amount of data grows. In addition, the team must orchestrate the application's transition from pointing to the old database to pointing to the new one in a way that minimizes the impact on the application's users. In most cases, rebuilding databases as an ongoing strategy, is suited only to the early development phase.
- 238. Chapter 7: Manually Deploying Databases from Source Control 238 Apply upgrade scripts A common way to upgrade a database is to create a set of upgrade scripts, often called change scripts, which we then apply to a specific database build to transition it to the next defined state. In this approach, every script we store in the VCS is a new upgrade script. Typically, each script will define a single change to a single object, whether to create that object, alter it, or even drop it. Alternatively, we might create one upgrade script that updates all objects, or create upgrade scripts for each object type (one for tables, one for views, and so on). Some teams store upgrade scripts only for tables, where each change must preserve the data, but use creation scripts for objects such as views and stored procedures, which are not dependent on the data. The versioning process requires a lot of thought and effort. The upgrade scripts should include any and all of the SQL needed to move a database from one version to the next. The team need to consider exactly how they wish every individual database modification to proceed, whether it's a straightforward update to a code object, or a table refactoring that requires a complex and intricate number of steps in order to preserve the existing data. They will run data definition language (DDL) statements, modify security and configuration settings, update reference data, or perform any other action necessary to upgrade the database. It is a time-consuming process and each step in the upgrade process will require rigorous testing. The other challenge with upgrade script versioning is that because every modification creates a new file in the VCS, it works against the VCS's natural diff and blame mechanisms. Developers often resort to opening every upgrade script, one by one, until finding the one that contains a "rogue" change. On the plus side, with all this work done, the deployment process becomes simple (in theory, at least), since it's merely a case of running the tested upgrade scripts in the right order. However, it still requires considerable care. As discussed in Chapter 4, the team need to ensure that that the right version of every script is in the VCS, that the build script executes each script in a precise order, that no change attempts fail or negatively impact other objects, that we don't miss any scripts or run any twice, and so on.
- 239. Chapter 7: Manually Deploying Databases from Source Control Also, of course, we need to verify that we apply the upgrade scripts to the right database build number. In the scheme illustrated below, we start with a clean build, which we've tagged in source control as v1.0. In the VCS, we create a subfolder for our upgrade scripts, and segregate them according to build. So we create a folder called v1.1 in which we store all the upgrade scripts needed to transition the database from build 1.0 to 1.1. When we apply the next set of upgrade scripts, we increment the build number to 1.2, and so on. 239 • Build database 1.0 • Apply 1.1 upgrade scripts to build 1.0. • Apply 1.2 upgrade scripts to build 1.1. • Apply 1.3 upgrade scripts to build 1.2. • Build database 2.0 • Apply 2.1 upgrade scripts to build 2.0. • Apply 2.2 upgrade scripts to build 2.1. • Apply 2.3 upgrade scripts to build 2.2. Since each set of upgrade scripts targets a specific build, it means that if we need to deploy a new version of build 1.3, starting from scratch, we would first build the 1.0 database, then apply the 1.1 upgrade scripts, next the 1.2 upgrade scripts, and finally the 1.3 upgrade scripts. However, when we implement build 2.0 (the revision tagged v2.0 in source control), we're essentially starting over. We apply the 2.1 upgrade scripts to 2.0, the 2.2 upgrade scripts to 2.1, and so on. The upgrade scripts should include any of the SQL needed to move a database from one version to the next. They can run data definition language (DDL) statements, modify security and configuration settings, update reference data, or perform any action related to updating the database.
- 240. Chapter 7: Manually Deploying Databases from Source Control The key is to run them in the correct order. Some teams actually include a number in the names of the upgrade scripts. Other teams create a master batch or PowerShell script that specifies which files to run and in what order. The approach we take depends on the methods we have in place to organize and run our files. Auto-generate a transition script An alternative to manually creating and maintaining a string of upgrade scripts for each database build is to store in the VCS only the latest CREATE script for each object. Each time we modify a database during development, we simply modify the existing CREATE script for that object and commit it to the VCS. For example, if we add a column to a table, we modify the table definition script file and check it into the VCS, rather than create a new upgrade script that alters the table. Running the latest CREATE script to get the latest version of an object implies that we can drop the existing object first, which is not possible outside the early development stages. Instead, teams can automate this process, both in development and when deploying to other environments, using a tool that compares the schema creation files in source control with the database that is to be upgraded. From this comparison, the tool generates one or more transition scripts to run against the target database, to synchronize it with the definition of the database in the VCS. In this way, we can compare any version of a database schema in source control with any database version. As a result, you can skip from build 1.0 to 1.3 without having to walk through the intermediate steps. If the database doesn't exist, the tool will create a new database. If it does exist, it will calculate the differences and apply the changes. We can also exert some manual control over exactly how the migration takes place. We cover this approach in detail in the next chapter. 240
- 241. Chapter 7: Manually Deploying Databases from Source Control Deploying a New Build of the Bookstore Database It's theory-to-practice time once again, and we're going to use a manual process to deploy the current version of our Bookstore database, which entails creating a master script file that runs the various SQL scripts necessary to build our database. Up to this point in the book, we've followed the approach of versioning the CREATE scripts for each object, so our master script will simply need to assemble and run these CREATE scripts in the right order, and then load any test/reference data. One of our database developers, Barb, will create our master script in PowerShell, after first adding a table that tracks build numbers. Saving database backup files Before starting the build and deployment process, create or designate a folder on the database server or network share for the database backup files. For the examples here, we'll use C:DataFilesBackups. 241 Create the build table We'll have Barb create a build table, DbBuild, as described in Listing 7-1. It records a three-part build number and includes a column that tracks the name of the master script file used to orchestrate the build process and a column that records the date we ran the build.
- 242. Chapter 7: Manually Deploying Databases from Source Control 242 USE Bookstore; GO CREATE TABLE dbo.DbBuild ( MajorNumber CHAR(2) NOT NULL , MinorNumber CHAR(2) NOT NULL , BuildNumber CHAR(3) NOT NULL , ScriptFile VARCHAR(50) NOT NULL , DateApplied DATETIME DEFAULT ( GETDATE() ) , CONSTRAINT pk_dbbuilds PRIMARY KEY ( MajorNumber, MinorNumber, BuildNumber ) ); GO Listing 7-1: The DbBuild table definition script. In SSMS, run the script to create the build table, and then save the DbBuild.sql script file to Barb's Tables folder in trunk (BookstoreProjecttrunkDatabasesBookstore Tables), and commit it to the VCS in the usual fashion. Create a tag for the latest Bookstore revision The next step is to create a tag that identifies the most current files as part of build v2.0. Make sure Barb's working folder is "clean" (updated and with no uncommitted changes) and then create the tag as follows: 1. Right-click the trunk folder, navigate TortoiseSVN | Branch/Tag. 2. Name the new tag v2.0 (in To path: enter /tags/v2.0). 3. Create the tag from the HEAD revision in the repository. Having created the new tag in the repository, update Barb's working folder to retrieve it. The tag identifies which files to use for the build. Going forward, any time the team want to build this database version, they need only reference the tag.
- 243. Chapter 7: Manually Deploying Databases from Source Control 243 Create the build script We're going to use PowerShell 3.0 to create the build script file, which will encapsulate the logic that assembles v2.0 of our Bookstore database. Of course, we could encap-sulate similar logic in any number of alternative forms, such as a batch file. However, we chose PowerShell because of its varied language elements, the ease with which we can incorporate flow control, and its capacity for error handling (although we don't include error handling in the example code). Listing 7-2 shows the PowerShell build script. # define variables # modify server instance and folder paths as necessary $instance = "localhostSqlSrv2012" $db = "Bookstore" $date = Get-Date -Format yyyyMMddHHmmss $file = "C:DataFilesBackups$($db)_$($date).bak" $folder = "C:SvnBarbBookstoreProjecttagsv2.0DatabasesBookstore" # variable to verify existence of database $id = Invoke-SqlCmd ` -ServerInstance $instance ` -Database "master" ` -Query "SELECT database_id FROM sys.databases WHERE name = '$db'" #check if database exists on server if ($id.database_id -gt 0) { # back up target database Backup-SqlDatabase ` -ServerInstance $instance ` -Database $db ` -BackupFile $file ` -Initialize # drop target database Invoke-SqlCmd ` -ServerInstance $instance ` -Database "master" ` -Query "DROP DATABASE $db" }
- 244. Chapter 7: Manually Deploying Databases from Source Control # create database Invoke-Sqlcmd ` -ServerInstance $instance ` -Database "master" ` -InputFile "$($folder)DatabaseBookstore.sql" # create tables Invoke-Sqlcmd ` -ServerInstance $instance ` -Database $db ` -InputFile "$($folder)TablesAuthor.sql" Invoke-Sqlcmd ` -ServerInstance $instance ` -Database $db ` -InputFile "$($folder)TablesBookCategory.sql" Invoke-Sqlcmd ` -ServerInstance $instance ` -Database $db ` -InputFile "$($folder)TablesBook.sql" Invoke-Sqlcmd ` -ServerInstance $instance ` -Database $db ` -InputFile "$($folder)TablesAuthorBook.sql" Invoke-Sqlcmd ` -ServerInstance $instance ` -Database $db ` -InputFile "$($folder)TablesDbBuild.sql" # create view Invoke-Sqlcmd ` -ServerInstance $instance ` -Database $db ` -InputFile "$($folder)ViewsAuthorsBooks.sql" # populate tables with test data Invoke-Sqlcmd ` -ServerInstance $instance ` -Database $db ` -InputFile "$($folder)LookupDataBookCategoryData.sql" 244
- 245. Chapter 7: Manually Deploying Databases from Source Control Invoke-Sqlcmd ` -ServerInstance $instance ` -Database $db ` -InputFile "$($folder)TestDataTestData.sql" # insert build number into DbBuild table Invoke-Sqlcmd ` -ServerInstance $instance ` -Database $db ` -Query "INSERT INTO DbBuild (MajorNumber, MinorNumber, BuildNumber, ScriptFile) VALUES ('2', '0', '0', 'BuildDatabase_2.0.0.ps1')" Listing 7-2: Creating the PowerShell database build script. The script contains a number of components, so let's walk through each one. The opening portion of the script creates a number of variables, as follows: • $instance and $db point to the SQL Server instance and the target database, respectively. Modify the former as appropriate for your instance. • $date and $file generate a file name for the database backup file. • $folder points to the v2.0 tags folder. Next, we create the $id variable. We use the Invoke-SqlCmd PowerShell cmdlet to populate $id with the target database ID from the master database. We're using the variable merely to test the existence of our target database, so we could have returned any column from the sys.databases view. If the database ID is greater than 0, we assume the database exists and the script backs up the database and then drops it. To back up the database, we use the Backup-SqlDatabase cmdlet and we use the Invoke-SqlCmd cmdlet to issue a DROP DATABASE statement. 245
- 246. Chapter 7: Manually Deploying Databases from Source Control Of course, dropping and recreating a database destroys the data, even if we do back it up first, and it's not a technique we'd use in production. We use it simply as a convenience, in case you don't have a SQL Server instance where Bookstore does not already exist. We discuss the right way to upgrade an existing database in the next section. The last half of the script is where all the action takes place, since it's where we build our database. The key is to run the SQL scripts in the correct order to ensure that the database builds correctly. In this case, we first run the database creation script, followed by the table creation scripts, in their proper order. Keep in mind that your script will create the database as specified in that script, in terms of the location and other properties of the data and log files. If deploying to an environment in which these or other properties need to be different, be sure to modify your database creation script as appropriate. The master script then creates the view and populates the tables (using the lookup and test data). For every command, we use the Invoke-SqlCmd cmdlet to run the appropriate script file. 246 Run the new build It's time to save our build script and test it out! First, save the script to Barb's DeployScripts folder in trunk as BuildDatabase_2.0.0.ps1, and commit it to the VCS as usual. PowerShell supports a number of ways to run a script. The simplest is to right-click the file and then click Run with PowerShell. This will launch the PowerShell console, run the script, and close the window. The only challenge with this approach is that, if informational messages are being displayed in the console window, you'll likely not see them, unless you're a very fast reader.
- 247. Chapter 7: Manually Deploying Databases from Source Control Another easy way to run the script is to open the PowerShell console and drag the file from Barb's working folder to the console. Then press Enter to launch the file. Or, if you actually want to edit the script, you can right right-click the file in the working folder and then click Edit to open the file in PowerShell ISE. PowerShell ISE gives you more options for testing and debugging the script and working with the file in general. If you need help getting started with PowerShell, a good place to begin is with Laerte Junior's article Getting Airborne with PowerShell (http://tinyurl.com/p92hsls). In the article, one of the issues that Junior discusses is how to set the execution policy in order to run a script file. However, to set the policy, we must run PowerShell as an administrator. As you start PowerShell, right-click the PowerShell link, and then click Run as administrator. Once over these small logistical hurdles, the PowerShell script should run fairly quickly, creating the backup, dropping the database, and building the new one. We can view the database properties in SSMS to verify the creation date. Also be sure to check the DbBuild table to ensure that it exists and contains our new build number. In addition, you can run the AuthorsBooks view to see whether it returns the expected data. The build script used in this example is simple, and lacks the normal error handling and control flow logic that defines the conditions under which to exit the script. However, it does demonstrate the basics of how to build and run a script that creates a new database at a particular version. 247
- 248. Chapter 7: Manually Deploying Databases from Source Control Upgrading the Bookstore Database Having deployed v2.0 of the Bookstore database to production, Barb and Fred continue working on the database in their development environment, making improvements that they will ultimately want to release as v2.0.1. This means that at some point they will want to apply these upgrades to the live v2.0 database. Up to now the team have versioned the latest CREATE script for each object. At this stage, they can either: 1. Continue this policy, and at deployment time use a schema comparison tool to generate a upgrade script that they can apply to the target database, or 2. Switch to saving each change as a separate upgrade script and then use a master build script that runs against v2.0 and applies all the upgrade scripts, in the right order, to v2.0. In this case, the team will adopt the second approach (Chapter 8 will cover the first one). Preparing the environment Before we get started with the database upgrade process, we need to set up a folder to store our upgrade scripts. Make sure Fred's working folder is fully updated with all previous changes, and then within C:SVNFredBookstoreProjectBookstoreDatabases Bookstore, create a new subfolder called ChangeScripts and within that add a subfolder called ChangeScripts_2.0.1. Add and commit these new folders in the usual way. 248
- 249. Chapter 7: Manually Deploying Databases from Source Control Creating the upgrade scripts We'll have Fred make a number of changes to the Bookstore database in the development environment: 1. Add the BookType table, which provides lookup data for the Book table 249 (AddBookTypeTable.sql). 2. Add lookup data to that table (AddDataToBookTypeTable.sql). 3. Add a BookTypeID column to the Book table to point to the new table (AddColumnToBookTable.sql). 4. Update the test data for the Book table (AddDataToBookTypeIDColumn.sql). 5. Add the BookTypeID column to our AuthorsBooks view (AddColumnToAuthorsBooksView.sql). Listing 7-3 shows, in a single listing, the scripts for all five changes, but we need to save each of these five changes as a separate upgrade script, in the ChangeScripts_2.0.1 folder. /*AddBookTypeTable.sql*/ USE Bookstore; GO CREATE TABLE dbo.BookType ( TypeID INT PRIMARY KEY , TypeName VARCHAR(30) NOT NULL ); GO /*AddDataToBookTypeTable.sql*/ USE Bookstore; GO
- 250. Chapter 7: Manually Deploying Databases from Source Control 250 INSERT INTO BookType VALUES ( 101, 'Paperback' ), ( 102, 'Hard cover' ), ( 103, 'Electronic book (ebook)' ), ( 104, 'Audio' ); GO /*AddColumnToBookTable.sql*/ USE Bookstore; GO ALTER TABLE dbo.Book ADD BookTypeID INT CONSTRAINT fk_BookTypeID REFERENCES BookType(TypeID); GO /*AddDataToBookTypeIDColumn.sql */ USE Bookstore; GO UPDATE Book SET BookTypeID = 101 WHERE BookID = 1; UPDATE Book SET BookTypeID = 101 WHERE BookID = 2; UPDATE Book SET BookTypeID = 103 WHERE BookID = 3; UPDATE Book SET BookTypeID = 102 WHERE BookID = 4; UPDATE Book SET BookTypeID = 102 WHERE BookID = 5; UPDATE Book SET BookTypeID = 101 WHERE BookID = 6; UPDATE Book SET BookTypeID = 104 WHERE BookID = 7; UPDATE Book SET BookTypeID = 101 WHERE BookID = 8; UPDATE Book SET BookTypeID = 103 WHERE BookID = 9; UPDATE Book SET BookTypeID = 102 WHERE BookID = 10; UPDATE Book SET BookTypeID = 104 WHERE BookID = 11; GO
- 251. Chapter 7: Manually Deploying Databases from Source Control 251 /*AddColumnToAuthorsBooksView.sql*/ USE Bookstore; GO ALTER VIEW dbo.AuthorsBooks AS SELECT ( CASE WHEN MiddleName IS NULL OR MiddleName = '' THEN a.FirstName + ' ' + a.LastName ELSE a.FirstName + ' ' + a.MiddleName + ' ' + a.LastName END ) AS FullName , b.Title AS BookTitle , bc.CategoryName AS BookCategory , ( CAST(b.BookID AS VARCHAR) + '_' + LOWER(LEFT(a.LastName, 2)) ) AS NewBookID , bt.TypeName AS BookType FROM Author a INNER JOIN AuthorBook ab ON a.AuthorID = ab.AuthorID INNER JOIN Book b ON ab.BookID = b.BookID INNER JOIN BookCategory bc ON b.BookCategoryID = bc.BookCategoryID INNER JOIN BookType bt ON b.BookTypeID = bt.TypeID; GO Listing 7-3: Five upgrade scripts. Preparing to deploy Once we have Fred commit all of these upgrade scripts to the VCS, Barb can update her working folder and then start preparing to deploy the v2.0.1 upgrade.
- 252. Chapter 7: Manually Deploying Databases from Source Control Create a new tag for the v2.0.1 build Barb's first step is to create a new tag that identifies all the files that comprise the v2.0.1 build: 1. Right-click the trunk folder, navigate TortoiseSVN | Branch/Tag. 2. Name the new tag v2.0.1 (in To path: enter /tags/v2.0.1). 3. Create the tag from the HEAD revision in the repository. Again, Barb must update the working folder to retrieve the new tag from the repository. Creating a Master upgrade script We'll again create a master PowerShell script for upgrading the database. The script will target the original build (2.0.0) with updates for the new build (2.0.1). In other words, the script is developed specifically to upgrade 2.0.0 to 2.0.1. Listing 7-4 shows the master PowerShell upgrade script. # define variables # modify server instance and folder paths as necessary $instance = "localhostSqlSrv2012" $db = "Bookstore" $date = Get-Date -Format yyyyMMddHHmmss $file = "C:DataFilesBackups$($db)_$($date).bak" $folder = "C:SvnBarbBookstoreProjecttagsv2.0.1DatabasesBookstore ChangeScripts 252 ChangeScripts_2.0.1" [string]$LastBuild = 200 # variable to verify existence of database $id = Invoke-SqlCmd ` -ServerInstance $instance ` -Database "master" ` -Query "SELECT database_id FROM sys.databases WHERE name = '$db'"
- 253. Chapter 7: Manually Deploying Databases from Source Control #check if database exists on server if ($id.database_id -gt 0) { # retrieve build number $CurrentBuild = Invoke-SqlCmd ` -ServerInstance $instance ` -Database $db ` -Query "SELECT (RTRIM(MajorNumber) + RTRIM(MinorNumber) + RTRIM(BuildNumber)) AS Build FROM DbBuild WHERE DateApplied = (SELECT MAX(DateApplied) FROM DbBuild)" # if wrong build, exit script if ($LastBuild -ne $CurrentBuild.Build) { cd c: Write-Host "Wrong target build. Update terminated." } else { # back up target database Backup-SqlDatabase ` -ServerInstance $instance ` -Database $db ` -BackupFile $file ` -Initialize # run update scripts Invoke-Sqlcmd ` -ServerInstance $instance ` -Database $db ` -InputFile "$($folder)AddBookTypeTable.sql" Invoke-Sqlcmd ` -ServerInstance $instance ` -Database $db ` -InputFile "$($folder)AddDataToBookTypeTable.sql" Invoke-Sqlcmd ` -ServerInstance $instance ` -Database $db ` -InputFile "$($folder)AddColumnToBookTable.sql" 253
- 254. Chapter 7: Manually Deploying Databases from Source Control Invoke-Sqlcmd ` -ServerInstance $instance ` -Database $db ` -InputFile "$($folder)AddDataToBookTypeIDColumn.sql" Invoke-Sqlcmd ` -ServerInstance $instance ` -Database $db ` -InputFile "$($folder)AddColumnToAuthorsBooksView.sql" # insert build number into DbBuild table Invoke-Sqlcmd ` -ServerInstance $instance ` -Database $db ` -Query "INSERT INTO DbBuild (MajorNumber, MinorNumber, BuildNumber, ScriptFile) VALUES ('2', '0', '1', 'UpdateDatabase_2.0.1.ps1')" cd c: Write-Host "Database updated to build 2.0.1." } } else { cd c: Write-Host "Database does not exist. Update terminated." } Listing 7-4: Creating the PowerShell database master upgrade script. The script once again begins by defining a number of variables. The only additional variable in this first section is $LastBuild. This contains the build number that our script targets (build 2.0.0, but without the periods). We'll use this to confirm that we're running our upgrade scripts against the correct database version, which we do in the later if statement. First, we define a variable named $CurrentBuild, which retrieves the current build number from the DbBuild table. Next, we compare the $LastBuild and $CurrentBuild values. If they are not equal, we display a message saying that we have the wrong target build and we take no other steps. 254
- 255. Chapter 7: Manually Deploying Databases from Source Control If the build numbers do correspond, we back up the database and then run each upgrade script in the appropriate order, accounting for dependencies. Finally, we add our build number and script file name to DbBuild. 255 Deploy the upgrade It's time to save our upgrade script and test it out! First, save the script as UpgradeDatabase_2.0.0.ps1 to the DeployScripts folder, in trunk, and commit it in the VCS. Run the script in PowerShell as described earlier. If you want to see the returned messages, then simply drag the file into the PowerShell console and press Enter. After you run the script, you should receive a message saying that the database has been updated to the new version. From there, verify that that the backup file has been created and that the DbBuild table contains the new build number. Also verify that the BookType table was created and populated, the Book table includes the new column, and the AuthorsBooks view displays the book type. Run the UpgradeDatabase_2.0.0.ps1 script again. This time through, it should display a message that you've run the script against the wrong target build. As you'll recall, the script will run correctly only against build 2.0.0, so we are reliant on the build if ever we want to re-create the 2.0.1 database. In other words, if we want to create the 2.0.1 database from scratch, we must first build the 2.0.0 database, and then apply the 2.0.1 upgrade scripts to that database, assuming we rely on upgrade scripts to upgrade the database.
- 256. Chapter 7: Manually Deploying Databases from Source Control Automating Deployments for Upgrade Script Versioning In this chapter, we've demonstrated a very basic example of this approach, saving each database modification to a designated folder as an upgrade script, and then using a simple PowerShell script to execute them in the right order during upgrades. However, it is still an overwhelmingly manual process, and therefore, especially for more complex upgrades than we've demonstrated here, time consuming and potentially error prone. To introduce better control and a degree of automation into this process, we could, as discussed in Chapter 4, adopt some of the ideas described and implemented in Alexander Karmanov's deployment framework (see http://tinyurl.com/pta9bcs). In addition, we can use tools such as dbdeploy (http://dbdeploy.com/) or migrator-dotnet (https://code.google.com/p/migratordotnet/) or Octopus deploy (https://octopusdeploy.com/) to help us automate deployments. Alternatively, we can adopt CREATE script versioning through the development and deployment cycle, and again use tools to help automate our processes, as discussed in the next and final chapter. Source Control and Database Deployment Source control and database deployment play integral roles in the world of database development. Source control provides the foundation necessary to successfully deploy your databases, yet it is by no means the only mechanism that contributes to an effective deployment strategy. You must have in place whatever policies, procedures and tools are necessary to ensure you can successfully build and upgrade your databases to whatever version you require. 256
- 257. Chapter 7: Manually Deploying Databases from Source Control In this chapter, we’ve looked at several aspects of database deployment and provided exercises to demonstrate a number of concepts. However, this information is meant only to help provide you with a conceptual overview of some of the factors to take into account when deploying a database. Database deployment is a large topic that could easily warrant its own book. In addition, the methods used to deploy databases are as diverse as the teams that deploy them. Our examples, like the information in general, are meant only to reinforce concepts. We are not suggesting that any one way to deploy your databases is the right way. The goal is to be able to deploy consistent, repeatable builds as efficiently as possible. How you achieve that goal is up to you. 257
- 258. Chapter 8: Automating Database Versioning and Deployment from Source Control Chapter 7 demonstrated a rather nuts-and-bolts approach to deploying a new database and upgrading an existing database from the VCS. Relying on nothing more than a build table and a master script, in PowerShell, we deployed a new version, v2.0, of the Bookstore database from the VCS. At this point the team continued development work, but switched from versioning the latest CREATE script for each database object to storing each individual database modification as a separate upgrade script. We then used a second PowerShell script to apply the changes to v2.0 of the deployed database, and transition it to v2.0.1. This upgrade script approach allows very precise control over each change we need to deploy to our databases, but we could regard this as both its strength and its weakness. It requires the developer to describe manually every single upgrade, whereas professional grade schema comparison tools are more than capable of automating the vast majority of our upgrades, while still allowing the developer to exert manual control over the harder ones. In this chapter, we'll discuss ways to streamline our database versioning and deployment processes to make them more reliable and less time consuming, using a set of automation tools. It will allow the team to adopt a unified, more integrated approach to database development and deployment, based solely around CREATE script versioning. Once again the chapter breaks down into theory and practice sections. The theory section takes a broader look at database versioning options and challenges, and how to choose the right versioning approach, and the tools and techniques that will allow us to start automating CREATE script versioning and deployment. The practice section demon-strates a particular automation approach, with the examples using tools from the Red Gate stable, such as SQL Source Control and SQL Compare. 258
- 259. Chapter 8: Automating Database Versioning and Deployment from Source Control Versioning the Database Our ultimate goal in versioning the database is to preserve a complete history of the database so that we can audit its evolution and, if need be, return to a past known state. In order to achieve this we need to version, primarily (but amongst many other things): • the database schema – every database object (table, view, function, procedure and so on) • static data – any "lookup" data that the application will assume exists (such as a list of all the states and provinces). When it comes time to build and deploy our database, the build tool must have available the correct version of every object, and all static data. Ideally, the tools or scripts we use to build and deploy the database will allow us to deploy together the required schema objects and tables of lookup data. This allows all the database content to be deployed within one transaction, and sequenced and validated together as a single unit. Simple tools will either not version the data at all or require you to launch a separate process to version data. More advanced tools allow you to select both. CREATE scripts versus upgrade scripts In the book so far, we've discussed two basic approaches to versioning a database in our VCS: 1. Save the modified CREATE script – We store one file per object and edit each file as appropriate to reflect each schema modification. In this approach, we are effectively versioning the current state of the schema. 2. Save a new upgrade script – For each object, we store the string of upgrade, or transformation, scripts (one change per script) that describe how to transition a database from an existing schema to the desired schema. 259
- 260. Chapter 8: Automating Database Versioning and Deployment from Source Control Table 8-1 summarizes some of the pros and cons of each approach. Technique Summary Pros Cons 260 CREATE scripts evolve over time Store CREATE scripts, modify these scripts over time Auditability – easy to see file evolution (what changed, and who changed it) using VCS diff and blame commands Harder to deploy upgrades, must infer differences Generally requires tools Upgrade scripts alter the schema in each script Store the com-mands necessary to transform schema to the desired state Easily deploy changes by running all scripts not run previously Does not account for version drift May not take the most efficient deployment path Much harder to track changes Table 8-1: Comparing database versioning strategies. When using the upgrade scripts versioning, developers store a new script for every change. If a developer wishes to make an "atomic" change to the database, such as adding a new table column and then making that column available in a view, he or she creates the necessary ALTER TABLE and ALTER VIEW scripts, verifies through testing that they transition the database to the correct new state, and then commits the two new upgrade scripts to the VCS.
- 261. Chapter 8: Automating Database Versioning and Deployment from Source Control The same atomic change with CREATE script versioning follows more or less the same path, except that after testing the developer will script out the CREATE script for each modified object, and save it to the VCS. In effect, the developer edits two existing scripts in the VCS, rather than creating two new ones. Either approach is viable, but the real differences between the two emerge when it comes time to deploy the versioned scripts. 261 Deployment challenges When using the upgrades script approach, other developers deploy these upgrades to their environment, or a build server applies them to a test environment, simply by running the scripts in the right order. It is a simple technique and allows fine-grained control over exactly how each upgrade should proceed. It can work well, as long as there are rigorous controls in place to determine exactly which scripts do and don't need to run in a particular deployment environment, and to prevent any direct modification of a deployed database. Deploying changes to a new environment is trickier with CREATE script versioning, since the developer (or build server) needs either to drop the existing object and run the CREATE script in the VCS, which is not recommended, or find a way to "infer" the alterations required to transition the existing object definition, in the target environment, so that it matches its definition in the VCS, while preserving any data. However, assuming the development team can overcome the deployment challenges (we'll discuss how in much more detail shortly), CREATE script versioning offers some major benefits, particularly in terms of improved auditing of project changes, and therefore easier team communication and project management. In either approach, if the process of packaging up all of the required files and deploying them is a manual one, then it is likely to be error prone and time consuming, especially for database deployments, where creating new objects and retiring old columns and tables must be done in a very specific order.
- 262. Chapter 8: Automating Database Versioning and Deployment from Source Control If this will quickly become tiresome for our developers, it could become catastrophic when the time comes to deploy to production servers. The live application can become quite unstable during the database deployment period, as both the new product version and the old product version presume very specific schema states. During the deployment, neither of these exist. We often just take the product out of service during this time. Imagine the panic-stricken DBA who is manually, urgently, carefully, quickly making all these changes while the business is at a stand-still, waiting for word of completion. It isn't long before we start reaching for a new solution, an automation solution, which both reduces the risk and speeds up deployment time. Choosing a versioning approach If the development team has a free hand in choosing a database versioning strategy, it is likely to be CREATE script versioning, especially if they have an easy and trusted way to generate the transition scripts that will enable sharing changes between developers' environments, and deploying them to testing. It is the approach that can best support their need to develop, test and deploy features quickly, while offering full change auditing through the VCS. However, it is unwise to make this choice without considering the needs of other relevant users and functions. For example, we ought to consider then needs of: • the build server – collects all project assets from the VCS to build and configure them and then package and deploy the product to a test server • operations staff – are responsible for the stability and performance of the product in the production environment • the product owner – manages the features and priorities of the product. Table 8-2 summarizes the main users and functions that have a stake in the VCS system and its versioning mechanism, and what needs they have. 262
- 263. Chapter 8: Automating Database Versioning and Deployment from Source Control User/function Primary need Primary tool Ideal workflow 263 Developer Easily publish and consume schema changes SQL Management Studio Publish changes from SQL Management Studio, apply others' changes when updating the codebase Build server Test schema and code changes TeamCity, Jenkins, Team Foundation Server, Cruise Control A build step applies the data-base migrations, testing the eventual deployment path Operations Minimize risk SQL Management Studio, database comparison tools, deployment auto-mation tools A non-technical user can deploy changes at any time by applying packages exhaus-tively tested previously Product owner Audit change history VCS Blame command, bug-tracking tools Easily discover new changes, the version that introduced a change, and identify when and by whom each change was made Table 8-2: Users and workflows. The operations team, for example, since they are responsible for the availability, performance and general well-being of the "live" application, will generally be averse to deploying any changes that have not been very carefully tested and rehearsed. If the organization must conform to rigorous up-time service-level agreements (SLAs) or rigorous audit requirements, they are likely to favor the upgrade scripts approach, because this technique lends itself to an upgrade path that is easy to rehearse and perfect. When the time comes to deploy, the risk of failure is reduced because the exact scripts that will affect production have been flexed in other environments.
- 264. Chapter 8: Automating Database Versioning and Deployment from Source Control Conversely, if the team use CREATE script versioning, with a "push button" tool to generate the necessary transition script, they may be more inclined to leave testing of that script to the last minute, in which case it's more likely that difficult and complex changes will cause disruption and downtime. Although it's important to consider everyone's needs as far as possible, it's also important not to get tied down in red tape, trying to build manual processes and policies to account for all the goals of all the users. We'd also generally advise against opting for the best-of-both- worlds approach of storing both the new CREATE script and the upgrade script, and each modification, in the VCS. In truth, it usually turns out to be the worst of both worlds, as it naturally creates a synchronization problem. What if the upgrade script doesn't yield the same final schema as the CREATE script? Which is right? What if the database developer forgot to commit one or the other? Since the previous chapter demonstrated upgrade script versioning, for deploying database upgrades, this chapter will now focus exclusively on CREATE script versioning. However, pick whichever approach is best suited to your primary workflows and users and stick to it, and then implement the necessary automation and procedures that will smooth the rough edges and mitigate risk for others. For example, if you choose CREATE script versioning, or even the upgrade script approach, then mitigate risk to the opera-tions team by deploying to environments that mimic production as early as possible in the development process, quickly spotting and fixing any potential issues with your migration scripts, as well as the product itself, long before it's time to go live. 264
- 265. Chapter 8: Automating Database Versioning and Deployment from Source Control Automation for CREATE Script Versioning and Deployment We'd like our chosen toolset to provide a consistent set of techniques to ease and automate team-based database development, and then deployment of the new database to various target environments. Ideally, the tools will allow developers to save off the CREATE scripts as database developers finish their tasks, right from within their devel-opment environment (SSMS), and to upgrade their environments with the CREATE scripts of others, as they're ready. The same tried and tested techniques to deploy between development and test environments should also be available to operations staff to safely deploy new versions to production. As discussed earlier, the major barrier when using CREATE script versioning is deploying schema changes, either to customers or to fellow developers. The CREATE scripts in the VCS represent the desired end-state of the database. If we're building a completely new version of the database from scratch, we can deploy them as-is, but what if a version of the database already exists in the target environment? To deploy them as-is, we'd need to tear down existing objects and replace them with their definition in the VCS. However, this is a bad idea for several reasons. It can become tricky and error prone when there are complex object dependencies, and it means we'll also need to re-create the data each time, exactly as it existed before. Alternatively, a developer may choose to make the changes manually, looking through the VCS log and manually updating each affected table, column, view, stored procedure, function, and so on. This is simply a non-sustainable approach, and it isn't long before the developer starts looking for an automation solution. Essentially, what we need instead is a "comparison engine" to automate the process of identifying differences between two schemas, and generate the transition script. This script is essentially a set of ALTER commands that will align the schema of the target database with that of a source database, or with the collection of CREATE scripts that form the latest revision in the VCS, while preserving all data. 265
- 266. Chapter 8: Automating Database Versioning and Deployment from Source Control The comparison engine could come in the form of a "roll-your-own" tool, or a third-party product. Whichever solution we choose, it will, ideally, allow manual intervention for delicate or complex database refactoring, such as splitting tables or changing a column's properties in a way that could affect the data it stores. This takes the form of a migration script, which will nudge the process away from the tool's defaults, when necessary. Upgrades, transitions, migrations… The terms for the various types of script can get a little confusing and are often used interchangeably so, to summarize their meaning in this book: • Upgrade script – a script describing a single database modification. • Transition script – an auto-generated script to align a database with a set of files in the VCS. • Migration script – a script used to steer the automated transition. We will, ideally, store the transition script in the VCS and then run it against the target schema. If there are any manual migration scripts, we could incorporate them into this transition script. When we have this process fully automated, the comparison engine will automatically compare the CREATE scripts and migration scripts stored in the VCS to the target server, infer the transition plan between them, and implement it. Whatever our exact strategy, it is important that the approach we use to transition a database from its current state to the desired state should be the same in all environ-ments. In other words, we should upgrade our database in development in exactly the same way as we would upgrade it in production, so that we fully understand and test the transition plan long before it comes time to execute it on a production system. Automating the transition script Our comparison engine, in whatever form it takes, needs to capture the database devel-oper's intent, and build a transition script that takes the shortest path to the goal, while 266 preserving user data.
- 267. Chapter 8: Automating Database Versioning and Deployment from Source Control Building our own tool to auto-generate a transition script from the CREATE scripts is not technically challenging, though some would argue that it's typically not cost-effective to build and maintain one to work robustly for every transition. Remember the panic-stricken DBA who is urgently trying to deploy this change while the software is offline? Nevertheless, a home-grown tool need only loop through each schema object on both sides, diffing the details. Tools such as SQL Server Management Objects (SMO) allow .NET programs to easily discover table details, views, stored procedure content, and so on. Looping first through each schema object type, then through each object, then through the details of each object can quickly produce a list of what's different between two databases. It is a bit more difficult to build this tool when we're pulling the desired schema state from source control, as it is in the form of a set of SQL CREATE scripts rather than an actual schema. To solve this, simply let SMO write CREATE scripts for the database, then use string comparison to identify differences. Alternatively run each CREATE script into a blank database, and use SMO to diff the two databases. However, we still need to apply our knowledge of the schema differences into a transition script that will transition a database into the desired state. If, we just DROP and CREATE each object, this doesn't bode well for preserving table data. Running ADD COLUMN and DROP COLUMN scripts for each difference is slightly better, but breaks down if the intent of the change is to rename a column. Ultimately, if undeterred by licensing costs or "not invented here" syndrome, the process of migrating a schema from one state to another is where the professional grade tools shine, and we have quite a number of choices for both of the automation strategies. If we're looking merely to infer the differences and then manually run the script, we'll look to tools such as Atlantis Compare, dbForge, Red Gate SQL Compare, and Visual Studio Data Tools. If we're looking for tools that fit into a continuous integration and/or delivery pipeline, we'll look towards tools such as SQL Source Control, Entity Framework Migra-tions, 267 and Rails Migrations.
- 268. Chapter 8: Automating Database Versioning and Deployment from Source Control Whichever tool we choose should have some or all of the following features: • Dump a database schema into CREATE scripts – in other words, we'd like the tool to generate a set of CREATE scripts automatically by analyzing an existing database. • Compare database schemas. • Compare a database to a set of SQL CREATE scripts. • Support versioning static lookup data. • Infer the differences between schemas and create a transition script. • Run the transition script, harvesting errors and reverting if necessary. • Allow manual overrides, when necessary, using migration scripts (covered in the next section) to instruct the comparison engine exactly how to perform a certain database upgrade. 268 Migration scripts There are times when any comparison engine can betray us. The goal of the engine is to infer the database developer's intent. It does this by analyzing the two schemas (or CREATE scripts and schema) and guessing what the developer had in mind. It calculates this direct transition path, and writes out the SQL script. For most cases, this direct path will be fine; we can simply test the generated script and run it. However, sometimes the most direct path is not the right path. For example, if we rename a column or rename a table, most schema comparison tools will auto-generate a transition script that drops the old column or table and creates a new column or table. This process doesn't preserve the user data. In these cases, we need to instruct the tool to perform the proper transition that will ensure preservation of data. In other words, we have to insert into the automated process a migration script that instructs the comparison engine on how to handle any unusual schema modifications.
- 269. Chapter 8: Automating Database Versioning and Deployment from Source Control For example, if we were to rename the Author table to Authors, we need to instruct our comparison engine to use sp_rename for that upgrade step, rather than drop Author and create Authors. Within these scripts, we'll not only execute the sp_rename but we'll also place an IF condition around it to ensure the transition is necessary. For example, what if the target database already had the destination Authors table? What if the source Author table didn't exist? In either case, the migration script doesn't apply, and we should not use it as we migrate the schema. To facilitate both deployment and the potential to roll back , many tools help us build a set of related migration scripts: one "up" script that upgrades the system to the new version, and one "down" script that downgrades the system or reverts the change. Though we'll want to account for this potential rollback process, the process of automating these "down" scripts is out of scope for this book. Instead we'll focus on creating and automating the "up" scripts, and inserting these migration scripts in the automation pipeline. Automated Versioning and Deployment for Bookstore It's theory-to-practice time again, for the final time in this book. The examples in the section will demonstrate just one possible toolset that will allow the team to: • Unify their versioning strategy around CREATE scripts, which will be used for deployment to all environments. • Improve developer interaction – saving a database developer's changes, and applying these changes to a co-worker's development environment, using Red Gate's SQL Source Control. 269
- 270. Chapter 8: Automating Database Versioning and Deployment from Source Control • Improve our previous manual deployment process – using Red Gate's SQL Compare to compare the CREATE scripts directory to the target server, and running the generated SQL script as a manual "push the button" process. • Allow the build server to deploy changes automatically to a integration server using JetBrain's TeamCity and Red Gate's SQL Automation Pack (brief overview only). Versioning databases with SQL Source Control Up to this stage in the book, we've interacted with the source control repository almost exclusively through the respective users' working folders, developing scripts on a live SQL Server development instance, but then saving them manually over to the working folder and committing them to the repository. Likewise, if a developer needed to apply the change made by another team member, he or she would either need to drop the existing table and re-create it using the latest CREATE script, and reimport any data (not recom-mended) or generate manually the ALTER command to implement the required change 270 (error prone). Here, we'll look briefly at an alternative, tool-based approach, where every developer in the team uses Red Gate's SQL Source Control source control client, integrated with SSMS, to commit their changes and get the changes of others. To work through the example in this section, you'll need to install this tool. SQL Source Control tool and references • Version used in this book: 3.6.3 • Download (with 14-day free trial): http://bit.ly/1gfUi8x • Documentation: http://bit.ly/1jffP0b • Enabling Migrations v2 (in beta at time of writing): http://bit.ly/1oHxtgs • Branching and merging support: http://bit.ly/1qONQKc and http://bit.ly/1p5nZJs
- 271. Chapter 8: Automating Database Versioning and Deployment from Source Control We'll assume each member of the team of developers works in their own sandbox copy of the database (the dedicated development model, see Chapter 3). Best practice: dedicated development databases With the right tools, I believe that it becomes a best practice for each developer to have their own sandbox database. We'd never open a Visual Studio solution from a shared network folder knowing a co-worker would also open the same solution from the same network folder. Why do this with your database? With the right tools, it becomes easy to keep the development database up to date without needing to call in a technician at every turn, and so the need to force database developers to experiment in a common development database lessens. Instead, put a copy of the database on each developer's machine, and use tools to keep it up to date. Link database to repository strategy In the most common mode of working with SQL Source Control, we link a database in a live SQL Server instance directly to the repository, using the appropriate repository URL. To do this, the first developer simply right-clicks the database and selects Link database to Source Control. SQL Source Control scripts out every database object, and when the developer commits, it auto-creates a specific folder structure in the repository, scripting every database object into a folder corresponding to its object type. Each subse-quent team member creates their own local copy of the latest version of the database by creating an empty database, linking the database to the same repository URL and performing a "Get Latest." Thereafter, each team member simply right-clicks on the database in SSMS, and commits their database modifications directly to the central source control repository, for access by other team members. In a single commit, this will version all outstanding local changes made by that user. Likewise, all team members can simply click Get Latest to retrieve others' changes, as shown in Figure 8-1. 271
- 272. Chapter 8: Automating Database Versioning and Deployment from Source Control Figure 8-1: Developer collaboration via SQL Source Control (repository strategy). In addition, each developer can perform a check-out of the repository to their working folder. This is necessary because we still currently need to perform certain source control operations, such as creating a branch or performing most types of merge, through a native client such as TortoiseSVN. For example, SQL Source Control supports linking to a branch, simply by unlinking from the current repository URL (e.g. the trunk) and relinking to the branch URL, or by creating a separate local database for the branch development work. However, we have to create and manage the branches through a native client. Likewise, SQL Source Control offers some limited support for merging, allowing developers to merge at the object level, either opting to keep "theirs" or "mine" in the case of a conflict. Recent versions of the tool do offer some line-by-line merging support by launching a third-party merge tool such as Beyond Compare (see the earlier Branching and merging support references). However, at the time of writing, line-by-line merging support was limited to stored procedures only. 272
- 273. Chapter 8: Automating Database Versioning and Deployment from Source Control Link database to working folder strategy Given that we still need to use the working folder for certain source control operations, and since in this book we've become accustomed to interacting with the repository through our working folder, we'll use SQL Source Control in a slightly different mode, as illustrated in Figure 8-2. Figure 8-2: Developer collaboration via SQL Source Control (working folder strategy). In this strategy, SQL Source Control merely synchronizes the database content with a chosen directory in the working folder, and then the user can choose which changes to commit, and when. This is incredibly helpful when we want to commit both database and application code in a single, atomic commit to ensure the build server captures all content associated with the change. If we committed each separately, we could easily have a broken build between the two commits. The working folder strategy is a perfect tool to separate the concern of scripting and updating the database, from the process of storing and versioning these scripts. 273
- 274. Chapter 8: Automating Database Versioning and Deployment from Source Control Versioning the existing Bookstore database We're now going to start the process of versioning the current version of the Bookstore database, both schema objects and static lookup data, as it existed at the end of Chapter 7, using SQL Source Control. Setting up the initial Bookstore database in Development If you've been following along with all the examples in the book so far then you'll already have a copy of the Bookstore database on your SQL Server instance. If not, in the code download for this chapter, we provide a backup of the database, as it existed at the end of Chapter 7, and you can simply restore it to your SQL Server instance to provide your starting point for the examples in this chapter. An alternative is to deploy the Bookstore database from the VCS to your development SQL Server instance. Again, if you haven't been working along, we provide, in the code download, a backup of the BookstoreProject repository as it existed at the end of Chapter 7. The first step is to restore this repo backup to an appropriate location, from the SVN command prompt, as described in the SVN merging precautions section of Chapter 6. For example: svnadmin load D:RepositoriesBookstoreProject < D:SVNBackups BookstoreProject.bak We can then deploy the database from the latest revision in the repository. We could do this using a PowerShell master build script, for example, as described in Chapter 7. Once we've worked through this section, discussing how to use SQL Source Control to link this database to our BookstoreProject working folder, then more automated deployment possibilities will be open to us, using a comparison engine to compare the latest revision in the repository to an existing database (empty or otherwise) and deploy the changes. 274
- 275. Chapter 8: Automating Database Versioning and Deployment from Source Control Linking Bookstore to working folders In Chapter 7, our team adopted a split versioning strategy, using CREATE scripts during development, but moving to upgrade script versioning after the first live deployment. Ordinarily, if a team decided to adopt CREATE script versioning for all deployments, they would move the old upgrade scripts into a "legacy" or "old" folder, and ultimately delete them. However, for the purposes of this book, we'll instead just leave all existing scripts in place, and create a new folder to which SQL Source Control can store the CREATE scripts for our existing Bookstore database. In previous chapters, our senior developer, Barb, checked out the BookstoreProject repository (https://<server>/ svn/BookstoreProject/) to C:SVNBarbBookstoreProject. The trunkDatabasesBookstore subfolder contained all the necessary content to create the database, tables, views, and data. Let's now have Barb create a new folder within C:SVNBarbBookstoreProjecttrunk Databases called Bookstore_SoC (where SoC forms a shorthand reference to SQL Source Control) and then commit that folder to the repository. Barb needs to link this folder to the Bookstore database in her development instance, after verifying that her live database represents precisely the latest revision in source control. To do this, fire up SSMS, right-click the Bookstore database and select Link database to Source Control (or, choose the same link from the Setup page of the SQL Source Control tab, which you can open from the Tools menu). Specify that we'll link the working folder, provide the URL to Barb's new Bookstore_SoC folder, and select the dedicated database development model, as shown in Figure 8-3. 275
- 276. Chapter 8: Automating Database Versioning and Deployment from Source Control Figure 8-3: Linking Bookstore to Barb's working folder. Now simply click Link to create the connection between source control and our Bookstore database. Back in SSMS Object Explorer, we'll see a small blue globe icon next to all the database objects, indicating that SQL Source Control has discovered objects that are new, yet committed to our Bookstore_SoC folder. On the Commit Change tab, we see all these objects listed as New. 276
- 277. Chapter 8: Automating Database Versioning and Deployment from Source Control Figure 8-4: Database changes to save to the working folder. Click Save Changes to write the scripts to our folder. Next, we can flip over to Windows Explorer to see exactly how SQL Source Control saved these scripts to Barb's working folder. SQL Source Control creates its own organizational structure for storing the scripts for each object, as shown in Figure 8-5. Note that it will create scripts only for the schema objects, not any reference data (but we'll fix that shortly). 277
- 278. Chapter 8: Automating Database Versioning and Deployment from Source Control Figure 8-5: The SQL Source Control-generated working folder. From Barb's working folder, let's have her commit all of this content to the repository. Developer collaboration: linking further users Our next user, Fred in this case, simply updates his working folder (C:SVNFred BookstoreProjectBookstoreDatabases) to capture the changes Barb committed. Fred now needs to create the Bookstore database in his own dedicated SQL Server instance based on the latest revision in his working folder. 278
- 279. Chapter 8: Automating Database Versioning and Deployment from Source Control Of course, at this stage in the book, Fred likely already has an existing Bookstore database, but of course generally this process of linking working folder to source control would happen right at the very start of the project. For the sake of this example, we'll have Fred create a new empty database called FredBookstore (see Listing 8-1) and then link this new database to his Bookstore_SoC folder. USE master; GO CREATE DATABASE FredBookstore ON PRIMARY ( NAME = N'FredBookstore_Data', FILENAME = N'<your path>FredBookstore.mdf' ) 279 LOG ON ( NAME = N'FredBookstore_Log', FILENAME = N'<your path>FredBookstore.ldf' ) GO Listing 8-1: Creating the FredBookstore database. Right-click the new database and select Link database to source control, specify the working folder using the URL C:SVNFredBookstoreProjectBookstoreDatabases Bookstore_SoC, and select the dedicated database development model, as described previously. Our first user, Barb, started with a live database containing all the required objects, and an empty working folder. Fred now has the opposite situation of a versioned schema in the repository, and an empty database. Switch to the Get Latest tab, and note all the schema objects that Barb committed and which Fred needs to apply to his live database. Click Apply Changes and in a short second, Fred should have a live development database with a schema identical to Barb's. As mentioned briefly earlier, although Barb and Fred have synchronized schemas, Fred's database will not contain any static/lookup data, since versioning the data is a separate step and Barb linked in only her schema objects. Let's fix that now.
- 280. Chapter 8: Automating Database Versioning and Deployment from Source Control 280 Versioning static data In our application, there are a few pieces of data that are static. Perhaps the application duplicates these as an enumeration, or perhaps we're just confident this data doesn't change. Throughout the book, we've added and modified a small volume of test data, as well as some static lookup data to our BookCategory and BookType tables, in order that we can display to users real book categories (such as "fiction") and book types (such as "paperback"), rather than meaningless IDs. In this example, we'll version only the lookup data, in the BookCategory and BookType tables, and not the test data. With such a small volume of test data, we could version that too in this case, but in more realistic cases the test data is likely to be of a volume that makes source controlling it impractical. Test data best practice Some teams choose to have a seed database that mirrors production data, perhaps anonymized to protect customer privacy, perhaps a subset of the full database to avoid storage costs. You may find it helpful to refresh this seed database periodically so developers can experience production-like data as they work. Having versioned this data, we can track changes to it, communicate with other team members, and restore to a point in time. In Barb's SQL Server instance, right-click the Bookstore database, and choose Other Source Control Tasks | Link or unlink static data, and then select the BookCategory and BookType tables, as shown in Figure 8-6.
- 281. Chapter 8: Automating Database Versioning and Deployment from Source Control Figure 8-6: Versioning the BookCategory and BookType lookup data. Click OK and then switch to the Commit Changes tab in the SQL Source Control pane, which should report that there is a Data link change type to save to the working folder. SQL Source Control filters You may have noticed the Edit filter rules tab in Figure 8-6. We won't cover filters here, but we can use them to exclude certain objects from source control (see http://bit.ly/1oIfir2). Click the Save Changes button. In Windows Explorer, Barb's Bookstore_SoC folder should register changes that she needs to commit, so perform that commit now, with a descriptive commit message. Now Fred can update his BookstoreDatabases folder and then, in SSMS, open the Get Latest tab in the SQL Source Control pane, and click Apply Changes in order to suck this lookup data into FredBookstore. 281
- 282. Chapter 8: Automating Database Versioning and Deployment from Source Control 282 Updating the test data Since we chose to version only the static data, Fred's new database won't contain any of the test data we added throughout the book. As discussed, most teams generate test data outside of the SQL Source Control process, perhaps using a seed database or perhaps using a data generation tool. In this example, Fred can easily generate test data for his FredBookstore database simply by using the latest version of the TestData.sql script in his BookstoreData-bases BookstoreTestData folder. However, here he will run into a slight issue. In the previous chapter, the team added a new BookTypeID to the Book table and some test data to populate that column. However, the team added the data as a change script, and it's not included currently in the TestData.sql file. This illustrates one of the possible issues of switching database versioning strategy in this way. To remedy this, Fred could update TestData.sql to incorporate the population of the BookTypeID column, and then run the script against his FredBookstore database. Modifying Bookstore Both of our developers now have a live Bookstore database, linked through their respective working folders which, in turn, link to a shared repository URL. They can now proceed with the development effort, committing changing and getting the changes of others, and start preparing to deploy a new version out to testing, then on to Staging and Production environments. Let's say that Dan, the new product owner tells Barb and Fred that the Bookstore appli-cation needs the ability to store the author's state and country of origin. Fred decides to add these columns to the Authors table of his FredBookstore database by running the script shown in Listing 8-2. Alternatively, right-click on the Authors table, and choose Design to add them through the Designer.
- 283. Chapter 8: Automating Database Versioning and Deployment from Source Control Generally, Fred would create a new AuthorLocation table for these state and country details and link it in to the Authors table, as we've done previously for book categories and types, but we'll keep things simple at this point, to focus on the collaborative source control processes. 283 USE FredBookstore; GO ALTER TABLE dbo.Author ADD State NVARCHAR(100), Country NVARCHAR(100); GO Listing 8-2: Add State and Country columns to the Author table in FredBookstore. At this point, Fred would run all the unit tests to ensure they pass, run the application, smoke-test critical pieces, and engage peer reviews to ensure the quality of the changes he's made. We'll presume all this is complete and that Fred is ready to share this change with Barb. All he needs to do is jump to the Commit Changes tab in the pane and he will see that SQL Source Control detected the change and generated the new version of the CREATE script to save to his working folder, as shown in Figure 8-7. Fred can click Save Changes and then, in Windows Explorer, commit the change with a descriptive commit message, in the usual fashion, to create a new revision in the repository. At this point, Barb should update her working folder in Windows Explorer, and then back in SSMS use the Get Latest tab to incorporate this schema change into her development database.
- 284. Chapter 8: Automating Database Versioning and Deployment from Source Control Figure 8-7: Saving the new CREATE script for the Authors table to Fred's working folder. Barb and Fred can proceed in this fashion, making changes to other schema objects or to the versioned static data as required, and saving the changes to their respective working folders directly from their sandbox SQL Server instances. They can visit the TortoiseSVN log to see the history of the database changes as it evolves. They can update their working folders and then apply the changes of other developers to their live database, as and when they are ready. If they hit any merge conflicts during updates, they can resolve them using TortoiseMerge, as described in detail in Chapter 6, or use the (currently limited) merge function-ality of SQL Source Control, as discussed earlier in this chapter. Again, if developers commit and update early and often, it will minimize the risk of merge conflicts. 284
- 285. Chapter 8: Automating Database Versioning and Deployment from Source Control Deploying databases with SQL Compare We've seen how a tool like SQL Source Control can help collaboration during devel-opment. Now, we're going to expand our toolset and see how to start automating the 285 deployment process. We're going to walk through an example of a basic database deployment of the latest revision of the Bookstore database in our VCS, directly to a new, empty database in the deployment environment, using a schema comparison engine. SQL Compare will infer the differences between the CREATE scripts stored in source control and the schema on the target database and generate a transition script. Later in the chapter, we'll see how to generate a database deployment package (such as a NuGet package) that our build server can use as part of a fully automated, continuous integration build and deployment. In order to perform this basic database deployment, one of our developers will need a license for SQL Compare (or just a free trial download). SQL Compare references • Version used in this book: 10.7 • Download (with 14-day free trial): http://bit.ly/1j2TcYl • Documentation: http://bit.ly/1kijpqz If you also installed the SSMS Integration Pack as part of the SQL Source Control or SQL Compare (or another Red Gate tool) download, then you can launch a SQL Compare (or Data Compare) deployment directly from within SSMS.
- 286. Chapter 8: Automating Database Versioning and Deployment from Source Control Automating deployments with the SQL Compare command line sqlcompare.exe can do deployments without user interaction when you purchase a SQL Compare Pro license. If you need to perform more complex database deployment, such as deploying the same database version to multiple targets, consider deploying using the command line and PowerShell scripts, as described here: http://tinyurl.com/nheehol. Deploying a new build of Bookstore As we did in previous chapters, the team should branch or tag the content before or after this deployment, to mark that all content up to and including this version is deployed. For example, following on from Chapter 7, the next minor release of our database could be v2.0.2. In which case, from Barb's working folder, she can create a new tag off of the trunk called v2.0.2 (in To path: enter /tags/v2.0.2), from the HEAD revision in the repository (and then update the working folder to retrieve the tag). Best practice: build numbers and deployments Always have an easy way to relate the deployed database to the source that created it. Either store the source code revision number in the target production database (avoid using the build server's build number as this is by definition transient), or store the deployed revision as a tag in source control, or both. This way, if an issue arises in production, a developer should be instantly able to retrieve from the VCS the exact content that comprises the deployed version of the database. Next, we need to create a "shell" database on the target SQL Server instance. To do this, Barb can connect to the target instance from SSMS and run Listing 8-1, adapted as appropriate to create a shell database on the target. In this example, she calls the new database ProductionBookstore. 286
- 287. Chapter 8: Automating Database Versioning and Deployment from Source Control 287 Deploy the schema Barb can use SQL Compare to auto-generate our transition script that will synchronize the current schema of ProductionBookstore (currently it is empty) with the latest revision of the database in source control. Open up SQL Compare and start a new project. On the New Project dialog, we define the source and target for the comparison. We want to use the most recent set of CREATE scripts as the source and our new ProductionBookstore as the target. When we select the Source Control radio button for the source, we then have the option to compare direct from the source control URL or from a scripts folder. As a DBA or developer performing the deployment, we might opt to deploy direct from a repository location. However, a member of the operations team is much more likely to work from a script folder sent to him or her by the developer or DBA. In this, case we're going to deploy from Bookstore_SoC, in Barb's working folder so, on the left side of the dialog, choose the Scripts folder option, and then enter the folder path C:SVNBarbBookstoreProjecttrunkDatabasesBookstore_SoC (or the same folder in the v2.0.2 tag, if you created it). On the right, choose the Database as the target, at the top, and then enter the details of the target server, specify the connection details, and select the ProductionBookstore database from the drop-down. Once complete, the screen should look something like what is shown in Figure 8-8.
- 288. Chapter 8: Automating Database Versioning and Deployment from Source Control Figure 8-8: SQL Compare with a script folder as source and a database as target. Click Compare Now and SQL Compare will generate a list of changes to synchronize the scripts folder and the target database. We should see six tables and a view that exist only in the scripts and not in ProductionDatabase. Select them all. We can click on each to find out exactly what changed and see the auto-generated script to perform the synchronization. You may see objects that exist only in the target, for example if the target instance auto-creates certain custom users on all new databases. If this happens, don't select those objects. With all necessary objects selected, click Deployment Wizard, at the top, and we get the options either to create a deployment script, or to Deploy using SQL Compare. We would generally choose the former, and then open up the script in SSMS to review and test it before running it. 288
- 289. Chapter 8: Automating Database Versioning and Deployment from Source Control However, here we're going to choose to deploy with the tool. Nevertheless, it's always a good idea to save the scripts that change the production database, so accept the option to save the transition script that SQL Compare generates to our DeployScripts folder. SQL Compare automatically assigns it a long name comprising the names of the source and target and the date. However, we'll simply call it BookstoreTransitionScript_2.0.1To2.0.2. Click Next again, and we see the auto-generated transition script. Scroll through this script and note how table creation is carried out in the specific order that ensures dependent tables are created before the foreign keys that depend on them, that tables are created before views, and that SQL Compare wraps everything in a transaction. Click Deploy Now to run this transition script against the ProductionBookstore database. In SSMS, connect to the target instance and verify that all the changes made it into ProductionBookstore. We didn't need to keep track of which things needed to be deployed or in which order we were to run these scripts. The tool inferred the changes, and built the SQL script. SQL Compare will also have saved the transitions script into Barb's DeployScripts folder, so she should commit it to the repository. 289 Deploy the static data Using the companion tool, SQL Data Compare, we can deploy the lookup data stored with the project. Earlier, we mentioned that with Red Gate's SSMS Integration Pack, we can launch schema or data comparisons right from within SSMS. Let's try that now for the static data (it you don't have the integration pack, simply launch SQL Data Compare separately). In SSMS, right-click Barb's Bookstore database, select Data Compare/deploy | Set as Source, choose Source Control as the source, set ProductionBookstore as the target and then launch SQL Data Compare.
- 290. Chapter 8: Automating Database Versioning and Deployment from Source Control After that the process is the same as for the schema, and you'll need to choose the same source scripts folder as last time (C:SVNBarbBookstoreProjecttrunkDatabases Bookstore_SoC). SQL Data Compare will generate a script to deploy the static data in the BookCategory and BookType tables to the target. Deploying upgrades to Bookstore Our database is now deployed and users are busily using it, and adding business data. The next time we deploy, to upgrade the ProductionBookstore database, we need to consider preservation of this business data. Meanwhile, back in development, Dan, the product owner, comes to Fred and identifies a new business requirement. They want to modify the Bookstore application UI so that it is mandatory for users to enter an author's country of origin, and to modify existing entries in the Authors table to add any missing country data. This is a change that requires a little thought. If the Bookstore application is not yet released internationally, we can presume all users in the system who haven't set a country yet are from the United States. In that case, we could just make the Country column not null, and set a default on it of "United States." However, as soon as the application releases internationally, the DEFAULT constraint becomes invalid. A better fix that accomplishes the long-term goal, is to make an atomic change that modifies existing data as required and then immediately alters the column to NOT NULL. However, this is clearly not a change that SQL Compare can handle alone, with manual prompting from a migration script. Enabling Migrations v2 in SQL Source Control and SQL Compare At time of writing, Migrations v2 was still in beta, and you will need to enable it in both tools in order to run this example. For SQL Source Control, see http://tinyurl.com/la7warn. For SQL Compare, start the tool, navigate Tools | Project Options and check the relevant option. 290
- 291. Chapter 8: Automating Database Versioning and Deployment from Source Control Therefore, we're going to generate the migrations script that SQL Compare will incor-porate into the transition script at deployment time. Unfortunately, at time of writing, SQL Source Control's migrations tooling only supports "up" (i.e. "apply this") scripts, and not "down" (i.e. "undo this") scripts. In SSMS, select Fred's FredBookstore database and then hop to the Migrations tab in SQL Source Control, choose Add Migration Script, and paste in the script in Listing 8-3. 291 DECLARE @ShouldRunMigrationScript BIT SET @ShouldRunMigrationScript = 1 IF NOT EXISTS ( SELECT 1 FROM Information_Schema.Columns WHERE table_schema = 'dbo' AND TABLE_NAME = 'Author' AND COLUMN_NAME = 'Country' ) BEGIN SET @ShouldRunMigrationScript = 0; PRINT 'Column [Country] in [dbo].[Author] could not be found - skipping migration.'; END IF @ShouldRunMigrationScript = 1 BEGIN UPDATE [dbo].[Author] SET [Country] = 'United States' WHERE [Country] IS NULL ALTER TABLE [dbo].[Author] ALTER COLUMN [Country] NVARCHAR(100) NOT NULL END Listing 8-3: The migration script. Give it a descriptive name, such as Set Author.Country Not Null, along with a description, and click Save and Close. Run the migration script on the FredBookstore database and then switch over to the Commit Changes tab.
- 292. Chapter 8: Automating Database Versioning and Deployment from Source Control You should see changes to the following objects: • The affected table – in this case, the Authors table. If it doesn't appear in the change list, you need to run the migration script against the local database. • MigrationHistory table-valued function – tracks which migration scripts are applied and which still need to be run. • RedGate schema – where the TVF is stored. We should version these items in source control and deploy them to all databases including production. Click Save Changes, then switch to Fred's working folder in Windows Explorer, and commit the changes. To update Barb's database using this migration, first update her working folder and then, back in SSMS, switch to the Get Latest tab, and click Apply Changes. This will run the migration script, and sets all the existing authors' countries to United States if they weren't specified. The final step is to deploy the changes with SQL Compare, exactly as described previously. This time, the transition script should note that the Author table exists on both source and target but is different, and that two objects (the TVF ad schema) only exist in source. Start the Deployment Wizard, and again we can either create a deployment script or perform the deployment directly through the tool. On the "objects you selected for deployment" screen, notice that it recommends deploying all dependencies (objects referenced by the changed object). Review the transition script, and note our "migration" should be one of the first actions in the script. On deploying the script, we should see the changes reflected in ProductionBookstore. 292
- 293. Chapter 8: Automating Database Versioning and Deployment from Source Control 293 Auditing changes Storing CREATE scripts that evolve over time makes auditing the database history as easy as automating the application code's history. We can use the SVN log, for example, to see which specific changes we made at each revision (see Chapter 4). We can take auditing to the next level using the Blame view. For example, right-click on the Author.sql file in C:SVNBarbBookstoreProjecttrunkDatabasesBookstore_ SoCTables and choose TortoiseSVN | Blame. For each line of code in the file, the Blame view shows us the last revision that modified that line, and who made the change. We can mouse over a line, to see the commit date and time, the commit message, and other inter-esting details. This is a wonderful tool for understanding who changed what, and when. It's much easier to pull open this single view than to pull open each of a dozen transition scripts looking for which one introduced or modified a particular column. Database Versioning and Deployment with Continuous Integration We've seen how tools help us to commit changes to schema and data, making it much easier for other developers to update their databases with these changes. We've also seen how easy it is to make a migration script that can nudge SQL Source Control to care for data when its default drop/create methodology is insufficient. We also simplified the manual database deployment process, by using SQL Compare to compare the latest CREATE scripts in the VCS with the "production" database, and infer the transition script. We can integrate these same tools and techniques into a continuous integration (CI) process whereby at regular intervals, such as at the end of each day (or even on every commit to our VCS, for some applications), the build server deploys a fresh build to the integration server for testing. In other words, instead of a developer running
- 294. Chapter 8: Automating Database Versioning and Deployment from Source Control SQL Compare to generate the transition script, the build engine will do so and use it, along with all the application source code and all other necessary build artifacts, all stored in the VCS, to run the deployment. Installing the CI server and agent(s), configuring application and deployment build steps, and other continuous integration methodologies are beyond the scope of this book, so this section offers only a brief and broad overview of one possible approach. Continuous integration tools and references • Installing and configuring TeamCity: http://bit.ly/1jyO8kr • Continuous Integration using Red Gate SQL Automation Pack: http://bit.ly/1o1uzmU • Automating database builds with TeamCity and SQL Automation Pack: http://bit.ly/RAC8md • Data synchronization with Team City: http://bit.ly/1j4Zhn7 With TeamCity installed, download and install the Automation Pack (you may need to restart TeamCity to load the new plugin). The reference above to Automating database builds with TeamCity and SQL Automation Pack provides worked examples for earlier and later versions of TeamCity, but in essence we need to open up a TeamCity build, add a new build step, choose the Red Gate runner type, and enter the CREATE scripts folder (/trunk/Databases/Bookstore_SoC) and the target database details. Since a CI build typically doesn't deploy directly to production, you could create a database called DevTestBookstore for this purpose. 294
- 295. Chapter 8: Automating Database Versioning and Deployment from Source Control With this build step in place, every time we commit code to the VCS, the build server will fire up to build, test, and deploy the content, including the database content. On each build, it will infer and execute the transition script. Together with unit and integration tests, this can give operations staff a high degree of confidence that the final deployment to production will work as expected. 295 Summary This chapter demonstrated how tools can help us improve and automate the process of versioning and deploying databases, alongside the application. Right from within our database development tool (SSMS) we were able to modify our development database, and then (after testing) commit it to the working folder, or directly to the VCS, with a just a few clicks. We were also able to discover and apply others' schema changes by inferring the differences between the scripts stored in source control and the developer's local database. With the same toolset, we could deploy new builds or upgrades directly to the target server, allowing our comparison engine to take care of most changes, but using migration scripts when we wanted to take specific steps to protect data integrity. We also explored how to incorporate such tools into a continuous integration and deployment pipeline. None of this would be possible, without a version control system that stored the right version of every asset required for a successful deployment, including, of course, all of the relevant database objects and static data. Choose the right VCS for your team, and then make the effort to find the right automation tools, whether home-grown, open source or commercial. Together, they can make the process of evolving, versioning and deploying a database far less painful.
- 296. About Red Gate You know those annoying jobs that spoil your day whenever they come up? Writing out scripts to update your production database, or trawling through code to see why it’s running so slow. Red Gate makes tools to fix those problems for you. Many of our tools are now industry standards. In fact, at the last count, we had over 650,000 users. But we try to go beyond that. We want to support you and the rest of the SQL Server and .NET communities in any way we can. First, we publish a library of free books on .NET and SQL Server. You’re reading one of them now. You can get dozens more from www.red-gate.com/books Second, we commission and edit rigorously accurate articles from experts on the front line of application and database development. We publish them in our online journal Simple Talk, which is read by millions of technology professionals each year. On SQL Server Central, we host the largest SQL Server community in the world. As well as lively forums, it puts out a daily dose of distilled SQL Server know-how through its newsletter, which now has nearly a million subscribers (and counting). Third, we organize and sponsor events (about 50,000 of you came to them last year), including SQL in the City, a free event for SQL Server users in the US and Europe. So, if you want more free books and articles, or to get sponsorship, or to try some tools that make your life easier, then head over to www.red-gate.com