Continuous Integration

images/logo
Developing, maintaining, and sharing software tools for research
Continuous integration
Danilo Pianini
danilo.pianini@unibo.it
Alma Mater Studiorum—Universit`a di Bologna
Ph.D. course in Data Science and Computation
June 7, 2018 - Bologna (Italy)
D. Pianini (UniBo) 04 - Continuous Integration June 7, 2018 1 / 40

images/logo
Outline
1 Introduction
2 Travis CI
3 Conﬁguration
Basics
Security
Deployment
Complete builds

images/logo
Introduction
Why continuous? I
Avoid the integration hell
Work in parallel
Don’t waste developers’ time with repetitive tasks
Don’t break stuﬀ
Time is money
Software development used to take several months for “integrating” a
couple of years of development [Fow]
Historically introduced by the extreme programming (XP) community
Today used by companies that do not adopt XP
IMVU [teab] delivers its software up to 50 times per day
Google and Mozilla release at least once a day

images/logo
Introduction
Why continuous? II
Higher frequency, lower risk [Fab]

images/logo
Introduction
Improve over classic development I
Protoduction [teaa]
When prototype code ends up in production
Classically used with a negative meaning
It’s time to rehabilitate it
Make it easy to access and use the latest prototype

images/logo
Introduction
Improve over classic development II
It’s compiling [Mun]
Make the building process fast!

images/logo
Introduction
Continuous Integration software
Software that promotes the practice of continuous integration
Runs a build for every change in the project
Prepares fresh environments where the builds are hosted
Notiﬁes the results, e.g. if a failure occurs
Provides tools for deploying the produced artifacts
Hosted CI with free plans for open source projects are blossoming:
Circle CI
Codefresh
Codeship
drone.io
Pipelines
Travis CI
Wercker
...

images/logo
Travis CI
Travis CI
Web based
Well integrated with GitHub
Build results are displayed in the repo without intervention
Automatic build of any pull request
Free for open source projects
Cronjobs
Build instances based on Docker
Project-local conﬁguration via YAML (in the .travis.yml ﬁle)
Out of the box support for Gradle
Dozens of deployment targets

images/logo
Travis CI
How it works
A web-hook can be registered to your GitHub repository that triggers
Travis CI at each new commit
Travis CI starts a pristine appropriate environment
Can be a container or a full virtual machine, depending on whether
sudo is required [CI]
The project gets cloned
The conﬁgured commands are executed
The conﬁgured deployments are performed
If necessary, project managers are informed of the build status

images/logo
Conﬁguration Basics
Outline
1 Introduction
2 Travis CI
3 Conﬁguration
Basics
Security
Deployment
Complete builds

images/logo
.travis.yml
Travis uses a project-local configuration
A .travis.yml file must be in your repository root
of course, it must be tracked in git
It is a YAML file, very human-readable and easy to learn 1
Also it is a superset of JSON, so any valid JSON is a valid YAML
Supports basically any language that can get built on Linux or
MacOS
No support for Windows builds
Support for build matrix
When your project can get built using different versions of different
tools, you may want to test all of them
It’s a cartesian product of configurations
Commit once, build on every supported environment
1
https://learnxinyminutes.com/docs/yaml/

images/logo
.travis.yml: the language section
Travis provides a number of default environments for the most
common languages
They differ by software installed by default
e.g. the C# compiler is not included if you run a Python build)
They behave differently
e.g. if Java is specified as language, the system automatically searches
for a build.gradle, a pom.xml (Maven), or an Ant build script
The first configuration required is a specification on which
environment to work in
Defaults to Ruby
For very simple projects, this might be enough of configuration
Example
language: python

images/logo
.travis.yml: custom behavior using the script section
The default conﬁguration may be not suitable for you
Either because you want to customize it
Or because you are using something that is not in the spectrum of
supported features
Bash commands can be conﬁgured to be executed in place of the
default behavior
Example
language: java
script:
- 'if [ "$TRAVIS_PULL_REQUEST" = "false" ]; then bash build.sh; fi'
- 'if [ "$TRAVIS_PULL_REQUEST" != "false" ]; then bash build_pull_req.sh; fi'

images/logo
.travis.yml: distribution selection in the dist section
By default, Travis CI builds in a Ubuntu Linux environment
Ubuntu LTS is generally used
The version of Ubuntu can be selected in a dist section
At the time of writing, trusty and precise are available
Mac OS X can be used by specifying os: osx in place of dist
Example
language: python
dist: trusty
script:

images/logo
.travis.yml: enabling super user access
By default, Travis CI builds in a docker container
It’s way faster than a VM, especially in terms of start up time
Docker does not allow for super-user access though
Sometimes it is required
e.g. for customizing the OS by installing packages
In such case, sudo: required switches the build to a full ﬂedged
VM with super user access
Example
language: python
dist: trusty
sudo: required
script:

images/logo
The build lifecycle in Travis
1 Install — Install any dependency required
1 apt addons — Optional
2 cache components — Optional
3 before install — Install additional dependencies in form of Ubuntu
packages using apt
4 install
2 Script — Run the build script
1 before script — Preparation for the build
2 script — Actual build
3 before cache — Optional
4 after success or after failure — Execute additional scripts
depending on the outcome of the build
5 before deploy — Optional, used to prepare resources to be uploaded
6 deploy — Optional, used to actually deploy the produced artifacts
7 after deploy — Optional, additional operations to be executed after
deployment
8 after script

images/logo
.travis.yml: Example with several phases
Example
language: java
dist: trusty
sudo: required
before_install:
- sudo apt-get -qq update
- sudo apt-get install -y graphviz
before_install: echo Begin actual build
script:
- ./gradlew clean build
- ./gradlew buildDashboard
after_success: echo Build successful
after failure: sudo mail -s "Build failure" admin@company.org < /dev/null
before_deploy: echo Preparing for deploy
after_deploy: echo Deployment phase concluded.

images/logo
Build variables I
Travis oﬀers a number of environment variables that allow for ﬁne tuning
the build process.
CI, TRAVIS, CONTINUOUS INTEGRATION, and
HAS JOSH K SEAL OF APPROVALa
a
Josh K. is a co-founder of Travis CI: https://twitter.com/j2h
Always set to true. Used for detecting if the build is running on the
Continuous integration environment
DEBIAN FRONTEND
Always set to noninteractive. Some scripts use it to determine whether
or not ask for user input.

images/logo
Build variables II
USER
Always set to travis. Do not depend on this value; do not override this
value.
HOME
Always set to /home/travis. Do not depend on this value; do not
override this value.
LANG and LC ALL
Always set to en US.UTF-8.
RAILS ENV, RACK ENV, MERB ENV
Always set to test

images/logo
Build variables III
JRUBY OPTS
Always set to "--server -Dcext.enabled=false
-Xcompile.invokedynamic=false"
JAVA HOME
Set to the appropriate value, depends on the selected JDK
TRAVIS ALLOW FAILURE
Set to true if the job is allowed to fail. Set to false if the job is not
allowed to fail.

images/logo
Build variables IV
TRAVIS BRANCH
For push builds, or builds not triggered by a pull request, this is the name
of the branch. For builds triggered by a pull request this is the name of the
branch targeted by the pull request. For builds triggered by a tag, this is
the same as the name of the tag (TRAVIS TAG). Note that for tags, git
does not store the branch from which a commit was tagged.
TRAVIS BUILD DIR
The absolute path to the directory where the repository being built has
been copied on the worker.
TRAVIS BUILD ID
The id of the current build that Travis CI uses internally.

images/logo
Build variables V
TRAVIS BUILD NUMBER
The number of the current build (for example, 4).
TRAVIS COMMIT
The commit that the current build is testing.
TRAVIS COMMIT MESSAGE
The commit subject and body, unwrapped.
TRAVIS COMMIT RANGE
The range of commits that were included in the push or pull request. Note
that this is empty for builds triggered by the initial commit of a new
branch.

images/logo
Build variables VI
TRAVIS EVENT TYPE
Indicates how the build was triggered. One of push, pull request, api,
cron.
TRAVIS JOB ID
The id of the current job that Travis CI uses internally.
TRAVIS JOB NUMBER
The number of the current job (for example, 4.1).
TRAVIS OS NAME
On multi-OS builds, this value indicates the platform the job is running
on. Values are linux and osx currently, to be extended in the future.

images/logo
Build variables VII
TRAVIS OSX IMAGE
The osx image value conﬁgured in .travis.yml. If this is not set in
.travis.yml, it is empty.
TRAVIS PULL REQUEST
The pull request number if the current job is a pull request, false if it’s
not a pull request.
TRAVIS PULL REQUEST BRANCH
If the current job is a pull request, the name of the branch from which the
PR originated. If the current job is a push build, this variable is empty.

images/logo
Build variables VIII
TRAVIS PULL REQUEST SHA
If the current job is a pull request, the commit SHA of the HEAD commit
of the PR. If the current job is a push build, this variable is empty.
TRAVIS PULL REQUEST SLUG
If the current job is a pull request, the slug (in the form
owner name/repo name) of the repository from which the PR originated.
If the current job is a push build, this variable is empty.
TRAVIS REPO SLUG
The slug (in form: owner name/repo name) of the repository currently
being built.

images/logo
Build variables IX
TRAVIS SECURE ENV VARS
Set to true if there are any encrypted environment variables. Set to
false if no encrypted environment variables are available.
TRAVIS SUDO
true or false based on whether sudo is enabled.
TRAVIS TEST RESULT
0 if all commands in the script section (up to the point this environment
variable is referenced) have exited with zero; 1 otherwise.
TRAVIS TAG
If the current build is for a git tag, this variable is set to the tag’s name.

images/logo
Build variables X
TRAVIS BUILD STAGE NAME
The build stage in capitalzed form, e.g. Test or Deploy. If a build does
not use build stages, this variable is empty.

images/logo
Conﬁguration Security
Outline
1 Introduction
2 Travis CI
3 Conﬁguration
Basics
Security
Deployment
Complete builds

images/logo
Sensible data in builds
It could be useful to access private data from within a build
Downloading a password-protected file
Decrypt a password-encrypted file
Open a keystore for signing a file
Store an API key for a service used e.g. for testing
Store a OAuth token for accessing a remote service
These data cannot be tracked on the repository (with the exception of
encrypted files, but the problem is simply moved to passing the decrypt
password to the build system).
These data must be provided in form of enviroment variables
Travis allows for inserting secure variables in the web interface

images/logo
Pull request attack
Usually, you want the integrator to build pull requests
You want to test the integration before committing it
What if the pull request changes the .travis.yml, printing all the
environment variables?
The developer of an open source project is defenseless
Travis CI does not allow access to secure variables when a pull request is
executed
As such, typically, the Travis build must be conﬁgured to detect
whether a pull request is being bult, and in case don’t perform tasks
that depend on secure variables

images/logo
Local Travis installation
Travis CI provides an installable module to help with several tasks
otherwise tedious:
Secure encryption of ﬁles
You may need your private key for automatic signing, but you want it
to be secret and only readable by builds you create
Secure encryption of global variables
You may need your password or username or other sensible data to
complete the deployment process, but you want it encrypted
In case of OAuth tokens, you also don’t want to waste time dealing
with it manually.
Install Travis CI locally:
1 Install RubyGems
On Arch Linux: pacman -Syu rubygems ruby-rdoc
2 Issue: gem install travis
3 Make sure your PATH includes the path where gems are installed

images/logo
Creating a secure variable
From the web interface:
Go to the settings page
Insert name and value
Select if it should be displayed on the build
Disable if the variable is meant to be secure
Use the environment variable in your build
From the local Travis CI application:
travis encrypt MY SECRET ENV=super secret
The secured variable will be printed on terminal
copy the secure="..." inside your .travis.yml

images/logo
Ecrypting a file
From the local Travis CI application:
travis encrypt-file my-super-secret-file
A new my-super-secret-file.enc file will be created
It must be added to track
The originating file must not be in track, and must never have been
(or it could be recovered): delete it immediately
copy the secure="..." inside your .travis.yml
Add the generated openssl command that appears on the terminal
to the correct phase of your build

images/logo
Don’t screw up: non-exhaustive list of advices
DO generate passwords, never use words related to the repository or
project name
DON’T use settings which duplicate commands to standard output,
such as set -x or set -v in your bash scripts
DON’T run env or printenv
DON’T echo "$SECRET_KEY"
DON’T use tools that print secrets on error output, such as php -i
DOUBLE CHECK before using git fetch or git push, as they might
expose tokens or other secure variables
DUOBLE CHECK for mistakes in string escaping
DUOBLE CHECK before using settings that increase verbosity
PREFER redirecting output to /dev/null when possible
e.g. git push url-with-secret >/dev/null 2>&1

images/logo
Conﬁguration Deployment
Outline
1 Introduction
2 Travis CI
3 Conﬁguration
Basics
Security
Deployment
Complete builds

images/logo
GitHub Releases
Travis CI can automate deployment of artifacts on GitHub releases
Example .travis.yml conﬁguration
deploy:
provider: releases
api_key:
secure: YOUR_API_KEY_ENCRYPTED
file: "FILE TO UPLOAD"
skip_cleanup: true
on:
tags: true
The authentication token for GitHub can be generated locally with:
travis setup releases
Remember to backup your travis ﬁle before running the command

images/logo
surge.sh
A free host for static websites (HTML + Javascript)
Install surge locally
Create an account (with email and password)
Create a new secret variable SURGE LOGIN
Create a new secret variable SURGE TOKEN
Obtain the value by using surge token
deploy:
provider: surge
project: ./build/docs/javadoc/
domain: myjavadoc.surge.sh
skip_cleanup: true
on:
all_branches: true

images/logo
Deploy to PyPI
The best place where to put your Python software modules!
Sign up to PyPI
deploy:
provider: pypi
user: "Your (possibly encrypted) username"
password:
secure: "Your encrypted password"

images/logo
Other targets
anynines – Appfog – Atlas – AWS CodeDeploy – AWS Elastic Beanstalk –
AWS Lambda – AWS OpsWorks – AWS S3 – Azure Web Apps – bintray –
BitBalloon – Bluemix CloudFoundry – Boxfuse – Catalyze – Chef
Supermarket – Cloud 66 – CloudFoundry – Deis – Engine Yard – GitHub
Pages – Google App Engine – Google Cloud Storage – Google Firebase –
Hackage – Heroku – Launchpad – Modulus – npm – OpenShift –
packagecloud.io – Puppet – Forge – PyPI – Rackspace Cloud Files –
RubyGems – Scalingo – Script – TestFairy – Ubuntu Snap Store –
Uploading Build Artifacts
Plus any target your build system can directly deal with (e.g. Maven
Central)

images/logo
Conﬁguration Complete builds
Outline
1 Introduction
2 Travis CI
3 Conﬁguration
Basics
Security
Deployment
Complete builds

images/logo
Example with Python
The default Python environment uses isolated virtualenvs
PyPy is supported out of the box
The script entry is mandatory
dependencies can be listed in a requirements.txt ﬁle
Travis automatically runs
pip install -r requirements.txt
during the install phase of the build

images/logo
Java examples
Examples of rich, multi-project, multi-language, multi-target deployments
are available at:
https://github.com/AlchemistSimulator/Alchemist
https://github.com/Protelis/Protelis

images/logo
References
References I
Travis CI.
The build environment.
https://docs.travis-ci.com/user/ci-environment/.
Accessed: 2017-05-08.
Darko Fabijan.
Why we need continuous integration.
https://semaphoreci.com/community/tutorials/continuous-integration.
Accessed: 2017-05-03.
Martin Fowler.
Continuous integration.
https://www.martinfowler.com/articles/continuousIntegration.html.
Accessed: 2017-05-02.
Randall Munroe.
xkcd: Compiling.
https://xkcd.com/303/.
Accessed: 2017-05-03.

images/logo
References
References II
The CodingHorror team.
New programming jargon.
https://blog.codinghorror.com/new-programming-jargon/.
Accessed: 2017-05-02.
The IMVU team.
Imvu: 3d avatar free chat, make new friends, dress up, shop.
https://www.imvu.com/.
Accessed: 2017-05-02.

Continuous Integration

More Related Content

Similar to Continuous Integration (20)

More from Danilo Pianini (20)

Recently uploaded (20)

Continuous Integration