Building a production-ready, fully-scalable Docker Swarm using Terraform & Packer on OpenStack.

Scaling With Docker Swarm using
Packer, Terraform & OpenStack
Bobby DeVeaux - March 28th 2017
https://joind.in/talk/a76ea

https://joind.in/talk/a76ea @bobbyjason #doxlon
• Created my first website at 9 Years old in 1995 😮
• Started coding PHP in 2001 - 16 years ago
• Developer, Team Leader, CTO, Director & Consultant
• Been using AWS for over 5 years
• Web Development, Message Queues, Automation, CI&CD
• Previously worked at SkyBet & DVSA
• Now a DevOps Consultant with UKCloud, Evangelising OpenStack
• Contributor to Terraform
• I ♥️ Docker, Terraform & Golang (or anything Hashicorp)
• #twitter: @bobbyjason
About Me ☁️
2

• I’m here to spread the awareness of UKCloud & OpenStack
• I want you to use Docker Swarm
• I want you to love Terraform
• I want to show you how to scale an app using all the above
Why Am I Here?
3

UKCloud at-a-glance
4

• Docker for AWS
• Docker for Azure
• UKCloud offer Cloud Native Infrastructure using Openstack
Why Openstack?
5

‘Final’ Demo
6

• Who’s using Docker yet?
• Who’s using Docker Swarm?
• Who’s using Terraform?
• Who’s using Packer?
• Who’s not played with any of them, and would love to?
Hands Up
7

https://joind.in/talk/a76ea @bobbyjason #doxlon 16

• Docker 1.12 & now 1.13 has Docker Swarm natively, “Swarm Mode”
• Cluster management integrated with Docker Engine
• Decentralized design
• Declarative service model
• Scaling
• Desired state reconciliation
• Multi-host networking
• Service discovery
• Load balancing
• Secure by default
• Rolling updates
Swarm Mode?
17

• docker swarm init —advertise-addr
• Initialises the node for Swarm mode
Docker Swarm Init
18

• docker service create --env APPLICATION_ENV=dev --update-
delay 10s --replicas 1 --network mynet --name php-fpm
bobbydvo/ukc_php-fpm:latest
• docker service create --update-delay 10s --replicas 1 -p 80:80 --
network mynet --name web bobbydvo/ukc_nginx:latest
• docker service update --image bobbydvo/ukc_nginx:1.81 web
Docker Service Create
19

• docker service scale web=5
Docker Service Scale
20

• docker stack deploy —compose-file docker-compose.yml mystack
Docker Stack Deploy (1.13 only)
21
version: "3"
services:
web:
tty: true
depends_on:
- php-fpm
image: bobbydvo/ukc_nginx:latest
ports:
- "80:80"
deploy:
mode: replicated
replicas: 2
update_config:
parallelism: 1
delay: 10s
failure_action: continue
monitor: 60s
max_failure_ratio: 0.3
php-fpm:
tty: true
build: ./
image: bobbydvo/dummyapp_php-fpm:latest
ports:
- "9000:9000"
environment:
- APPLICATION_ENV=prod
deploy:
mode: replicated
replicas: 1
update_config:
parallelism: 1
delay: 10s
failure_action: continue
monitor: 60s
max_failure_ratio: 0.3

• We’re about to embark onto the interesting stuff
• Any questions?
…and Pause.
22

• Packer
• Terraform
• Docker Swarm
• Jenkins For Release
How Do We Scale on Real Infrastructure
23

• Terraform is a tool for building, changing, and versioning
infrastructure safely and efficiently. Terraform can manage existing
and popular service providers as well as custom in-house solutions.
• Infrastructure as Code: Infrastructure is described using a high-level
configuration syntax. This allows a blueprint of your datacenter to be
versioned and treated as you would any other code. Additionally,
infrastructure can be shared and re-used.
• Execution Plans: Terraform has a "planning" step where it generates
an execution plan. The execution plan shows what Terraform will do
when you call apply. This lets you avoid any surprises when
Terraform manipulates infrastructure
What Is Terraform?
24

• How long do your builds & deployments in travis / Jenkins take?
• What’s acceptable?
• ‘Quick’ is relative, and depends on your requirements.
• When I say quick deployments, I’m referring to efficient
deployments using Foundation Images.
Who Likes Quick Deployments?
29

• Ansible / Puppet / Chef means that lots of projects now build from
the base box image, i.e. CentOS6 or Ubuntu 14.04 etc.
• Do you want to be building this each build? Some of you are clever,
and don’t. Some of you are clever, but didn’t consider an
alternative, or didn’t know how. Maybe some of you don’t even use
automated builds…
• Using Packer and your provisioner of choice, you can export the
artefact and store it as a Docker Container or Image in your cloud
provider (Amazon AMI, OpenStack Glance, etc).
Foundation Images
30

• Tool for creating identical machine images
• Supports multiple platforms
• Supports many provisioners (Ansible, Chef, Puppet, Bash.. etc.)
• Can export image in multiple formats AMIs for EC2, VMDK/VMX
files for VMware, OVF exports for VirtualBox, etc.
What Is Packer?
31

Packer template.json
32
{
"variables": {
"aws_access_key": "",
"aws_secret_key": ""
},
"builders": [{
"type": "amazon-ebs",
"access_key": "{{user `aws_access_key`}}",
"secret_key": "{{user `aws_secret_key`}}",
"region": "us-east-1",
"source_ami": "ami-fce3c696",
"instance_type": "t2.micro",
"ssh_username": "ubuntu",
"ami_name": "packer-example {{timestamp}}"
}]
}

Ansible Provisioning
33
"provisioners": [
{
"type": "shell",
"inline": [
"rpm -iUvh http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-7.noarch.rpm",
"yum -y update",
"yum -y install ansible",
"ansible --version"
]
},{
"type": "ansible-local",
"playbook_file": "./ansible/playbook.yml",
"role_paths": [
"./ansible/roles/init",
"./ansible/roles/server",
"./ansible/roles/mongodb",
"./ansible/roles/php7",
"./ansible/roles/nginx",
"./ansible/roles/supervisord",
"./ansible/roles/redis"
],
"group_vars": "./ansible/common/group_vars"
}

The cool bit;
Putting it all together!

Build Docker Images Regularly
43
#!/bin/bash
VERSION=1
CONTAINER=$1
BUILD_NUMBER=$2
if [[ $CONTAINER == 'all' ]];
then
for CONTAINER in php-fpm nginx dynamodb consul;
do
docker build ./$CONTAINER -t bobbydvo/ukc_$CONTAINER:latest
docker tag bobbydvo/ukc_$CONTAINER:latest bobbydvo/ukc_$CONTAINER:$VERSION.$BUIL
done
exit
fi
docker build ./$CONTAINER -t bobbydvo/ukc_$CONTAINER:latest
docker tag bobbydvo/ukc_$CONTAINER:latest bobbydvo/ukc_$CONTAINER:$VERSION.$BUILD_NUMBER
docker push bobbydvo/ukc_$CONTAINER:latest
docker push bobbydvo/ukc_$CONTAINER:$VERSION.$BUILD_NUMBER

How NOT to Build Your Jenkins Job
44
echo $BUILD_NUMBER
docker -v
whoami
sudo docker login -u bobbydvo -p Lr6n9hrGBLNxBm
sudo ./build.sh $CONTAINER $BUILD_NUMBER

How to Build Your Jenkins Job
45
docker login -u bobbdvo -p Lr6n9hrGBLNxBm
.docker/config.json:
{
"auths": {
"https://index.docker.io/v1/": {
"auth": “************************************”
}
}

46
echo $BUILD_NUMBER
docker -v
whoami
#sudo docker login -u bobbydvo -p Lr6n9hrGBLNxBm
sudo ./build.sh $CONTAINER $BUILD_NUMBER

47

Building DummyPHP Docker Image
48
FROM bobbydvo/ukc_php-fpm:latest
WORKDIR /srv
COPY . /srv/
WORKDIR /srv
RUN composer install
CMD ["/usr/bin/supervisord","-n","-c","/etc/supervisord.conf"]

On Merge Jenkins Hook
49
set -e
DUMMY_VERSION=$BUILD_VERSION
NGINX_VERSION='latest'
sudo docker-compose build
sudo docker run -i bobbydvo/dummyapp_php-fpm /srv/vendor/bin/phpunit -c /srv/app/phpunit.xml
# tag & push only if all the above succeeded (set -e)
sudo docker tag bobbydvo/dummyapp_php-fpm:latest bobbydvo/dummyapp_php-fpm:$DUMMY_VERSION
sudo docker push bobbydvo/dummyapp_php-fpm:$DUMMY_VERSION
sudo docker push bobbydvo/dummyapp_php-fpm:latest
ssh -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null core@51.179
ssh -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null core@51.179

Infrastructure Next
50

Terraform Build Infrastructure - Manager1
51
resource "openstack_compute_instance_v2" "swarm_manager" {
name = "swarm_manager_0"
count = 1
#coreos-docker-alpha
image_id = "0fe61d2f-0f9b-4dc8-8706-b45771f86d10"
flavor_id = "7d73f524-f9a1-4e80-bedf-57216aae8038"
key_pair = "${openstack_compute_keypair_v2.test-keypair.name}"
security_groups = ["${openstack_compute_secgroup_v2.example_secgroup_1.name}"]
user_data = "${data.template_file.cloudinit.rendered}"
network {
name = "${openstack_networking_network_v2.example_network1.name}"
floating_ip = "${openstack_networking_floatingip_v2.example_floatip_manager.address}"
}

Terraform - Cloudinit.sh
52
data "template_file" "cloudinit" {
template = "${file("cloudinit.sh")}"
vars {
application_env = "dev"
}
}

Terraform - Cloudinit.sh
53
docker swarm init
docker swarm join-token --quiet worker > /home/core/worker-token
docker swarm join-token --quiet manager > /home/core/manager-token
docker stack deploy --compose-file /home/core/docker-compose.yml mystack > /dev/nu

Terraform - Master X
54
resource "openstack_compute_instance_v2" "swarm_managerx" {
name = "swarm_manager_${count.index+1}"
count = 2
#coreos-docker-beta
flavor_id = "7d73f524-f9a1-4e80-bedf-57216aae8038"
user_data = "${data.template_file.managerinit.rendered}"
network {
}
}

Terraform - Managerinit.sh
55
data "template_file" "managerinit" {
template = "${file("managerinit.sh")}"
vars {
swarm_manager = "${openstack_compute_instance_v2.swarm_manager.access_ip_v4}
}
}

Terraform - ManagerInit.sh
56
# Copy Tokens from master1 => masterX
sudo scp -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o U
# Copy docker-compose.yml file
sudo scp -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o U
sudo docker swarm join --token $(cat /home/core/manager-token) ${swarm_manager}

Adding Workers
58
resource "openstack_compute_instance_v2" "swarm_slave" {
name = "swarm_slave_${count.index}"
count = "${var.swarm_node_count}"
#coreos-docker-beta
flavor_id = "c46be6d1-979d-4489-8ffe-e421a3c83fdd"
user_data = "${data.template_file.slaveinit.rendered}"
network {
}
}

Adding Workers - slaveinit.sh
59
data "template_file" "slaveinit" {
template = "${file("slaveinit.sh")}"
vars {
swarm_manager = "${openstack_compute_instance_v2.swarm_manager.access_ip_v4}"
node_count = "${var.swarm_node_count + 3}"
}
}

Adding Workers - slaveinit.sh
60
sudo scp -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null
sudo docker swarm join --token $(cat /home/core/worker-token) ${swarm_manager}
ssh -o StrictHostKeyChecking=no -o NoHostAuthenticationForLocalhost=yes -o UserKnownHostsFile=/dev/null -i /ho
## Forces redistribution across all nodes

Adding Workers - How Many?
61
variable "swarm_node_count" {
default = 1
}
variable "swarm_node_count" {
default = 5
}

• Docker will bring up another container. Lets try.
What If A Container Dies?
62

• Provisioned Docker Containers
• Infrastructure as Code
• Automated Deployments for CI / CD
• Scalable Architecture
• Openstack + UKCloud
There you have it!
63

‘Final’ Demo - A Release
64

Thank you :-)
Bobby DeVeaux
@bobbyjason
https://joind.in/talk/a76ea

Building a production-ready, fully-scalable Docker Swarm using Terraform & Packer on OpenStack.

More Related Content

What's hot (18)

Similar to Building a production-ready, fully-scalable Docker Swarm using Terraform & Packer on OpenStack. (20)

More from Outlyer (20)

Recently uploaded (20)

Building a production-ready, fully-scalable Docker Swarm using Terraform & Packer on OpenStack.

Editor's Notes