"Continuously delivering infrastructure using Terraform and Packer" training material
Download as PPTX, PDF8 likes2,989 views
The document presents an overview of implementing infrastructure as code using Terraform and Packer, highlighting key concepts, commands, and best practices for usage in AWS environments. It covers the evolution of Terraform, its comparison with CloudFormation, and advanced features like variables, modules, and remote states. Additionally, it discusses team collaboration, CI/CD workflows, and provides resources for further learning.
![TERRAFORM COMMANDS
$ terraform
Usage: terraform [--version] [--help] <command> [args]
Common commands:
apply Builds or changes infrastructure
console Interactive console for Terraform interpolations
destroy Destroy Terraform-managed infrastructure
env Environment management
fmt Rewrites config files to canonical format
get Download and install modules for the configuration
graph Create a visual graph of Terraform resources
import Import existing infrastructure into Terraform
init Initialize a new or existing Terraform configuration
output Read an output from a state file
plan Generate and show an execution plan
push Upload this Terraform module to Atlas to run
refresh Update local state file against real resources
show Inspect Terraform state or plan
taint Manually mark a resource for recreation
untaint Manually unmark a resource as tainted
validate Validates the Terraform files
version Prints the Terraform version
All other commands:
debug Debug output management (experimental)
force-unlock Manually unlock the terraform state
state Advanced state management](https://image.slidesharecdn.com/training-cdinfrastructureusingterraform-170601112718/85/Continuously-delivering-infrastructure-using-Terraform-and-Packer-training-material-16-320.jpg)
![TERRAFORM - RESOURCES
Links:
https://www.terraform.io/docs/configuration/resources.html
resource "aws_autoscaling_group" "application" {
name = "${var.name}"
launch_configuration = "${aws_launch_configuration.application.name}"
vpc_zone_identifier = ["${module.public_subnet.subnet_ids}"]
depends_on = ["module.s3_artifacts"]
tag {
key = "Name"
value = "${var.name}"
propagate_at_launch = true
}
lifecycle {
create_before_destroy = true
ignore_changes = ["desired_capacity"]
}
}](https://image.slidesharecdn.com/training-cdinfrastructureusingterraform-170601112718/85/Continuously-delivering-infrastructure-using-Terraform-and-Packer-training-material-26-320.jpg)
![TERRAFORM - DATA SOURCES
Links:
https://www.terraform.io/docs/configuration/data-sources.html
data "aws_ami" "ami" {
most_recent = true
filter {
name = "name"
values = ["ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-*"]
}
owners = ["099720109477"] // Canonical
}
resource "aws_launch_configuration" "application" {
image_id = "${data.aws_ami.ami.image_id}"
}](https://image.slidesharecdn.com/training-cdinfrastructureusingterraform-170601112718/85/Continuously-delivering-infrastructure-using-Terraform-and-Packer-training-material-27-320.jpg)
![TERRAFORM - CONDITIONS
Links:
https://blog.gruntwork.io/terraform-tips-tricks-loops-if-statements-and-gotchas-f739bbae55f9
module "application" {
is_feature = "${replace(replace(terraform.env, "/^[^(feature)].*/", "false"), "/^feature.*/", "true")}"
}
# Example: If ... then
resource "foo" "bar" {
count = "${var.enable_ssl}" # true => 1, false => 0
}
# Example: If not ... then
resource "foo" "bar" {
count = "${1-var.enable_ssl}" # true => 1, false => 0
}](https://image.slidesharecdn.com/training-cdinfrastructureusingterraform-170601112718/85/Continuously-delivering-infrastructure-using-Terraform-and-Packer-training-material-31-320.jpg)
"Continuously delivering infrastructure using Terraform and Packer" training material
- 2. Hello! I AM ANTON BABENKO I enjoy AWS, DevOps, solutions architecture & web-development. github.com/antonbabenko linkedin.com/in/antonbabenko
- 3. 0. AGENDA
- 4. 0. AGENDA 1. State of things 2. Basics of Terraform and Packer Getting started demo 3. More advanced concepts in Terraform Practice 4. Working as a team CI/CD pipeline with Terraform and Packer Practice 5. Resources
- 5. 1. STATE OF THINGS Tools for AWS & Infrastructure as code
- 6. AVAILABLE TOOLS AWS CloudFormation, Google Deployment Manager Puppet, Chef, Ansible, Salt… AWS API, libraries (Boto, Fog) Terraform & Packer by HashiCorp
- 8. TERRAFORM Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. www.terraform.io
- 9. Version: 0.6.8 (released 2.12.2015) Open-source, written in Golang. Very active development: CHANGELOG.md (ca. 1 release per month) GitHub Issues (ca. 5-15 issues resolving daily) Growing community (IRC, Mailing list, Stack Overflow) TERRAFORM FACTS (2015)
- 10. Latest version: 0.9.4 (released 26.4.2017) Open-source, written in Golang. Very active development: CHANGELOG.md (ca. 3 releases per month) GitHub Issues (10+ issues resolving daily) Growing community (IRC, Mailing list, Stack Overflow, Slack channels, Gitter, etc) TERRAFORM FACTS (2017)
- 12. Year 2015 CloudFormation Terraform Configuration format JSON HCL/JSON State management No Yes Execution control No Yes! Logical comparisons Yes Limited Supports iterations No Yes Manage already created resources No Yes (hard) Providers supported Only AWS 20+ (incl. AWS, GCE, Azure)
- 13. Year 2017 CloudFormation Terraform Configuration format YAML/JSON HCL/JSON State management Kind of Yes Execution control Yes Yes! Logical comparisons Yes Yes Supports iterations Yes Yes Manage already created resources No Yes! Providers supported Only AWS 60+ (incl. AWS, GCE, Azure)
- 14. CloudFormation (2015) Terraform 0.6.8 (2015) Terraform 0.9.4 (2017) AWS resource types 121 103 280 Resource properties and operations completeness 90% Work in progress Work in progress :) Handle failures Optional rollback Fix it & retry Exit faster. Fix it & retry Contribute? No Yes! Yes! AWS SPECIFICS
- 16. TERRAFORM COMMANDS $ terraform Usage: terraform [--version] [--help] <command> [args] Common commands: apply Builds or changes infrastructure console Interactive console for Terraform interpolations destroy Destroy Terraform-managed infrastructure env Environment management fmt Rewrites config files to canonical format get Download and install modules for the configuration graph Create a visual graph of Terraform resources import Import existing infrastructure into Terraform init Initialize a new or existing Terraform configuration output Read an output from a state file plan Generate and show an execution plan push Upload this Terraform module to Atlas to run refresh Update local state file against real resources show Inspect Terraform state or plan taint Manually mark a resource for recreation untaint Manually unmark a resource as tainted validate Validates the Terraform files version Prints the Terraform version All other commands: debug Debug output management (experimental) force-unlock Manually unlock the terraform state state Advanced state management
- 17. TERRAFORM INIT Initialize a new or existing Terraform environment by creating initial files, loading any remote state, downloading modules, etc. *.tf Your infrastructure terraform.tfstate S3, Atlas, Consul, etcd, HTTP
- 18. TERRAFORM PLAN Generates an execution plan for Terraform *.tf Your infrastructure terraform.tfstate
- 19. TERRAFORM APPLY Builds or changes infrastructure according to Terraform configuration files *.tf Your infrastructure terraform.tfstate
- 20. TERRAFORM etc # Draw dependency graph (require “graphviz”) terraform graph -draw-cycles | dot -Tpng -o graph.png # Show help terraform --help
- 21. TERRAFORM & PACKER DEMO1 Code inside {terraform,packer}/demo1: https://github.com/antonbabenko/cd-terraform-demo
- 23. TERRAFORM AHEAD Variables Modules States Backends Data sources, providers, provisioners Conditions
- 24. TERRAFORM - MODULES Modules in Terraform are self-contained packages of Terraform configurations that are managed as a group. Links: https://github.com/terraform-community-modules/ Lots of github repositories (588) module "network_security" { source = "git::[email protected]:myself/tf_modules.git//modules/network/security?ref=v1.0.0" vpc_cidr = "${var.vpc_cidr}" }
- 25. TERRAFORM - VARIABLES Terraform != programming language Types: string, number, boolean, list, map Interpolation functions: length, element, file … Interpolation is not allowed everywhere Links: https://www.terraform.io/docs/configuration/syntax.html variable "iam_users" { description = "List of IAM users to create" type = "list" } resource "aws_iam_user" "users" { count = "${length(var.iam_users)}" name = "${element(var.iam_users, count.index)}" }
- 26. TERRAFORM - RESOURCES Links: https://www.terraform.io/docs/configuration/resources.html resource "aws_autoscaling_group" "application" { name = "${var.name}" launch_configuration = "${aws_launch_configuration.application.name}" vpc_zone_identifier = ["${module.public_subnet.subnet_ids}"] depends_on = ["module.s3_artifacts"] tag { key = "Name" value = "${var.name}" propagate_at_launch = true } lifecycle { create_before_destroy = true ignore_changes = ["desired_capacity"] } }
- 27. TERRAFORM - DATA SOURCES Links: https://www.terraform.io/docs/configuration/data-sources.html data "aws_ami" "ami" { most_recent = true filter { name = "name" values = ["ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-*"] } owners = ["099720109477"] // Canonical } resource "aws_launch_configuration" "application" { image_id = "${data.aws_ami.ami.image_id}" }
- 28. TERRAFORM - OUTPUTS Links: https://www.terraform.io/docs/configuration/outputs.html output "application_name" { value = "${var.name}" } output "vpc_id" { value = "${module.vpc.vpc_id}" }
- 29. TERRAFORM - STATES & BACKENDS Terraform keeps state of managed infrastructure and configuration in “terraform.tfstate”. Links: https://www.terraform.io/docs/state/index.html https://www.terraform.io/docs/backends/index.html terraform { backend "s3" { bucket = "my-tf-states" key = "staging/eu-west-1/shared" region = "eu-west-1" lock_table = "terraform_locks" } }
- 30. TERRAFORM - REMOTE STATES Links: https://www.terraform.io/docs/providers/terraform/d/remote_state.html data "terraform_remote_state" "shared" { backend = "s3" config { bucket = "my-tf-states" region = "eu-west-1" key = "staging/eu-west-1/shared" encrypt = true } } output "vpc_id" { value = "${data.terraform_remote_state.shared.vpc_id}" }
- 31. TERRAFORM - CONDITIONS Links: https://blog.gruntwork.io/terraform-tips-tricks-loops-if-statements-and-gotchas-f739bbae55f9 module "application" { is_feature = "${replace(replace(terraform.env, "/^[^(feature)].*/", "false"), "/^feature.*/", "true")}" } # Example: If ... then resource "foo" "bar" { count = "${var.enable_ssl}" # true => 1, false => 0 } # Example: If not ... then resource "foo" "bar" { count = "${1-var.enable_ssl}" # true => 1, false => 0 }
- 32. TERRAFORM DEMO2 Code inside terraform/demo2 : https://github.com/antonbabenko/cd-terraform-demo
- 33. 4. TERRAFORM Working as a team...
- 34. ● How to structure your configs? Reduce radius blast Size matters a lot Structure based on teams (infrastructure team-members = network; developers = modules owners) Separate repositories for modules and infrastructure Infrastructure can share same repository as application ● How to continuously test infrastructure using Terraform? Validate, plan, env Test modules independently, include working examples and README Test Kitchen, Inspec, Serverspec… Full run with smaller (yet, sane!) values TERRAFORM HOW?
- 35. TERRAFORM WORK FLOW Init, plan, apply, apply, plan, apply… Executors: Single developer Multiple developers Requires remote backend configuration (locks for lengthy operations) CI system Notes: MFA? Module versioning is important Group code by both - region and environment (staging, prod)
- 36. TERRAFORM WORK FLOW Init, plan, apply, apply, plan, apply… Open a Pull request: Validation (terraform validate) Optionally: Create new ephemeral (short-lived) Terraform environment (“terraform env new feature-branch”), run automated tests (kitchen-terraform, for example) and destroy it after Run plan and display output for review (git comment) Branch merged into master: Terraform apply to staging Optionally: terragrunt apply-all Branch tagged (release): Terraform apply to production
- 37. TERRAFORM - EXAMPLE 1 (pseudo) ● Developer commits application code ● CI system: ○ Run tests, builds artifact ○ Packer: Bake AMI ○ Terraform: Plan and apply with just created AMI id to create deployment ○ Run integration, performance tests ○ Deploy to staging
- 38. TERRAFORM - EXAMPLE 1 - feature ● Developer commits application code to a feature branch name feature-123 ● CI system: ○ Run tests, builds artifact using Packer ○ Run Packer: Bake AMI and tag it with branch=feature-123 ○ Run Terraform: ■ Plan the infrastructure for test environment, where AMI id lookup is using data source ami by tag branch=feature-123 ■ Optionally, save plan to a file, prompt git user in UI, post comment to github PR ■ Apply the plan ○ Run integration, performance tests ○ Deploy to staging
- 39. TERRAFORM DEPLOYMENTS Rolling deployments Using provider’s mechanisms: ECS (or other scheduler) CloudFormation Using custom mechanisms: DIY scripts combined with ‘-target’ arguments Blue-green deployments No provider’s mechanisms for this DIY
- 41. TERRAFORM RESOURCES Books and blog posts: Getting Started with Terraform by Kirill Shirinkin Terraform: Up and Running: Writing Infrastructure as Code by Yevgeniy Brikman Infrastructure as Code: Managing Servers in the Cloud by Kief Morris Using Pipelines to Manage Environments with Infrastructure as Code by Kief Morris Tools: https://github.com/gruntwork-io/terragrunt https://github.com/dtan4/terraforming https://github.com/coinbase/terraform-landscape https://github.com/newcontext-oss/kitchen-terraform https://github.com/kvz/json2hcl Other relevant repositories:
- 42. THANK YOU! All code from this talk: https://github.com/antonbabenko/cd-terraform-demo
Editor's Notes
- #3: Organizer of AWS user group norway AWS certified solution architect and sysops Doing web-development, devops for the last 10+ years. Doing AWS for the last 5 years. open-source, team leadership windsurfing, sailing, paragliding
- #7: Who is using AWS API directly or using libraries (like Troposphere written in Python) ?
- #13: State management - TF has local tfstate file describing metadata of created resources Execution control = well controlled. Plan => output file or limit by targets => apply with confidence. CF can only validate syntax. Logical comparisons = more, less, equal value. In TF you can use “count=0” or “count=1” resource parameter instead of boolean true/false to control resource creation. Manage already created resources like EIP, S3 buckets, VPC is not possible without deleting them first.
- #14: State management - TF has local tfstate file describing metadata of created resources Execution control = well controlled. Plan => output file or limit by targets => apply with confidence. CF can only validate syntax. Logical comparisons = more, less, equal value. In TF you can use “count=0” or “count=1” resource parameter instead of boolean true/false to control resource creation. Manage already created resources like EIP, S3 buckets, VPC is not possible without deleting them first.
- #15: Some resource properties (for example, ec2 keypair) can be created using AWS API, but not available in CloudFormation. Terraform uses AWS API, so you can get/update missing properties in many cases. update_rollback_failed = contact customer service --- Handle failures => Partial State and Error Handling If an error happens at any stage in the lifecycle of a resource, Terraform stores a partial state of the resource. This behavior is critical for Terraform to ensure that you don't end up with any zombie resources: resources that were created by Terraform but no longer managed by Terraform due to a loss of state.
- #18: Atlas, Consul, etcd, S3 or HTTP Terraform will automatically update remote state file once where are any changes in it. There are also ways to pull and push to remote state file.
- #19: Refresh state locally and generate execution plan based on tf configs
- #20: Apply the changes required to reach the desired state of the configuration. Or the pre-determined set of actions generated by a terraform plan execution plan.
- #22: Atlas, Consul, etcd, S3 or HTTP Terraform will automatically update remote state file once where are any changes in it. There are also ways to pull and push to remote state file.
- #33: Atlas, Consul, etcd, S3 or HTTP Terraform will automatically update remote state file once where are any changes in it. There are also ways to pull and push to remote state file.