The challenge of application distribution - Introduction to Docker (2014 dec 03)
Download as PPTX, PDF5 likes1,599 views
This document discusses the challenges of application distribution and how Docker addresses these issues by providing a portable and efficient containerization solution. It contrasts traditional deployment methods with Docker's capabilities to simplify development, testing, and deployment processes across different environments. The document also covers various aspects of Docker's ecosystem, including how containers work, their benefits, and use cases in real-world applications.
The challenge of application distribution - Introduction to Docker (2014 dec 03)
- 1. The challenge of application distribution Introduction to Docker
- 2. Credits & references Intros slides borrowed from • dotCloud’s « Docker introduction » slideshow • AWS re:Invent 2014 session APP303, « Lighting Fast Deploys with Docker Containers and AWS »
- 3. Agenda • The application distribution challenge • The current solutions • Introduction to Docker, Containers, and the Matrix from Hell • Why people care: Separation of Concerns • Technical Discussion • Ecosystem, momentum • How to build Docker images • How to make containers talk to each other, how to handle data persistence • Demo 1: isolation • Demo 2: real case - installing Go Math!, tail –f containers, unit tests
- 5. The challenge: apps have fundamentally changed ~2000 2014 Long lived Development is iterative and constant Monolithic and built on a single stack Built from loosely coupled components Deployed to a single server Deployed to a multitude of servers
- 6. Everybody already heard/said it:
- 7. The challenge: apps have fundamentally changed Static website User DB Redis + redis-sentinel Web frontend Queue Analytics DB Background workers API endpoint nginx 1.5 + modsecurity + openssl + bootstrap 2 postgresql + pgv8 + v8 hadoop + hive + thrift + OpenJDK Ruby + Rails + sass + Unicorn Python 3.0 + celery + pyredis + libcurl + ffmpeg + libopencv + nodejs + phantomjs Python 2.7 + Flask + pyredis + celery + psycopg + postgresql-client Development VM QA server Public Cloud Disaster recovery Contributor’s laptop Production Servers Multiplicity of Stacks Multiplicity of hardware environments Production Cluster Customer Data Center Do services and apps interact appropriately? Can I migrate smoothly and quickly?
- 8. The Matrix From Hell Static website Web frontend Background workers User DB Analytics DB Queue ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? Dev VM QA Server Single Prod Server Onsite Cluster Public Cloud Contributor’s laptop Customer Servers
- 10. Current solutions • ‘Configuration’ tools like Ansible, Chef, Puppet, … • VM tools : Vagrant • Container tools: Google’s LMCTFY
- 11. Goods Multiplicity of transporting/storing Multipilicity of methods for Do I worry about how goods interact (e.g. coffee beans next to spices) Can I transport quickly and smoothly (e.g. from boat to train to truck) Cargo Transport Pre-1960
- 12. Also a matrix from hell ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
- 13. Multiplicity of Goods Multiplicity of methods for transporting/storin g Do I worry about how goods interact (e.g. coffee beans next to spices) Can I transport quickly and smoothly (e.g. from boat to train to truck) Solution: Intermodal Shipping Container …in between, can be loaded and unloaded, stacked, transported efficiently over long distances, and transferred from one mode of transport to another A standard container that is loaded with virtually any goods, and stays sealed until it reaches final delivery.
- 14. Containers applied to software
- 15. Static website User DB Web frontend Queue Analytics DB Dev VM QA server Public Cloud Contributor’s laptop Docker is a shipping container system for code Multiplicity of Stacks Multiplicity of hardware environments Production Cluster Customer Data Center Do services and apps interact appropriately? Can I migrate smoothly and quickly …that can be manipulated using standard operations and run consistently on virtually any hardware platform An engine that enables any payload to be encapsulated as a lightweight, portable, self-sufficient container…
- 16. Static website Web frontend Background workers User DB Analytics DB Queue Dev VM QA Server Single Prod Server Onsite Cluster Public Cloud Contributor’s laptop Customer Servers Docker eliminates the matrix from Hell
- 17. Good for developers • Build once…(finally) run anywhere* • A clean, safe, hygienic and portable runtime environment for your app. • No worries about missing dependencies, packages and other pain points during subsequent deployments. • Run each app in its own isolated container, so you can run various versions of libraries and other dependencies for each app without worrying • Automate testing, integration, packaging…anything you can script • Reduce/eliminate concerns about compatibility on different platforms, either your own or your customers. • Cheap, zero-penalty containers to deploy services? A VM without the overhead of a VM? Instant replay and reset of image snapshots? That’s the power of Docker * Anywhere = any x86 server running a modern Linux kernel (3.2+ generally. 2.6.32+ for RHEL 6.5+, Fedora, & related)
- 18. Good for (Dev)Ops • Configure once…run anything • Make the entire lifecycle more efficient, consistent, and repeatable • Increase the quality of code produced by developers. • Eliminate inconsistencies between development, test, production, and customer environments • Support segregation of duties • Significantly improves the speed and reliability of continuous deployment and continuous integration systems • Because the containers are so lightweight, address significant performance, costs, deployment, and portability issues normally associated with VMs
- 19. Why it works—separation of concerns • Dan the Developer – Worries about what’s “inside” the container • His code • His Libraries • His Package Manager • His Apps • His Data – All Linux servers look the same • Oscar the Ops Guy • Worries about what’s “outside” the container • Logging • Remote access • Monitoring • Network config • All containers start, stop, copy, attach, migrate, etc. the same way
- 20. Under the hood • To learn more about how it works, some background: –Linux namespaces • PID • IPTables • UTS (~hostname) • Mount • IPC (inter-process) –CGroups –LXC Containers
- 21. Containers vs. VMs App A Bins / Libs Bins / Libs Bins / Libs Hypervisor (Type 2) Host OS Server Guest OS App A’ Gues t OS App B Gues t OS App A’ Docker Host OS Server App A Bins/Libs Bins/Libs App B App B’ App B’ App B’ VM Container Containers are isolated, but share OS and, where appropriate, bins/libraries Guest OS Guest OS …result is significantly faster deployment, much less overhead, easier migration, faster restart
- 22. Why are Docker containers lightweight? VMs Containers App A Bins / Libs Original App (No OS to take up space, resources, or require restart) App Δ Bins / App A Bins / Libs App A’ Bins / Libs Gues t OS Modified App Copy on write capabilities allow us to only save the diffs between container A and container A’ VMs App A Bins / Libs Guest OS Every app, every copy of an app, and every slight modification of the app requires a new virtual server App A Copy of App No OS. Can Share bins/libs Guest OS Guest OS
- 24. Terminology Docker concept OOP analogy Dockerfile Source code Image Class Container Instance
- 25. What are the basics of the Docker system? Dockerfile For A Source Code Repositor y Docker Engine Docker Container Image Registry Build Docker Host 2 OS (Linux) Container A Container B Container C Container A Push Search Pull Run Host 1 OS (Linux)
- 26. Changes and Updates App A’’ Bins / Libs Docker Engine Docker Container Image Registry Push Update Docker Engine App A Bins / Libs App Δ Bins / Base Container Image Host is now running A’’ Container Mod A’’ App Δ Bins / App A Bins / Libs Bins / Host running A wants to upgrade to A’’. Requests update. Gets only diffs Container Mod A’
- 27. A new open source project… or the next big thing? • Compatible with Jenkins, Travis, Chef, Puppet, Vagrant and OpenStack… • Ecosystem: lots of tools, even by serious contributors (Google – Kubernetes, cAdvisor - NetFlix …), new IAAS provider emerge (like Tutum), • Supported by Amazon, Google, Microsoft, Rackspace, Digital Ocean, Joyent, RedHat, … • Momentum... See pull requests numbers and frequency to get an idea • Used in production… even by big players (HP, Ebay, New Relic, Etsy, Yelp, Spotify, …)
- 28. Ecosystem Support • Operating systems – Virtually any distribution with a 2.6.32+ kernel – Red Hat/Docker collaboration to make work across RHEL 6.4+, Fedora, and other members of the family (2.6.32 +) – CoreOS—Small core OS purpose built with Docker • OpenStack – Docker integration into NOVA (& compatibility with Glance, Horizon, etc.) accepted for Havana release • Private PaaS – OpenShift – Solum (Rackspace, OpenStack) – Other TBA • Public PaaS – Deis, Voxoz, Cocaine (Yandex), Baidu PaaS • Public IaaS – Native support in Rackspace, Digital Ocean,+++ – AMI (or equivalent) available for AWS & other • DevOps Tools – Integrations with Chef, Puppet, Jenkins, Travis, Salt, Ansible +++ • Orchestration tools – Mesos, Heat, ++ – Shipyard & others purpose built for Docker • Applications – 1000’s of Dockerized applications available at index.docker.io
- 29. Use Cases • Ted Dziuba on the Use of Docker for Continuous Integration at Ebay Now – https://speakerdeck.com/teddziuba/docker-at-ebay – http://www.youtube.com/watch?feature=player_embedded&v=0Hi0W4gX--4 • Sasha Klizhentas on use of Docker at Mailgun/Rackspace • http://www.youtube.com/watch?feature=player_embedded&v=CMC3xdAo9RI • Sebastien Pahl on use of Docker at CloudFlare • http://www.youtube.com/watch?feature=player_embedded&v=-Lj3jt_-3r0 • Cambridge HealthCare • http://blog.howareyou.com/post/62157486858/continuous-delivery-with-docker-and- jenkins-part-i • Red Hat Openshift and Docker • https://www.openshift.com/blogs/technical-thoughts-on-openshift-and-docker
- 30. Use Cases—From the Community Use Case Examples Link Clusters Building a MongoDB cluster using docker http://bit.ly/1acbjZf Production Quality MongoDB Setup with Docker http://bit.ly/15CaiHb Wildfly cluster using Docker on Fedora http://bit.ly/1bClX0O Build your own PaaS OpenSource PaaS built on Docker, Chef, and Heroku Buildpacks http://deis.io Web Based JiffyLab – web based environment for the instruction, or lightweight use of, Environment for Python and UNIX shell Instruction http://bit.ly/12oaj2K Easy Application Deployment Deploy Java Apps With Docker = Awesome http://bit.ly/11BCvvu How to put your development environment on docker http://bit.ly/1b4XtJ3 Running Drupal on Docker http://bit.ly/15MJS6B Installing Redis on Docker http://bit.ly/16EWOKh Create Secure Sandboxes Docker makes creating secure sandboxes easier than ever http://bit.ly/13mZGJH Create your own SaaS Memcached as a Service http://bit.ly/11nL8vh Automated Application Deployment Multi-cloud Deployment with Docker http://bit.ly/1bF3CN6 Continuous Integration and Deployment Next Generation Continuous Integration & Deployment with dotCloud’s Docker and Strider http://bit.ly/ZwTfoy Testing Salt States Rapidly With Docker http://bit.ly/1eFBtcm Lightweight Desktop Virtualization Docker Desktop: Your Desktop Over SSH Running Inside Of A Docker Container http://bit.ly/14RYL6x
- 31. How to build an image? Two ways to create an image: • You work in a container, then commit it for exploratory work • You build a new image from a Dockerfile for automated, repeatable process
- 32. 1 container ?= (1process|1task|other) ? Best practices appear • Do not force you to 1 process per container • Ideal case: 1 ‘task’ per container, with separation of concerns. • The task might require 1 single process • Or 0 (for data-only containers) • Or several (ex: nginx+php-fpm, monitoring, …)
- 33. Containers are isolated… then how to make them talk to each other? • Private network between containers • Linking to make them easy to talk to a friend container • Expose ports • Mount volumes (rw or ro) • Advanced/not so trivial other solutions –Service discovery –Enter into the container’s namespace
- 34. How to handle data persistence • Data only containers, sharing data volumes • Mounting this data volumes in the running containers
- 35. How to handle data persistence How data volume works
- 36. How to handle data persistence How data volume works
- 37. Demo time! • Quick look at the Docker commands • Isolation demo • Multi containers demo: installing and launching Go Math! Academy in seconds. • Running unit tests in a container
- 39. Warning •Docker is great –It reduces dependencies, provides isolated and controlled context •But it brings new tools, and new technologies, hence a potential complexity increase –libswarm, systemd, etcd, ambassadord, fleet, etc
- 40. Some references • Start by the online tutorial and documentation, they are great! • Here’s some links kept while learning… but so much available! –Eight Docker Development Patterns –Panamax and Docker and Some Best Practices –Tutum blog –The 5 Most Important Things I’ve Learned From Using Docker –Understanding Docker Volumes –Docker's Youtube videos
- 41. More references First steps • dotScale 2013 - Solomon Hykes - Why we built Docker (video) • Twitter University – Solomon Hykes – Introduction to Docker (video) Going further • Docker layers cost (blog), on how to optimize your images layer size. • Twitter University - Docker at Spotify (video) • Top 10 open-source Docker developer tools (blog, incomplete but gives ideas) • Docker In-depth: Volumes (blog) Food for thought / Stimulate your imagination • AWS re:Invent 2014 - Lightning fast deploys with Docker containers and AWS (video, slides) • FutureStack14 - Docker and the Future of Modern Software (New Relic blog, video) • AWS re:Invent 2014 - Infrastructure as Code (video, more related to Cloudformation than to Docker but very inspiring if you consider the two technologies) • Alternative? Rocket by CoreOS… started a debate (like Joyent answer)
- 42. Tools for dev, day-to-day • For everyone • On Mac – Kitematic UI to manage your images/containers • On Windows – Try the new XShell5 beta to easily deal with all your current containers • So many others… and new ones every week • Fig (Fast isolated environments, by Docker) repo, site • A Better Development Environment with Docker and Fig (blog) • Lots of tools to manage multi containers apps: Kubernetes, Shipyard, Panamax,
- 43. Thank you
Editor's Notes
- #39: Placeholder for demos, has I always click next! Switch to shell now