Troubleshooting Tips from a Docker Support Engineer
6 likes1,145 views
The document discusses various troubleshooting techniques for Docker including using tools like socat and curl to characterize networking and TLS issues, checking container processes and permissions, using volumes to store persistent data, and resolving issues with incorrect localhost references between containers. It also provides examples of troubleshooting issues with a Minecraft server, Ruby application, and Nginx proxy configuration.
![$ docker build -t my_sinatra_app .
$ docker run -p 4567:4567 --name webdev
-v /home/jane/code:/app/code my_sinatra_app
23:30:18 [rerun] Code launched
/usr/local/bundle/gems/sqlite3-1.3.13/lib/sqlite3/database.rb:9
1:in `initialize': no such table: config
…
Jane's Ruby App](https://image.slidesharecdn.com/troubleshootingtipsfromadockersupportengineer-dockercon2017-170420154645/85/Troubleshooting-Tips-from-a-Docker-Support-Engineer-27-320.jpg)
![FROM python:3-alpine
RUN pip install bottle
ADD . /code
WORKDIR /code
EXPOSE 8000
CMD ["python", "app.py"]
Application Dockerfile](https://image.slidesharecdn.com/troubleshootingtipsfromadockersupportengineer-dockercon2017-170420154645/85/Troubleshooting-Tips-from-a-Docker-Support-Engineer-41-320.jpg)
![$ source env.sh
$ docker-compose up -d
ERROR: SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate
verify failed (_ssl.c:590)
compose TLS issue](https://image.slidesharecdn.com/troubleshootingtipsfromadockersupportengineer-dockercon2017-170420154645/85/Troubleshooting-Tips-from-a-Docker-Support-Engineer-67-320.jpg)
![$ curl -s --unix-socket /var/run/docker.sock
http::/containers/json | jq '.[].Names[0]'
"/focused_tesla"
"/exciting_einstein"
"/web"
"/app"
docker ps with curl | jq](https://image.slidesharecdn.com/troubleshootingtipsfromadockersupportengineer-dockercon2017-170420154645/85/Troubleshooting-Tips-from-a-Docker-Support-Engineer-88-320.jpg)
Troubleshooting Tips from a Docker Support Engineer
- 1. Troubleshooting Jeff Anderson Developer Support Engineer at Docker @programm3rq
- 2. Troubleshooting Basics Common Issues ○ Volumes ○ Networking ○ TLS Advanced Troubleshooting Techniques Troubleshooting
- 5. 1. Characterization 2. Hypothesis 3. Test & Observe Troubleshooting Basics
- 8. Minecraft Server ● Single Java Process ● Stores game world state on disk ● Listens on port 25565 Enthusiast/Power User/Tinkerer Bob
- 9. FROM java:7 ADD minecraft_server.1.10.2.jar / RUN mkdir -p /opt/minecraft RUN echo "eula=true" > /opt/minecraft/eula.txt EXPOSE 25565 WORKDIR /opt/minecraft CMD java -jar /minecraft_server.1.10.2.jar Minecraft Dockerfile
- 10. $ docker build -t mc:1.10.2 . $ docker run -d --name old -p 25565:25565 mc:1.10.2 Minecraft Build and Run
- 12. FROM java:7 ADD minecraft_server.1.11.2.jar / RUN mkdir -p /opt/minecraft RUN echo "eula=true" > /opt/minecraft/eula.txt EXPOSE 25565 WORKDIR /opt/minecraft CMD java -jar /minecraft_server.1.11.2.jar Minecraft Dockerfile (updated)
- 13. $ docker build -t mc:1.11.2 . $ docker stop old $ docker run -d --name new -p 25565:25565 mc:1.11.2 Minecraft Build and Run (updated)
- 15. Where did my stateful minecraft data go?! Bob
- 16. Storing important data A volume is a directory on the host that is made available to a container. Docker does this with a bind mount. Volumes
- 17. $ mount -o bind /opt/source /opt/destination $ touch /opt/source/test $ ls -li /opt/source/* /opt/destination/* 497080 -rw-r--r-- 1 root root 0 Apr 9 01:37 /opt/destination/test 497080 -rw-r--r-- 1 root root 0 Apr 9 01:37 /opt/source/test $ ls -lid /opt/source/ /opt/destination/ 500424 drwxr-xr-x 2 root root 4096 Apr 9 01:37 /opt/destination/ 500424 drwxr-xr-x 2 root root 4096 Apr 9 01:37 /opt/source/ Bind Mount
- 18. Three Types 1. Host volume "I want my data to be here specifically" 2. Named Volume "I want to refer to my data later easily" 3. Anonymous Volume "I just want a volume" Volumes
- 19. # Host Volume $ docker run -v /opt/hostpath:/container/data … # Named Volume $ docker run -v important_stuff:/container/data … # Anonymous Volume $ docker run -v /container/data … Volume Types
- 20. Bob Minecraft data should go in a volume.
- 21. $ docker diff old … C /opt/minecraft A /opt/minecraft/server.properties A /opt/minecraft/world A /opt/minecraft/world/region A /opt/minecraft/world/region/r.0.0.mca … Put data in a volume
- 22. $ docker volume create minecraft $ docker create --name new -p 25565:25565 -v minecraft:/opt/minecraft mc:1.11.2 $ docker cp old:/opt/minecraft minecraft $ docker cp minecraft new:/opt/ $ docker start new Put data in a volume
- 23. Use volumes to designate where stateful data goes Bob
- 24. Local dev environment Ubuntu 16.04 desktop Wants to use Docker in her development workflow Ruby Developer Jane
- 25. Useful for local development Jane uses RubyMine Wants code auto-reload with the rerun gem Host Volumes
- 26. FROM ruby RUN gem install sinatra sqlite3 rerun COPY . /app/code WORKDIR /app/code EXPOSE 4567 CMD rerun 'ruby server.rb -o 0.0.0.0' Ruby App Dockerfile
- 27. $ docker build -t my_sinatra_app . $ docker run -p 4567:4567 --name webdev -v /home/jane/code:/app/code my_sinatra_app 23:30:18 [rerun] Code launched /usr/local/bundle/gems/sqlite3-1.3.13/lib/sqlite3/database.rb:9 1:in `initialize': no such table: config … Jane's Ruby App
- 28. Useful for local development This development environment needs a test database. By default, it creates an sqlite3 file called test.db This can be initialized with the 'init.sql' file in the project Host Volumes
- 29. $ sqlite3 -bail test.db < init.sql Error: near line 1: attempt to write a readonly database Jane's Ruby App
- 31. Permission and ownership issues are dealt with in the same way with and without docker. The numeric uid is what matters. Permissions and Ownership
- 32. $ sqlite3 -bail test.db < init.sql Error: near line 1: attempt to write a readonly database $ ls -lin … 6721104 -rw-r--r-- 1 1000 1000 163 Apr 18 2017 init.sql 6721145 -rw-r--r-- 1 0 0 0 Apr 18 2017 test.db Jane's Ruby App
- 33. Characterization and Hypothesis ● Files created by the container are owned by uid 0 ● The image's default user is uid 0 ● test.db file permissions are 0644 ● sqlite3 is running as uid 1000 (jane) Hypothesis: this is a normal permissions/ownership issue. Permissions and Ownership
- 34. Characterization and Hypothesis Do these: ● chown 1000 test.db ● run container as uid 1000 Avoid these: ● chmod 777 ● sudo sqlite3 Permissions and Ownership
- 35. examples of containerized process writing files ● database files ● pid files ● bytecode caching ● in-app file uploads ● plugin/theme installation ● log files Permissions and Ownership
- 36. Docker for Mac Docker for Mac shares files from macos host to hyperkit VM This file sharing mechanism will ensure files written by containers will always match your macos user id Host Volumes
- 39. Working on a small Python web application. Early stages of development. Ready to Dockerize the project.Web Developer Small Company Josh
- 40. from bottle import route, run, template import socket @route('/') def index(): return str(socket.gethostname()) + 'n' run(host='0.0.0.0', port=8000) Application Code
- 41. FROM python:3-alpine RUN pip install bottle ADD . /code WORKDIR /code EXPOSE 8000 CMD ["python", "app.py"] Application Dockerfile
- 42. $ docker build -t app . $ docker run -d --name web -p 8000:8000 app $ curl http://localhost:8000 d8939bc62a36 Running the python code
- 43. server { listen 80; server_name localhost; location / { root /usr/share/nginx/html; index index.html index.htm; proxy_pass http://localhost:8000/; } } nginx config file
- 44. FROM nginx:alpine RUN rm -f /etc/nginx/conf.d/default.conf ADD nginx.conf /etc/nginx/conf.d/default.conf nginx Dockerfile
- 45. $ docker build -t mynginx . $ docker run -d --name nginx -p 80:80 mynginx $ curl http://localhost/ <html> <head><title>502 Bad Gateway</title></head> <body bgcolor="white"> <center><h1>502 Bad Gateway</h1></center> <hr><center>nginx/1.11.10</center> </body> </html> Running nginx
- 46. Web Developer Small Company Josh Unexpected 502 Error
- 47. 502 - app.py and nginx Networking localhost eth0 nginx localhost eth0 - 172.18.0.6 nginx - 0.0.0.0:80 0.0.0.0:80 web localhost eth0 - 172.18.0.5 app.py - 0.0.0.0:8000 0.0.0.0:8000
- 48. 502 Characterization and Hypothesis ● curl localhost:8000 does not work from nginx container (connection refused) ● curl localhost:8000 works from the app container ● curl 172.18.0.5:8000 works from the nginx container ● curl 172.18.0.5:8000 works from the app container Networking
- 49. 502 - app.py and nginx Networking localhost eth0 nginx localhost eth0 - 172.18.0.6 nginx - 0.0.0.0:80 0.0.0.0:80 web localhost eth0 - 172.18.0.5 app.py - 0.0.0.0:8000 0.0.0.0:8000 curlcurl
- 50. 502 Characterization and Hypothesis Hypothesis: nginx using the 'localhost' upstream is incorrect Test: update the nginx config file with the container ip. Networking
- 51. server { listen 80; server_name localhost; location / { root /usr/share/nginx/html; index index.html index.htm; proxy_pass http://172.18.0.5:8000/; } } nginx config file
- 52. $ curl http://localhost d8939bc62a36 Running the python code
- 53. 502 - app.py and nginx Networking localhost eth0 nginx localhost eth0 - 172.18.0.6 nginx - 0.0.0.0:80 0.0.0.0:80 web localhost eth0 - 172.18.0.5 app.py - 0.0.0.0:8000 0.0.0.0:8000
- 54. 502 - app.py and nginx Networking localhost eth0 nginx localhost eth0 - 172.18.0.6 nginx - 0.0.0.0:80 0.0.0.0:80 web localhost eth0 - 172.18.0.5 app.py - 0.0.0.0:8000
- 55. Network Service Discovery How will nginx discover the IP going forward? Docker runs a resolver at 127.0.0.11. It resolves container ips by their --name or --net-alias Networking
- 56. server { listen 80; server_name localhost; resolver 127.0.0.11; location / { root /usr/share/nginx/html; index index.html index.htm; proxy_pass http://web:8000/; } } nginx config file updated
- 59. Docker EE Docker Datacenter Deploys internal apps Devops Team at a big company Working on the Docker Project Steven
- 61. Universal Control Plane ● Implements the Docker Daemon API on port 443 ● There is a web GUI as well ● You connect to it with a "client bundle" TLS
- 62. $ ls … ca.pem cert.pem key.pem … env.sh $ cat env.sh export DOCKER_TLS_VERIFY=1 export DOCKER_CERT_PATH="$(pwd)" export DOCKER_HOST=tcp://ucp.example.com:443 Client Bundle
- 63. $ source env.sh $ docker run --rm -it alpine echo hello dockercon hello dockercon $ docker service create -p 80:80 nginx:alpine ellhziigdmo2hae2z7wxuv4qt Client Bundle
- 65. Installed New Certs ● Chrome no longer complains about the self signed certificate ● docker run and docker service still work as they did before TLS
- 66. Steven User reports TLS error
- 67. $ source env.sh $ docker-compose up -d ERROR: SSL error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590) compose TLS issue
- 68. TLS issue reported after cert install ● TLS error when using compose ● Same endpoint works in browser ● Same endpoint works with `docker` CLI Hypothesis: compose has different TLS client expectations from this TLS endpoint TLS
- 69. TLS issues don't need to be scary Cheat sheet (check the following): ● Subject/Alt name match ● Full Chain of Trust ● Chain Root is trusted TLS
- 70. TLS issues don't need to be scary Cheat sheet (check the following): ● Subject/Alt name match correct ● Full Chain of Trust ● Chain Root is trusted TLS
- 71. openssl x509 -noout -text < 0.pem | grep 'Subject:|Issuer:' Issuer: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3 Subject: CN=ucp.example.com cert Subject and Issuer
- 72. openssl x509 -noout -text < 1.pem | grep 'Subject:|Issuer:' Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3 Subject: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3 cert Subject and Issuer
- 73. TLS issues don't need to be scary Cheat sheet (check the following): ● Subject/Alt name match correct ● Full Chain of Trust missing root ● Chain Root is trusted TLS
- 74. openssl x509 -noout -text < 2.pem | grep 'Subject:|Issuer:' Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3 Subject: O=Digital Signature Trust Co., CN=DST Root CA X3 cert Subject and Issuer
- 75. root: Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3 Subject: O=Digital Signature Trust Co., CN=DST Root CA X3 intermediary: Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3 Subject: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3 certificate: Issuer: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3 Subject: CN=ucp.example.com cert chain
- 77. TLS issues don't need to be scary Cheat sheet (check the following): ● Subject/Alt name match correct ● Full Chain of Trust correct ● Chain Root is trusted TLS
- 78. TLS issues don't need to be scary Cheat sheet (check the following): ● Subject/Alt name match correct ● Full Chain of Trust correct ● Chain Root is trusted correct TLS
- 79. $ source env.sh $ docker-compose up -d … Creating network "acme_default" with the default driver Creating acme_tomcat_1 Creating acme_apache_1 docker-compose working
- 80. TLS issue when using compose ● TLS works when using compose ● Same endpoint works in browser ● Same endpoint works with `docker` CLI Python TLS client wants the certificate authority it trusts to be a root certificate. TLS
- 81. TLS Pro Steven
- 83. Amber keeps up pace by being proactive She has several general troubleshooting tactics that help characterize issuesWorks at a big company Has been a sysadmin, developer, network admin Currently technical lead on the devops team Amber
- 84. Tools - command line utilities ● socat - bidirectional communication over tcp, udp, stdio, pipes, unix domain sockets, etc ● curl - make web requests ● jq - parse, filter, create json text ● regular network tools - iptables, ipvsadm, route, ip, arp, tcpdump, ifconfig ● nsenter - enter a namespace Amber's Toolbox
- 85. Tools - command line utilities ● Nico Kabar's netshoot container: ○ https://github.com/nicolaka/netshoot ○ docker pull nicolaka/netshoot ● Jérôme Petazzoni's nsenter ○ https://github.com/jpetazzo/nsenter Amber's Toolbox
- 86. $ socat -v tcp4-listen:5566,bind=127.0.0.1,reuseaddr,fork unix-connect:/var/run/docker.sock $ docker -H 127.0.0.1:5566 ps MITM docker socket traffic
- 87. $ socat -v tcp4-listen:5566,bind=127.0.0.1,reuseaddr,fork unix-connect:/var/run/docker.sock > 2017/04/16 10:38:09.400245 length=131 from=115 to=245 GET /v1.26/containers/json HTTP/1.1r Host: 127.0.0.1:5566r User-Agent: Docker-Client/17.03.0-ce (darwin)r Accept-Encoding: gzipr r < 2017/04/16 10:38:09.401486 length=197 from=199 to=395 HTTP/1.1 200 OKr Api-Version: 1.26r Content-Type: application/jsonr Date: Sun, 16 Apr 2017 15:38:09 GMTr Docker-Experimental: truer Server: Docker/17.03.0-ce (linux)r Transfer-Encoding: chunkedr … MITM docker socket traffic
- 88. $ curl -s --unix-socket /var/run/docker.sock http::/containers/json | jq '.[].Names[0]' "/focused_tesla" "/exciting_einstein" "/web" "/app" docker ps with curl | jq
- 89. $ PID=$(docker inspect --format {{.State.Pid}} happy_tesla) $ nsenter -n -t $PID iptables -nL $ nsenter -t `pidof dockerd` -m nsenter --net=/var/run/docker/netns/ingress_sbox ipvsadm -l $ for i in /var/run/docker/netns/* ; do nsenter -t `pidof dockerd` -m nsenter --net=$i ifconfig; done nsenter
- 90. Techniques Host A container networking is working Host B container networking is not They are seemingly identical How to identify the differences? graphical diff! Amber's Toolbox
- 93. Techniques - How to Ask a Question Amber's Toolbox <statement of observation> |---------------------------| | demonstration of relevant observations |---------------------------| <question>
- 94. Techniques - How to Ask a Question Amber's Toolbox <statement of observation> |---------------------------| | demonstration of relevant observations |---------------------------| <question> Characterization Hypothesis
- 95. Techniques - How to Ask a Question Amber's Toolbox I'm getting a 502 error when I hit the staging acmecorp endpoint $ curl -vkL https://staging.internal.acmecorp.com/_ping/ … Is there a deploy happening now?
- 96. Becoming a Troubleshooting Pro ● Docker Forums https://forums.docker.com/ ● Docker Community Slack https://dockr.ly/community What you can do
- 97. THANK YOU Be a troubleshooting pro! @docker #dockercon Jeff Anderson @programm3rq