Simon Laws – Apache Flink Cluster Deployment on Docker and Docker-Compose
5 likes10,687 views
This document provides instructions for deploying an Apache Flink cluster on Docker and Docker Compose. It describes setting up the necessary tools like VirtualBox and Ubuntu, installing Docker and Flink, building Docker images from the Flink source code, and running Flink containers locally. It then explains how to push the images to IBM Bluemix and run the Flink cluster within Bluemix containers, including creating the JobManager and TaskManager containers through the Bluemix CLI.
Simon Laws – Apache Flink Cluster Deployment on Docker and Docker-Compose
- 1. IBM Confidential 1 Apache Flink Cluster Deployment On Docker and Docker-Compose Simon Laws [email protected]
- 2. © 2014 IBM Corporation What We’re Going To Build 2 Windows 7 IBM Bluemix VirtualBox 4.3.18 Ubuntu 14.04 Docker 4.3.18 Cygwin ssh Flink Jobmanager container Flink Taskmanager container Cloudfoundry Client Docker Flink image Flink Jobmanager container Flink Taskmanager container Flink image
- 3. © 2014 IBM Corporation Docker 3 docker.com
- 4. © 2014 IBM Corporation Docker 4 docker.com/ whatisdocker
- 5. © 2014 IBM Corporation IBM Bluemix Sign up for a free account and experiment 5 bluemix.net
- 6. © 2014 IBM Corporation IBM Bluemix Signup Sign up for a free account and experiment Various options for academics, startups etc. Prices here for when you go over free allocation https://console.eu- gb.bluemix.net/pricing/ 6 https://console.ng.bluemix.net/registration/
- 7. © 2014 IBM Corporation Apache Flink and Docker 7 github.com/apache/ flink/tree/master/ flink-contrib/ docker-flink
- 8. © 2014 IBM Corporation Install Ubuntu Using VirtualBox 8 Windows 7 IBM Bluemix VirtualBox 4.3.18 Ubuntu 14.04 Docker 4.3.18 Cygwin ssh Flink Jobmanager container Flink Taskmanager container Cloudfoundry Client Docker Flink image Flink Jobmanager container Flink Taskmanager container Flink image
- 9. © 2014 IBM Corporation Setting Up The Host Operating System If you’re on Linux you’re probably good to go If you’re on Windows or OSX you need to run Linux on a VM As an alternative you could do this on a VM in the new VM service in Bluemix. It’s in Beta at the moment. Get VirtualBox Create a Linux VM (I used Ubuntu) Install Docker Download Flink Source Try a Flink example 9
- 10. © 2014 IBM Corporation Get VirtualBox 10 virtualbox.org
- 11. © 2014 IBM Corporation Get Linux Distro (I went for Ubuntu 14.04) 11 releases.ubuntu.com/14.04
- 12. © 2014 IBM Corporation Create VM from ISO in VirtualBox (and set port forwarding) 12 End result is that I can now SSH into the instance of Ubuntu running inside VitualBox. Everything for this point on happens inside this Ubuntu VM To start with make sure that Ubuntu is up to date > sudo su - > apt-get update sudo su - apt-get updatesudo su - apt-get update
- 13. © 2014 IBM Corporation Install Docker and Flink 13 Windows 7 IBM Bluemix VirtualBox 4.3.18 Ubuntu 14.04 Docker 4.3.18 Cygwin ssh Flink Jobmanager container Flink Taskmanager container Cloudfoundry Client Docker Flink image Flink Jobmanager container Flink Taskmanager container Flink image
- 14. © 2014 IBM Corporation Install Docker and Docker-Compose 14 github.com/apache/ flink/tree/master/ flink-contrib/ docker-flink
- 15. © 2014 IBM Corporation Download Flink Source 15 git clone https://github.com/apache/flink.git cd flink/flink-contrib/docker-flink The base Docker image source Builds the “base” and “flink” images The flink Docker image source Docker-Compose configuration
- 16. © 2014 IBM Corporation Docker Images and Containers 16 Source Docker Image Docker Container docker build docker run What Images Do I have? docker images What Containers Do I have running? Push elsewhere (Docker Hub) To share with others ssh or docker exec docker psDockerfile
- 17. © 2014 IBM Corporation Dockerfiles – The Recipe for a Docker Image 17
- 18. © 2014 IBM Corporation Build Docker Images 18 Windows 7 IBM Bluemix VirtualBox 4.3.18 Ubuntu 14.04 Docker 4.3.18 Cygwin ssh Flink Jobmanager container Flink Taskmanager container Cloudfoundry Client Docker Flink image Flink Jobmanager container Flink Taskmanager container Flink image
- 19. © 2014 IBM Corporation Building The Images From the Docker Files 19 ./build.sh If you look in the script it use the “docker build” command to build a “base” image and a “flink” image This failed for me on the first attempt with the base build reporting chpasswd: (user root) pam_chauthtok() failed, error: System error chpasswd: (line 1, user root) password not changed It turned out I had an old version of Docker so I updated to latest and it worked
- 20. © 2014 IBM Corporation The Built Images 20
- 21. © 2014 IBM Corporation Run Docker Containers 21 Windows 7 IBM Bluemix VirtualBox 4.3.18 Ubuntu 14.04 Docker 4.3.18 Cygwin ssh Flink Jobmanager container Flink Taskmanager container Cloudfoundry Client Docker Flink image Flink Jobmanager container Flink Taskmanager container Flink image
- 22. © 2014 IBM Corporation Running Linked Images Manually (To See What’s Going On) 22 docker run -d -p 48080:8080 -p 48081:8081 --expose 6123 --name jobmanager flink /usr/local/flink/bin/config-flink.sh jobmanager docker run -d --expose 6121 --expose 6122 --link jobmanager --name taskmanager1 flink The job manager Add a task manager This failed at first with a strange “Exec format exception”. I made a local fix by removing the comment from the top of the flink/config-flink.sh file leaving #!/bin/bash As the first line
- 23. © 2014 IBM Corporation Flink up and Running 23
- 24. © 2014 IBM Corporation Running a Test 24 I had previously uploaded flink-java-examples-0.9.1-WordCount.jar from my local Windows install of Flink
- 25. © 2014 IBM Corporation Accessing Containers with Docker Exec 25
- 26. © 2014 IBM Corporation Making Things Simpler With Docker-Compose 26 docker-compose up –d docker-compose scale taskmanager=3 docker-compose kill
- 27. © 2014 IBM Corporation Connecting To Bluemix 27 Windows 7 IBM Bluemix VirtualBox 4.3.18 Ubuntu 14.04 Docker 4.3.18 Cygwin ssh Flink Jobmanager container Flink Taskmanager container Cloudfoundry Client Docker Flink image Flink Jobmanager container Flink Taskmanager container Flink image
- 28. © 2014 IBM Corporation Running the Images On Bluemix 28 Log in to Bluemix: 1.cf login Run the IBM Containers cf CLI plug-in. 2.cf ic login Create a Dockerfile in the root directory of your app source files. Learn more about Dockerfiles. Build an image from your Dockerfile. The command returns an image ID. 3. docker build -t image_name Tag the image with your private namespace in the IBM Containers registry. 4. docker tag image_name registry.stage1.ng.bluemix.net/sluk/image_name:image_tag The image name is optional. If it is not specified, the image is tagged with latest. Push this image to the IBM Containers registry: 5. docker push registry.stage1.ng.bluemix.net/sluk/image_name:image_tag You can create a container from this image in the Bluemix Catalog, or with the following command: 6. cf ic run --name container_name registry.stage1.ng.bluemix.net/sluk/image_name:image_tag
- 29. © 2014 IBM Corporation Install The Cloud Foundary Client Software 29 In the Ubuntu VM we have been using to run Flink using Docker get and install the Cloud Foundary client tool wget --output-document=cfcli.deb https://cli.run.pivotal.io/stable?release=debian64&version=6.12.4&source=githu b-rel dpkg -i cfcli.deb In the Ubuntu VM we have been using to run Flink using Docker get and install the Cloud Foundary client cf install-plugin https://static-ice.ng.bluemix.net/ibm-containers-linux_x64
- 30. © 2014 IBM Corporation Log Into Bluemix – Web User Interface 30
- 31. © 2014 IBM Corporation Log Into Bluemix – Command Line Using CF Tool 31
- 32. © 2014 IBM Corporation Log Into Bluemix Containers – Command Line Using CF Tool 32
- 33. © 2014 IBM Corporation Push Image Up To Bluemix 33 Windows 7 IBM Bluemix VirtualBox 4.3.18 Ubuntu 14.04 Docker 4.3.18 Cygwin ssh Flink Jobmanager container Flink Taskmanager container Cloudfoundry Client Docker Flink image Flink Jobmanager container Flink Taskmanager container Flink image
- 34. © 2014 IBM Corporation Tag and Push Up Our Images 34 docker tag flink registry.ng.bluemix.net/sluk/flink docker push registry.ng.bluemix.net/sluk/flink
- 35. © 2014 IBM Corporation List Images Now Up On Bluemix 35 cf ic images
- 36. © 2014 IBM Corporation Run Containers On Bluemix 36 Windows 7 IBM Bluemix VirtualBox 4.3.18 Ubuntu 14.04 Docker 4.3.18 Cygwin ssh Flink Jobmanager container Flink Taskmanager container Cloudfoundry Client Docker Flink image Flink Jobmanager container Flink Taskmanager container Flink image
- 37. © 2014 IBM Corporation Creating Containers in Bluemix via User Interface 37
- 38. © 2014 IBM Corporation Creating Containers in Bluemix via CF Command Line 38 Option 1) Leverage this Cloud Foundry IBM Containers plugin without affecting the local docker environment: Example Usage: cf ic ps cf ic images Option 2) Leverage the docker CLI directly. In this shell, override local docker environment to connect to IBM Containers by setting these variables, export DOCKER_HOST=tcp://containers-api.ng.bluemix.net:8443 export DOCKER_CERT_PATH=/root/.ice/certs export DOCKER_TLS_VERIFY=1 Example Usage: docker ps docker images
- 39. © 2014 IBM Corporation39 Create jobmanager Container in Bluemix via CF Command Line cf ic run -p 22:22 -p 8080:8080 -p 8081:8081 -p 6123:6123 --expose 6123 --name jobmanager registry.ng.bluemix.net/sluk/flink /usr/local/flink/bin/config-flink.sh jobmanager Assign a public IP so you can get to the console
- 40. © 2014 IBM Corporation40 Create taskmanager Container in Bluemix via CF Command Line cf ic run -p 22:22 -p 6121:6121 -p 6122:6122 --expose 6121 -- expose 6122 --name taskmanager1 --link jobmanager:jobmanager registry.ng.bluemix.net/sluk/flink /usr/local/flink/bin/config- flink.sh taskmanager
- 41. © 2014 IBM Corporation Running The Flink Cluster 41
- 42. © 2014 IBM Corporation QUESTIONS? 42