![12
Send message
http://docs.celeryproject.org/en/latest/userguide/calling.html
kick.apply_async([‘neuron_x’])
kick.apply_async([‘neuron_x’], routing_key = ‘neuron_h1’)](https://image.slidesharecdn.com/buildingdistributedsystemwithceleryondockerswarm-pyconjp2016-160925085320/85/Building-Distributed-System-with-Celery-on-Docker-Swarm-PyCon-JP-2016-12-320.jpg)
![33
Deploy parallel computing
• Deploy containers over Docker Swarm
– deploy containers
• docker-compose up –d
• docker-compose scale celery= n (for example)
• Send task messages to worker containers
results = [ doSomething.apply_async([data]) for data in dataset ]
• But, Docker Swarm needs
root password-less SSH](https://image.slidesharecdn.com/buildingdistributedsystemwithceleryondockerswarm-pyconjp2016-160925085320/85/Building-Distributed-System-with-Celery-on-Docker-Swarm-PyCon-JP-2016-33-320.jpg)
![36
Summary
• Easy to use
– doSomething.apply_async ([xxx], routing_key = ‘host_NY’)
– getData.apply_async(routing_key = ‘host_NY’) .get()
• Adventages of a distributed system
– Decoupling, DI/IC (dependency injection / inversion of control )
– Distributed = Shared
– Load-Balancing
• Caution:
– See ref: Distributed systems theory for the distributed systems engineer
• Take a look of Canvas in Celery
– Chain
– Group](https://image.slidesharecdn.com/buildingdistributedsystemwithceleryondockerswarm-pyconjp2016-160925085320/85/Building-Distributed-System-with-Celery-on-Docker-Swarm-PyCon-JP-2016-36-320.jpg)
Building Distributed System with Celery on Docker Swarm - PyCon JP 2016
- 1. Building Distributed System with Celery on Docker Swarm Wei Lin PyCon JP 2016 2016/09/22
- 2. 2 About Me • Wei Lin @Wei_1144 [email protected] https://github.com/Wei1234c • Mostly worked in the fields of: – marketing – strategy-planning – project-management • Materials of this talk: https://github.com/Wei1234c/PyCon-JP-2016-talk
- 7. 7 Neural network h1 hh2 h3
- 9. Celery
- 11. Send message to a specific queue
- 16. Assign an unique worker for a specific queue
- 17. 17 Starting the Workerhttp://docs.celeryproject.org/en/latest/userguide/workers.html https://github.com/celery/celery/issues/3065 $ celery -A IoT worker -n neuron_h1 -Q neuron_h1 --concurrency=1
- 18. Docker
- 22. 22 Celery worker containers as neurons h h1 h2 h3 kick.apply_async( routing_key = ‘neuron_h1’) getData.apply_async(routing_key = ‘neuron_h1’) .get()
- 23. 23 “Celery + Docker-Swarm” So ? I think…
- 25. 25 Think OO • “From” to Inherit • Dockerfile to Encapsulate • Docker Image as Class • Docker Container as Object
- 26. 26 Docker Swarm & Celery doSomething.apply_async (routing_key = ‘host_NY’) Tokyo Taipei New York getData.apply_async(routing_key = ‘host_NY’) .get()
- 28. 28 OOAD Station C Station B Station A
- 31. 31 parallel computing ⊆⊆⊆⊆⊆⊆⊆⊆ distributed computing
- 32. 32 How to do Parallel Computing 1. Run and deploy Docker containers across Docker Swarm cluster, with a Celery worker process running in each container. 2. When initiate worker processes, don’t assign “Queue” parameter, so ALL the workers will listen to the default queue in broker. 3. Sending bunch of messages to broker without setting routing_key parameter, the messages will be routed to the default queue. 4. In a parallel fashion, all workers will pick up messages from the default queue and do the tasks.
- 33. 33 Deploy parallel computing • Deploy containers over Docker Swarm – deploy containers • docker-compose up –d • docker-compose scale celery= n (for example) • Send task messages to worker containers results = [ doSomething.apply_async([data]) for data in dataset ] • But, Docker Swarm needs root password-less SSH
- 34. 34 SETI@home
- 35. 35 One for all, All for one https://github.com/Wei1234c/OneForAll_AllForOne/blob/master/celery_projects/OneForAll_AllForOne.md • run a Docker image to join distributed parallel computing cluster, that’s all. docker run –d wei1234c/musketeer (for example)
- 36. 36 Summary • Easy to use – doSomething.apply_async ([xxx], routing_key = ‘host_NY’) – getData.apply_async(routing_key = ‘host_NY’) .get() • Adventages of a distributed system – Decoupling, DI/IC (dependency injection / inversion of control ) – Distributed = Shared – Load-Balancing • Caution: – See ref: Distributed systems theory for the distributed systems engineer • Take a look of Canvas in Celery – Chain – Group
- 37. 37 Source Code • Celery on Docker Swarm – using “Word Count” as an example of parallel computing. – https://github.com/Wei1234c/CeleryOnDockerSwarm/tree/master /celery_projects • IoT as Brain – simulate an artificial neural-network of 6 neurons. – https://github.com/Wei1234c/IOTasBrain/tree/master/celery_proj ects • One for all, all for one – follow the SETI@home paradigm. – on each machine, running a single Docker container to join the cluster. – https://github.com/Wei1234c/OneForAll_AllForOne/tree/master/c elery_projects
- 38. 38 References References : • Distributed systems theory for the distributed systems engineer – http://the-paper-trail.org/blog/distributed-systems-theory-for-the-distributed-systems- engineer/ • Celery user guide – http://docs.celeryproject.org/en/latest/userguide/ • Docker document – https://docs.docker.com/ • MQTT Message Type and Flows – http://blog.maxkit.com.tw/2014/02/mqttmessage-type-and-flows.html • Celery on Docker Swarm – https://github.com/Wei1234c/PyCon-JP-2016- talk/blob/master/Celery%20on%20Docker%20Swarm%20- %20PyCon%20JP%202016.md • IoT as Brain – https://github.com/Wei1234c/PyCon-JP-2016- talk/blob/master/IoT%20as%20Brain%20-%20PyCon%20JP%202016.md