Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009
38 likes6,649 views
The document discusses distributed programming using RabbitMQ, emphasizing its role in concurrent computing by dividing tasks among multiple nodes. It covers challenges such as managing worker nodes, message acknowledgment, and control scripts, while promoting RabbitMQ's capabilities for high-performance messaging. Additionally, it highlights the use of Erlang and EventMachine to facilitate efficient, non-blocking operations in a distributed environment.
Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009
- 1. “Divide and conquer riding rabbits and trading gems” - a tale about distributed programming - • http://www.slideshare.net/hungryblank • http://github.com/hungryblank/rabbit_starter Paolo Negri @hungryblank
- 3. About me Time GNU/Linux - Dbs Perl “systems” PHP Python Ruby
- 4. Summary:
- 7. Control
- 8. The problem Given a dictionary of 1.000.000 search phrases, compare for each one the first page of results on google.com and bing.com
- 9. Don’t do it. It’s a violation of terms of service! (Sorry, I needed an example that required little or no explanation)
- 10. Comparison
- 12. Great! but... • Fetching, parsing and analyzing 2.000.000 pages of results will take a long time • we want to collect the statistics in a limited time and this sequential approach is not quick enough
- 13. Distributed Computing Is a method of solving computational problem by dividing the problem into many tasks run simultaneously on many hardware or software systems (Wikipedia)
- 14. Map Reduce "Map" step the master node takes the input, chops it up into smaller sub-problems, and distributes those to worker nodes. (Wikipedia)
- 15. Problems: • How many nodes? • How many workers? • Distribution mechanism to feed the workers?
- 16. What about queuing? • the master node takes the input, chops it up into smaller sub-problems, and publishes them in a queue • workers independently consume the content of the queue
- 17. Here comes • RabbitMQ is an implementation of AMQP, the emerging standard for high performance enterprise messaging • It’s opensource • Written in Erlang
- 18. Erlang? • general-purpose concurrent programming language designed by Ericsson • first version written by J. Armstrong in 1986 • distributed • fault tolerant • soft real time • high availability
- 19. Erlang - is coming back • Projects • CouchDB - RESTful document storage • WebMachine - REST toolkit • Nitrogen - web framework • Mochiweb - web framework • Ruby shops using it • 37 Signal - Campfire • Heroku
- 20. + Erlang It’s messages all the way down
- 21. Install it • sudo apt-get install rabbitmq • sudo gem install tmm1-amqp Note: rabbitMQ must be v1.6.0 and amqp gem v 0.6.4 to follow the code in the slides
- 22. Do it! - master node
- 23. Do it! - worker node
- 24. Get for free • Decoupling master/worker • Workers take care of feeding themselves • Flexible number of workers
- 25. Behind the scenes Worker1 Master msg A Exchange Queue Worker2 Worker3
- 26. Behind the scenes Worker1 Master Exchange Queue Worker2 msg A Worker3
- 27. Behind the scenes Worker1 Master Exchange Queue msg A Worker2 Worker3
- 28. Behind the scenes msg A Worker1 Master Exchange Queue Worker2 Worker3
- 29. RabbitMQ • Multiple exchanges (and multiple types of exchange) • Multiple queues • Queues are connected by bindings to exchanges • Exchanges route messages to queues
- 30. RabbitMQ • Exchanges and queues have names • BUT direct exchanges created implicitly are not public and don’t have name • Queues and messages are resident in RAM and can be persisted on disk (at a performance price)
- 31. What and where Master (ruby) RabbitMQ (Erlang) Worker TCP/IP (ruby) Worker (ruby) Exchange Queue Worker (ruby)
- 32. Problem #1 If a worker has a problem we might lose one or more messages http://www.flickr.com/photos/danzen/2288625136/
- 33. Solution - ACK in worker
- 34. Acknoledgement • messages - 1 or more depending by prefetch settings - are passed to the client • the client processes and acknowledges the messages one by one • if the client connection closes => unacknowledged messages go back in the queue
- 35. http://www.flickr.com/photos/streetfly_jz/2770586821/ Problem #2 No convenient way to control the workers
- 36. System queue - worker
- 37. System queue - control • save it as system_command.rb • ruby system_command.rb halt
- 38. System queue Queue1 Worker1 Control script msg A Exchange Queue2 Worker2 Queue3 Worker3
- 39. System queue Queue1 Worker1 Control script Exchange Queue2 Worker2 msg A Queue3 Worker3
- 40. System queue msg A Queue1 Worker1 Control script Exchange msg A Queue2 Worker2 msg A Queue3 Worker3
- 41. EventMachine
- 42. EventMachine Is an implementation of Reactor Pattern • Non blocking IO and lightweight concurrency • eliminate the complexities of high- performance threaded network programming
- 43. without EM with EM Ruby process Ruby process Time code code network network operation Free operation use network use network operation operation Callback result result Free Free
- 44. EventMachine amqp gem is built on top of EventMachine => you’re in a context where you can leverage concurrent programming
- 45. EM - Deferrables “The Deferrable pattern allows you to specify any number of Ruby code blocks that will be executed at some future time when the status of the Deferrable object changes “
- 46. EM - Deferrables
- 47. EM - Deferrables
- 48. EM - Deferrables
- 49. EM - Deferrables
- 50. Deferrables without deferrables with deferrables GooglePage GooglePage BingPage BingPage Time
- 51. http://www.flickr.com/photos/philocrites/341850461/ Problem #3 How many of them? what are they doing?
- 54. Heartbeat queue Worker1 msg A Worker2 Exchange Queue Monitor Worker3 msg B
- 55. Heartbeat queue Worker1 msg A Worker2 Exchange Queue Monitor msg B Worker3
- 56. Heartbeat queue Worker1 Worker2 Exchange Queue msg B msg A Monitor Worker3
- 57. Clustering RabbitMQ - node A RabbitMQ - node B TCP/IP TCP/IP RabbitMQ - node C TCP/IP
- 58. async vs sync Syncronous clients on github • famoseagle/carrot • celldee/bunny
- 59. Rabbits on github • danielsdeleo/moqueue Tests/stubbing • Check out ezmobius/nanite “self assembling fabric of ruby daemons” • Keep an eye on tonyg/rabbithub implementation of pubsubhubbub (PubSub over REST) • auser/alice web app to monitor RabbitMQ
- 60. More rabbits on github • tmm1/em-spec • eventmachine/eventmachine • tmm1/amqp • macournoyer/thin
- 61. Q&A ?
- 62. Thanks! (Thanks Mark!) Paolo Negri / hungryblank