Making Symfony Services async with RabbitMq (and more Symfony)
Download as PPTX, PDF0 likes1,136 views
This document discusses using RabbitMQ and Symfony to generate Microsoft Office documents asynchronously from XML content. Currently, using just LibreOffice is slow, unreliable, and does not scale well. The proposed solution is to use RabbitMQ with Symfony services to queue document generation jobs and process them in parallel with multiple worker processes. This improves performance, reliability and allows the process to scale. Some challenges that still need to be addressed are network security, throughput and determining if a existing solution could be used instead of a custom one.
Making Symfony Services async with RabbitMq (and more Symfony)
- 1. It’s all about eXperience Gaetano Giunta PHP London July 2015 Making Symfony Services async with RabbitMq (and more Symfony) PechaKucha style!
- 2. A case study: generating M$Office docs 2 The needs • Produce content in Microsoft Office formats • Many formats for each document • Both Word and Excel docs • XML content generated by CMS • LibreOffice used to generate MS Office and PDF versions • Has anyone tried generating OOXML by hand?
- 3. A case study: generating M$Office docs 3 Reality bites • Slooow • Rest assured that some of the docs will be pretty big • Unreliable • Depending on version, LO crashes from a-bit to almost- always • Race-prone • Two conversion jobs in parallel might step on each other
- 4. A case study: generating M$Office docs 4 …and the results are: • Ugly code: lots of polling, copying of files around, manual locking • Does not scale at all: only one conversion process active at any given time Web server PHP Libre Office Waiting processes
- 5. TextBook scenario for introducing queues 5 Web server Worker ? Rabbit MQ Libre Office PHP
- 6. TextBook scenario for introducing queues 6 Web server Worker ? Rabbit MQ Libre Office PHP
- 7. TextBook scenario for introducing queues 7 Web server PHP ! Rabbit MQ Libre Office PHP
- 8. TextBook scenario for introducing queues 8 Web server PHP ! Rabbit MQ Libre Office PHP
- 9. TextBook scenario for introducing queues 9 Web server Symfony !!! Rabbit MQ Libre Office PHP
- 10. TextBook scenario for introducing queues 10 Web server Symfony !!! Rabbit MQ Libre Office PHP
- 11. The rationale 11 • Same library handling both sides of the connection • Easier to troubleshoot • Sf Console component is lovely • Everything developed in a single repository • No need to learn new languages • Everything which I can rewrite in PHP, I eventually will* * = 3rd whimsical law of Gaetano
- 12. The details 12 • Json used as payload encoding • The worker has been made thread-safe • Again, thanks to Symfony: the « Process » Component • It spins up a new php process each time it receives a command • The new process executes the command • Bonus: a ‘watchdog’ to restart workers via crontab if they die Symfony listener Symfony listener Symfony listener Symfony worker Libre Office Libre Office W
- 13. The evolution 13 • What is exactly a « queued command » ? • For starters, each existing Symfony console command! • Why not every existing Symfony Service? • Allows to run existing services asynchronously • As long as their arguments are serializable to json • Not a project-specific tool any more • Could be turned into a Bundle • … • Profit!
- 14. The problems 14 • Assumes good security on the network • Low throughput • Spinning up a new php worker process takes time • Could we use php-fpm to communicate to workers instead? • Am I reinventing the wheel? • Others? • Suggestions are welcome
- 15. I’ll be here all night 15 THANK YOU https://github.com/kaliop-uk/kueueingbundle • @gggeek • www.kaliop.co.uk