Celery for internal API in SOA infrastructure
7 likes14,545 views
The document discusses using Celery as an internal API for service-oriented architectures, describing how Celery can provide a low-level interface that defines only task names and handles serialization, routing, and error handling through message passing via a broker like RabbitMQ. It also outlines the benefits of Celery for building asynchronous and parallel APIs, while noting its complexity and that the broker should never be exposed publicly for security reasons.
Celery for internal API in SOA infrastructure
- 1. Celery for internal API in SOA infrastructure Roman Imankulov PyCon Russia, Feb 2013
- 2. 1. Internal API. What's the deal SOA service architecture example
- 3. 1. Internal API. What's the deal ● DCOM: too Microsoftish ● CORBA: too Enterprisy ● SOAP: XML inside ● RESTful: too sloppy Celery to the rescue?
- 4. Agenda 1. Internal API. What's the deal 2. How to use Celery for internal API 3. Organizing Celery-based API. Routing 4. Celery-based API. Benefits, pitfalls and security notes
- 5. 2. How to use Celery for internal API Celery Worker
- 6. 2. How to use Celery for internal API Classic celery client imports task function
- 7. 2. How to use Celery for internal API Classic celery client imports makes you: ● install everything ○ slows down installation ● import everything ○ python processes grow in size ○ circular dependencies ○ tight coupling ● share everything ○ ... with outsource developers
- 8. 2. How to use Celery for internal API "Task implementation is not to be exposed without necessity" William of Ockham
- 9. 2. How to use Celery for internal API Low-level interface defines only task name
- 10. 2. How to use Celery for internal API Convenient API requires some boilerplate code
- 11. 2. How to use Celery for internal API Make it even more convenient with introspection
- 12. 2. How to use Celery for internal API Introspection magic with celery-api http://github.com/imankulov/celery-api
- 13. 3. Celery-based API. Routing
- 14. 3. Celery-based API. Routing Creating API endpoint 1. Common package for all API endpoints 2. Every worker imports celery instance 3. Expose tasks for worker with CELERY_IMPORTS 4. Launch worker with celery worker -A module_name -Q foo -n foo
- 15. 3. Celery-based API. Routing Send tasks with add.apply_async(queue='foo', args=(1, 1))
- 16. 3. Celery-based API. Routing Define routing rules
- 17. 3. Celery-based API. Routing Task in the wire { "body": <base64 encoded string with task name and args>, "properties": { "body_encoding": "base64", "delivery_info": { "priority": 0, "routing_key": "foo", "exchange": "foo" }, "delivery_mode": 2, "delivery_tag": <UUID> }, ...}
- 18. 3. Celery-based API. Routing Task in the wire "delivery_info": { "priority": 0, "routing_key": "foo", "exchange": "foo" }
- 19. 3. Celery-based API. Routing AMQP for dummies
- 20. 3. Celery-based API. Routing Define all queues (API endpoints) you have
- 21. 4. Celery-based API. Benefits High level protocol. Takes the responsibility for data serialization routing exception handling
- 22. 4. Celery-based API. Benefits Support for wide range of brokers. You can switch between brokers with no major changes in your code Brokers: RabbitMQ Redis SQLAlchemy Django MongoDB Amazon SQS CouchDB Beanstalk
- 23. 4. Celery-based API. Benefits Built-in extra stuff which you were afraid to ever dream of. asynchronous execution parallel execution asynchronous parallel execution delayed execution throttling automatic retrying of failed tasks limiting the time of execution autoscaling API inspection public key cryptography (message signing)
- 24. 4. Celery-based API. Pitfalls Python-centric. Celery is written in Python and meant to be used with Python code exclusively
- 25. 4. Celery-based API. Pitfalls Increasing complexity. Celery codebase 46k LOC Kombu codebase 19k LOC Compare Flask codebase 10k LOC Werkzeug codebase 28k LOC Jinja2 codebase 16k LOC
- 26. 4. Celery-based API. Security note Never ever expose Celery broker to the Web!
- 27. 4. Celery-based API. Security note 1. Try >>> redis_url = 'redis://:[email protected]' >>> celery = Celery(broker=redis_url, result=redis_url) 2. Get the contents of /etc/pycon_secret 3. Exchange it for a secret prize ;)