Django Celery
18 likes12,829 views
Celery is an open source asynchronous task queue/job queue based on distributed message passing. It allows tasks to be executed concurrently, in the background across multiple servers. Common use cases include running long tasks like API calls or image processing without blocking the main process, load balancing tasks across servers, and concurrent execution of batch jobs. Celery uses message brokers like RabbitMQ to asynchronously queue and schedule tasks. Tasks are defined as Python functions which get executed by worker processes. The workflow involves defining tasks, adding tasks to the queue from views or management commands, and having workers process the tasks.
![retry / rate
from celery.task import Task
tasks.p
class UpdateStatus(Task):
name = "twitter.updatestatus"
routing_key = 'twitter.updatestatus'
ignore_result = True
default_retry_delay = 5 * 60
max_retries = 12 # 1 hour retry
rate_limit = ‘10/s’
def run(self, tweet, **kwargs):
try:
post_to_twitter(tweet)
except Exception, exc:
# If twitter crashes retry
self.retry([tweet,], kwargs, exc=exc)
from twitter.tasks import UpdateStatus views.p
UpdateStatus.delay(tweet=‘hello world’)](https://image.slidesharecdn.com/djangocelery1-100727052925-phpapp02/85/Django-Celery-19-320.jpg)
Django Celery
- 1. celery distributed task @matclayton
- 2. warning
- 3. background
- 5. what is celery? "celery is an open source asynchronous task queue/ job queue based on distributed message I. It is focused
- 6. what is celery? "celery is an open source asynchronous task queue/ job queue based on distributed message passing. It is but what does that
- 7. you can run • distributed • concurrently • in the background
- 8. use cases • external api calls (twitter) • long running tasks (transcoding) • concurrent execution (batch image resize) • load balancing (across servers)
- 9. sql django celery ORM carro AMQP SQL or STOMP
- 10. result • database • AMQP • cache • tokyo tyrant • redis • mongodb
- 11. components 1. views.py / management command 2. broker – RabbitMQ 3. workers
- 15. setting up the $ sudo apt-get install rabbitmq- server $ sudo pip install celery $ rabbitmqctl add_user myuser mypassword $ rabbitmqctl add_vhost myvhost $ rabbitmqctl set_permissions -p
- 16. setup INSTALLED_APPS += ("djcelery", ) settings.p BROKER_HOST = "localhost" BROKER_PORT = 5672 BROKER_USER = “myuser" BROKER_PASSWORD = “mypassword" BROKER_VHOST = “myvhost" CELERY_QUEUES = { "regular_tasks": { "binding_key": "task.#", }, "twitter_tasks": { "binding_key": "twitter.#", }, "feed_tasks": { "binding_key": "feed.#", }, } $ python manage.py celeryd -B
- 17. hello from celery.decorators import task tasks.p @task def add(x, y): return x + y >>> result = add.delay(4, 4) >>> result.wait() # wait 8
- 18. post to from celery.task import Task tasks.p class UpdateStatus(Task): name = "twitter.updatestatus" routing_key = 'twitter.updatestatus' ignore_result = True def run(self, tweet, **kwargs): post_to_twitter(tweet) from twitter.tasks import UpdateStatus UpdateStatus.delay(tweet=‘hello world’) views.p
- 19. retry / rate from celery.task import Task tasks.p class UpdateStatus(Task): name = "twitter.updatestatus" routing_key = 'twitter.updatestatus' ignore_result = True default_retry_delay = 5 * 60 max_retries = 12 # 1 hour retry rate_limit = ‘10/s’ def run(self, tweet, **kwargs): try: post_to_twitter(tweet) except Exception, exc: # If twitter crashes retry self.retry([tweet,], kwargs, exc=exc) from twitter.tasks import UpdateStatus views.p UpdateStatus.delay(tweet=‘hello world’)
- 20. podcast from celery.task import PeriodicTask tasks.p class FeedImportPeriodicTask(PeriodicTask): run_every = timedelta(hours=1) routing_key = 'feed.periodic_import' def run(self, **kwargs): logger = self.get_logger(**kwargs) logger.info("Running Periodic Feed Import task!") update_podcasts(silent=False)
- 21. class FeedImporter(Task): name = "feed.import" routing_key = 'feed.import' tasks.p ignore_result = True default_retry_delay = 5 * 60 # retry in 5 minutes max_retries = 72 # 6 Hours to cover major outages def run(self, podcast_id, **kwargs): try: logger = self.get_logger(**kwargs) # The cache key consists of the task name and the MD5 digest of the feed id. lock_id = "%s-lock-%s" % (self.name, podcast_id) is_locked = lambda: str(cache.get(lock_id)) == "true" acquire_lock = lambda: cache.set(lock_id, "true", 300) # memcache delete is very slow, so we'd rather set a false value # with a very low expiry time. release_lock = lambda: cache.set(lock_id, "nil", 1) logger.debug("Trying to import feed: %s" % podcast_id) if is_locked(): logger.debug("Feed %s is already being imported by another worker" % podcast_id) return acquire_lock() try: import_feed(logger, podcast_id) finally: release_lock() except Exception, exc: logger.error(exc)
- 22. typical • running out of disk space == rabbitmq fail • queue priorities, difficult • non-pickle-able errors • crashing consumers
- 23. other cool • tasksets / callbacks • remote control tasks • abortable tasks • eta – run tasks at a set time • HttpDispatchTask • expiring tasks • celerymon • celeryev • ajax views
- 24. finding • http://github.com/ask/celery • http://github.com/ask/django- celery • irc.freenode.net #celery (asksol owner, always helpful and about)