Celery

Celery, What is it?

• • Message in the queue maintains correct order as produced

  • The message is persisted even during downtime unless consumed, hence no tasks are missed even when there is downtime

  • The system is distributed, able to scale up or down based on requirements, and has no single point of failure (SPOF)

• For the message (task) queue, the following database can be used: Redis, AWS SQS, or RabbitMq

Keywords to know

• Producer: service that emits the message is called producer worker

• Consumer: the service that listens and reacts to messages is called the consumer worker. This is how communication happens

Additional functionalities

• task:

• queue:

• delay vs async:

• specifying priorities

• specifying number of cores

Flower

• Celery comes with in-build visualisation functionality named flower

• Flower UI is easy for monitoring on tasks, queues and processing of different queued tasks

Implementation

Example to explain use case:

• used for async tasks, the best way to explain it is through a web scraper

• If one wants to scrap 1000 pages and need sleep time of 2 min for each page.

  • The synchronous task takes min time of 2000 min to process all pages

  • But the async operation, 64 processes at a time can bring down the execution time to (1000/64)*2 minutes to process

Reference for Further Reading

Python :Celery timing task is fine