> For the complete documentation index, see [llms.txt](https://ankit-apdc.gitbook.io/system-design/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ankit-apdc.gitbook.io/system-design/building-blocks/message-queue/celery.md). # Celery ## Celery, What is it? * * Message in the queue maintains correct order as produced * The message is persisted even during downtime unless consumed, hence no tasks are missed even when there is downtime * The system is distributed, able to scale up or down based on requirements, and has no single point of failure (SPOF) * For the message (task) queue, the following database can be used: Redis, AWS SQS, or RabbitMq ## Keywords to know * Producer: service that emits the message is called producer worker * Consumer: the service that listens and reacts to messages is called the consumer worker. This is how communication happens **Additional functionalities** * task: * queue: * delay vs async: * specifying priorities * specifying number of cores ## Flower * Celery comes with in-build visualisation functionality named flower * Flower UI is easy for monitoring on tasks, queues and processing of different queued tasks ## Implementation ### Example to explain use case: * used for async tasks, the best way to explain it is through a web scraper * If one wants to scrap 1000 pages and need sleep time of 2 min for each page. * The synchronous task takes min time of 2000 min to process all pages * But the async operation, 64 processes at a time can bring down the execution time to (1000/64)\*2 minutes to process ## Reference for Further Reading {% embed url="" %}