Message Queue
Basics, its types and use cases
Last updated
Was this helpful?
Basics, its types and use cases
Last updated
Was this helpful?
A message queue is an asynchronous service-to-service communication used in serverless and microservices architectures.
Putting the same thing in simple words, it is a process that puts a message onto a message queue and does not require an immediate response to continuing processing
Message queuing allows applications to communicate by sending messages to each other
The message queue provides temporary message storage, the purpose of it can be:
queue the tasks when the processor is busy or not connected
queue the tasks so that they can be processed in parallel (and/or distributed) - (example celery)
on-the-fly and real-time processing of data as it arrives (example Kafka as message queue)
Instead of building one large application, it is beneficial to decouple different parts of the application and communicate between them asynchronously using message queues
is very much in use now, few benefits of decomposing an application into different smaller services are:
Modularity: This makes the application easier to understand, develop, test, and become more resilient to architecture erosion
Scalability: Since microservices are implemented and deployed independently of each other, i.e. they run within independent processes, they can be monitored and scaled independently.
Integration of heterogeneous and legacy systems: The process for Software modernization of legacy applications is done using an incremental approach through microservice architecture
Distributed development: it parallelizes development by enabling small autonomous teams to develop, deploy and scale their respective services independently. Microservice-based architectures facilitate continuous integration, continuous delivery and deployment
Different types of message queues are in place for different purposes. I have encountered two types of message queues so far, Kafka is one type and Celery (using Redis, RabbitMQ, or AWS SQS) is another type
We can classify based on three properties:
Message replay is allowed
We can prioritize a particular task
Routing is possible; direct or regular expression-based routing allows messages to reach specific queues without additional code
When to use which service:
Kafka is used more for streaming that is "on-the-fly and real-time processing of data as it arrives", message replay is possible in Kafka that is message is not removed once it’s consumed. But, Kafka lack the ability to prioritize tasks or routing the incoming messages
Celery with RabbitMQ, Redis, or AWS SQS does not have the option to replay the messages, once the message is acked, it’s removed from the queue. However, it has ability to prioritize the message queues or route the incoming message
Therefore, two message queues have different use cases, and in many cases, two queues compliment each other to add extra capabilities to the system when used efficiently