What Are Message Queues in Event-Driven Architecture?

Message queues are a type of middleware that enables asynchronous communication between the different components of event-driven architecture (EDA). Events are occurrences or changes in the state of data or processes. These may include things like an order placed by a customer, a package received at a location, an alert sent by an IoT sensor, a workload completed by a cloud service, an update to a CRM record, or suspicious activity on a network that may indicate a data breach. Events are things that happened in the past that can trigger other things in your system. Message queues act as an intermediary between the systems that generate events (known as event producers) and the systems that receive event data and act on it (event consumers or subscribers.)

Event-driven architecture is a software design pattern that enables you to quickly detect, process, respond to, and share information about events. The components of EDA include:

• Event producers (also known as publishers). Event producers are systems or technologies designed to send messages or notifications when an event occurs. Event producers may be things like user interfaces that issue notifications about a user’s actions, or IoT devices that generate events when certain conditions occur. Databases, workflow engines, external APIs, and application monitors are also examples of event producers.
• Event routers (message queues, event brokers, event buses). Event routers receive event messages from producers and make them available to event consumers. Each type of event router functions differently and offers different benefits to event-driven architecture.
• Event consumers (also known as event subscribers or listeners). Event consumers receive event data, process messages, and act on them in some way. Actions may include things like sending an email confirmation, updating an inventory database, alerting security teams to a possible breach, triggering a complex workflow, and many other types of tasks or processes.

When a message queue receives data about an event from a producer, it stores the data in a queue that can be accessed and processed by an event consumer. Message queues promote asynchronous communication, since producers can continue to send messages to a queue without needing a response from a consumer, and consumers can access the queue and read or ingest messages on their own time frame.

Message queues are also responsible for routing, ensuring reliable message delivery to the right consumers. Typically, message queues follow a first-in, first-out (FIFO) order, in which messages are deleted once consumers retrieve them.

Alternatives to message queues include:

• Event bus. An event bus is a system that facilitates communication about events by acting as a channel to which publishers can send event messages and from which consumers can receive them. Event buses are typically used for publish/subscribe (pub/sub) messaging patterns, in which information is sent immediately to consumers after it’s received from producers, and messages are typically deleted after they are consumed.
• Event stream. An event stream sends a continuous flow of event messages, and messages are retained for a specific time. This allows consumers to access messages retroactively. Event streams are also used in applications that process live data feeds, such as in financial transactions, social media feeds, or IoT systems.
• Event broker. Event brokers are designed to handle more complex tasks such as filtering messages, or translating or transforming messages from one protocol to another. Event brokers are also typically used for complex event processing, in which consumers access and simultaneously analyze messages from multiple streams.

Some well-known providers and frameworks for message queueing and message services include:

• Apache Kafka, a distributed streaming platform for real-time data streams.
• RabbitMQ, a popular message broker that supports multiple protocols, including Advanced Message Queuing Protocol (AMQP).
• AWS Simple Queue Service (SQS), a fully managed service offered by Amazon Web Services.
• Apache ActiveMQ, an open-source message broker that supports multiple protocols.
• Redis, an open-source in-memory data structure that can be used as a message queue.
• Apache Pulsar, an open-source distributed pub/sub messaging system that offers both message queue and event streaming capabilities.

• Message queues are an important part of asynchronous communication in event-driven architecture, in which various components and distributed systems can generate, communicate, receive, and process tasks independently of other components. This enables greater responsiveness and parallel processing.
• Message queues also promote greater resilience. When a component fails or becomes unavailable, messages can still be stored in a queue until the component is back online and can retrieve messages again.
• Message queues may help with load balancing in a system by distributing event messages across multiple instances of a component to prevent bottlenecks.