Summary

Enterprise messaging and event streaming are distinct technologies that serve complementary roles in building asynchronous, loosely coupled, and scalable applications, with each excelling in different scenarios based on their unique processing, consumption, and delivery characteristics.

Abstract

The article "Comparing Enterprise Messaging and Event Streaming" elucidates the differences and complementary nature of enterprise messaging and event streaming. It emphasizes that while both technologies enable the development of asynchronous and scalable applications, they are not interchangeable. Enterprise messaging handles discrete messages independently, while event streaming focuses on aggregating and analyzing data over time to discover patterns. The consumption style also differs, with messaging systems destructively consuming messages, whereas streaming systems allow consumers to access the entire message history. Additionally, the article highlights the fine-grained subscription capabilities of messaging systems and the "at least once" delivery guarantee of streaming systems, contrasting with the "exactly once" and transactionally coordinated delivery guarantees of messaging systems. The conclusion suggests that the choice between messaging and streaming should be guided by the specific use case, with general-purpose message brokers like Apache ActiveMQ or RabbitMQ suitable for messaging, and distributed event streaming solutions like Apache Kafka or Amazon Kinesis ideal for streaming applications.

Opinions

The author posits that enterprise messaging and event streaming are often mistakenly used interchangeably, despite having distinct architectures and problem-solving approaches.
It is the author's view that the complementary nature of these technologies is evident when examining their message processing styles: messaging systems deal with discrete messages, while streaming systems are designed for continuous data flow analysis.
The article suggests that the destructive consumption of messages in messaging systems versus the non-destructive consumption in streaming systems is a significant differentiator, impacting the ability to access message history.
The author believes that the ability to subscribe to specific message types in messaging systems is advantageous for applications that require fine-grained control over message filtering.
The author concludes that the decision to use either technology should be based on the delivery guarantees required by the application, with messaging systems being more suitable for "exactly once" and transactionally coordinated scenarios, and streaming systems excelling in "at least once" delivery scenarios.

Comparing Enterprise Messaging and Event Streaming

They are complementary technologies instead of competing technologies

Enterprise messaging and event streaming are two technologies you can use to implement asynchronous, loosely coupled, and highly scalable applications. Today, many people use them interchangeably, not knowing that are differences exist.

When you look at their inner architectures to understand why they exist and what problems they are trying to solve, you’ll realise that these two are complementary technologies rather than competing technologies.

In this article, we are going to compare and contrast these technologies across several dimensions. Towards the end, you can see a summary of use cases that each technology will be ideal.

Example — A thermostat application

For a better understanding, consider the following thermostat application.

A temperature sensor publishes readings to a messaging system every 10 seconds. A thermostat application consumes these readings from the messaging system and decides whether to turn on/off a heater.

A typical message published to the messaging system looks like the following. It has the sensor ID, room where the sensor is located, timestamp, and the reading value in Celcius.

Dimension 1 — Message processing style

The messaging approach

One way is to look at each message and make decisions independently.

For example, if the reading is less than 20, the thermostat turns on the heater. It repeats the same logic the next time a message comes. Even though this is unnecessary, that is how the messaging works.

In messaging, each message is discrete. A consumer can understand each message in isolation, and they can directly act upon them

The streaming approach

We can also aggregate readings over a time window to calculate the average value of reading. Then make a decision based on that.

For example, the thermostat can calculate the average reading over the last minute and decides whether to turn on the heater. The average seems a realistic measure here.

Streaming is a good fit here because you are not making decisions by looking at discrete messages. You try to discover patterns by looking at an incoming stream of messages (an event is a proper term).

In streaming, an individual message doesn’t make much sense. You need to aggregate and perform certain operations on them to gain more insights

Messages can be aggregated over a time window or based on the count. Then perform aggregate operations(sum, average, mean, count), filtering, and transformations to uncover patterns.

Messaging and streaming employ different processing styles

Dimension 2 — Message consumption style

Message consumption is a destructive operation in messaging systems. Messages are stored in the messaging system until they are consumed. Then get deleted to make way for new messages.

In contrast, consumers never delete messages in streaming systems. The messaging system removes messages independently from the consumers, irrespective of their consumption status.

Dimension 3 — Access to message history

Consumption style leads to another difference in both technologies.

In messaging systems, messages are deleted after they are consumed. Therefore, newly joined consumers can not see the older messages. Some pub/sub systems can keep messages for an extended period. But that is not the point here.

In streaming systems, new and old consumers have access to a complete history of messages. Newly joined consumers can catch up with the older messages as they move forwards and backward to read messages.

That will add benefits from many perspectives. For example, you can debug a consumer by replaying the messages from the beginning that it had consumed. Apart from that, when a new version of a consumer deploys, it can quickly synchronise to the latest state by consuming the stream from the beginning.

In our example, a new consumer can be introduced to analyse past readings, and visualise the temperature variance. When it deploys, it will consume messages from day one to produce a dashboard.

Dimension 4 — Fine-grained subscription to messages

Messaging systems provide a fine-grained subscription to messages with subscription filters. With hierarchical topics, consumers can subscribe to messages that they are only interested in. The messaging system performs event filtering there.

Conversely, streaming consumers will receive all the events in a partition.

The consumer’s logic has to decide which events to process and which events to discard.

In our example, the thermostat in the living room can tune-in to receive temperature readings that only relate to the living room. It is made possible by subscribing to the topic temperature/rooms/living to filter out any unrelated messages.

The Living Room thermostat can choose to ignore the readings that are irrelevant to the living room by subscribing to **temperature/rooms/living** topic

Dimension 5 — Message delivery guarantees

Message delivery guarantee is another dimension to compare these two technologies. Both have been designed and built to serve different levels of delivery guarantees.

Different levels of delivery guarantees are listed below.

At least once delivery

Messages are not lost in the event of a failure, but any message may be duplicated. Applications need to be designed to ignore or handle duplicate messages.

Exactly once delivery

Messages are not lost in the event of a failure and each message can be processed individually to prevent possible duplication. Applications reliably know that each message will be processed only once, almost independently of application logic.

Transactionally coordinated delivery

An application can coordinate the processing of a message with a specific update to an external resource, such as a database. Applications can update multiple resources without fear of them ending up out of step.

Based on each other’s design criteria, we can come to following conclusion.

Messaging systems are good at handling exactly once and transactionally coordinated delivery use cases while streaming is good at handling at least once delivery scenarios.

Conclusion

Messaging and streaming are complementary technologies rather than competing technologies. They serve different purposes in the messaging industry. Deciding which technology to pick should be based on the use case. I hope you can use the above dimensions as guidelines.

The following figure provides a summary of the discussion we have had so far. When you are looking at building a messaging solution, you can consider a general-purpose message broker like Apache ActiveMQ or RabbitMQ. Apache Kafka and Amazon Kinesis are good candidates when developing a distributed, durable, and highly-scalable event streaming solution.

A visual version of this comparison can be found below.