How pub/sub messaging works, and why it matters today
While pub/sub has been a standard messaging approach for decades, it's arguably more important than ever for developers to understand what it is, how it works and why it matters.
Thanks to the rise of things like distributed computing, real-time data processing, event-driven architecture and modular development, the way organizations manage communication between software components is more important than ever. Unfortunately, these advancements have simultaneously made communication more complex to manage than ever.
As such, an increasing number of development teams are forced to leave the comfort of dependable, one-to-one exchanges in favor of more dynamic approaches to messaging.
The publisher/subscriber (pub/sub) messaging pattern is hardly a new concept in software architecture. But it remains relevant in the world of cloud-native development, where complex software component relationships present significant communication challenges. Today, pub/sub enables organizations to build robust applications and systems in the cloud that dependably stay connected, regardless of how distributed the individual services are.
Let's look at how pub/sub works, the kind of problems it can pose and the places development teams can look for tooling that supports pub/sub implementations.
What is pub/sub messaging?
Pub/sub messaging is nothing new. For instance, those who have worked with legacy paradigms such as service-oriented architecture are likely familiar with the pub/sub pattern through the lens of the enterprise service bus and message-oriented middleware. While the methodology has evolved to meet demands for software approaches like microservices and event-driven architecture, the basic idea remains the same.
In the pub/sub pattern, the software-based entities that send and receive messages are referred to as "publishers" and "subscribers," respectively. Publishers send messages to a queue -- or other messaging broker -- where those messages wait for processing. To route them correctly, the broker assigns each message to particular "topics" that logically segment the related operations. The services and components subscribed to that topic receive an alert that the message is in the queue and ready for processing.
With this approach, the publishing services are merely responsible for placing the message in the queue, and do not have to wait for a response before moving on to the next task. Likewise, subscribing components can access messages at will, since they are no longer on the hook to immediately send a response. This decoupling enables asynchronous communication, which builds the foundation for complex applications that are both distributed and scalable.
Limitations of pub/sub
While pub/sub is a perfectly viable pattern for many large-scale enterprise application communication needs, there are some cases where its weaknesses are conspicuous and prohibitive.
Pub/sub is one part of system architecture that will not be outdated any time soon.
One area where pub/sub struggles is when messages must follow a "first-in, first-out" (FIFO) order. The pub/sub pattern isn't ideal for FIFO operations, as messages are strictly processed in the order that they're received. Some of the major cloud platforms (Google Cloud, for instance) do offer pub/sub message ordering as a configurable feature. However, the inherent decoupling means that FIFO order is never a guarantee.
The pattern also poses some problems when it comes to message-related error handling. With pub/sub, there will inevitably be times that messages in the queue never get successfully processed. This can happen for any number of reasons, ranging from invalid message data to unavailable services. At some point, those failed messages need to be dealt with, which might require development teams to perform manual code reviews. While this might be preferable to losing unprocessed messages entirely, it illustrates the point that sometimes pub/sub can be a little more work to maintain than traditional, one-to-one messaging.
Pub/sub tools and services
Just as service-to-service communication has become a bigger priority for many businesses, pub/sub messaging is taking center stage in conversations around enterprise application architecture. To that end, there is a growing multitude of both independent and proprietary tools that make it easy to get started and implement pub/sub at scale. However, most of these capabilities are wrapped within larger collections of tools, rather than standalone pub/sub offerings.
Organizations can find pub/sub services from any major, proprietary cloud platforms they use. These include things like Amazon Simple Notification Service, Google Cloud Pub/Sub and Azure Web PubSub. This route makes the most sense for enterprises that are already invested in these major cloud platforms, not only from a cost perspective, but from a compatibility and ease-of-use perspective, as well.
An example of pub/sub messaging using Amazon Simple Notification Service (SNS).
Of course, developers are not beholden to proprietary cloud vendors when it comes to pub/sub support. Many open source development toolkits such as Redis, Ably Realtime, Dapr and Apache Pulsar offer capabilities that support pub/sub messaging. The no-cost entry point of these open source tools makes it fairly simple to experiment with them to see if they meet your needs. This is still true for enterprises already locked into a major cloud provider, as many of these tools integrate readily with those platforms and are readily available through places like GitHub and the AWS Marketplace.
At any rate, pub/sub is one part of system architecture that will not be outdated anytime soon. In today's containerized world, service-to-service communication is of utmost importance. Luckily, the growing multitude of pub/sub services is only making it easier to start implementing pub/sub at any scale.
