What should we make of ESB architecture in a distributed world?

When I started to follow software development and application architecture in 2015, vendors, analysts and experts had all proclaimed the death of the ESB architecture. Everyone favored distributed, flexible microservices, and they were doomed if they didn't break up their monoliths.

The world these so-called experts prophesized was one of self-contained and independent services. No longer would enterprise software shops need to maintain bulky, monolithic data stores, since each service will carry all the data, logic and processing orders it needs in its own backpack.

But the need for a central system never truly went away. Anyone who thinks they can operate without some sort of single source of truth -- at least as a failover support system -- seems to be as misguided as the vendors who push the fanciful promises of a totally dynamic and distributed software architecture.

Consider what Kim Clark, an integration and process architect at IBM, says in his 2018 article on the fate of the ESB:

"If you look back to somewhere just before the year 2000, integration was almost exclusively asynchronous, using files and messages to integrate between systems of record. Writing code in or around each system for each and every point-to-point integration was expensive and resulted in a complex web of interactions. The inevitable result was to introduce an integration hub that sat between all systems."

There is a fair point to be made: asynchronous communication is more sophisticated today than it was in 1999. Docker containers, for one, provide not just applications but distilled application services to run on their own, completely self-contained from the system. And there's no question that the shift to total cloud application domination makes asynchronous communication more of a necessity than an option.

But even according to the "Ten Commandments of Microservices" written by architect and tech analyst Janakiram MSV, microservices require some sort of central service registry (which can admittedly be cloud based), a federated way to check service rules and constraints, and some standardized method of logging and alert messaging.

That's the catch. Can an enterprise ecosystem really run cohesively without some sort of single, powerful agent keeping these extremely important tasks in line? And once you add any legacy applications into the mix, it really starts to seem like an impossible goal.

A pure message broker, such as Apache Kafka, might be able to use event-driven messaging to do what an ESB does without the heavy load. But maybe the goal shouldn't be to throw the ESB away, but rather adapt the ESB architecture for today's needs.

As Ken Tien, who wrote a blog for Mulesoft in 2016 on microservices vs. ESBs, puts it:

"As for ESBs, no; they are still alive. We must adapt our concept of this architecture and can no longer understand the Integration Bus as a centralized and inflexible concrete structure for the whole company."

It is arguable that the ESB architectural pattern is more important than ever. It's simply a pattern that needs to find a better place in a new world -- which in many ways, it already has. Keep in mind, what functions do an iPaaS serve that really vary so differently from the ESBs?

There are plenty of critical problems unforeseen by creators of the ESBs, such as issues that involve security, service availability and protocol compatibility. But why throw away the fundamental ESB architecture? Think about the ESB as not just something that keeps services under lock and key, but rather, an arbiter that provides the support that services need to run as independently as possible.