Construct an event-driven architecture for real-time operations
Event-driven architecture is one tool to tap into the potential of real-time analytics. Organizations should understand the structure and challenges of EDA to get started.
Business needs for both real-time and widespread access to data are growing. This leaves enterprise technology teams under pressure to deliver.
Enterprise technology teams must examine their existing IT infrastructure to identify where it supports real-time access to data, as well as data democratization. If or when they find that it doesn't support those two things, IT teams might implement event-driven architecture (EDA).
"There is a push toward real-time data, not just near-real-time, and that's where event-driven architecture comes in," said Scott Sugar, practice lead for data and analytics at tech advisory and services company ProServeIT.
EDA is a framework that enables a specific way to move data about events between systems. With EDA, the events trigger business processes and how they flow, which can yield business benefits.
"EDA delivers some critical aspects of digital business: real-time intelligence and responsiveness, extensibility of applications, global scale, and 'lossless' business activity analysis," reported tech research firm Gartner in its Maturity Model for Event Driven Architecture.
Business and IT leaders seem to concur with Gartner's conclusions. Of 840 businesses and IT people surveyed, 85% said they recognized the critical business value of adopting EDA, in a 2021 survey from research firm Coleman Parkes and event management software company Solace.
What is EDA?
EDA is built around events. An event is something that happened and generated a notification of some kind. Examples include a dropped network connection detected by monitoring software, or an open warehouse door detected by an IoT sensor. The event is known as a change in state.
"An event is anything I can detect within an IT system, with hardware or software, that can be captured and represented as a piece of data," said Gary Olliffe, an analyst at Gartner.
Events typically trigger other events as part of a business process. For example, a dropped network connection could trigger another system to troubleshoot and then reset the network. The system should only reset the network if prompted by a failure event.
EDA recognizes event producers, consumers and brokers, which are sometimes called event routers or buses. In EDA, a producer publishes an event to the broker, which then pushes that information to any and all event consumers programmed to receive information about that particular event. This design essentially decouples the event producer from the event consumer.
EDA has been in limited use for many years because most enterprise IT environments weren't designed for the event-driven paradigm nor did organizations have the demand for and ability to use data seen today. Today, organizations have large portions of their IT environment in the cloud. They've broken down old monolithic application architectures and created microservices and implemented serverless computing.
"EDA allows coordination between those microservices," said Dominic Duggan, a Stevens Institute of Technology associate professor who teaches EDA as part of a course on cloud computing.
EDA technologies
Many open source and commercial technologies can create EDAs.
IT teams can use products from cloud providers -- Microsoft's Azure Service Bus and Event Grid, AWS' Simple Queue Service (SQS) and EventBridge, and Google's Pub/Sub are options. There are also open source choices in the EDA space including RabbitMQ, Apache Kafka and NATS.
Event-driven APIs are also key components to construct an EDA. Event-driven APIs are also sometimes called event-based, streaming or real-time APIs. They're in contrast to synchronous REST APIs -- which are sometimes called RESTful APIs -- for use in a representational state transfer architecture. Protocols for use with event-driven APIs include WebSockets and Webhooks.
Benefits and challenges
EDA brings a variety of benefits. One of the top benefits is the loose coupling of systems, Duggan said, noting it creates resiliency, flexibility and agility.
"If your infrastructure isn't tightly coupled together, and something fails, it doesn't have a cascading effect," he said. "Coupling it together too tightly will make it too fragile."
The loose coupling enables systems to scale and update independently. This is how EDA delivers flexibility and agility.
EDA's on-demand nature means organizations don't pay for continuous polling to check for events. Without needlessly using up network bandwidth and other compute resources, the EDA design might save on IT operational costs.
Organizations can store events, so they can analyze accumulated data to look for patterns that point to trends and other information to use for decision-making, Olliffe said.
EDA also lets processing happen asynchronously, so the processing time is driven by business needs rather than technology constraints, he said. This architecture improves speed and scalability by distributing load.
[Some] think it's a universal solution to all the interactions that take place in their systems. EDA is just a set of patterns in an engineer's or architect's toolbox.
Gary OlliffeAnalyst, Gartner
Despite such benefits, experts stress that EDA is not a paradigm meant for universal use or to replace other options.
"[Some] think it's a universal solution to all the interactions that take place in their systems. EDA is just a set of patterns in an engineer's or architect's toolbox," Olliffe said, noting that not all systems within an enterprise IT environment use events to communicate.
Products from EDA software vendors abstract much of the work of implementing EDA. Enterprise IT teams don't have to build and secure all the components. However, it isn't just the flip of a switch.
"It's still not plug-and-play. There is still some management and maintenance to it," Sugar said. "If there isn't a need for real-time data, event-driven architecture in most cases will create more complexity than required."
With EDA, the event data that systems, such as various event consumers, use across the enterprise won't be universally updated all at the same time.
"It's not always consistent, although it's eventually consistent," Olliffe said. This is the concept in EDA, distributed databases and other IT systems known as eventual consistency.
In an eventual consistency setup, IT must work with the business to understand the level of tolerance the business process has for data that does not all match at the same time, he said. If a particular business process can't tolerate this consistency issue, EDA is not the way to go.
