Kit Wai Chan - Fotolia
5 components of IoT streaming architecture
Software architects must create the basic IoT streaming architecture model for event flows before they can add unique features to their IoT applications.
Designing IoT applications is akin to building a house: Organizations need a strong foundation to support application variations. For IoT, that foundation is streaming architecture.
Software architects must follow a blueprint to ensure they don't lose track of the overall flow when considering components along the IoT data path. IoT streaming applications process data in real time to draw insights for uses such as log analysis, process control and material handling applications, where event handling is critical to guarantee efficient plant operations. The applications are characterized by an uncontrolled message flow -- a stream of events -- which describes most of IoT.
To face the challenges that IoT streaming architecture poses, software architects must first understand what IoT data streaming is: It's the event processing flow in which a set of sources generate events. The sources are sensitive to processing delays, and they are technologically and geographically diverse. Five components make up the IoT streaming architecture.
1. Event receiver
IoT streaming architecture begins with the event receiver where IoT sensors send events. The event receiver takes various data formats in a stream and converts them to a standard structure that a single layer of components can process. If an event doesn't contain its own time stamp, the event receiver stamps the event.
The event receiver is a layer, rather than a single component. It accommodates new devices and old ones with obsolete data structures that need support for widespread use. IoT streaming architecture often has event receivers for each collection of sources, and they're customized to support the event formats of the source community they serve.
2. Local controller
Sometimes software architects combine the event controller with a local controller. Many streaming IoT events require closed control loops between an event source and an IoT device that controls a real-world process, such as turning on lights or opening gates. A local controller can relax latency constraints on real-time operations later in the stream process.
3. Event classifier and serializer
The event classifier and serializer classify events in as much detail as necessary to assign them processing priorities and move them along the correct processing path. The classification added to each event's data structure based on standardized event data models would follow the event through the remainder of the flow.
The serialization piece handles the collection of multiple event streams into a single contextual stream for processing. Serialization is essential when events come from multiple local domains but must correlate across domains for proper processing. If the application doesn't need cross-domain processing, then it may not need serialization. For example, an application doesn't use serialization if each local domain creates only transactional records for historical analysis.
Some IoT deployments will log or queue the results of this step for use in later steps. When data requires further real-time processing, this step initiates a process workflow based on the classification and proceeds to the next step in IoT streaming architecture.
4. Event correlation and transaction
The event correlation and transaction step converts event signals into actionable messages, described as an extract, transform and load process. Messages have three destinations:
- They go to another process workflow as a stream.
- They can be formatted as transactions and entered into a transaction processing legacy application the organization already uses.
- They can be stored in a database for later analysis and processing.
A single correlated event could go to any or all destinations. Organizations may require real-time analytics with an emphasis on performance, such as NoSQL.
5. Event post-process and analytics
The event post-process and analytics step supports any applications that don't require real-time processing and analytics. Some IoT applications don't involve any extension of the control loop beyond responding locally to local events, such as opening a gate. Other applications require at least a record of the event. Some require post-processing of the event and actions related to it. These steps and analytics tools extend traditional IT applications and databases, rather than event processing.
