The advent of IoT radically changed the enterprise data lifecycle, and the implementation of AI on the edge is changing it again to meet real-time analysis needs.
IT pros have little room for guesswork in IoT infrastructure planning. An organization's digital architecture can shift dramatically in less than two years. Behind that shift are two huge game changers in the gathering and handling of data: IoT and AI.
The IoT data lifecycle, as a topic of planning, design and management, doesn't get nearly the discussion it warrants in most enterprise IT departments or the C-suite. And that's unfortunate because the flow of data in and out of the enterprise may be the single biggest driver of institutional change.
Traditional data lifecycle varied less
Before there were clouds, the enterprise data lifecycle was simple and essentially invariable and circular. IT pros could easily implement, master and maintain the data lifecycle. Data was more structured, less varied and traveled on only a handful of channels to a few destinations. Each step in the traditional data lifecycle is conceptually straightforward:
Plan. Determine the data required to support existing business processes. Use the most current data available for operational planning and process support.
Acquire. Input data into application systems via data entry and integration with external systems.
Process. Validate through authentication and error-check the data, enriching it as needed.
Analyze. Apply the data to the appropriate business processes and study it for implications and insights that support decisions.
Integrate. Merge the results of data analysis into decision-making processes and subsequent analyses.
Store. Store the data in transactional systems for access by ongoing applications and processes and long-term archives.
The simple, orderly scheme had already been groaning under the weight of the internet before the advent of the cloud. With clouds came new possibilities for connecting the IT core of the enterprise to a geometric explosion of new devices, and manufacturers wasted no time in taking advantage of that opportunity.
Edge devices drive IoT data lifecycle change
The global IoT device population will pass 20 billion this year. That doesn't just stress the old data lifecycle, it shatters old-school architecture like an eggshell. Even the easy connectivity of clouds doesn't begin to open enough channels for the data generated by IoT. The massive increase in traffic calls for new architecture.
The IoT data lifecycle splits the problem into three manageable pieces: the edge, where IoT gathers data; the enterprise front end -- also the cloud -- where corporate processes and communications happen, including traditional enterprise data architecture minus long-term storage; and the enterprise back end, where data is stored long term and will be mined and analyzed.
The steps in the traditional data lifecycle are now distributed and sometimes repeated among the edge, front end and back end.
The data lifecycle starts at the edge, the local-proximity network containing IoT devices generally beyond the company firewall. Data acquisition now includes a gateway layer of field servers dedicated to supporting the IoT network and providing secure access to the enterprise cloud, as well as the many modalities used by the IoT device population. Edge nodes aggregate data from IoT devices and the gateway layer for preprocessing. The nodes prepare and convert the data for transport to cloud systems. The nodes then route the data to the cloud and manage IoT network traffic and security.
In the cloud, the inbound data undergoes processing where it is validated and enriched. The data is then analyzed with an emphasis on edge application performance evaluation. The resulting edge analytics insights are integrated into business processes and further evaluated to determine if the data supports any process changes. Any data needed for application control on the edge is disseminated.
In the back end, data may undergo additional analysis and data mining or continue its journey to storage or long-term archives.
IT pros find the IoT data lifecycle at the edge tougher to accommodate. Managing infrastructure is more challenging and security more difficult. Updating and patching a wide range of IoT devices becomes a science itself. With all these pieces in place, this lifecycle is manageable.
And then along comes AI.
AI shapes the IoT data lifecycle of tomorrow
The lure of deploying AI on the edge is not just irresistible, it's a competitive necessity. Smart buildings, augmented reality and real-time facial recognition systems are winding their way into the cost of doing business.
With AI, there is a looming architectural problem. IoT networks live on the edge, and applications increasingly require real-time AI-based decisions. But the compute resources that support AI typically live in the cloud, where they serve automation and predictive processes. The latency caused by IoT application packets making multiple round trips to the cloud for real-time analysis is a deal breaker.
The deployment of AI on the edge offers some relief. For example, the presence of edge nodes -- processing resources near IoT networks -- can serve the dual purpose of running the models that provide the intelligence. Edge nodes can support more complex routing and retain some IoT data for AI support in the edge network while pushing whatever data the cloud systems need when convenient.
The IoT data lifecycle with AI at the edge closely follows the current data lifecycle with the edge but adds data processing steps.
Devices at the edge acquire data, then perform a preprocess step in which each node aggregates data with the addition of sophisticated filtering and real-time analytics. AI processing at the edge helps include decision support for retention or distribution of specific IoT data before it is routed to the next stage. IoT applications can monitor and adjust their own performance based on the data they produce. Performance data is then integrated and can be merged into local analytics and machine learning without having to traverse a network, all before data is sent to the cloud.
With each evolutionary surge, the IoT data lifecycle becomes something new that requires additional infrastructure and upgraded methodologies.
Preprocessing at the edge applies AI to real-time data on local devices, enabling applications to react more quickly to ingested data without the need to send that data over the network. However, devices at the edge may still need to send some data to the cloud to be processed and analyzed further. Cloud systems can merge inbound IoT data into existing models to refine performance, then return application control data to the edge where updates are applied.
Finally, the back end stores data, where further mining and analysis can occur.
In this IoT data lifecycle model, both edge and cloud infrastructure scale as needed. Organizations minimize bottlenecks through dynamic routing. They distribute and appropriately platform machine learning resources for fluctuating demand. Both the edge and cloud process data and apply analytics for continuous improvement.
What's tomorrow's enterprise technology stack?
With each evolutionary surge, the IoT data lifecycle becomes something new that requires additional infrastructure and upgraded methodologies. AI becomes increasingly mission-critical and edge-centric, which means a highly flexible and extensible technology stack in the enterprise edge network becomes essential. The stack must have an emphasis on resources for real-time AI-based support for tools, such as facial recognition systems that must function in real time, yet perpetually learn. Real-time decisions pose a significant expense, but organizations should plan for it.
Organizations will need to push more data back to the edge from the cloud in support of AI. Real-time response in intelligent IoT systems will require considerable modeling and simulation before deployment. As the edge expands with more IoT devices and data, the resources and IT infrastructure needed to support the IoT landscape must also grow.
While all of this is happening, automation will proliferate to accommodate the rapid response required by the new intelligent ecosphere, removing human bottlenecks in processes wherever possible. Software and infrastructure to support that automation will be essential or the efforts to strengthen functionality on the edge will largely be in vain. Each of these challenges requires unprecedented innovation and creativity from everyone involved -- and that's something no IoT or AI can provide.
