Addressing the fundamental challenges to IoT data management

Problem 1: Integrating IoT data

Today, we’re seeing many technologies that offer analytics on enterprise IoT data — but this only solves part of the equation. Before there was IoT, companies were still generating enterprise data from log files, message queues, transactional data, etc. This data is still very important and needs to be in consideration with IoT data.

In fact, to get the most value from IoT data, companies need to integrate and correlate it with its other enterprise data, wherever it’s being stored, so that organizations have a holistic view to make critical business decisions.

The challenge in integrating IoT data with enterprise data is the sheer number of different technologies at play. Companies need to look at everything within their current architecture, such as databases, ETL and batch, data lakes, messaging systems, etc., and determine how to infuse their legacy technologies with the new wave of IoT devices. The goal is to enable all these technologies to work seamlessly together in order to fulfill business needs.

Problem 2: Managing IoT data

Another primary issue with IoT is the amount of data it’s generating and the speed at which it’s happening. A 2017 IDC white paper forecasts that by 2025, the world will generate 160 zettabytes of data, up from 16 zettabytes today. The report all notes that by 2025, about 25% of all data will be real time in nature. Of this 25%, 95% of it will be generated by IoT.

The kicker here is that based on all this data being generated, only a small percentage of it can be stored (between 3-15%, depending on the source) — there’s literally not enough physical storage on the planet to hold all of this data.

With a majority of companies dealing with IoT data management processes through batch capture, this causes a major dilemma on how to interact with data in the future.

Problem 3: Securing IoT data

Finally, it should come as no surprise that security is one of the top challenges in the IoT space. As organizations continue to utilize connected devices for enterprise use, it becomes increasingly more difficult to monitor and react to digital threats; it’s not uncommon for enterprise companies to have 100+ security technologies in place.

Recently, ForeScout put out a report finding that more than half (52%) of surveyed security leaders are experiencing a high level of anxiety as a result of IoT security. One of the most shocking findings was that, when asked who should be responsible for securing IoT, 82% of the respondents did not have a clear answer of ownership.

The combination of a convoluted security technologies market and the growing number of connected devices needing support ultimately results in inevitable gaps throughout the ecosystem. With the level of security that IoT demands, companies need a streamlined approach to monitor its data and react appropriately and effectively when there is suspicious behavior on the network.

Solution: A streaming-first architecture

The key to addressing the integration, manageability and security of IoT data (as well as other enterprise data) is through a streaming-first architecture. We’re entering a time where CPU and memory are finally more affordable for companies. Instead of capturing data, storing it and analyzing it later, we can now gain insights from our data the moment it’s born by processing and analyzing it in-memory, before it hits disk, in a streaming fashion.

Being able to continuously collect data in real time is a critical first step. Using change data capture to transform databases into data streams enables companies to integrate with legacy technologies while reaping the benefits of a modern data infrastructure.

So how does this help with our three main problems with IoT data management?

Regarding the integration of IoT data with other enterprise data, one of the main hurdles is infusing legacy technologies with modern IoT devices to work seamlessly. By enabling legacy technologies to become streaming sources, companies now have the ability to integrate and correlate all data assets, in any digital environment, without operating on siloed information, enhancing decisions to be made based on context and time, while the data is still operational.

Also, processing and analyzing data in real time can help address the data crisis that we are already starting to experience. Tools such as filtering, aggregation and correlation used on data while in-flight can help determine what is the most relevant to store and what can be discarded, alleviating the need to store every piece of generated data. Batch capture was a result of our technological limitations — dealing with data in real time is the evolutionary next step in data management.

For IoT security, a streaming-first architecture can enable you to analyze multiple endpoint security systems’ logs and different infrastructure components’ logs in real time to identify issues or breaches that are not obvious from a single security technology. Where individual security technologies look for certain exploits, using tools such as anomaly detection and pattern matching can evaluate all streams to find irregularities and possible malicious intrusions.

Towards a streaming future

Establishing that IoT data needs to become part of your overall business ecosystem, a modern data architecture enables companies to expand its digital transformation efforts by:

• Having the flexibility and scalability to connect everything from legacy historians and devices, through modern sensors, to innovations such as blockchain and AI
• To work seamlessly in multiple digital environments — on-premises, at the edge and in the cloud
• Smartly and effectively handling the huge volumes of IoT data, storing only that which is necessary while still being able to respond immediately
• Continually integrating and correlating IoT and other enterprise data to monitor and proactively respond to cybersecurity threats

To learn more about some of these advantages, you can watch my entire Global IoT DevFest presentation by clicking here.

All IoT Agenda network contributors are responsible for the content and accuracy of their posts. Opinions are of the writers and do not necessarily convey the thoughts of IoT Agenda.