Understand IoT data storage options for data deluge
Cloud is the favored storage choice for many IoT uses, but vendors have stepped up their offerings to meet the growing variety of needs, including custom-built options.
As organizations draw more data from the rapidly expanding number of connected devices in the world, moving data from endpoints to the cloud won't always make sense.
Nearly 42 billion devices will be connected to the internet by 2025 as part of the ever-expanding IoT ecosystem, according to a forecast from research firm IDC. Those devices will generate 79.4 zettabytes of data, a staggering volume that organizations will need to capture, store and process, according to the report "Worldwide Global DataSphere IoT Device and Data Forecast, 2019-2023."
In most use cases, moving data from endpoint devices to the cloud is the obvious choice. Cloud usually offers the elasticity required to match organizational demand changes and the managed infrastructure that IT relies on. Most organizations can use cloud to store the data generated by IoT devices and to run the applications needed to process device-generated data.
"If the storage amounts are low and all you're doing is collecting and storing data, and it's not a real-time analytical application, it's perfectly fine to store the data in the cloud," said Massimo Russo, a managing director and senior partner at Boston Consulting Group. "But the answer is not always the cloud, because it can be expensive or have latency issues for some use cases."
Organizations that have endpoints generating tremendous volumes of data often find it's too expensive to send all of it to the cloud. That's particularly true if they don't need to analyze all data to reach their objectives. It's even more true if those organizations are keeping all data just in case they will need it later on.
Understand alternatives to the cloud
Alternatives exist for organizations that can't move all or even some of the data generated from their endpoint devices to the cloud.
A newer alternative more specific to IoT is the use of edge gateways for IoT data storage.
One alternative is the endpoint device itself. Devices generally have some capacity to store and process data, though it varies by class of device and by vendor. Devices' storage capacity is low and the compute power can handle only lightweight applications, said Mahmoud Daneshmand, a professor of business intelligence and analytics at the Stevens Institute of Technology and co-founder the IEEE Internet of Things Journal. For example, a connected thermometer that takes frequent readings might have the capacity to analyze those readings for temperatures outside a predetermined acceptable range, but the device itself couldn't handle more complex analytics.
Another option is the use of on-premises servers located near the endpoint devices. Although conventional on-premises servers can reduce latency and address security and privacy concerns by moving data off-site, conventional servers still come with the same drawbacks -- such as high capital costs and lack of elasticity -- that pushed IT to adopt cloud for organizational uses.
A newer alternative more specific to IoT is the use of edge gateways for IoT data storage. Edge gateways sit between IoT devices in the field and the organization's central data center, whether that is a public or private cloud. Gateway devices vary in power and capacity. They handle storage and computing needs closer to the device before moving selected data.
Established vendors, such as Amazon, Hewlett Packard Enterprise and Microsoft, have gateway device offerings. Other vendors and their products include the Cisco IC3000 Industrial Compute Gateway and Dell Technologies' Edge Gateway for IoT. Newer entries include a platform from FogHorn Systems, which offers on-site data processing and real-time analytics, as well as machine learning and AI capabilities.
Even here, IoT leaders need to consider the volume of data they're processing, how much they need to analyze and store at the edge, and how much each option will cost when building out the architecture to support their IoT ambitions.
"They have to decide [which data] to keep on the edge and which to move to the corporate data center. Typically, they don't want to keep all the data there on the edge, and they don't want huge capacity out on the edge because it can make it very expensive," said James Staten, vice president and principal analyst at Forrester Research.
Micro data centers offer custom-built options
IT can also opt to use micro data centers for cases when moving data from endpoint devices to a conventional cloud service isn't an option because of issues such as distance-related latency. Micro data centers could be custom built to handle endpoint data or conventional colocation data centers that are geographically close to the endpoint devices and offer a platform to handle the data locally.
"Leading vendors in the colocation business have been positioning themselves as an option," Staten said. "And the [telecommunication companies] are trying to position their data centers as micro data centers, where you don't have to take up a whole building or sign a five-year contract. You can use these centers as you need it for as long as you need it. And you don't have to set it up; they'll provide the services for you."
IT leaders will likely have even more options in the near future, said Abhijit Sunil, a Forrester analyst serving infrastructure and operations professionals.
Sunil said he sees the increasing number of IoT deployments and the varying storage, computing and networking needs driving demand for custom form factors. That is something vendors are now delivering. In a recent post, Sunil cited the startup Pensando Systems as an example, which is developing a custom programmable processor optimized for edge computing.
