IoT data analysis may require hybrid approach

Where should companies analyze their IoT data, and what is AnalyticOps? The Data Mill reports.

One of the questions CIOs and businesses will have to wrestle with when building out an internet of things project is this: Where should the analysis of data happen -- at the edge or in the cloud?

Analyzing data at the edge gives a company speed. Rather than send every ping or a batch of pings to the cloud, data points are collected and analyzed on the device. But not all internet of things (IoT) use cases require the kind of speed provided by edge analytics.

"These are challenges customers have to think about: What sort of data analytics do I need? Where do the analytics run? ... What is the time latency associated with this? Do I need a response every millisecond as soon as the event occurs? Or is it OK if I get to know about an issue once a day?" said Adarsh Narasimhamurthy, senior engineer at MathWorks, a mathematical computing software company in Natick, Mass., during a panel discussion at the recent Predictive Applications and APIs event in Cambridge, Mass.

Consider the transportation of temperature-sensitive -- and, in some cases, time-sensitive -- pharmaceuticals. Today, temperature is measured using devices that aren't connected to the internet, according to Krenar Komoni, CEO at Tive Inc., a supply chain IoT startup. Deviations or "temperature excursions" aren't known until the shipment reaches a distribution center.

Tive's IoT device, on the other hand, is a connected device, providing an almost real-time look into the shipping conditions. "When [drug manufacturers] get real-time data, they'll know when a temperature excursion occurred," Komoni said. "And they can reorder the drug so that the patient can get it on time."

Tive's IoT device is small enough to fit in the palm of your hand. But because of its size and the fact that it's battery-powered, it doesn't have the capacity to do data analysis. The same goes for a connected heart monitor, said Ed Featherston, a senior enterprise architect and director at Collaborative Consulting LLC in Burlington, Mass.

"There's only so much computational power they can build in the little thing they put in my chest," he said, speaking from experience. "It's only capable of doing so much."

In both examples, the data produced by the IoT devices is simple, and performing analytics on the device itself is neither feasible nor is it necessary. But what's the data processing and analytics strategy when the device -- say, a connected car -- is both complex and massive? That's what one audience member wanted to know: Rather than either a cloud or an edge approach, is it possible for IoT data to be analyzed at the edge and in the cloud?

"Yes, definitely," said Harley Stowell, CEO and founder of Sea Street Technologies Inc. in Boston. He suggested using an architecture that prioritizes where, when and how data is processed so that mission-critical data is analyzed at the edge and less critical data gets pushed out to the cloud.

MathWork's Narasimhamurthy agreed, suggesting the decisions a connected car makes about itself should be processed locally; data associated with the global environment can be processed in the cloud. "This all ties back to resiliency," he said. "Even if the cloud is not available, the car should be able to function completely as a self-sufficient unit on its own."

Move over DevOps

DevOps is trending. But Stuart Bailey, CTO at the Open Data Group in River Forest, Ill., suggested that businesses relying on analytics also need to consider AnalyticOps, a function that sits between the data scientists and the business.

"These are the folks who have to curate the designed analytics and then feed them to the business application and make sure that it's running," Bailey said.

The AnalyticOps function maintains the analytics, which are used to find meaningful patterns in the data. The analytical models are typically built by data scientists, but aren't necessarily maintained by them. If the data changes or the model needs to be updated, the AnalyticOps function will ensure the work gets done in a timely manner, he said.

Introducing AnalyticOps to the analytics workflow "is an organizational change; it's not a technical change," Bailey said.

Say what?!?

"One of the other things I always talk about with IoT, it's also the pedigree of the data. It's coming from all over the world, it's crowdsourced, it's other devices. How do I know you're getting legitimate information? That presents a huge challenge for the enterprise, because you've never had to deal with it at that scale before." -- Ed Featherston, Collaborative Consulting

"Like computer science was in the '80s and '90s, data science is quite mature from a technical perspective, but still new in terms of its action in the marketplace." -- Stuart Bailey, partner and CTO, Open Data Group

"It's not no information, it's some information. That's what we're doing here: How do we reason from partial information? How do we get as much information out of the data we have?" -- Richard Tibbetts, CEO, visiting scientist at MIT Probablistic Computing, on BayesDB, an open source project to visually explore databases

Next Steps

Data science and the business: Friends or foes?

When hiring a data scientist, think outside of the engineering box

Six missteps that could kill your data science project

Dig Deeper on