IoT data: Where is it stored?
In my previous column I mentioned that IoT is something that has been a part of our lives for quite some time, without realizing it. I talked about the example of the electricity meter technician that came to my house to measure my usage. Through his device he could see my previous meter reading and fill in the new value so that automatically the connected billing system could create my final bill.
I also mentioned that today’s IoT is also a consumer play with many (smart) devices gathering data and using that IoT data for different services. A smart fridge might alert you from a future need to go shopping (or automatically do the shopping for you online), but it can also scan promotions and might notify you that your favorite brand is running a 3 + 1 free promotion at a certain store.
I also talked about the problem that occurred with the electricity meter technician because the backend system was having an outage and needed to be fixed.
Over the years, as employees, we got used to outages and while this is annoying for us — and costly for the company — we learned to live with it.
I also discussed the need of having availability at the back-end of the service and a good availability solution should be in place to guarantee this.
There’s more to IoT data storage
But let’s look at it from the consumer point of view for a second here and the expectation consumers have when it comes to outages. Whenever you offer services to a consumer, you have to know that outages are not accepted and can cause more than just money loss and some annoyances. Consumers are more demanding for your services than your employees. In many cases, when you experience downtime, there will be reputation damage, lots of (bad) social media activity and in the long run you might miss potentially a lot of new or recurring customers because of it.
As I said before, you need availability at the back-end, but no matter what, you have to take outages or interruptions into play. Even a two-minute interruption could be damaging. So how can you handle this?
The answer to this question is by storing IoT data at two places. At the back-end, and at the device itself. No, you don’t need to store the entire data at that device; the cost of the hardware would become very high if you want to do that. What you need to define is a subset of data that you want to have offline or cached at the device.
Let’s look at an example: a health tracker. Health trackers will gather a large amount of data while a user is wearing it; data such as heartbeat, amount of steps taken, running/biking/swimming activity and much more. If you only store that data at the back-end, it would mean the device is uploading the data to the back-end service all the time. The moment you have a small outage, that data cannot be uploaded and your customer would have a gap in his timeline. Also, imagine that customer wants to review the average running stats before starting a new run but can’t because your service is down, he would be disappointed and start with a bad feeling on his run.
Having (limited) IoT data at the device itself will be necessary for many types of devices. The amount of data and what data you store on it should be seriously considered upfront and could make a huge difference in your hardware design process. In the example of the health tracker, you can image things such as the average of the last week, amount of steps for a limited number of days, caching of the current activity and so on.
This technique also offers a second advantage. Even when your service is online, it doesn’t mean that the consumer is online. It is a wrong assumption to believe that everyone is connected all the time to your service. As a frequent traveler, I am many hours per month not connected to the internet when I am in an airplane. As a water sports enthusiast, I am often in areas where there is no Wi-Fi, and even data connectivity (or cellphone connectivity) is limited or non-existent at times.
Conclusion
When designing your solution and when it comes to the point of IoT data location, you need to keep a few things in your mind:
- Outages can occur and can cause serious damage to your business
- Your consumers might (and will) not always be connected to your service
- Expectations of consumers are very high and want to see “data” at all times and certainly not loose data
Besides having all the data at the back-end, your strategy should also include a caching mechanism at the device and a small set of data also to review, even when not connected to your service.
All IoT Agenda network contributors are responsible for the content and accuracy of their posts. Opinions are of the writers and do not necessarily convey the thoughts of IoT Agenda.