The infinite struggle of testing IoT
Last year, Nintendo brought out a new console. My sons and I are fond of gaming, so we had to get this new console — the Switch. If I am not writing blogs, articles, books or doing any other work, I will be gaming.
The great thing about consoles nowadays is that they are crammed full of sensors and actors that enhance the gaming experience. In-game effects are enhanced and gameplay is expanded into the physical realm. Compared with the old consoles I started on in the ’80s, we now have infinitely better controllers; the controllers of yore are nothing compared with the things we now hold in our hands. The controllers of today are almost consoles themselves!
From infinite gaming to infinite IoT
Physical controllers are one thing. But it is now possible to control games with voice recognition, gestures or running around through your neighborhood. There are infinite possibilities of controlling games even within one type of control. Gaming connectivity, collection of data and the overall gaming experience is an IoT solution.
Let’s take a step into the development world of the internet of things. Creating IoT products and testing them is a big challenge. It’s safe to say that with IoT technologies, the amount of possibilities to test is infinite. We just cannot test it all. Separating a functional part of an IoT product helps with testing. This functional part can be described in some form of requirement. It serves as input for a test case. The four elements that make up a test case are:
- Precondition (desired situation you want to start with)
- Steps to follow through the system under test (manipulation of the starting condition)
- Expected result (describe the situation you expect the system to be in after the steps taken)
- Clean up (make sure the system ends up in a known and safe situation)
Test design techniques produce a set of steps and expected results. Test design techniques are selected because of specific risks are involved and a certain coverage is needed.
This all works in the safe and controlled environment of the functional part of the IoT product. Combining all parts of the IoT technology, and thus combining all tests available, gives a broader test coverage of the system. Not necessarily the right coverage of the system is achieved.
The combination of all functional parts creates infinite possibilities for an IoT product. The next step is making choices on dealing with infinite possibilities. We cannot test it all.
IoT layers steer away from functional testing
Break down an IoT product in layers of functionality and you will get:
- Thing layer (physical thing with actors and sensors)
- Bridge layer (the communication part, think Wi-Fi, LoRa, 4G)
- Data platform layer (data storage, cloud and possible business intelligence)
- App layer (an interface that could be a website or even a touchscreen on the thing itself)
Each layer can now be tested in splendid isolation. Coverage can be reached and a good sense of confidence is gained on each individual layer. Since we now know each layer is functioning well, the integrated product can be looked at. The focus can now shift from a functional view to more non-functional characteristics. Think on aspects like performance, user experience, interoperability or installability.
IoT products characterize themselves as solutions with added value. They have to create a business value on top of an existing function. In that sense, functionality is not top priority anymore. This is something that must work anyway. Take a coffeemaker at home. This machine makes coffee. The IoT version of the coffee machine collects data on your coffee-drinking behavior and learns when to brew what kind coffee. It still brews coffee. The added value here is that you think it makes great coffee because the timing is always right.
From this example, we learn that functionality of coffee making still has to work. Testing the IoT product must focus on the added value of unnoticed brewing of the right coffee at the right time. It is more about (in this case) the IoT coffee experience. Testing an experience is something different then testing a functionality. You might start using techniques like storytelling or crowd-testing. IoT testing is less functional testing and more non-functional testing!
Time to execute
With the focus of testing shifted away from functional testing, it is time to execute the tests you have planned. Test execution requires the IoT system and a test environment. I like to picture the IoT test environment as a set of concentric circles (like an onion).
- Circle 1: The inner core; one physical IoT product (the thing itself)
- Circle 2: The thing and the rest of the IoT technology
- Circle 3: A set of the same things working together within the IoT solution
- Circle 4: IoT solutions of the same product family working together (only domestic appliances of one brand)
- Circle 5: All IoT solutions from one brand must coexist correctly
- Circle 6: A defined set of IoT solutions of different brands, but of the same type (for example, all connected cars must travel the roads safely)
- Circle 7: All IoT solutions out there
- Circle 8: Future IoT solutions that could possible connect to your product
Of course, the entire set of circles may be applicable to you. Risk estimation can lead to testing only in test environments that cover circles 1 to 4. The exact definition of these circles may vary because of the architecture of your IoT product. Maybe you think you need more detailed circles for defining test environments. The idea of the concentric circle model for IoT environments is here to help you define your IoT test environment. With the IoT layers, a structured approach to testing IoT solutions takes shape.
Your IoT testing strategy
It is time for action. This article defines the elements needed to build your IoT testing strategy that copes with the infinite test combinations problem. With IoT layers, the right functional tests are executed (not too much!). Focus shifts to the non-functional quality attributes for IoT technologies. Specifically, the IoT experience needs coverage. With a good definition of your test environment, everything is in place to start with IoT test execution.
No need for infinite test time to cover infinite possibilities! As a side effect, the time-to-market for IoT products goes down. This can be a new bottleneck in IoT testing, by the way. I will keep that discussion for another time.
With this knowledge, I think the next generation of game consoles will be great and perfectly tested when they enter my home. I am going to play a game right now. Let me know how you test your IoT product!
All IoT Agenda network contributors are responsible for the content and accuracy of their posts. Opinions are of the writers and do not necessarily convey the thoughts of IoT Agenda.