Embedded vision in IoT

From computers to robots to artificial intelligence, multiple key technology advances have been inspired by the need to replicate or emulate human intelligence, sensing capabilities and behavior.

Various sensors, such as acoustic, vision and pressure, have been inspired from human hearing, vision and pressure-sensing abilities.

Undoubtedly one of the most important human sensing ability is vision. Vision allows humans to see the environment and interpret, analyze and take action.

Human vision is a remarkably involved complex, intelligent “machine” that occupies a significant part of the brain. Neurons dedicated for vision processing in the brain take up close to 30% of the cortex area.

Making devices, objects and things “see” the environment visually, as well as analyze and interpret, has been a key research area for a number of years.

Technological complexity, massive computation power requirements and prohibitive costs previously restricted vision sensing and intelligence to security and surveillance applications using surveillance cameras. However, the situation has changed dramatically today; the vision sensor market has exploded. Cameras are getting embedded everywhere and in all sorts of devices, objects and things, both moving and static. In addition, computation power available on the edge and on the cloud has increased dramatically. And this has trigged the embedded vision revolution.

The right sensor/camera price point, various technological advances in vision sensors resolution, dynamic range and amount of computation power available to process the video/imaging are leading to mindboggling growth and diverse applications.

Vision intelligence enabled through a combination of classical image processing and deep learning, has become possible in today’s world of connected embedded systems, devices and objects, taking advantage of both edge computing power on the device itself and also cloud computing power.

This has triggered rapid growth in self-driving vehicles, drones, robots, industrial applications, retail, transportation, security and surveillance, home appliances, medical/healthcare, sports and entertainment, consumer augmented and virtual reality, and, of course, the ubiquitous mobile phone. Vision and its intelligence is a storm in the IoT world, and it’s only going to increase.

In fact, no other sensor has had such a dramatic impact. So prevalent has video become in day-to-day life that most people take it for granted. From streaming live video systems to video on demand to video calling, it’s easy to forget the dramatic impact vision sensors have had in a world of internet-connected environments and devices; it is truly the unsung hero sensor of IoT. Combine it with vision intelligence and the whole market takes a new dimension.

The growth in prevalence of embedded vison has its roots in the explosive growth of mobile phones with embedded cameras. Prior to the mobile phone revolution, video/cameras and intelligence remained associated with security and surveillance. But then mobile phones with embedded cameras arrived and this aligned with simultaneous massive growth in computation power for video analytics and intelligence, both on the edge and the cloud. This explosive combination has led to the remarkable growth, and vision sensors started getting embedded everywhere from robot and drones to vehicles, industrial machine vision applications, appliances and more.

There are various types of vision sensors, but complementary metal-oxide semiconductor, or CMOS, sensors by far have had the largest impact and have led this explosive growth of vision sensors in various embedded systems and smartphones.

Sensors are everywhere — and are numerous. Self-driving cars today have more than 10 video cameras, drones have three to four video cameras, security surveillance cameras are everywhere, mobile phones are streaming live video. Video data from these sources is streamed for further intelligence in the cloud, while real-time edge processing is happening on devices and things themselves.

Vision Sensor resolution, dynamic range and number of vision sensors continues to scale up with no end in sight. With massive amounts of video data getting produced by these sensors, naturally amount of computation power required is huge, as its transmission and storage requirements.

Previously, there was a rush to stream video to the cloud for real-time or stored vision analytics. Cloud offered immense computation power, but bandwidth necessary for transmission even after compression was high. Huge amount of storage, latencies involved, and security and privacy concerns are making customers rethink the cloud and instead consider vision analytics at the device/object level and then doing offline video processing in the cloud.

With low-latency, high-speed 5G connectivity promises, there is a thought to distribute real-time video processing between the edge and the cloud. However, it remains to be seen how much of this is possible — if it at all — and whether it makes sense to transmit real-time compressed video to the cloud from millions of endpoints hogging transmission bandwidth.

The importance of edge analytics has market enabled various system-on-a-chip (SoC), graphics processing units (GPU) and vision accelerators. Cloud with GPU acceleration is getting used for non-real-time video analytics, or for training neural networks on large amount of test data while real-time inference is happening on the edge with the accelerators.

With deep learning and optimized SoCs now available, along with vision accelerators for classic image processing, the edge analytics trend is likely to continue, with additional events, parameters and intelligence pushed to cloud for further analysis and correlation. The cloud will continue to remain important for offline stored video analysis, while some systems can still do real-time analysis there.