Using today's advanced AI systems to run machine learning on smaller devices like microprocessors offers benefits, but also limits, which experts are working to surmount.
One exciting avenue in the world of AI research and development is finding ways to shrink AI algorithms to run on smaller devices closer to sensors, motors and people. Developing embedded AI applications that run machine learning on microcontrollers comes with different constraints around power, performance, connectivity and tools.
Embedded AI already has various uses: identifying types of physical activity with smartphone sensors, responding to wake words in consumer electronics, monitoring industrial equipment and distinguishing family members from strangers in home security cameras.
A range of new tools, such as TinyML and TensorFlow Lite, can simplify the development of smaller, more power-efficient AI algorithms.
"The rise of TinyML deployed on microcontrollers enables intelligence to be distributed into more connected products in the physical world, whether they be smart home gadgets, toys, industrial sensors or otherwise," said Jason Shepherd, vice president of ecosystem at edge-computing platform Zededa.
The release of Arm's AI/ML-optimized Cortex M55 core last year is already catalyzing increasingly sophisticated and even more lightweight microcontrollers with embedded coprocessing to optimize both overall processing capability and power consumption. New AI tools also make it easier for developers without deep experience in embedded software to train, optimize and deploy AI models on microcontroller-based hardware.
Making machine learning small
The biggest difference between CPUs and microcontrollers is that microcontrollers are often directly connected to sensors and actuators. This reduces latency, which is essential in safety-critical applications like controlling brakes and industrial equipment or responding to people.
"The big trend in the AI industry is moving machine learning inference to the edge, where the sensor data is generated," said Sang Won Lee, CEO of Qeexo, an AI platform for embedded systems.
New AI tools make it easier for developers without deep experience in embedded software to train, optimize and deploy AI models on microcontroller-based hardware.
Running inference on the edge immediately provides valuable benefits such as reducing latency, bandwidth and power usage. Also, there is higher availability from not having to depend on the cloud or a centralized server. Lee observed that running inference on microcontrollers typically consumes less than 5 milliwatts compared to 800 milliwatts for sending data to the cloud over a cellular network.
However, various microcontroller limits also present new challenges for traditional AI workflows. Top constraints include limited power, memory, hardware and software environments.
David Kanter can attest to this in his role as executive director at MLCommons, an industry consortium developing benchmarks, data sets and best practices for machine learning. He said industry groups are starting to establish benchmarks to help developers shortlist the appropriate combinations of microcontrollers, development tools and algorithms for various tasks such as MLPerf Tiny.
What is a microcontroller?
Microcontrollers preceded the development of CPUs and GPUs and are embedded in virtually every kind of modern device with sensors and actuators. They are a vital consideration for enterprises interested in weaving AI into physical devices, whether to improve the user experience or enable autonomous capabilities.
For example, AI Clearing has developed a drone platform that automatically captures construction site progress. Jakub Lukaszewicz, head of AI for AI Clearing, said that microcontrollers have been especially important for his team since they are often the main computers on drones responsible for flying and communicating with the operator.
He thinks of microcontrollers as low-end CPUs with limited processing capability. There are many microcontroller varieties on the market with various architectures and functionalities, but they all have two crucial advantages over high-end CPUs: low cost and low power consumption.
The low cost makes them ideal for adding interactive functionalities to traditional devices like toys or home appliances. In recent years, microcontrollers allowed those devices to have color screens and multimedia capabilities. Low-power consumption will enable microcontrollers to be used in wearables, cameras or devices that run for a long time on a small battery.
AI on a low-power microcontroller
Lukaszewicz has been following a new trend of creating microcontrollers with integrated neural processing units (NPUs), which are specialized units designed to run machine learning models on microcontrollers efficiently.
"Every major microcontroller producer is preparing a product outfitted with such a device," he said.
These generally come with specialized SDKs that can transform neural networks prepared on a computer to fit onto an NPU. These tools generally support models created with frameworks like PyTorch, TensorFlow and others thanks to the ONNX format. Also, various third-party tools are emerging from companies like Latent AI and Edge Impulse to simplify AI development across different microcontrollers.
But these toolkits do not support all operations available on bigger CPUs that have more RAM, Lukaszewicz observed. Some models are too big, while others use unsupported operations. Often, engineers need to prune a model or adjust its architecture for the NPU, which requires a lot of expertise and extends the development time.
Donncha Carroll, a partner in the revenue growth practice of Axiom Consulting Partners leading its Data Engineering and Science team, said developers also need to weigh the tradeoffs between the lower cost of microcontrollers compared with CPUs or GPUs and their flexibility. It's harder to reconfigure or retrain embedded systems quickly.
"A centralized solution using microprocessors will sometimes make more sense," he said.
Tiny AI use cases
There are many use cases for microcontrollers, Carroll said. For example, microcontrollers can:
monitor your home kitchen while you're at work and notify you when your dog has jumped onto the counter to eat the leftover chicken;
combine information from a variety of sensors (e.g., noise, temperature, vibration) to determine when a complex piece of equipment (e.g., wind turbine) needs maintenance;
determine if an unauthorized individual is present in a secure location;
determine if some combination of ambient conditions (e.g., humidity, temperature, sunlight, air quality) is a likely threat to agricultural output on a farm; and
activate a device based on a trigger such as someone raising their voice above a certain decibel level.
Planning for a tiny future
The limits of running machine learning on microcontrollers are also inspiring new AI system designs.
"Microcontrollers are so computationally constrained, they are driving some of the most interesting work in model compression," said Waleed Kadous, head of engineering at distributed computing platform Anyscale.
Previously employed by Google, Kadous worked on the sensor hub in Android phones that uses an ML model to determine if someone is standing still, walking, running, commuting or biking. He believes this represents a typical use case for thinking about how low-power embedded sensing can be distributed throughout the environment.
One line of research is exploring ways to shrink big models to run on much smaller devices without losing too much accuracy. Another explores cascading complexity models, which combine quick models that decide if there is anything of interest with more complex models for deeper analysis. This could allow an application to detect anomalies and then engage another processor to take an action, such as uploading data to a cloud server.
Down the road, Kadous expects to see more general hardware for ML model execution move down into microcontrollers. He also hopes to see better tools for model compression that complement the improvements made in compilers for microcontrollers.
Eventually, this will lead to tools for improving a given microcontroller's performance and not just what's in its environment. "I think ML will move into the execution of the microcontroller itself for things like power management, and to squeeze the last few milliwatts of power out. ML will also slightly improve the microcontroller's operational efficiency," Kadous said.
