Evolving from IoT to edge AI system development

Courtesy: Avnet

The advancement of machine learning (ML) along with continued breakthroughs in artificial intelligence (AI) are funnelling billions of dollars into cutting-edge research and advanced computing infrastructure. In this high-stakes atmosphere, embedded developers face challenges with the evolution of Internet of Things (IoT) devices. Teams must now consider how to implement ML and edge AI into IoT systems.

Connected devices at the edge of the network can be divided into two broad categories: gateways and nodes. Gateways (including routers) have significant computing and memory resources, wired power and reliable high-bandwidth connectivity. Edge nodes are much smaller devices with constrained resources. They include smartphones, health wearables, and environmental monitors. These nodes have limited (often battery) power and may have limited or intermittent connectivity. Nodes are often the devices that provide real-time responses to local stimuli.

Most edge AI and ML applications will fall into these two categories. Gateways have the capacity to become more complex, enabled by access to resources such as wired power. What we currently call IoT nodes will also evolve functionally, but are less likely to benefit from more resources. This evolution is inevitable, but it has clear consequences on performance, energy consumption, and, fundamentally, the design.

Is edge AI development more challenging than IoT?

There are parallels between developing IoT devices and adopting edge AI or ML. Both involve devices that will be installed in arbitrary locations and unattended for long periods of time. Data security and the cost of exposure are already key considerations in IoT when handling sensitive information. Edge gateways and nodes retain more sensitive data on-device, but that doesn’t eliminate the need for robust security. Beyond their size and resources, there are significant differences between the two paradigms, as outlined in the table.

Comparing IoT and edge AI applications

What’s at stake in edge AI?

IoT devices are data-centric, providing local control along with actionable data that is typically passed to a cloud application. AI and ML at the edge replace cloud dependency with local inferencing. Any AI/ML application starts with sourcing and validating enough data to train a model that can infer meaningful and useful insights.

Once the data is gathered and the source model has been trained, its operation must be optimized through processes such as model pruning, distillation and quantization, to drive simpler inference algorithms on more limited edge AI hardware.

Every time a new set of model parameters is passed to the edge AI device, the investment in training is at risk. Every time an edge node delivers a result based on local inference, it reveals information about the local model. Edge AI devices are also open to physical tampering, as well as adversarial attacks intended to corrupt their operation or even poison the inferencing models they rely upon.

Adaptive models and federated learning

While IoT devices can be maintained using over-the-air (OTA) updates, edge AI systems use adaptive models to adjust their inferencing algorithms and decision-making processes locally in response to external stimuli. This adaptation is valuable because it helps systems deliver resilient performance in evolving situations, such as predictive maintenance for manufacturing processes or patient monitoring in healthcare devices.

Adaptive models also enable edge AI devices to evolve functionally without exposing input data or output decisions. Over time, a source model implemented on an edge AI device should become steadily more valuable as it adapts to real-world stimuli.

This value becomes more apparent when adaptive models are used as part of a federated learning strategy. While edge AI devices using adaptive models keep their learning to themselves, under federated learning strategies, they share.

Each edge AI device adapts its model to better serve its local situation and then, periodically, sends its updated model parameters to the cloud. The submitted parameters are averaged and used to update the source model to reflect this experience. The upside is a source model that is regularly enhanced by field experience from multiple contexts. The challenge is that increasingly valuable model parameters must traverse a network, making them vulnerable to theft or misuse.

Balancing security and model performance

Security measures, such as encryption, help protect model parameters but may be at odds with the design goal of creating a low-resource, highly responsive edge AI device. This challenge can be made more difficult by some strategies used to compress source models to run on low-resource devices.

For example, quantization works by reducing the resolution with which model parameters are expressed, for example, from 32-bit floating-point representations during training to 8- or even 4-bit integers in the compressed version. Pruning disregards nodes whose parameters have a negligible influence on the rest of the model. Distillation implements “teacher–pupil” learning, in which the embedded model learns by relating the inputs and outputs of the source model.

Each of these techniques simplifies the model and makes it more practical for running on resource-constrained hardware, but at the cost of protection against security challenges. For example, quantized or pruned models may offer less redundancy than the source model, making them more efficient, but leave them less resilient to adversarial attacks.

The implementation of security features, such as encrypted parameter storage and communication, or periodic re-authentication, can create processing overheads that undercut the gains achieved by model compression. As the value at risk in edge AI devices rises, embedded engineers will need to weave security concerns more deeply into their development processes. For example, pruning strategies may have to be updated to include “adversarial pruning” in which different parts of a model are removed to see what can be discarded without making the remainder more vulnerable.

Keeping edge AI systems up to date

Embedded edge AI developers will need to be extremely flexible to accommodate rapidly changing ML algorithms, rapidly evolving AI processor options, and the challenge of making the hardware and software work together.

On the hardware side, multiple semiconductor start-ups have been funded to develop edge AI chips. Although the architectures differ, the design goals are often similar:

• Minimize data movement during computation to save power
• Implement extremely efficient arrays of multipliers to do the matrix math involved in ML inference
• Wrap the inferencing engine up in an appropriate set of peripherals for the end application

On the algorithmic side, although ML inferencing involves many standard operations, such as calculating weights and biases in layers or handling back-propagation, the exact way these operations are deployed is evolving rapidly.

Not every combination of ML operations will run well on every processor architecture, especially if the toolchains provided for new chips are immature.

The verification challenge

Design verification may also be a challenge, especially in situations where the implementation of security forces changes to the inferencing models, to the point that they must be retrained, retested and revalidated.

This may be especially important in highly regulated sectors such as healthcare and automotive. Engineers may have to rely heavily on hardware-in-the-loop testing to explore real-world performance metrics such as latency, power, accuracy, and resilience that have been formulated during the development process.

Embedded system designers must be ready to adapt rapidly. Edge AI algorithms, hardware strategies and security issues threatening efficacy are all evolving simultaneously. For edge AI devices, the well-understood challenges of IoT device design will be overlaid with concerns about the value of AI data at rest and on the move.

The impact of security features on inferencing performance must be balanced with the rate of progress in the field. Embedded AI system designers should mitigate these challenges by adhering to standards, choosing established toolchains where possible, and picking well-proven tools to handle issues such as scalable deployment and lifecycle management.