The growing role of FPGAs for accelerating AI workloads

What is an FPGA?

An FPGA is an integrated circuit that users can reconfigure after manufacturing, hence the term field-programmable. This reconfigurability is the defining characteristic of FPGAs, distinguishing them from hardware devices whose internal configuration can't be meaningfully changed once shipped to customers.

FPGAs achieve this flexibility by letting users change logic blocks and interconnects, which effectively means that developers can modify how the device processes data. This makes it possible to optimize the hardware for various use cases, such as minimizing latency or prioritizing energy efficiency.

Typically, developers apply changes to FPGAs by writing and deploying code that configures device operations. Thus, developers don't have to change the physical hardware itself to modify how the device works; they can simply edit the software.

Advantages of using FPGAs for AI

FPGAs were not originally intended for AI deployments. They were conceived decades ago to let engineers experiment with different circuit configurations without creating new physical hardware each time they wanted to modify a design. This makes FPGAs useful for hardware prototyping and situations where an application's optimal hardware configuration evolves over time as the application is changed.

However, in recent years, FPGAs have also become popular as AI accelerators. AI projects can use FPGAs for two main purposes:

• Model development. The ability to update hardware configurations is useful during model development because model designs often change over time. With an FPGA, engineers can update hardware configurations to suit the model as the model changes.
• Inference. As data processing devices go, FPGAs are lightweight and energy-efficient, and the ability to customize them based on unique model designs can further improve efficiency. For this reason, FPGAs are sometimes used for inference, especially in deployment scenarios where hosting AI software using traditional servers or data centers is not feasible -- for example, running an AI service on a device without reliable internet connectivity.

Although AI developers can also use GPUs, NPUs and even standard CPUs for both model development and inference, the ability to customize hardware configurations gives FPGAs a level of flexibility that other types of AI accelerators lack.

Challenges of using FPGAs for AI

While FPGAs offer some advantages for AI projects, they also present several potential challenges:

• Programming complexity. Programming an FPGA requires specialized tools and knowledge; it's more complicated than choosing from preset options or scripting in a common language like Python. Adding to the complexity, programming software often varies across FPGA vendor platforms. Thus, taking full advantage of FPGAs' customizability can involve a steep learning curve.
• Limited AI optimizations. Because FPGAs are generally intended for hardware prototyping, most aren't inherently optimized for AI workloads. Engineers can find clever ways to configure FPGAs to support AI applications effectively, but these devices aren't well suited for AI out of the box.
• Low resource capacity. Most FPGAs are lightweight by design. While this makes them energy-efficient, it's a challenge when running AI workloads that require high amounts of processing power, as individual FPGAs can't crunch numbers quickly. Developers can distribute workloads across multiple FPGAs if a single device isn't sufficient, but that's more complicated than using a single high-capacity device or a small, powerful cluster.
• Smaller, siloed ecosystems. Compared with widely adopted devices like Nvidia GPUs, the FPGA ecosystem offers fewer software libraries, machine learning frameworks and other tooling. When such tools are available, they are often siloed within specific vendor ecosystems, making it hard for developers to switch among FPGA devices or use multiple types at the same time.

FPGAs vs. GPUs for AI

Due to FPGAs' limitations, developers often view GPUs as an overall better option for AI development and deployment.

This is in part because a typical GPU provides much more processing power than a typical FPGA. And while developers can't optimize a GPU's internal logic for a specific model, they can design a model tailored to the GPU they intend to use for training or inference. In this way, developers can still optimize performance by modifying software rather than hardware when using a GPU -- the difference is that the software changes occur at the level of model architecture.

In addition, GPUs integrate easily with popular open source machine learning libraries like PyTorch and TensorFlow, offering a familiar and well-supported set of tools and developer community. This makes it easy for developers to use GPUs for model training and deployment without having to learn specialized tools or work within a siloed vendor ecosystem.

That said, FPGAs offer several benefits that GPUs lack. For one, they typically cost less as individual units -- although, measured in terms of cost relative to processing power, GPUs might be more cost-effective. Second, they generally consume less energy, making them better suited for efficiency-sensitive applications. How much these differences matter can vary from one AI project to another.

Chris Tozzi is a freelance writer, research adviser, and professor of IT and society who has previously worked as a journalist and Linux systems administrator.