How to choose the best GPUs for AI projects

The role of GPUs in AI

GPUs are critical to AI projects -- specifically, those that use machine learning to process large quantities of data.

Most machine learning models train by performing a huge amount of calculations. Although each individual calculation is often simple, such as evaluating a small data unit, the sheer volume of calculations makes the process extremely time-consuming if the computer must perform one calculation before it can begin the next -- in other words, if it operates sequentially rather than in parallel.

This is where GPUs come in; they excel at parallel processing, enabling them to perform many calculations simultaneously. GPUs achieve this efficiency through their architecture, which can comprise thousands of cores, each capable of handling calculations independently of the others.

In contrast, traditional CPUs typically feature only a few dozen cores at most, making them much less efficient at performing massive numbers of calculations at the same time. It's not that CPUs don't have substantial computing power; they just can't leverage that power as efficiently as GPUs due to their much smaller number of cores.

In addition to model training, GPUs are also valuable for inference -- the process by which a trained model interprets real-world data. GPUs' performance advantage over CPUs here varies, depending mostly on how many calculations take place during inference and how many of those happen in parallel. But for models with extensive internal processing and parallelization, GPUs are typically much better for inference, just as they are for training.

GPU features to consider for AI workloads

While all GPUs can speed up AI operations thanks to their high capacity for parallelization, the extent to which a GPU will benefit a given model depends on the specific features of that GPU.

When evaluating GPU options for AI, key points to consider include total core count, total memory, memory clock speed, GPU clock speed and AI-specific hardware optimizations. Understanding how much each of these features matters for your project can help you decide which GPU is right for you.

Total core count

In general, the most important GPU feature for shaping AI performance is total core count. That's because GPUs' value for model training and inference mainly stems from their ability to execute massive numbers of calculations in parallel. The more cores a GPU has, the greater its parallel processing capacity.

That said, the benefits of adding more cores varies by project. For example, a model designed to evaluate a small data set, or one using a simple internal algorithm with few layers, might perform just as well on a GPU with fewer cores as it would on a higher-core device.

Total memory

Most modern GPUs come with built-in memory, known as video random access memory (VRAM), which provides temporary storage for data that GPU cores process. Because the read/write speeds from GPU cores directly to VRAM are very fast, storing data in VRAM is much more efficient than using the system's general RAM or, worse, a hard disk.

Generally, the more VRAM a GPU has, the better it will perform for AI workloads -- but there are exceptions. The benefits of additional VRAM depend on how much data each GPU core needs to store temporarily during training or inference, as well as how much data needs to be shared among cores. Simpler models and those where the results of one calculation don't affect others might not require as much memory.

Memory clock speed

In addition to total memory, memory clock speed also plays a key role in overall AI model performance. Memory clock speed measures how fast GPU cores can read from and write to VRAM. Large amounts of memory are less useful if clock speeds are low because slow data transfer can become a bottleneck. This matters less for models that don't generate substantial temporary data or require frequent sharing of data among GPU cores.

GPU clock speed

GPU clock speed refers to how fast the cores inside a GPU can process information. Faster GPU clock speeds almost always yield better model performance.

However, it's important not to overemphasize clock speed, especially when using a GPU for AI as opposed to applications like gaming, where clock speed matters more. Because individual calculations during model training and inference are usually relatively simple, the overall number of cores -- and thus the ability to execute parallel computations -- is often more important than the amount of processing power per core.

In addition, many modern GPUs let users modify the clock speed within the supported range for a given GPU. Increasing clock speed is one way to improve performance in cases where a model is underperforming. But be cautious not to aggressively overclock a GPU, as excessive clock speed can lead to overheating.

AI-specific hardware optimizations

Some GPUs include specialized hardware components optimized for specific tasks. For example, tensor cores are designed to accelerate machine learning processes.

However, these specialized hardware features are generally only helpful for models designed to utilize them. For instance, tensor cores only benefit models that support mixed precision, a technique where computations are performed using a combination of different numerical precisions. Thus, it's important to ensure that your AI project can actually take advantage of any specialized hardware that a given GPU provides.

Other factors to consider when choosing a GPU for AI projects

In addition to the essential GPU considerations described above, it's also useful to look at the following factors when comparing GPU options:

• Cost. GPU prices can vary widely, ranging from under $100 to several thousand. While more expensive GPUs generally perform better, don't overspend on a GPU offering power and features that your model won't fully utilize.
• Vendor software support. The level of active development and software support a GPU vendor provides for its devices significantly affects ease of deployment. This includes essential components like software drivers that enable operating systems to interface with the GPU, as well as machine learning libraries or modules optimized for certain GPU architectures.
• Heat generation. GPUs that generate large amounts of heat require more advanced cooling systems. Be sure that your computer or server can dissipate the heat your GPU produces to prevent overheating and maintain performance.
• Motherboard integration. Although most GPUs connect to systems using standard PCIe slots, some require specialized connections, such as Server PCI Express Module slots. Ensure that your motherboard has compatible expansion slots for the GPU you intend to use; otherwise, you won't be able to install it.

GPU as a service: An alternative to purchasing your own GPUs

If you're unsure which GPU is right for your project or if buying a GPU isn't within your budget, consider renting a GPU in the cloud via a GPU-as-a-service platform.

GPUaaS is a type of cloud service that provides on-demand access to GPUs. Often, GPUaaS providers offer a range of GPU options, letting users choose the best one for a given AI workload. In addition, because GPUaaS eliminates the need for a large upfront investment in GPUs, it's a good option if you only need access to GPUs temporarily or occasionally -- for example, if you plan to use GPUs for model training, but CPUs for inference.

Chris Tozzi is a freelance writer, research adviser, and professor of IT and society who has previously worked as a journalist and Linux systems administrator.