TFLOPS Aren’t Everything: The Many Dimensions That Shape GPU Performance

By Pawel Rzeszucinski, PhD

Introduction

The rapid advancement of AI algorithms, capable of fully leveraging GPUs’ capabilities, has transformed this hardware into an indispensable element of the current AI revolution. While we typically judge hardware quality by its raw ‘speed’, with GPUs it’s not that straightforward — it’s much more nuanced and interesting.

When working in AI, choosing the right GPU isn’t simply about picking the fastest one. Many initially believe that raw speed is the key factor and that more TFLOPS always equate to better performance. However, the deeper one dives into the subject, the clearer it becomes that GPU performance is multi-dimensional. A variety of parameters beyond raw compute power influence AI performance differently based on specific workloads.

This guide aims to help organize knowledge on the topic. We’ll explore Compute Power (TFLOPS), Memory (VRAM and Memory Bandwidth), Interconnect Bandwidth, and specialized hardware like CUDA and Tensor cores. By understanding each element, AI practitioners can make informed decisions when selecting GPUs. Finally, we’ll examine practical examples demonstrating that raw speed alone doesn’t determine optimal performance.

NVIDIA A100 Specifications — Can You Speak English, Please?

Take a look at NVIDIA’s official A100 spec sheet. If the terminology seems overwhelming, don’t worry. This guide will break it down, and by the end, these concepts will be much clearer.

Compute Power (TFLOPS)

First, let’s discuss the most common GPU metric: TFLOPS (teraflops), which indicates the number of floating-point operations a GPU can process per second. Initially, it appears that higher TFLOPS means better performance. However, real AI workloads are more nuanced.

GPUs perform computations at different precision levels, each suited for different tasks:

• FP64 (Double Precision): Highly accurate but slow and inefficient for AI workloads.
• FP32 (Single Precision): Common in AI training, offering a balance between speed and accuracy.
• FP16/BF16 (Half Precision): Optimized for deep learning, reducing computation time and memory usage.
• INT8 (Integer Precision): Primarily used for model inference, focusing on speed rather than precision.

AI model training prioritizes higher precision (e.g., FP32, FP16), but for optimized inference (execution of trained models), lower-precision formats like INT8 are often preferred to improve efficiency.

GPU Memory (VRAM and Memory Bandwidth)

GPU memory is critical for handling AI workloads. There are two key considerations:

• VRAM (Video RAM): The working memory for the GPU, storing training data, model parameters, and input batches.
• Memory Bandwidth: Determines how quickly data flows between VRAM and the GPU cores. A high bandwidth prevents bottlenecks during training.

GPUs with High Bandwidth Memory (HBM) offer superior performance compared to those using traditional GDDR6 memory. As AI models grow in size, GPUs with high memory capacity and bandwidth become essential to prevent slowdowns caused by data movement constraints.

Interconnect Bandwidth

Many AI tasks require multiple GPUs working together. The efficiency of multi-GPU communication depends on interconnect bandwidth, which determines data exchange speed.

• PCIe: The most widely used connection, but limited in speed. Suitable for small-scale setups.
• NVLink: NVIDIA’s high-speed interconnect, designed for efficient data exchange within a single server.
• InfiniBand: Required for large-scale AI clusters spanning multiple servers, essential for training massive models like GPT-series.

Choosing the right interconnect technology is crucial. Small-scale projects can rely on PCIe, while high-performance computing demands NVLink or InfiniBand to minimize latency.

Specialized GPU Hardware: CUDA vs. Tensor Cores

• CUDA Cores: General-purpose parallel processors handling a variety of tasks efficiently.
• Tensor Cores: Hardware specialized for matrix multiplications, accelerating deep learning computations.

While CUDA cores manage most processing, Tensor cores are essential for deep learning workloads, using Fused Multiply-Add (FMA) operations to drastically reduce computation time in AI training.

It’s Not All Speed Then, Is It?

Optimizing GPU performance means aligning GPU specifications with specific workloads:

• Real-time inference on edge devices (e.g., autonomous vehicles): TFLOPS are crucial for rapid inference, while memory and interconnect bandwidth are less relevant.
• Training massive models: Sufficient VRAM and memory bandwidth are more critical than pure TFLOPS. Without enough memory, even powerful GPUs suffer from data transfer bottlenecks.
• Multi-GPU or multi-node training (e.g., large-scale AI models): The speed of interconnects (NVLink, InfiniBand) becomes the limiting factor rather than individual GPU performance.

Understanding these requirements helps AI practitioners choose the right GPU hardware for optimal efficiency.

NVIDIA A100 Specifications — Let’s Test Our Knowledge!

NVIDIA’s A100 GPU comes in two variations:

• PCIe GPUs: Offer versatility and moderate power consumption.
• SXM GPUs: Deliver higher performance in high-density data centers but require advanced cooling solutions.

AI users must also consider the trade-off between computational precision and speed:

• Higher precision (FP64): Essential for scientific computing but inefficient for AI workloads.
• Lower precision (TF32, BF16, FP16): Optimized for AI tasks, maximizing computational efficiency.

An additional feature, Multi-Instance GPU (MIG), allows partitioning a single GPU into multiple virtual GPUs—similar to a virtual private server (VPS)—enabling resource-sharing across workloads.

Overall, tailoring GPU selection to workload requirements ensures the best balance of performance and efficiency.

Final Thoughts

AI engineers often focus on algorithmic improvements, but as demonstrated, GPU selection plays an equally significant role. GPU performance is shaped by a multidimensional set of factors, including compute power, memory, bandwidth, and interconnect technology.

• TFLOPS alone don’t determine performance.
• The workload dictates the most relevant GPU characteristics.
• Strategic GPU selection leads to optimal efficiency and cost savings.

Whether developing models on a personal machine or training state-of-the-art AI models across distributed clusters, understanding these factors ensures the best outcomes. TFLOPS may grab the headlines, but real performance lies in the details.