← All explainers

CHIPS & COMPUTING · EXPLAINER

Why AI accelerators need high-bandwidth memory

How stacked memory feeds AI processors, why capacity and bandwidth are different, and when faster arithmetic still has to wait for data.

An AI accelerator can perform enormous numbers of calculations, but it cannot calculate with data it has not received. Model weights, inputs, intermediate results and the growing working state of an AI service all have to move through memory. When that movement is slower than the arithmetic, expensive compute units wait.

That is why accelerator announcements now put memory beside compute in the headline specifications. This week, AMD introduced another memory-capacity option in its planned MI400 family. The numbers are large, but they answer two different questions: how much information can fit close to a processor, and how quickly that information can move.

High-bandwidth memory, or HBM, is built to improve both. It is not a promise that every AI workload will run at a product’s advertised peak speed.

Capacity and bandwidth solve different problems

Memory capacity is the amount of data a device can hold. Memory bandwidth is the rate at which data can travel between memory and the processor. A warehouse analogy helps: capacity is the floor space inside; bandwidth is the number and width of loading doors. A huge warehouse with one narrow door can still leave workers waiting.

AMD says its MI450X will have 288 gigabytes of HBM4 memory, while the MI455X will have 432 GB. The company presents the smaller option as a way to match memory capacity to the workload while retaining the family’s core compute and connectivity capabilities. More capacity can keep larger models, longer context state or more simultaneous work on the device. It does not automatically make a small job finish faster. AMD: MI400 Series update

Bandwidth answers the speed question. AMD lists up to 23.3 terabytes per second of peak theoretical memory bandwidth for the MI455X. “Peak theoretical” matters: it describes a hardware ceiling under defined conditions, not the sustained rate of every application. Software access patterns, caches and the kind of calculation determine how much of that path is useful.

Why ordinary memory placement is not enough

HBM puts several dynamic random-access memory dies into vertical stacks. Tiny electrical paths called through-silicon vias connect the layers. The stacks sit in the same advanced package as the accelerator and use a very wide interface to move many pieces of data in parallel.

Samsung’s HBM description identifies the two defining features: vertically stacked memory dies and TSV interconnections. Micron similarly describes HBM as a specialized three-dimensional stacked memory architecture rather than separate flat memory packages spread across a circuit board. Samsung Semiconductor: HBM · Micron: high-bandwidth memory

The accelerator and memory stacks are commonly connected through a dense package layer called an interposer. The short, numerous connections matter because data movement consumes time and energy. HBM increases the width of the path instead of relying only on a faster signal through a comparatively narrow path.

This construction is also why HBM belongs to the advanced-packaging story. Producing the memory dies is only one step. Manufacturers have to stack, connect, test and integrate them beside the compute device. A chip announcement therefore depends on memory supply and packaging capacity as well as fabrication of the accelerator itself.

When the processor waits for memory

Some operations do many calculations for every byte they move. Others repeatedly load and store data while doing relatively little arithmetic. The second group can become memory-bound: the calculation units are ready, but data movement sets the pace.

NVIDIA’s performance guide explains that total time can be limited by math, memory bandwidth or latency. Its guide to memory-limited neural-network layers notes that normalization, activation and pooling operations often depend primarily on memory transfer time. Faster arithmetic alone does not remove that limit. NVIDIA: GPU performance background · NVIDIA: memory-limited layers

AMD’s software documentation makes the same distinction: a memory-bound kernel is limited by how quickly data can move between HBM and the processor’s on-chip caches. It also shows why layout matters. If nearby threads request nearby data, hardware can combine their requests into fewer physical transactions. Poor access patterns can waste part of the advertised bandwidth. AMD ROCm: understanding GPU performance

That is the limit of the warehouse analogy. Computer memory traffic is scheduled in tiny, repeated transactions. The way software arranges those transactions can be as important as the width of the door.

Training and inference use memory differently

During training, a system may need model parameters, intermediate activations, gradients and optimizer state. These do not always have to remain on one accelerator, but moving them elsewhere adds communication. More local memory can change how the job is divided across devices and reduce some transfers.

During inference, the model weights still need a home, and a service also maintains temporary state for active requests. One important example is the key-value cache used by many transformer models. Longer prompts, longer generated responses and more simultaneous users can increase this working state. AMD explicitly connects MI400 memory capacity to models, context windows and KV caches. That is a product claim about intended capability, not a guarantee of response time for a particular service. AMD CDNA architecture

Capacity can therefore affect whether a workload fits comfortably, how many devices it needs and how much state can remain close to the processor. Bandwidth affects how quickly data can be supplied once it is there. Latency—the delay before a transfer begins—is another limit. One large bandwidth number does not describe all three.

HBM does not replace the rest of the memory system

HBM is fast and close to the accelerator, but it is not the only place data lives. On-chip caches hold smaller amounts even closer to the compute units. System memory, solid-state storage and networked storage provide larger or more persistent capacity. Multiple accelerators also exchange data over scale-up links inside a system and networks between systems.

Each boundary can become the bottleneck. A model may fit in HBM while GPUs wait for one another across a network. A storage pipeline may fail to feed a training cluster. A cache-friendly implementation can outperform another program on the same hardware. The broader Grid & Silicon explainer, What a GPU does inside an AI system, follows those layers together.

HBM also adds heat inside an already dense package. The processor, memory and their interconnects must operate within the electrical and thermal design of the accelerator module and server. More memory capacity is useful only when packaging, power delivery and cooling support it.

How to read the next HBM claim

Start by separating capacity from bandwidth. Gigabytes or terabytes describe how much memory is available. Gigabytes or terabytes per second describe a transfer rate. Neither is a count of calculations.

Next check the boundary. Is the figure for one memory stack, one accelerator, one server or a complete rack? Adding device figures can describe installed hardware, but it does not prove that every device can use all memory as one seamless pool.

Then look for qualifiers such as peak, theoretical, projected and up to. AMD’s MI400 figures describe a product family planned for future deployment, and its performance comparisons come from specified company tests. They should not be relabeled as independent operating measurements.

Finally, match the benchmark to the real job. Ask which model, numerical format, batch size, context length, software version and accuracy target were used. A result can be valid and still answer a narrower question than the headline suggests.

HBM matters because an accelerator is only useful when data reaches its arithmetic units. Stacking memory close to the processor and widening the path helps keep those units busy. The honest conclusion is not that more HBM always means a faster AI system. It is that capacity, bandwidth and software have to fit the workload together.

Sources and further reading

Glossary — plain definitions for words used here.