Definition: Memory bandwidth is how fast a processor can move data between its memory and its compute cores, measured in terabytes per second — and it is the real ceiling on how quickly an AI model answers, because generating each token requires reading the model's parameters out of memory.
TL;DR: AI text generation is limited by memory speed, not arithmetic speed. Producing one token means reading the model's weights out of memory once, so the cores usually sit waiting on data rather than the other way round. This is why HBM capacity and bandwidth decide how many users one chip can serve. Build an app free →
The Least Intuitive Fact in AI Infrastructure
Everyone assumes AI is limited by how fast chips can calculate. For training, that is broadly true. For the generation you actually experience, it is not.
Inference has two phases with opposite hardware profiles, and conflating them is the most common technical error in this area:
| Phase | What it does | Bound by | What it sets |
|---|---|---|---|
| Prefill | Reads your whole prompt in parallel | Compute | The pause before the first word |
| Decode | Produces one token per pass | Memory bandwidth | Generation speed and cost |
During decode the arithmetic is trivial — a single token's worth. The expensive part is pulling the model's parameters and cached state out of memory and into the cores. The math is fast. The fetching is slow.
The loop that produces every word you read. The wide arrow is the memory read; the arithmetic is the small part.
Why This Explains So Much
Once you know generation is memory-bound, several otherwise-puzzling things become obvious:
- Why batching is the business model. One read of the weights can serve many users at once. A GPU answering one person wastes most of its capacity; the same GPU answering fifty is nearly the same amount of memory traffic. This is why serving cost per user falls as a service gets busier.
- Why long context costs more. The KV cache grows with conversation length and lives in the same fast memory as the weights, so a longer context window means more bytes moved per token.
- Why memory makers matter as much as chip designers. Bandwidth is supplied by high-bandwidth memory, and its capacity and speed set the practical ceiling on serving capacity.
- Why smaller models feel disproportionately fast. Fewer parameters means fewer bytes to read per token, so speed improves faster than parameter count would suggest.
Roofline Intuition: Two Ceilings, One Roof
Hardware people reason about all of this with one mental chart, the roofline model, and the intuition transfers in a paragraph. A chip has two ceilings: a compute ceiling (how many operations per second its cores can execute) and a memory ceiling (bandwidth times arithmetic intensity — how many operations you perform per byte fetched). Your actual throughput is capped by whichever ceiling is lower for your workload.
Where does AI generation sit? An H100 can execute on the order of a thousand trillion operations per second but read only 3.35 TB/s — so a workload needs to do roughly 300 operations per byte it fetches to keep the cores busy. Single-stream decode does about 2 per byte: each weight is read, used in one multiply-add pair, and discarded until the next token. That is two orders of magnitude under the line — the cores could, in principle, serve a hundred-plus users with the arithmetic they waste waiting on one.
Which is exactly what batching does: serve a hundred streams per weight read and the intensity rises a hundredfold, sliding the workload along the roof toward the compute ceiling. Every serving trick in this article — batching, quantisation (fewer bytes per weight), mixture-of-experts (fewer weights per token), speculative decoding (more tokens per read) — is the same move on the same chart: raise the work done per byte moved.
The ceiling also gives you a free sanity check on any speed claim. 70 GB of 8-bit weights over 3.35 TB/s caps a single stream near 48 tokens per second on one H100; a bigger number is batched throughput, a different precision, or a smaller model.
The Memory Hierarchy
Every tier trades capacity against speed, and the closer to the cores, the more expensive per byte.
| Tier | Bandwidth class | Latency | What lives here |
|---|---|---|---|
| Registers / SRAM | Fastest | ~1 ns | Active working values |
| HBM | ~TB/s | ~100 ns | Model weights, KV cache |
| Host DRAM | ~100 GB/s | ~100 ns+ | Staging, overflow |
| Local NVMe | ~GB/s | ~100 µs | Checkpoints, hot datasets |
| Network storage | Slowest | ~ms | Training corpora, archives |
What You Can Do About It
Most of this is the provider's problem, but two levers reach the application layer:
- Prompt caching. Repeated context can skip prefill, which is the single largest lever for document-heavy workloads.
- Right-sizing the model. A smaller model that is good enough moves fewer bytes per token and answers faster. Model routing makes that choice automatically per request.
Architectures like mixture-of-experts attack the problem directly by activating only part of the network per token, reducing bytes read without reducing total model size.
For most teams the honest lever is one level up: shape the workload, not the hardware. Multi-step AI agent runs feel bandwidth limits most, because per-call latency compounds across the chain — keeping context lean keeps each call's memory traffic small. And work with no human waiting belongs in scheduled automations, where it can run dense and batched instead of competing for interactive capacity.
Related Concepts
- HBM — where the bandwidth comes from
- KV Cache — the other tenant of that memory
- Tokens Per Second — the user-visible result
- GPU · Inference
- Data Center — the building that powers the reads
- Model Parameters — what is being read
- Inference Cost
Frequently Asked Questions About Memory Bandwidth
Why is AI inference memory-bound rather than compute-bound?
Because generating each token requires reading the model's parameters out of memory, and modern accelerators can perform arithmetic far faster than they can be fed. During the decode phase the compute units frequently idle while waiting for weights to arrive, so the memory system rather than the arithmetic units sets the pace.
Is training also memory-bound?
Generally no. Training processes many tokens in parallel, so each read of the weights is amortised across a large amount of arithmetic. That makes training more compute- and interconnect-bound, while single-stream generation is memory-bound. This is one reason training and inference favour different hardware trade-offs.
Does more memory bandwidth make a model smarter?
No. It makes the same model answer faster and serve more users concurrently. Model quality comes from architecture, data, and training compute. Bandwidth is a throughput and cost property, not a capability one.
How does this affect what I pay?
Directly. Because bandwidth limits how many requests one chip can serve at once, it sets the denominator on inference cost. It is also why output tokens are priced higher than input tokens: output comes from the memory-bound decode phase, input from the parallel prefill phase.
What is the roofline model?
A one-chart way to see whether a workload is limited by a chip's arithmetic or its memory. Plot achievable throughput against arithmetic intensity — operations performed per byte fetched — and the chip traces a slanted roof: a rising bandwidth-limited slope that flattens into a compute ceiling. Low-intensity work like single-stream decode sits far out on the slope.
What is arithmetic intensity?
The number of operations a workload performs per byte it moves from memory. Single-stream text generation manages only about two — each weight is fetched, used once, and discarded — while an H100 needs roughly 300 to keep its cores fed. That gap, not slow arithmetic, is why generation idles compute.
Can software overcome a memory bandwidth limit?
It can raise the work done per byte, which amounts to the same thing. Batching shares one weight read across many users, quantisation shrinks each weight, mixture-of-experts reads fewer weights per token, and speculative decoding extracts several tokens per read. What software cannot do is move bytes faster than the HBM allows — the physical ceiling stands.