Definition: HBM (high-bandwidth memory) is DRAM stacked vertically and placed directly beside the processor on the same package, connected by thousands of microscopic vertical channels — a design that trades manufacturing complexity for several times the bandwidth of conventional memory.
TL;DR: HBM is the memory that sits next to an AI accelerator instead of across a circuit board. It exists because AI generation is limited by memory bandwidth rather than arithmetic, so how much HBM a chip has and how fast it runs decides how many users that chip can serve. Build an app free →
Skyscrapers Instead of Suburbs
Conventional memory lies flat on a board a few inches from the processor. Every byte makes a comparatively long trip.
HBM does two things differently. It stacks memory dies vertically, eight or twelve high, and drills thousands of vertical connections straight through the silicon. Then it puts that stack on the same package as the processor, millimetres away rather than inches.
The result is a very wide, very short path — a skyscraper of memory downtown instead of suburbs of memory across a highway.
Distance and width are the whole design. HBM is closer and much wider than conventional memory.
Why AI Specifically Needs It
Because inference is memory-bound. Generating a single token requires reading the model's parameters out of memory, so a chip's usable throughput is set by how fast it can be fed rather than how fast it can calculate.
Two things compete for that memory:
- Model weights, which must be read for every token generated.
- The KV cache, the conversation's working state, which grows with context length and on long conversations can occupy more memory than the weights themselves.
That competition is the concrete reason a longer context window costs more and why serving capacity is quoted in concurrent requests rather than raw teraflops.
| Property | HBM | Conventional DRAM |
|---|---|---|
| Physical position | On the processor package | On the board |
| Arrangement | Stacked vertically | Flat |
| Interface width | Very wide | Narrow |
| Bandwidth | ~TB/s | ~100 GB/s |
| Relative cost per byte | Much higher | Lower |
| Typical use | Accelerator working memory | Host system memory |
Why Capacity per Stack Caps Model Size per GPU
An accelerator's memory ceiling is set by two multiplied numbers: how much fits in one HBM stack, and how many stacks fit around the die. Package real estate allows roughly eight stacks beside a flagship processor, so the per-stack capacity of the current memory generation directly decides the largest model one chip can hold.
| Generation | Capacity per stack | Bandwidth per stack | Eight-stack ceiling |
|---|---|---|---|
| HBM2e | 8–16 GB | ~0.4 TB/s | ~128 GB |
| HBM3 | 16–24 GB | ~0.8 TB/s | ~192 GB |
| HBM3e | 24–36 GB | ~1.2 TB/s | ~288 GB |
| HBM4 | 48–64 GB | ~2 TB/s+ | ~512 GB |
HBM3e ships at 24 GB for an 8-high stack and 36 GB for a 12-high one (Micron); a B200-class part carries 192 GB across its stacks with roughly 8 TB/s of aggregate bandwidth. Run the model math against that ceiling: a 70-billion-parameter model needs ~140 GB at 16-bit — fitting only with quantisation or by leaving no room for the KV cache — while a 400-billion-parameter model at any useful precision cannot fit on one chip at all. It must be sliced across many, and every generated token then pays interconnect traffic between them. Each HBM generation quietly raises the size of model that can be served simply, which is why memory roadmaps and model roadmaps move together.
A Short History: A Graphics Part That Became the Bottleneck of AI
HBM was standardized by JEDEC in 2013 and first shipped on a consumer graphics card, AMD's Fiji GPU, in 2015 — a niche, expensive answer to GDDR's bandwidth plateau. NVIDIA's datacenter parts adopted it from the P100 (2016) onward, and the transformer era turned it from packaging curiosity into the industry's scarcest commodity: by the mid-2020s HBM had become the dominant profit driver for the three memory makers able to produce it, with supply allocated a year or more ahead. The lesson compounds through this whole category — in AI infrastructure, the bottleneck is rarely the processor; it is feeding the processor.
Why It Constrains the Whole Industry
HBM is difficult to manufacture — stacking and bonding dies at scale has low tolerance for defects — and only a small number of manufacturers can produce it at volume. Capacity is typically allocated well in advance.
The practical consequence for anyone building on AI is indirect but real: memory supply is one of the constraints on how much serving capacity exists, which feeds into availability and pricing of the models you call.
On a hosted platform the constraint reaches you only as economics you do not manage. When your AI agents hold long context or your automations fan out across many runs, you are consuming slices of this memory by the request — metered in credits rather than in stack allocations, with the capacity planning absorbed by the platform. The full path from a memory stack to your answer is traced in how AI data centers work.
Related Concepts
- Memory Bandwidth — what HBM is bought for
- KV Cache — the other thing occupying it
- GPU — what it sits beside
- Data Center — the building the packages fill
- Tokens Per Second — what the bandwidth becomes
- Inference · Inference Cost
- Model Parameters · Context Window
Frequently Asked Questions About HBM
What does HBM stand for?
High-bandwidth memory. It is a memory standard defined by stacking DRAM dies vertically and connecting them to a processor on the same package through a very wide interface, rather than placing memory chips flat on a circuit board.
Why not just use more regular memory?
Because the limit is bandwidth, not capacity. Conventional memory can be added in quantity but connects through a comparatively narrow interface. AI generation needs to move enormous volumes of data per second, and width plus proximity is what delivers that.
Does HBM make AI models better?
No. It makes a given model faster to serve and lets one chip handle more concurrent users. Capability comes from architecture, data, and training. HBM is a throughput and economics property.
How does HBM affect what I pay for AI?
It sets how many requests a single accelerator can serve at once, which is the denominator underneath per-token pricing. It also explains the pricing asymmetry between input and output tokens, since output comes from the memory-bound generation phase.
How much HBM does an AI GPU have?
Flagship accelerators in 2026 carry between 80 GB and roughly 300 GB: an H100 has 80 GB of HBM3, a B200-class part 192 GB of HBM3e. The number matters twice — it caps the model size one chip can hold, and whatever is left over after weights becomes room for KV caches, which is serving capacity.
Who makes HBM?
Only three companies produce it at volume: SK hynix, Samsung, and Micron. The stacking and through-silicon-via bonding are hard enough that no other supplier has qualified at scale, which is why HBM capacity is allocated to chip makers a year or more in advance and why memory supply shapes GPU availability.
Is HBM why AI GPUs cost so much?
It is a large part. The memory stacks and the advanced packaging that bonds them beside the die are among the most expensive components on the board, cost far more per byte than conventional DRAM, and come from a three-supplier market running at capacity. A meaningful share of a GPU's price is really a memory price.