Definition: The KV cache is the stored attention keys and values for every token a model has already processed, held in fast memory beside the accelerator so that generating each new token only requires computing that one token's contribution rather than re-reading the entire conversation.
TL;DR: The KV cache is a model's working memory for the current conversation. Without it, generating token 500 would mean re-processing tokens 1–499 from scratch; with it, each new token costs work proportional to the context length instead of a full re-read of the prefix. It lives in the same scarce memory as the model weights, which is why long context costs more. Build an app free →
What Problem It Solves
The attention mechanism works by comparing each new token against every previous one. Done naively, producing the 500th token would mean recomputing the representation of all 499 tokens before it — and the 501st would redo all 500. Generation would get slower with every word.
The KV cache stores those intermediate representations once. Each new token computes only its own keys and values, then reads the cached ones for everything before it.
Each new token adds one entry and reads the rest — per-token work stays proportional to the context length instead of re-running the whole prefix through the model.
Where It Lives, and Why That Costs Money
The cache sits in the accelerator's high-bandwidth memory — the same scarce, expensive memory holding the model's weights. It has two properties that drive AI economics:
- It grows with conversation length. Every token added to the context window adds an entry.
- On long conversations it can occupy more memory than the model itself.
Because that memory is finite, the cache directly limits how many conversations one accelerator can hold simultaneously. Fewer concurrent conversations means higher cost per conversation. This is the concrete mechanism behind "long context is expensive" — you are renting fast memory for the duration of the exchange.
The Memory Math for a 128k-Context Request
The cache's size is plain arithmetic. Per token, a model stores two vectors — one key, one value — for each of its layers and each of its KV heads:
bytes per token = 2 × layers × KV heads × head dimension × bytes per value
Take a 70-billion-parameter-class model with grouped-query attention: 80 layers, 8 KV heads, head dimension 128, 16-bit values. That is 2 × 80 × 8 × 128 × 2 ≈ 320 KB per token. Now scale by context:
| Context length | KV cache size | Share of an 80 GB accelerator |
|---|---|---|
| 4,000 tokens | ~1.3 GB | ~2% |
| 32,000 tokens | ~10 GB | ~13% |
| 128,000 tokens | ~41 GB | ~51% |
One maxed-out 128k request occupies half the GPU's memory before the ~70 GB of 8-bit weights are even counted — so it simply does not fit alongside them on a single 80 GB card, and it evicts dozens of short conversations that could have shared the hardware. That is the entire economics of long context in one row of a table.
Two design responses follow directly. Grouped-query attention exists precisely to shrink this: with classic multi-head attention (64 KV heads instead of 8) the same request would need eight times the memory. And serving systems now treat cache memory the way operating systems treat RAM — vLLM's PagedAttention (2023) allocates it in small pages instead of contiguous blocks, which cut the fragmentation waste that used to leave much of the cache space unusable.
The Two Phases It Connects
| Phase | Relationship to the cache | Bound by |
|---|---|---|
| Prefill | Builds the cache from your prompt, in parallel | Compute |
| Decode | Reads the cache, appends one entry per token | Memory bandwidth |
This is also why prompt caching is such a large lever. If the same long preamble is sent repeatedly, its cache entries can be retained and reused, letting the request skip prefill over that portion entirely — saving both the latency and the compute.
What This Means in Practice
Three practical consequences for anyone building on AI:
- Front-load stable context. Put unchanging instructions and documents at the beginning of a prompt, where they can be cached, and put the variable part at the end.
- Long conversations get more expensive per turn, not less. The cache carries forward, so each turn reads a larger cache than the last — and the quality often degrades at the same time, which is context rot. Cost up, accuracy down is the signature of a conversation that should have been reset.
- Trimming history is a real cost lever. Summarising an old conversation rather than resending it verbatim reduces cache size directly.
This is why architectures that keep durable facts outside the transcript win twice. A Taskade AI agent reads the project it is pointed at fresh on each request instead of dragging a growing history, and every automation run starts a new, short-context session on trigger — so the cache stays small whether it is the first run or the thousandth. The step-by-step version of that setup is in attaching knowledge to an agent.
Related Concepts
- Attention Mechanism — what the cache stores the output of
- Context Window — the budget the cache fills
- Prompt Caching — reusing cache across requests
- Memory Bandwidth · HBM
- Inference · Inference Cost
- Tokens Per Second — the speed the cache read sets
- Context Rot — the quality cost of the same growth
- Transformer
Frequently Asked Questions About the KV Cache
What does KV stand for?
Keys and values — two of the three components of the attention mechanism, alongside queries. Keys and values are cacheable because they describe tokens already processed and do not change as generation continues. Queries are computed fresh for each new token.
Why does the KV cache make long context expensive?
Because it lives in the accelerator's limited high-bandwidth memory and grows with every token in the conversation. A larger cache means fewer conversations fit on one chip at the same time, and more bytes must be read per generated token. Both effects raise cost per request.
Is the KV cache the same as prompt caching?
No, though they are related. The KV cache is the in-memory state for a single request. Prompt caching is a provider feature that retains part of that state across requests so a repeated preamble does not need reprocessing. Prompt caching is a way of reusing KV cache work.
Does clearing conversation history reduce cost?
Yes. A shorter conversation means a smaller cache, fewer bytes read per token, and more room for concurrent requests. Summarising older turns instead of resending them verbatim is one of the most effective cost levers available at the application layer.
How big does a KV cache get?
For a 70-billion-parameter-class model with grouped-query attention, roughly 320 KB per token — about 1.3 GB at a 4k context and about 41 GB at 128k, which is half of an 80 GB accelerator for a single request. Models without grouped-query attention store several times more per token.
Can the KV cache be compressed?
Yes, and it is an active engineering front. Grouped-query and multi-query attention shrink it architecturally, quantising cached values to 8 or 4 bits halves or quarters it, paged allocation (vLLM's PagedAttention) eliminates fragmentation waste, and some systems evict or summarise entries for old tokens. Each trades a little fidelity or complexity for concurrency.
Why do providers charge more for long context?
Because the cache is the marginal cost. A long-context request holds tens of gigabytes of fast memory for its whole duration, crowding out other users, and every generated token must read that larger cache, consuming more memory bandwidth. Long-context price tiers pass those two physical costs through.