Definition: Tokens per second (TPS) is the rate at which a model streams its answer out — the throughput half of perceived AI speed, distinct from how long you wait before the first word appears.
TL;DR: Tokens per second measures generation speed after the reply starts. Below roughly 40 TPS a stream reads sluggishly; comfortable reading speed sits well under that, so faster mainly helps long answers and agent loops. TPS is set by memory bandwidth, model size, and batch size — not by how "smart" the model is. Build an app free →
Two Different Speeds
People say "fast AI" and mean two unrelated measurements. Separating them is the first step to reasoning about latency at all.
| Metric | What it measures | Set by | What it feels like |
|---|---|---|---|
| Time to first token (TTFT) | Delay before the first word | Prefill — reading your prompt | The pause after you hit enter |
| Tokens per second (TPS) | Rate of the stream after that | Decode — memory bandwidth | How fast text scrolls |
A long prompt hurts TTFT. A long answer hurts total time via TPS. Optimising the wrong one is a common mistake: attaching a large document slows the pause, not the stream.
Two clocks, two causes. The first is about what you sent; the second is about what you get back.
What Counts as Fast
For a human reading a reply, the useful threshold is lower than people assume — comfortable reading speed is only a handful of tokens per second, so anything above roughly 40 TPS reads as instant.
Where higher TPS genuinely matters is where no human is reading in real time:
- Long-form generation, where total time is dominated by output length.
- Agentic loops, where one instruction triggers many model calls in sequence and each one's latency compounds.
- Reasoning models, which generate substantial internal reasoning before the visible answer.
The compounding is worth making concrete. A five-step agent chain in which each step produces 600 tokens spends 75 seconds in pure decode at 40 TPS, but 25 seconds at 120 TPS — the same work, the same model calls, 50 seconds apart in feel. No single step looked slow; the chain was.
Interactive vs Batch: Two Different Targets
"How fast is fast enough?" has two answers, because there are two kinds of consumer of the stream — a person watching it, and a pipeline waiting on it.
| Workload | What matters | Practical target |
|---|---|---|
| Chat, a human reading | Perceived smoothness | ~20–50 TPS per user, plus a short first-token wait |
| Voice assistants | The pause before speech | Time-to-first-token above all; TPS barely matters |
| Agent chains and reasoning | Total wall-clock across many calls | As high as the hardware allows |
| Batch jobs (digests, backfills, evals) | Cost per token only | None — overnight is fine, so providers sell discounted batch tiers |
The two targets pull against each other on shared hardware: packing more users into a batch raises the total tokens a chip produces while lowering each individual stream. Provider-scale numbers make the gap vivid — Microsoft quotes about 865,000 tokens per second for a single rack in its Fairwater AI datacenters, an aggregate figure thousands of interactive streams wide, not a speed any one user sees.
The practical takeaway: match the work to the tier. Anything a person is not watching — a nightly digest, a weekly report, a data backfill — belongs in scheduled automations rather than an interactive session, where it costs less and competes with nobody.
What Sets the Number
Four things, and only one of them is about the model's capability:
- Memory bandwidth. Every token requires reading the model's weights out of memory. This is the hard ceiling.
- Model size. More parameters means more bytes read per token, so larger models generate more slowly at equal hardware.
- Batch size. Serving many users at once raises total throughput but can lower any single user's TPS — a genuine tension between cost efficiency and individual latency.
- Context length. A larger KV cache means more state read per token, so long conversations stream more slowly than short ones.
Architectural choices push against this. Mixture-of-experts models activate only part of the network per token, reducing bytes read. Speculative decoding drafts several tokens with a small model and verifies them with the large one, effectively producing more than one token per expensive pass.
Reading Published Benchmarks Carefully
TPS figures are only comparable when three things are stated: batch size, context length, and hardware. A number measured at batch size 1 on an empty context is a best case that no production system operates in. If a benchmark omits those, it is a marketing figure rather than a measurement.
There is also a hard physical ceiling to sanity-check any claim against. Single-stream generation cannot read weights faster than the memory allows: a 70 GB set of 8-bit weights over an H100's 3.35 TB/s of bandwidth caps one stream near 48 tokens per second, no matter the software. Claims far above that for a dense model of that size are describing batched aggregate throughput, speculative decoding, a mixture-of-experts architecture reading fewer bytes per token — or a smaller model than implied.
Where TPS Actually Reaches You
If you use a hosted platform, you never pick a TPS figure — you feel the routing decisions made on your behalf. Taskade routes each request across 15+ frontier models so that quick interactive steps go to fast models and heavyweight reasoning goes to capable ones, and AI agents running multi-step work benefit most, since their latency compounds per call. Work with no human watching runs as automations on triggers and schedules, where throughput is invisible and only the outcome matters. Choosing depth per task is covered in thinking modes.
Related Concepts
- Memory Bandwidth — the ceiling
- GPU · HBM — the hardware behind the ceiling
- Inference — where TTFT and TPS come from
- KV Cache · Context Window
- Test-Time Compute · Reasoning Effort
- Inference Cost · Model Routing
Frequently Asked Questions About Tokens Per Second
What is a good tokens-per-second rate?
For a person reading a reply, anything above roughly 40 tokens per second reads as instant, because comfortable human reading speed is well below that. Higher rates matter most for long outputs, agent loops, and reasoning models, where total generation time rather than perceived streaming speed is what you are waiting on.
What is the difference between tokens per second and time to first token?
Time to first token is the delay before the reply starts, set by the prefill phase reading your prompt. Tokens per second is the rate of the stream afterwards, set by the decode phase. A long prompt slows the first; a long answer is governed by the second.
Why do larger models generate more slowly?
Because generating each token requires reading the model's parameters out of memory, and more parameters means more bytes to read. Generation speed tracks memory bandwidth divided by model size far more closely than it tracks raw arithmetic throughput.
Does a higher tokens-per-second rate mean better answers?
No. It is a throughput property of the hardware and model size, not a measure of quality. A smaller model will usually stream faster than a larger one while producing less capable output, which is exactly the trade-off model routing is designed to make per request.
Why does my AI stream more slowly during busy hours?
Because you share hardware with everyone else. Serving systems batch concurrent requests to keep the chip efficient, and a fuller batch divides the same memory bandwidth across more streams — total throughput rises while each individual stream slows. Your slower afternoon stream is the provider's cost efficiency working as designed.
What is batch inference?
Running requests that nobody is waiting on — digests, backfills, evaluations — grouped together for maximum hardware efficiency instead of minimum latency. Providers typically discount it heavily, often around half the interactive price, because dense batches extract far more tokens per GPU-hour than latency-sensitive traffic can.
Does streaming make AI generate faster?
No. Streaming shows tokens as they are produced instead of waiting for the full answer, so the wait feels shorter, but the generation rate is identical. What streaming genuinely improves is time-to-usefulness: you can start reading, and often stop the generation early, which does save real time and tokens.