Definition: Inference cost is what it actually costs to run one AI request — billed per token sent in and per token generated out, with output tokens typically priced several times higher than input tokens because they come from a slower, harder-to-batch phase of the work.
TL;DR: Inference cost is the recurring expense of any AI product, and unlike training it never ends. Production inference averaged roughly $0.77 per million tokens across providers in April 2026, with budget tiers from about $0.075 per million input tokens. Output is priced 4–6× input. The right unit to manage is cost per completed task, not cost per token. Build an app free →
Why Output Costs More Than Input
The asymmetry surprises people, and it comes directly from hardware.
Inference has two phases. Prefill reads your entire prompt at once, in parallel, saturating the accelerator's arithmetic units — efficient work that batches well. Decode produces one token per pass and is limited by memory bandwidth rather than arithmetic, so it is far harder to amortise across users.
| Prefill (input) | Decode (output) | |
|---|---|---|
| What it does | Reads the prompt | Writes the answer |
| Parallelism | High | One token at a time |
| Bound by | Compute | Memory bandwidth |
| Batches well | Yes | Less well |
| Relative price | 1× | 4–6× |
The self-loop is the expensive part, and it runs once per word of the answer.
What Actually Drives the Bill
Five variables, roughly in order of impact:
- Model choice. The spread between budget and frontier tiers is more than an order of magnitude. Model routing exists to spend frontier capability only where it changes the answer.
- Context length. Long prompts multiply prefill work and inflate the KV cache, which raises cost for every subsequent token in the conversation.
- Output length. Priced at the higher rate, and directly controllable by asking for less.
- Number of calls. Agentic loops turn one user instruction into many model calls. This is usually the dominant term and the one people forget to count.
- Cache hit rate. Prompt caching lets a repeated preamble skip prefill.
A Worked Example: 100 Users on One H100
Abstract per-token prices become concrete the moment you price a single GPU. Here is a deliberately round back-of-envelope for serving a 70-billion-parameter-class model to 100 concurrent users on one NVIDIA H100 — every input is stated, so you can swap in your own.
| Assumption | Value | Basis |
|---|---|---|
| GPU rental | $2.50/hour | On-demand H100 rates clustered around $2–3/hr on specialized clouds in 2026 (IntuitionLabs pricing survey) |
| Model weights | ~70 GB at 8-bit | One byte per parameter |
| Memory bandwidth | 3.35 TB/s | H100 SXM published spec |
| Aggregate output | ~1,500 tokens/s | Batched decode, realistic mid-range |
| Concurrent users | 100 | Each sees ~15 tokens/s |
The chain: decode is memory-bound, so the chip can read its 70 GB of weights at most 3.35 TB/s ÷ 70 GB ≈ 48 times per second. Batching lets one read serve every user at once, which is how 100 streams share one chip. At ~1,500 aggregate tokens per second the card produces about 5.4 million output tokens per hour, so the raw hardware cost is $2.50 ÷ 5.4M ≈ $0.46 per million output tokens — under the $0.77 cross-provider average, before power, networking, idle capacity, and margin close the gap.
The catch is memory. One hundred conversations also means one hundred KV caches competing with the weights for the same 80 GB of HBM. Give each user a long context and the batch no longer fits, concurrency drops, and cost per user rises — which is the physical mechanism behind long-context pricing.
Common Misconceptions
- "Token prices are falling, so my bill will fall." Per-token prices fall; tokens per finished task rise, because reasoning models think longer and agent loops multiply calls. Bills routinely go up while unit prices go down.
- "A bigger context window is free capacity." Every token you carry is billed on every subsequent call and enlarges the KV cache. Unused context is a recurring cost, not idle headroom.
- "Speed and cost are separate problems." They are the same problem. Tokens per second and cost per token are both set by how many bytes move through memory, which is why the fixes below improve both at once.
Cost Per Token Is the Wrong Unit
The most useful framing is cost per completed task, not cost per token. Per-token prices have fallen steadily while tokens consumed per finished piece of work have risen — because reasoning models think longer and agents loop.
A per-token dashboard can look flat while spend rises, because it misses the growing call count. Anyone managing an AI budget should measure the task.
Two industry-level facts worth knowing when reading pricing claims:
- Estimates of how fast cost per fixed quality is falling range from roughly 50× to 200× per year depending on the benchmark and accounting basis. Any single figure is a choice, not a fact.
- Depreciation schedules are a large swing factor in the economics underneath published prices — every year of assumed useful life added or removed moves reported costs substantially, and the industry has not converged on four, five, or six years.
How to Reduce It
- Right-size the model per request rather than defaulting to the largest.
- Cache stable context and put it at the front of the prompt.
- Bound the loop. Cap agent iterations; most runaway bills are loop-count problems. In Taskade, automations run as bounded workflows with a defined trigger and end, which makes spend predictable in a way an open-ended chat is not.
- Ask for less output. Output is the expensive half.
- Summarise history instead of resending full transcripts. Pointing an AI agent at a maintained project brief instead of a growing chat log does this structurally — see attaching knowledge to an agent.
- Move work off-peak where latency does not matter. A digest that runs overnight has no reason to pay interactive-tier rates or compete for interactive capacity.
Related Concepts
- Inference — the process being priced
- Token · Tokenizer — the billing unit
- KV Cache · Prompt Caching
- Context Window · Test-Time Compute
- Memory Bandwidth — the hardware reason for the pricing shape
- Tokens Per Second — the speed side of the same physics
- GPU · Data Center — the assets being amortised
- Model Routing · Model Access
Frequently Asked Questions About Inference Cost
Why are output tokens more expensive than input tokens?
Because they come from the decode phase, which produces one token per pass and is limited by memory bandwidth rather than arithmetic. Input tokens come from prefill, which processes the whole prompt in one parallel pass and batches efficiently across users. The typical ratio is four to six times.
What does AI inference cost per million tokens?
Production inference averaged roughly $0.77 per million tokens across providers in April 2026, with budget tiers starting near $0.075 per million input tokens and frontier tiers well above the average. Prices move frequently, so treat any figure as dated.
Is inference more expensive than training?
For most products, yes, in aggregate. Training a model is a large one-time expense; inference recurs on every request forever. As usage grows, inference dominates total spend even though a single training run is a much larger single number.
How do I estimate the cost of an AI agent workflow?
Count calls before tokens. Take one representative run, multiply the number of model calls by the average input and output tokens per call, and price input and output separately at the model's rates. Agent loops are usually dominated by call count, not by any single long prompt, so capping iterations is the highest-leverage estimate input.
Why did my AI bill go up when token prices went down?
Because spend is price times volume, and volume is rising faster than price is falling. Reasoning models generate long internal deliberation, and agentic workflows turn one instruction into many calls. A per-token dashboard hides this; a cost-per-completed-task view exposes it immediately.
What is the cheapest way to run AI at scale?
Batch everything that is not interactive. One read of the model's weights can serve an entire batch of requests, so aggregate throughput per dollar rises steeply with batch size — the mechanism in the worked example above. Providers pass this through as discounted batch tiers for work that can wait minutes or hours.
How does Taskade handle inference cost?
Taskade routes each request across 15+ frontier models from OpenAI, Anthropic, Google, and open-weight providers, applies caching transparently, and bills in normalized credits rather than per-model token prices — so model selection and price differences are absorbed by the platform rather than managed per request. Automations keep agent runs bounded, which keeps spend bounded too.