AI Inference Explained (2026): How Every AI Answer Is Made and Priced

Every time you ask an AI a question, a machine in a data center does a few billion multiplications to write your answer, one small piece of a word at a time. That work has a name: inference. It is the step you pay for every day, long after a model has finished training.

Most people meet inference only as a price on a pricing page ("$0.50 per million output tokens") or as a pause before a chatbot starts to type. Both come from the same physics. This guide follows one request from your prompt to a GPU and back, then explains why the price of that work keeps falling, who sells it, and what it means for a team that has to pick AI tools and pay for them.

TL;DR: AI inference is the work a model does each time you use it: prefill reads your prompt, then decode writes the answer token by token. Cached input can cost 10 to 30 times less than fresh input, but cost per task decides your bill. Taskade runs no GPUs. Pro and up can bring their own key. See the models →

Quick answer: what is AI inference? AI inference is the step where a trained model answers new input. For a large language model, LLM inference runs in two phases: prefill reads the whole prompt at once, and decode writes the reply one token at a time while a KV cache keeps the work already done. Training happens once. Inference runs on every request, so it sets the speed and the cost of every AI tool.

In this guide: What inference is · Inference vs training · Types and places · Timeline · The request journey · Token by token · Prefill and decode · KV cache · Cost anatomy · Optimization · 2026 changes · Batching · Routing · Falling prices · Serving a 2.8T model · Who sells it · Open vs frontier · Choosing tools · BYOK · Glossary

🧠 What Is AI Inference?

AI inference is the act of running a trained AI model on new input to produce an output. For a language model, the input is your prompt and the output is text, written one token at a time. Every chatbot reply, every AI agent step and every AI automation is one or more inference calls, and each call is billed by the tokens it reads and writes.

A model has two lives. In training, engineers show it huge amounts of text and adjust its billions of weights until it predicts the next token well. That job runs once per model version and can take months. In inference, the finished model sits in GPU memory and answers requests, in seconds, all day. The inference wiki entry covers the formal definition. The table below covers the practical one.

The short version: training decides how smart a model is, and inference decides how fast and how cheap each answer is. Because inference repeats for every use, it is the cost that grows with your team's adoption. The inference cost wiki entry and our guide to AI cost per task go deeper on the bill itself.

🧩 AI Inference Examples You Already Use

Every AI feature you touch in a day is an inference call, from a spam filter to a chat reply. Image recognition and fraud scores make one prediction per input. A large language model makes one prediction per token, so a single answer is hundreds of inference steps in a row.

The last two rows are where most teams meet inference at work. An AI agent runs many inference calls per task, and an automation runs them without anyone watching, so the cost per task matters more than the speed of one call. IBM Technology's short explainer covers the general idea, from training data to a live prediction:

🏋️ Inference vs Training: How Do Compute, Cost and Hardware Differ?

Inference vs training comes down to one big job against millions of small ones. Training a model runs once on a huge cluster for weeks. Inference runs a forward pass for every token of every answer, for as long as the model is in use. A common rule of thumb from the 2020 OpenAI scaling-laws paper puts training at about 6 × N × D operations and inference at about 2 × N per token, where N is the parameter count and D the number of training tokens.

The rule makes the trade-off easy to compute. Take a dense 70-billion-parameter model trained on 15 trillion tokens:

TRAINING VS INFERENCE COMPUTE  (rule of thumb, dense model)
N = 70 billion parameters        D = 15 trillion training tokens
training  ≈ 6 × N × D  = 6 × 70e9 × 15e12   ≈ 6.3 × 10^24 operations, once
inference ≈ 2 × N      = 2 × 70e9           ≈ 1.4 × 10^11 operations per token
break-even = training ÷ inference per token
= 6.3e24 ÷ 1.4e11                ≈ 45 trillion tokens  (= 3 × D)
at   1 trillion tokens served per day  →  inference matches training in ~45 days
at 100 billion tokens served per day   →  ~450 days
For a mixture-of-experts model, use the ACTIVE parameters for the
per-token cost: Kimi K3 reads ~104B of its ~2.8T parameters per token.

The chart shows the same arithmetic over time. The flat line is the one-time training run. The bars are inference compute added up day by day at 1 trillion tokens a day.

Real fleets point the same way. Google researchers led by David Patterson reported in 2022 that inference used about three fifths of Google's machine learning energy, and training about two fifths, across sample weeks in 2019, 2020 and 2021. Reasoning models push the share further toward inference, because they spend extra test-time compute on hidden tokens before they answer.

The model training wiki entry covers the training side, and the scaling laws entry explains where the 6 × N × D rule comes from. The difference matters to a buyer for one reason: a lab pays for training once, but your team pays for inference on every use, through credits, a subscription or a provider bill.

🗺️ What Are the Types of AI Inference, and Where Does It Run?

AI inference comes in three delivery types and runs in four kinds of place. Real-time inference answers one request while a person waits. Streaming inference shows the answer as the model writes it. Batch inference processes many requests later, often at half the price. The place, from a cloud data center to a phone, sets the speed, the privacy and the bill.

The batch discount is written into both labs' documentation. OpenAI's Batch API guide states a 50% discount against its synchronous API and a 24-hour completion window. Anthropic's Message Batches documentation states the same 50% and says most batches finish in less than an hour. A background job that nobody watches is the natural place to use it.

IBM's overview of AI inference lists the same five chip families. For almost every team that buys AI tools, inference means the cloud row and the GPU row, which is the case the rest of this guide follows. Our guide to how AI data centers work covers the buildings those GPUs sit in.

📅 AI Inference Timeline: Key Dates From 2017 to 2026

Modern AI inference took shape in under a decade, from the transformer paper in 2017 to a 2026 market where single providers serve tens of trillions of tokens a day. Most of the big cost drops came from serving software, not only from new chips. Continuous batching, paged KV memory and prefill/decode separation each changed what a GPU-hour can produce.

Two cautions about this table. First, a falling price per token is not the same as a falling bill. The section on cost per task explains why. Second, the 40 trillion figure is the company's own statement, reported by CNBC in July 2026, and no third party has measured it.

🛤️ How Does an AI Request Travel From Your Prompt to a GPU?

An AI request passes through six stops: your app, a provider's API, a router, a queue, a GPU that runs prefill and decode, and a stream back to your screen. The first word you see marks the end of prefill. Every word after it is one turn of the decode loop. Most of what you pay for happens inside the GPU stop.

Here is the whole journey in one diagram. Read it top to bottom.

YOUR SCREEN
   │  "Summarize this contract in five bullets"  (+ the contract, + app instructions)
   ▼
APP (for example, an AI workspace)
   │  adds system instructions, picks a model, sends an API call
   ▼
PROVIDER API ──────────────► bills you per million tokens in and out
   │
   ▼
ROUTER ── which copy of the model has room, and which one already
   │      holds this prompt's start in its KV cache?
   ▼
QUEUE ── waits for a free slot in the next batch
   │
   ▼
GPU NODE (weights already loaded in high-bandwidth memory)
   │  1. TOKENIZE   text  ──►  token IDs
   │  2. PREFILL    read every prompt token at once     (compute-bound)
   │  3. KV CACHE   store keys and values for each token (memory)
   │  4. DECODE     write 1 token, append, repeat        (memory-bound)
   │       └─ loop runs once per output token
   ▼
STREAM ── tokens turn back into text and appear on screen
             wait before the first word  = TTFT  (time to first token)
             speed after the first word  = tokens per second

The same flow as a timed exchange shows where each wait comes from.

Two facts fall out of this picture. A long prompt makes you wait longer before the first word, because prefill must read all of it. A long answer makes the request take longer and cost more, because decode runs once per output token. That is why a pricing page lists input and output tokens separately.

For the hardware under the GPU stop, our guide to how AI data centers work follows the power from the grid to the rack. For the deep serving mechanics, including the break-even math a provider runs per GPU node, read the companion guide Inference-as-a-Service (IaaS).

The machines at the GPU stop: NVIDIA H100 accelerators. The memory stacked next to each chip, more than the math units, decides how many people one card can serve at once. Photo: Wikimedia Commons / 极客湾Geekerwan / CC BY 3.0

⏱️ How Does LLM Inference Work? One Request, Token by Token

LLM inference works as a short prefill followed by a long loop: the model reads the whole prompt once, then repeats one decode step per output token until it writes a stop token. To make that concrete, this section follows one request with real timings. The numbers come from the Lambda benchmark cited below: an 8-GPU NVIDIA HGX B200 node with vLLM, 32 users at once, 8,192 input tokens per request, a mean 1,186 ms to the first token and 28.50 tokens per second per user.

The request: an operations lead pastes a contract and asks for five bullets. With the app's instructions and the contract, the prompt is 8,192 tokens. The answer will be 400 tokens. One decode step at 28.50 tokens per second takes about 35 milliseconds.

ONE REQUEST, TOKEN BY TOKEN   (Lambda B200 benchmark rates, 32 users per node)
t = 0.000 s   send 8,192-token prompt
t = 1.186 s   token   1  appears          ← TTFT: queue + prefill
t = 1.221 s   token   2                   ← every later token ≈ 35 ms apart
t = 1.502 s   token  10
t = 4.661 s   token 100
t = 8.171 s   token 200
t = 15.19 s   token 400  + end token      ← end-to-end latency
time spent before the first word   ≈  1.2 s   ( 8% of the request)
time spent writing the answer      ≈ 14.0 s   (92% of the request)

The split surprises most people. The prompt was 20 times longer than the answer, yet 92% of the clock time went to writing the answer, because prefill handles thousands of tokens in one parallel pass and decode handles one token per pass. The cumulative clock looks like this:

🔬 What Happens Inside One Decode Step?

One decode step pushes a single new token through every layer of the model, then turns the last layer's output into a probability for every token in the vocabulary. In each layer, the attention part compares the new token with every earlier token through the KV cache, and the feed-forward part (or the chosen experts, in a mixture-of-experts model) transforms it. At the end, a sampler picks the next token, guided by settings such as temperature.

Three details explain behavior people notice. First, the model never "plans" the whole answer in one step: it commits to one token at a time, which is why next-token prediction is the right mental model. Second, the sampler is why the same prompt can give different answers: at a temperature above zero, it draws from the probabilities instead of always taking the top one. Third, a tokenizer splits text into sub-word pieces, so "tokens per second" is not "words per second": an English word averages about 1.3 tokens.

Grant Sanderson of 3Blue1Brown walks through attention, the part of each step that reads the KV cache, with the clearest visuals available:

📏 The Five Numbers That Describe Inference Speed

Inference speed is described by five numbers, and each one answers a different question. The worked request above produces all five. Vendors quote different ones, so it pays to know which number a benchmark reports before you compare two hosts.

NVIDIA's inference glossary defines goodput as throughput measured while the system keeps its TTFT and TPOT targets. A provider can always raise raw throughput by packing more users into a batch, so goodput counts only the tokens that arrived fast enough to be useful.

⚡ What Are Prefill and Decode?

Prefill and decode are the two phases of every AI answer. Prefill reads the whole prompt in one parallel pass and is limited by compute. Decode writes the answer one token at a time and is limited by memory bandwidth, because each new token needs another read of the model's weights. The pause before the first word is prefill. The steady typing after it is decode.

An analogy helps. Prefill is a reader who scans a whole page at once. Decode is a writer who must reopen a heavy reference book before every single word. The writer is slow, not because thinking is hard, but because fetching the book takes time. On a GPU, "the book" is tens or hundreds of gigabytes of weights in high-bandwidth memory.

A real benchmark shows both numbers at once. Lambda ran DeepSeek-V4-Pro with vLLM on one NVIDIA HGX B200 node (8 GPUs), with 32 requests in parallel, 8,192 input tokens and 2,048 output tokens each. It measured 911.92 output tokens per second in total, 28.50 tokens per second per user, and a mean wait of 1,186 milliseconds before the first token. The same node writes about 32 times faster in total than it does for any one person, because it serves 32 people at once.

That gap between per-user speed and per-server speed is the whole economics of inference in one line. A person feels the per-user number. The provider pays rent with the per-server number. Our tokens per second wiki entry explains both measures and how vendors quote them.

Citation capsule: In Lambda's benchmark of DeepSeek-V4-Pro on one 8-GPU NVIDIA HGX B200 node with vLLM, 32 parallel requests produced 911.92 output tokens per second in total but 28.50 tokens per second for each user, with a 1.19-second mean time to first token. Per-server throughput pays for the hardware, and per-user speed is what a reader sees.

🧮 Why Is Decode Memory-Bound? The Back-of-Envelope Math

Decode is memory-bound because every new token requires reading every active weight from GPU memory, and a GPU can do math far faster than it can move those bytes. A one-line formula gives the ceiling on per-user speed for a single user with nobody else in the batch: memory bandwidth divided by the bytes read per token.

DECODE SPEED CEILING, ONE USER ALONE ON THE GPU
max tokens per second  ≈  memory bandwidth  ÷  bytes of weights read per token
70B dense model in FP8 (1 byte per weight)  →  ~70 GB read per token
  1 × H100   3.35 TB/s  ÷ 70 GB  ≈   48 tokens per second, at best
1 × B300   8.0  TB/s  ÷ 70 GB  ≈  114 tokens per second, at best
the same model in FP4 (0.5 byte per weight)  →  ~35 GB per token
1 × B300   8.0  TB/s  ÷ 35 GB  ≈  228 tokens per second, at best
Real speeds land below the ceiling: the KV cache is read too,
and kernels never reach 100% of peak bandwidth.

The same formula explains why batching is so cheap. With 32 users in a batch, the GPU still reads the 70 GB of weights once per step, but it writes 32 tokens instead of 1. The math units, which sat idle while bytes arrived, finally have work to do. Engineers call this raising the arithmetic intensity: more operations for each byte moved. It is also why HBM, the stacked memory beside each GPU, matters more for inference than raw compute does.

High-bandwidth memory stacks sit right next to the GPU die so that decode can read weights fast. Photo: C. Spille/pcgameshardware.de / Wikimedia Commons / CC BY-SA 4.0

Mark Moyou of NVIDIA explains the same prefill and decode workload, and how batch size and sequence length change it, in this PyTorch conference talk:

🏎️ How Hosts Raise Per-User Speed: Speculative Decoding

Speculative decoding raises per-user speed by letting a small draft model guess several tokens ahead, which the large model then checks in a single pass. Decode is slow because every token needs another read of the weights and the cache from memory. If the large model accepts four guessed tokens in one step, the reader gets four tokens for about the price of one memory read. The answer does not change, because the large model throws away any guess it would not have written itself.

All three methods train the draft on the target model's hidden states. That means a host builds its best speculator after the model is already serving real traffic, which is one reason a host's speed on a new model can keep rising after launch day. Baseten lists all three as speculator options in its day-zero write-up on Kimi K3.

Same weights, different speed. On 2026-10-20, Artificial Analysis listed median output speeds for Kimi K3 on selected hosts. LithosAI's Ultra Chat endpoint led at about 422 tokens per second. Nebius ran about 192, Amazon Bedrock about 142, Fireworks about 112, Baseten about 96, Together AI about 57, and Moonshot's own API about 42. That is a ten-times spread between the fastest host and the lab itself, for identical weights. The chips, the parallelism, the batch sizes and the speculators make the difference. These are live figures that change week to week, so re-check them before you pick a host. Our tokens per second guide explains what each speed feels like to a reader.

🗄️ What Is the KV Cache, and Why Does Caching Cut the Bill?

The KV cache is the stored result of the work a model has already done on the tokens it has read. It keeps the attention "keys" and "values" for each token in GPU memory, so the model does not recompute them for every new word. When a new request starts with exactly the same text as an earlier one, a provider can reuse that cache and charge the repeated input tokens at a deep discount.

Without a KV cache, the model would reread your entire prompt before every word it writes. With it, the model reads the prompt once, then each decode step adds one entry. The KV cache wiki entry shows the mechanics. What matters for cost is the second trick, prompt caching: keeping the cache after a request ends, labeled by the exact text that produced it.

AI tools resend the same opening text constantly. A chat app resends the whole thread on every turn. An agent resends its instructions and tool list on every step. A team assistant opens every conversation with the same company instructions. If the start of the prompt matches, the provider skips most of the prefill. Here is what that discount looked like on public price lists, checked on 2026-10-18:

The provider's side of the same story is just as large. In its Open Source Week write-up, DeepSeek reported that over one 24-hour window, 342 billion of its 608 billion input tokens, or 56.3%, hit its on-disk KV cache. More than half of the reading work was already done before the request arrived.

One rule makes or breaks caching: the match is exact, from the first token. Change one word at the very start of a prompt, such as a timestamp in the instructions, and everything after it becomes fresh input again. Good AI tools keep the stable part of a prompt first and the changing part last. Our guides to prompt caching and how to reduce LLM costs show how to structure prompts so that the cache hits.

📐 KV Cache Explained in Numbers: How Big Does It Get?

A KV cache grows by a fixed number of bytes for every token in the conversation, so its size is a simple product. For a standard transformer, multiply 2 (one key and one value) by the number of layers, the number of key-value heads, the size of each head and the bytes per number. Then multiply by the tokens in the conversation and by the users in the batch.

Meta's Llama 3 70B is a good worked case, because its configuration is public: 80 layers, 64 query heads but only 8 key-value heads (a design called grouped-query attention, or GQA), and a head size of 128.

KV CACHE SIZE  =  2 × layers × KV heads × head size × bytes per value × tokens
Llama 3 70B, 16-bit cache:
2 × 80 × 8 × 128 × 2 bytes   =  327,680 bytes  ≈ 320 KiB per token
  one   8,192-token chat        ≈  2.5 GiB
one 128,000-token document    ≈ 40   GiB   (half of one H100's memory)
How many 8,192-token chats fit beside the weights on 8 × H100 (640 GB)?
640 GB − ~70 GB of FP8 weights  ≈ 570 GB free
570 GB ÷ 320 KiB per token      ≈ 1.7 million tokens of cache
1.7 million ÷ 8,192             ≈ 210 chats at the same moment

That last line is the real limit on batch size, and therefore on price. Each design trick that shrinks the per-token cache lets one node serve more people at once. Grouped-query attention cut Llama 3 70B's cache eight times, compared with giving every one of its 64 query heads its own keys and values. Multi-head latent attention (MLA), which DeepSeek introduced and Kimi K3 uses on 24 of its layers, stores one compressed vector per token instead.

The full multi-head figure is a what-if for the same 70B shape with 64 key-value heads. The Kimi K3 figure covers only its 24 MLA layers. Its other 69 layers keep a fixed state instead, which the next section explains. The context window wiki entry and our history of the context window show why longer contexts made this arithmetic the center of serving design.

🧮 Hybrid Models Keep Two Caches

A hybrid model such as Kimi K3 keeps two kinds of cache, because it mixes two kinds of attention layer. Of its 93 layers, 24 are Gated MLA (multi-head latent attention) layers, full attention that keeps a compact per-token cache like the one above. The other 69 are Kimi Delta Attention (KDA) layers, a form of linear attention that keeps one fixed-size state per conversation, however long the conversation grows. The stack repeats three KDA layers, then one MLA layer.

KIMI K3: CACHE MEMORY FOR ONE CONVERSATION
24 MLA layers  ── per-token cache ── 576 values per layer per token
≈ 13.5 KiB per token in FP8
≈ 13.5 GiB at the full 1,048,576-token context
69 KDA layers  ── fixed state ────── ≈ 217 MB in BF16 for the whole model
≈ 27 MB per GPU when split 8 ways (TP8)
the same at token 10 and at token 1,000,000

The two caches change how prompt caching works. An MLA cache can resume at the exact token where two prompts stop matching. A KDA state is a running summary, so it cannot be cut at an arbitrary token. The engine saves copies of the state at checkpoints, and a cached prefix resumes from the nearest checkpoint before the mismatch, then rereads the gap. Caching still skips most of the prefill, but it is coarser than on a pure-attention model, and the serving engine runs two memory allocators side by side, one for each kind of cache.

KDA is the latest step in a short line of linear attention research. Katharopoulos and colleagues in 2020 swapped softmax for a simple feature map, so each attention head keeps one fixed d × d state. DeltaNet added the delta rule, which edits the value stored under a key instead of only adding to it. Yang and colleagues made that rule train in parallel (NeurIPS 2024, arXiv 2406.06484). Gated DeltaNet added a forget gate (ICLR 2025, arXiv 2412.06464). KDA (arXiv 2510.26692, October 2025) lets each channel of the state forget at its own rate. The payoff is a cache that stays small at a million tokens, which is why Moonshot uses it for most of the stack.

📦 What Is Batching, and Why Does It Make Tokens Cheap?

Batching means one GPU serves many requests in the same decode step. Each step reads the model's weights from memory once, and that one read can produce a token for one user or for dozens. Because the memory read is the slow part, adding users to a batch costs little until the GPU runs out of memory or compute. Batching is the main reason a token costs fractions of a cent.

Early servers used static batches: a group of requests started together and the GPU waited until the longest one finished. In July 2022 the Orca paper at OSDI introduced iteration-level scheduling. Requests join and leave the batch at every single step, so a finished answer frees its slot at once. That idea, now called continuous batching, is standard in open-source servers such as vLLM. In September 2023, the PagedAttention paper behind vLLM added a second gain: it stores each request's KV cache in small fixed pages, the way an operating system pages memory, and reported 2 to 4 times higher throughput than earlier systems at the same latency.

The limit is memory. Every user in the batch needs their own KV cache in GPU memory for the life of their request. The model weights take their share first, and only what is left holds caches. When it fills, new requests wait in the queue. That is why newer chips often add memory and bandwidth rather than raw math.

Batching also explains why scale matters so much to a provider. Here is an illustration with round numbers, not any company's real costs. Assume an 8-GPU node rents for $6 per GPU-hour ($48 per hour) and the provider earns $7 for every million output tokens it serves, counting the input tokens that come with them. Revenue per hour then depends only on throughput:

At the 912 tokens per second of the Lambda benchmark above, this node would earn about $23 an hour against $48 of rent. It breaks even only near 1,900 tokens per second of steady demand, roughly twice that output, every hour of the day. The gap closes through bigger batches, more cache hits, faster kernels and fewer idle hours. That is the margin game every inference provider plays, and it is why the companion deep dive spends a full section on it.

🔀 What Does Routing Mean in AI Inference?

Routing in AI inference means choosing where a request goes, and the word covers three different choices. A model router picks which model answers. A provider router picks which company serves a given model. A replica router picks which copy of one model, on which GPU server, handles the request. Each choice changes speed, price and cache hits, so it helps to keep them apart.

The replica level is where caching meets routing. A plain load balancer spreads requests evenly across servers. For AI, that wastes work: your second message can land on a server that never saw your first, so the whole conversation is prefilled again from scratch. A cache-aware router sends you back to the server that already holds your prompt's start in its KV cache, unless that server is full. The open-source llm-d project, launched by Red Hat with CoreWeave, Google Cloud, IBM Research and NVIDIA in May 2025, does this on Kubernetes. Its site cites its own benchmarks of up to 3 times higher output throughput and 2 times faster time to first token.

A non-technical team only ever touches the top level. When an AI tool offers an Auto model setting, it is a model router: it decides which model fits each task so you do not have to read benchmark tables. The other two levels sit with the tool and its providers. They still show up in your experience as speed and as the price the tool can afford to charge.

🛠️ Which Techniques Make LLM Inference Faster and Cheaper?

LLM inference gets faster and cheaper through a dozen techniques that each attack one bottleneck: compute in prefill, memory bandwidth in decode, or memory size for the KV cache. None of them changes what the model knows. They change how many bytes move per token, how many tokens share each byte, and how much work is skipped. Most serving engines turn on several of them at once.

Quantization deserves a closer look, because it shows up on model cards as "FP8" or "MXFP4". It stores each weight in fewer bits. A 70-billion-parameter model needs about 140 GB in 16-bit form, 70 GB in 8-bit form and 35 GB in 4-bit form. Fewer bytes mean faster decode and more room for the cache, which is why Kimi K3 trained with 4-bit weights from the fine-tuning stage onward instead of shrinking them after training. Model distillation takes the other route: it trains a smaller student model to copy a larger teacher.

🔄 What Happens to a Request Inside a Serving Engine?

A serving engine such as vLLM moves each request through a short life: it waits in a queue, gets scheduled into a batch, prefills, decodes, and ends in one of three ways. The scheduler re-plans the batch at every step. If GPU memory for the cache runs out, the engine can pause a running request, free its cache and put it back in the queue, a step called preemption.

🧰 Which Inference Engines Do Providers Use?

Most AI inference in 2026 runs on a handful of open-source serving engines, tuned by each provider. The engine sits between the model weights and the API and runs the batching, caching and parallelism described in this guide.

KodeKloud's walkthrough connects these pieces, from the GPU to vLLM to the llm-d router, in one sitting:

📉 Why Do AI Tokens Keep Getting Cheaper?

AI tokens get cheaper because four levers move at once: serving software, hardware, model efficiency and competition. Epoch AI found in March 2025 that the price to reach a fixed level of performance fell between 9 and 900 times per year, depending on the task. Sticker prices fell too: GPT-4 launched in March 2023 at $30 per million input tokens, and in October 2026 OpenAI listed GPT-6 Luna at $0.10.

The price drop is easy to see on a chart. These are list prices per million tokens for three OpenAI models, from the March 2023 GPT-4 launch to the price list checked on 2026-10-18. The tiers differ in capability, so read this as the cost of entry, not a like-for-like comparison.

Mixture-of-experts deserves a special note because it attacks the decode bottleneck directly. A mixture-of-experts model has many weights in total but reads only a fraction of them for each token. Less memory traffic per token means more tokens per second from the same chip. Most large open-weight models in 2026 use this design.

Fireworks AI CEO Lin Qiao put a number on the next stretch. On the 20VC podcast in July 2026, she forecast roughly a 10 times cost reduction over three years that would drive far higher usage. Treat it as a forecast from a seller of inference, recorded days after a funding round in which the podcast's own fund invested. The more useful part of her argument was the unit: she said the industry should "evaluate the token economy per task", because one model can be twice as cheap per token and twice as wordy, and then the two cost the same.

🧱 Total Parameters, Not Active Ones, Set the GPU Bill

A mixture-of-experts model saves memory traffic on each token, but it does not save GPU memory: the total parameter count, not the active count, decides how many GPUs a host must rent. Kimi K3 has about 2.8 trillion parameters and uses about 104 billion of them, roughly 3.7%, on each token. Each token picks 16 of 896 routed experts plus 2 shared experts. A host still keeps all 896 experts loaded.

Batching is the reason. One user's token touches only 16 experts, but a production batch holds tokens from many users at once, and across that batch nearly every expert is busy at some point. Swapping experts in and out of GPU memory on demand would add a slow read to a decode step that should take milliseconds, and the host would miss the latency it promises. So the rule of thumb is: active parameters set the speed per token, total parameters set the hardware bill.

💸 Does a Cheaper Token Mean a Cheaper AI Bill?

A cheaper token does not guarantee a cheaper bill, because the bill is cost per task, not cost per token. Cost per task is the price per token multiplied by the tokens one job uses, plus retries and failed attempts. Reasoning models and AI agents can spend many more tokens on one job, so the price per token falls while the cost per finished task can rise.

Three numbers move in different directions at once:

1. Sticker price per token has fallen on most tiers since 2023, as the chart above shows.
2. Price per unit of intelligence has collapsed. Epoch AI's 9 to 900 times per year is the cost of holding a fixed benchmark score.
3. Tokens per task have grown. A reasoning model thinks in hidden tokens before it answers, and an agent runs many steps, each with its own prompt.

COST PER TASK  =  [ (input tokens × input price)
                  + (cached tokens × cached price)
                  + (output tokens × output price, including hidden reasoning) ]
               ×  (attempts until the task is done)
Example shape for one agent job:
10 steps × 20,000 input tokens   = 200,000 input   (most of it cacheable)
10 steps ×  1,000 output tokens  =  10,000 output
1 retry                          =  ×2 if nothing is cached

This is why the AI cost per task guide recommends a 20-task test on your own work before you compare tools, and why our TSK benchmark scores models on real builds instead of on price lists. A model that finishes a job in one try at a higher token price often beats a cheaper model that needs three tries.

🧾 AI Inference Cost: What Is Inside the Price of One Token?

AI inference cost has two sides: what a provider spends to make a token, and what a customer pays per finished task. On the provider side, GPU time dominates, and everything that keeps a GPU busy with paying work lowers the cost of each token. On the customer side, the price list is only the start: caching, output length, reasoning tokens and retries decide the bill.

🏗️ The Provider Side: Where the Money Goes

Providers do not publish their cost breakdowns, so this table lists the parts of the cost and the lever that moves each one, not invented percentages. Our Inference-as-a-Service deep dive runs the break-even math for one node.

🧮 The Customer Side: A Worked Bill for One Agent Job

The same agent job can cost more than 50 times as much in one setup as in another, depending on the model and on caching. Here is one job priced on the OpenAI list prices checked on 2026-10-18: GPT-6.1 Sol at $2.00 per million input tokens, $0.10 cached and $10.00 output, and GPT-6 Luna at $0.10, $0.01 and $0.50. The job has 10 steps, reads 200,000 input tokens in total and writes 10,000 output tokens.

ONE 10-STEP AGENT JOB   (200,000 input tokens, 10,000 output tokens)
GPT-6.1 Sol, no cache hits
input   200,000 × $2.00 / 1M   = $0.400
output   10,000 × $10.00 / 1M  = $0.100
total                          = $0.500
GPT-6.1 Sol, 90% of input cached (instructions and history reused)
fresh    20,000 × $2.00 / 1M   = $0.040
cached  180,000 × $0.10 / 1M   = $0.018
output   10,000 × $10.00 / 1M  = $0.100
total                          = $0.158   (3.2× cheaper)
GPT-6 Luna, 90% cached           = $0.0088
GPT-6 Luna, 90% cached, 3 tries  = $0.026   (still 6× under Sol cached)
The cheap model wins only if it finishes the job.
If it fails the task, its cost per FINISHED task is unbounded.