Definition: Inference-as-a-Service (IaaS) is the business of running trained AI models and selling their output through an API, billed per million tokens. You send a prompt and get tokens back. The provider runs the hardware and does the batching, caching, and scaling for you.
TL;DR: Inference-as-a-Service is the AI-era successor to Infrastructure-as-a-Service. It keeps the same acronym and moves one layer up the stack. Cloud IaaS rents a server by the hour. Inference-as-a-Service rents the output of a model by the token. Model labs, hyperscalers, GPU clouds, inference platforms, and custom-chip companies all sell it. Build an app free →
Why Inference-as-a-Service Matters
Every AI reply is a purchase from this market. A chat answer, an agent step, and an AI automation run each send tokens to a provider and get tokens back. A lab trains a model once. Inference occurs on every request, so the price per token sets the running cost of every AI product.
The category also explains why most AI apps own no GPUs. A GPU node costs the same per hour when it serves one user or a hundred. An app that buys tokens pays only for the work it uses. The seller carries the idle time, and it earns its margin when each GPU stays busy.
For the full market story, with history and margins, read Inference-as-a-Service: How AI Is Sold in 2026. For the buyer's view of speed, price, and provider choice, read AI Inference Explained.
How Inference-as-a-Service Works
Each request has two phases. Prefill reads the whole prompt in one parallel pass and fills the KV cache. Decode then writes the answer one token at a time. Prefill sets the wait before the first word appears. Decode sets the tokens per second of the stream.
The provider does two things to sell more tokens from each GPU-hour:
- Continuous batching. The server packs many requests onto one GPU and adds or removes requests at each decode step.
- Prompt caching. The server keeps the KV cache of a repeated prompt prefix and reuses it. A cache hit skips prefill for that part, so providers bill cached input at a lower price.
The Five Kinds of Seller
Five kinds of company sell Inference-as-a-Service. Only two of them treat the serving of other people's models as their whole business.
| Seller kind | Examples | What gives them an advantage | Main limit |
|---|---|---|---|
| Model labs | OpenAI, Anthropic, DeepSeek | They know their own model best and set the reference price | One catalog, and limited compute |
| Hyperscalers | AWS Bedrock, Azure AI, Google Vertex AI | Enterprise contracts, compliance, and cloud credits | Serving is one product among thousands |
| Neoclouds (GPU clouds) | CoreWeave, Nebius, Crusoe, Lambda | Low-cost GPU-hours at large scale | Mostly sell raw capacity, not tuned token APIs |
| Inference platforms | Fireworks, Together AI, Baseten, DeepInfra, Novita | Software tuning, many models, fast launches | Compete on price with the lab and with each other |
| Custom silicon | Groq, Cerebras | Per-user speed far above GPUs | Narrow model coverage, slow to add new architectures |
From Hosting to Inference
Each era of computing had a "rent it, do not own it" layer. Each new layer moved the unit of sale closer to the result that the customer wants.
Raw capacity becomes a commodity, and the margin moves to the layer that customers touch.
The Unit Economics on One GPU Node
The unit of cost is the GPU-hour. The product is tokens. This example uses one 8-GPU B200 node that serves DeepSeek V4 Pro at the third-party list price. It assumes two input tokens for each output token.
| Step | Value |
|---|---|
| Node rent | 8 × |
| List price | $1.74 per 1M input tokens, $3.48 per 1M output tokens |
| Revenue per 1M output tokens of work | 2 × $1.74 + $3.48 = $6.96 |
| Break-even demand | $48 ÷ $6.96 ≈ 6.9M output tokens per hour ≈ 1,900 to 2,000 output tokens per second |
| Basic vLLM, measured (Lambda, 8× B200, 8,192 input and 2,048 output tokens per request) | ~912 output tokens per second |
The Lambda run used 4 input tokens for every output token, a heavier mix than this example. At the 2:1 mix, 912 tokens per second earn less than half of the rent. At the 4:1 mix of the benchmark, the same run earns about $34 per hour, or about 70% of the rent. At either mix, a basic open-source setup does not pay its rent. Caching, larger batches, custom kernels, and custom chips close that gap. The inference cost page explains how these levers appear on the price list.
Inference-as-a-Service vs Infrastructure-as-a-Service vs SaaS
The three models sell different things, and the cost of one more customer behaves differently in each.
| Infrastructure-as-a-Service | Inference-as-a-Service | Software-as-a-Service | |
|---|---|---|---|
| What you rent | Virtual machines, storage, networks | A trained model's output | Finished software |
| Unit of sale | Per hour | Per million tokens | Per seat, per month |
| Who runs the software | You | The provider | The vendor |
| Cost of one more customer | Your own servers | Real GPU time for every token | Close to zero |
| Example buyer question | "How many servers do I need?" | "What does this task cost in tokens?" | "How many seats do I need?" |
How Taskade Uses Inference-as-a-Service
Taskade runs no GPUs of its own. It buys inference from model providers and routes each request across frontier models from top AI labs. On Taskade AI, you see the cost as credits, not as raw tokens.
You can also bring your own key (BYOK) from a provider, so that the provider bills your own account:
| Provider key | What runs on it | Plans |
|---|---|---|
| OpenAI, Anthropic, Google Vertex AI | Custom AI agents, Ask Agent, and automation AI steps | Pro and up |
| xAI, Groq, Mistral, Together AI | Custom AI agents and Ask Agent | Business and up |
- Your provider bills the tokens. An agent chat on your key uses no Taskade credits. An automation step on your key uses one fixed step fee in Taskade credits.
- Taskade EVE runs on Taskade AI, not on a customer key.
Setup steps are in Bring Your Own AI Keys. The key providers page shows which models each key can reach.
Related Concepts
- Inference: the runtime that IaaS sells
- Inference Cost: what each token costs to serve
- KV Cache: the memory that limits users per GPU
- Prompt Caching: the discount for repeated prefixes
- Tokens Per Second: the speed that providers sell
- GPU · Memory Bandwidth
- Open Source Models: the weights that third-party hosts serve
Frequently Asked Questions About Inference-as-a-Service
What is Inference-as-a-Service?
Inference-as-a-Service (IaaS) is the business of running trained AI models and selling their output through an API, billed per million tokens. You send a prompt, and the provider returns tokens. The provider runs the hardware, the batching, the caching, and the scaling.
Is Inference-as-a-Service the same as Infrastructure-as-a-Service?
No. Infrastructure-as-a-Service rents raw servers, storage, and networks by the hour, and you run your own software. Inference-as-a-Service rents the result of a model run, billed by the token. The acronym is the same because the job is the same, one layer up: it turns expensive hardware into a metered utility.
Who sells Inference-as-a-Service?
Five kinds of company sell it: model labs such as OpenAI and Anthropic, hyperscalers such as AWS and Azure, GPU clouds such as CoreWeave and Lambda, inference platforms such as Fireworks and Together AI, and custom-silicon companies such as Groq and Cerebras.
How do inference providers make money?
They earn the gap between the rent of a GPU-hour and the tokens that they sell from it. An 8-GPU B200 node rents for about $48 per hour. At 2026 list prices for DeepSeek V4 Pro, it needs about 2,000 output tokens per second of steady demand to break even.
Does Taskade run its own GPUs?
No. Taskade buys inference from model providers and shows the cost as credits. On Pro and up, you can connect your own OpenAI, Anthropic, or Google Vertex AI key. On Business and up, you can also connect an xAI, Groq, Mistral, or Together AI key. An agent chat on your key uses no Taskade credits, and an automation step on your key uses one fixed step fee in Taskade credits.