Definition: A GPU (graphics processing unit) is a processor built from thousands of small cores that all perform the same operation on different data at once — a shape that happens to match the matrix arithmetic inside a neural network almost exactly.
TL;DR: A GPU trades the flexibility of a CPU for raw parallel throughput: thousands of simple cores instead of a handful of complex ones. That design was built for rendering graphics and turned out to be the right shape for deep learning. Today a flagship AI rack holds 72 of them wired to behave as one machine. Build an app free →
The Chef and the Line Cooks
A CPU is a head chef: a small number of very capable cores that handle complex, branching, sequential work. It runs your operating system and makes decisions.
A GPU is ten thousand line cooks: thousands of simple cores that all chop the same onion at the same instant. Each one is individually unimpressive. Collectively they perform an enormous number of identical operations per second.
Neural networks are billions of identical multiply-and-add operations. One head chef cannot do that in reasonable time. Ten thousand line cooks can.
The same work, two architectures. The GPU wins whenever the work is wide and identical.
Why This Was an Accident
GPUs were designed to shade millions of pixels independently — a wide, identical, embarrassingly parallel problem. When researchers found that training a deep learning model was the same kind of problem, the hardware already existed and was already cheap because gamers had funded it.
That coincidence, and the software layer built on top of it to make general-purpose math accessible, is the foundation of the modern AI industry. The full story is in the history of NVIDIA.
What a GPU Is Not Good At
Being a GPU has costs, and they explain most of the practical limits you meet in AI products:
- Branching. Work that takes different paths per item wastes most of the cores.
- Small jobs. One user's single request leaves most of the chip idle, which is why serving software batches many requests together.
- Being fed. This is the big one. A GPU's arithmetic units are so fast that the real limit is usually memory bandwidth — getting weights out of memory and into the cores. During text generation the cores frequently sit waiting.
GPUs Inside an AI Rack
A modern rack-scale system holds 72 accelerators wired by a high-speed internal fabric so they share one address space and behave as a single machine. This matters because a frontier model does not fit on one chip: its parameters are sliced across many, and every generated token requires those chips to exchange intermediate results.
| Property | Typical direction |
|---|---|
| Cores | Thousands of simple ones |
| Best at | Wide, identical, parallel math |
| Worst at | Branching and sequential logic |
| Usual bottleneck | Memory bandwidth, not arithmetic |
| Memory beside it | High-bandwidth memory |
| Power per rack | ~120 kW for 72 of them |
Why a New GPU Is Not 15× the Old One
Every generation launches with an enormous multiplier — Blackwell arrived with claims of up to 30× the inference performance of the H100 generation. The number is not false, but it compares a full 72-GPU rack against individual older chips, at a new lower-precision number format (FP4) the old generation lacked, on a workload chosen to showcase the interconnect. Chip against chip, at matched precision, the picture is tamer:
| Property | H100 (SXM) | B200 | Ratio |
|---|---|---|---|
| Memory | 80 GB HBM3 | 192 GB HBM3e | 2.4× |
| Memory bandwidth | 3.35 TB/s | ~8 TB/s | ~2.4× |
| Power (TDP) | 700 W | 1,200 W | 1.7× |
Because generation is memory-bound, real-world serving gains track the bandwidth row — roughly 2–2.5× per chip — not the headline. The rest of the marketed multiple comes from stacking improvements: more chips per rack, faster interconnect between them, lower-precision math, and better serving software, each real, none a property of one GPU.
The practical reading skill: when a performance claim crosses a generation, check the denominator. Chip versus rack, FP4 versus FP8, batch throughput versus single-stream — each mismatch quietly multiplies the number. Per watt, the honest generational gain is meaningful but ordinary; the extraordinary totals come from deploying far more watts.
Do You Need to Think About GPUs?
If you use a hosted AI product, no. Model routing and capacity planning are absorbed by the platform, and you pay per token rather than per GPU-hour. The place GPU economics surface in ordinary use is indirect: long context is expensive because it multiplies memory traffic, and agentic loops are expensive because they multiply requests.
That is the actual trade when you run AI agents on a platform instead of your own fleet: the batching, routing, and generation upgrades happen underneath you, and multi-step agent work simply gets faster and cheaper over time. The same applies to automations — a scheduled workflow is a stream of GPU work you never had to size a cluster for. If you want the fuller story of how a request travels from your keyboard to a rack of these chips and back, it is in how AI data centers work.
Related Concepts
- Memory Bandwidth — the actual limit on GPU throughput
- HBM — the memory that sits beside it
- Data Center — the building designed around it
- Neural Network · Deep Learning — the math it accelerates
- Transformer — the architecture that made the demand explode
- Inference · Model Training
- Tokens Per Second · Inference Cost — the user-visible outputs of GPU economics
Frequently Asked Questions About GPUs
What is the difference between a CPU and a GPU?
A CPU has a small number of complex cores optimised for sequential, branching work — running an operating system, making decisions. A GPU has thousands of simple cores that execute the same instruction across different data simultaneously. CPUs are better at variety; GPUs are better at volume of identical work.
Why does AI need GPUs specifically?
Because neural network training and inference are dominated by matrix multiplication — billions of identical multiply-and-add operations that can all happen at once. That is precisely the workload GPUs were built for, even though they were originally built for graphics rather than AI.
Are GPUs the only chips that run AI?
No. Custom accelerators designed specifically for machine learning are widely deployed, and CPUs can run smaller models. GPUs remain dominant for frontier work largely because of the mature software ecosystem built around them over two decades.
Is the GPU the expensive part of an AI request?
It is the expensive asset, but only about 42% of the energy of a served request is the accelerator itself. Host processors, idle reserve capacity, and facility overhead account for the rest, according to Google's published measurements of its own serving stack.
How many GPUs does it take to run a large AI model?
Whatever it takes to hold the weights plus working state in fast memory. A 70-billion-parameter model at 8-bit needs about 70 GB and fits on one 80 GB accelerator with little room to spare; a 400-billion-parameter model at 16 bits needs around 800 GB and must be sliced across several. Frontier models occupy multiple chips, sometimes multiple racks.
Why are AI GPUs so expensive?
Because the product is not just the processor die. A flagship accelerator packages the die with stacks of high-bandwidth memory using advanced bonding, both supplied by a small number of manufacturers running near capacity — and demand from AI buyers has exceeded supply for most of the decade. Scarcity, packaging complexity, and memory cost set the price as much as the silicon does.
What is the difference between an H100 and a B200?
A generation, mostly expressed as memory. The B200 carries 192 GB of HBM3e at roughly 8 TB/s against the H100's 80 GB of HBM3 at 3.35 TB/s — about 2.4× on both counts — while drawing 1,200 W to the H100's 700 W. For memory-bound generation work, that bandwidth ratio is the honest per-chip speedup.