Definition: Next-token prediction is the single operation behind every large language model: given all the text so far, the model scores every token it knows, picks one, appends it to the text, then runs the whole transformer again from the beginning and does it once more.
TL;DR: A language model does not write a sentence. It writes one token, re-reads everything including that new token, and writes the next one. A 500-word reply is roughly 650 separate passes. Everything that looks like reasoning is assembled out of that one repeated move. Build an app on it free →
You already do a version of this. Someone says "could you pass the" and three or four endings arrive in your head before they finish the sentence. Salt at a dinner table, wrench in a garage, remote on a sofa. You are not consulting a plan. You are picking the ending the situation makes most likely. A model does that once per token, with the whole conversation as the situation, thousands of times in a row.
Why Next-Token Prediction Matters in 2026
Every behaviour people call AI reasoning is produced by one mechanism: choosing the next small piece of text, over and over. There is no separate planning module and no private scratchpad where the model works something out before speaking. Once you know that, a lot of otherwise puzzling behaviour becomes predictable. It explains why asking a model to show its work raises accuracy, because chain-of-thought steps are the only place thinking can happen. It explains why the same prompt can return two different answers, because the pick is drawn from a ranked list rather than looked up in a table, which is the subject of non-determinism. It explains why a model states a false fact in the same confident tone as a true one, since hallucinations are just plausible next tokens. It even explains cost: every token you send is re-read on every pass through the context window.
How Next-Token Prediction Works
The loop has five moves and it never varies. Text goes in, one token comes out, the token is appended, and the model starts over on the longer text.
- Your text is split into tokens. A tokenizer breaks the prompt into the pieces the model actually reads. Common words survive whole, rarer ones break into fragments, and punctuation counts. The ratios are in token.
- The model reads all of them at once. A transformer does not walk the text left to right. It weighs how strongly every token relates to every other token using the attention mechanism, so the last line of a long prompt can be shaped by the first.
- It scores every token it knows. The output of a pass is not a word. It is a score for each of the tens of thousands of tokens in the model's vocabulary, computed from the fixed numbers set during training and explained in model parameters.
- One token is chosen. Usually a high scorer, but not always the very top one. A small amount of controlled randomness is what keeps writing from sounding mechanical, and it is why two identical prompts can drift apart. See non-determinism.
- The token is appended and the whole thing runs again. The model keeps nothing between passes. It is stateless, so every pass re-reads the entire text, including the words it wrote a moment ago. That repeated pass is what inference means.
- It stops when it produces an end-of-reply token or reaches the length limit it was given.
Watching four passes makes the append-and-repeat shape concrete:
PASS 1 "The capital of France is" -> " Paris"
PASS 2 "The capital of France is Paris" -> ","
PASS 3 "The capital of France is Paris," -> " and"
PASS 4 "The capital of France is Paris, and" -> " it"
^ every pass re-reads the whole thing, including its own last word
Nothing is held over from one line to the next except the text itself. The growing draft is the entire memory.
Why the Tokens Are the Thinking
Every pass gets the same fixed amount of computation, no matter how easy or hard the question is. "What is 2 plus 2" and "plan a three-city product launch" each get one pass per token. That is the constraint that shapes everything else about how AI models behave.
It means the only way a model can spend more effort on a harder problem is to write more tokens. Ask for the answer alone and it has one pass to get there. Ask for the steps, and each step it writes becomes part of the text the next pass re-reads, so the work compounds. This is why chain-of-thought prompting lifts accuracy without changing the model at all, and why the visible reasoning is not decoration: the tokens are literally where the thinking happens.
Everything built on top follows from the same fact. Reasoning models are trained to write a long private chain before answering. Test-time compute is the name for buying accuracy with extra tokens. Self-consistency runs the chain several times and takes the majority answer. Careful prompt engineering works because the prompt is re-read on every single pass, so a clear instruction keeps paying for itself.
Next-Token Prediction vs Phone Autocomplete
"It is just autocomplete" is the most common way this gets dismissed, and the comparison is closer than critics think in mechanism and much further apart in practice. Both rank candidate continuations. The difference is how much text is considered, how many candidates are scored, and what set the scores in the first place.
| Trait | Phone keyboard autocomplete | Next-token prediction |
|---|---|---|
| How much it reads | The last word or two | Every token in the conversation, via attention |
| Candidates ranked per step | A handful | Tens of thousands |
| What sets the scores | A word-frequency list | Billions of model parameters fitted to a huge corpus |
| Follows an instruction | No | Yes, the instruction is part of the text it re-reads |
| Stays consistent over a page | No | Yes, the whole draft stays in view |
| What comes out | One suggested word | Working prose, code, and structured output |
The mechanism is humble. The behaviour that emerges from running it at this scale, on this much text, is not. That gap is the thing worth holding in your head.
Connection to Taskade
You never see the token loop in Taskade, but it is running underneath everything an agent does. Each Taskade AI Agent wraps next-token prediction in an agent loop: the model writes tokens, some of those tokens are a request to run one of its built-in tools, Taskade runs the tool, and the result is appended to the text before the next pass. That is how text prediction turns into finished work, like a record updated or an email drafted for review.
Model choice is handled for you. The Auto setting routes each job to one of 15+ frontier models from OpenAI, Anthropic, Google, and open-weight providers, so you never pin a version. Your workspace supplies the text those passes read: the projects, records, and files an agent is pointed at are the Memory half of Workspace DNA, where Memory feeds Intelligence and Intelligence triggers Execution. When an automation fires, the same loop runs unattended, and 100+ bidirectional integrations give it real events to read and real places to write.
What You Would Build in Taskade
Think about the weekly report you assemble by hand. You open four places, skim what changed, decide what matters, and write a paragraph a manager will actually read. Every one of those judgement calls is the kind of thing next-token prediction is good at, once it has the right text in front of it.
In Taskade that becomes a status digest you describe in a sentence. New activity lands in a project, an agent reads what changed since the last run, writes a short summary in your house voice, flags the two items that slipped, and posts it every Monday morning. You review it in whichever of the 7 project views you think in, a board, a table, or a calendar. The agent keeps writing it between check-ins, and because the reasoning is written out rather than hidden, you can read why it flagged what it flagged and correct it in plain language.
Describe yours and build it free →
Related Concepts
- Token: the unit being predicted
- Tokenizer: how your text becomes tokens
- Transformer: the architecture that runs each pass
- Model Parameters: the fixed numbers that set the scores
- Non-Determinism: why the same prompt varies
- Chain-of-Thought: why writing more tokens helps
- Inference: what one pass actually costs
- Context Window: how much text a pass can read
- AI Agents in Taskade: the loop wrapped around prediction
Frequently Asked Questions About Next-Token Prediction
How do large language models actually work?
They predict one token at a time. The model reads all the text so far, scores every token in its vocabulary, picks one, appends it, and repeats from the top. A full reply is thousands of those passes. There is no other mechanism underneath, which is what makes large language models so simple to describe and so hard to predict.
Is next-token prediction just autocomplete?
The mechanism is related, the scale is not. Phone autocomplete ranks a handful of candidates from the last word or two. A large language model ranks tens of thousands of candidates while re-reading the entire conversation through the attention mechanism, using numbers fitted to an enormous body of text. Same shape, different universe of behaviour.
Does the model plan the whole sentence before writing it?
No. It commits to one token at a time and cannot take one back. Anything resembling a plan has to be written out as tokens first, which is exactly what chain-of-thought prompting and reasoning models do. The plan is visible in the output because the output is the only place it can exist.
Why does the same question give a different answer each time?
Because the pick at each step is a draw from a ranked list, not a lookup. A little controlled randomness is deliberate: it is what keeps writing from sounding mechanical. One different token early on sends the rest of the reply down a different path. Non-determinism covers when this helps and how to reduce it.
Why does telling a model to think step by step improve the answer?
Because each pass gets a fixed amount of computation. Asking for the answer alone gives the model one pass to reach it. Asking for the steps lets it spend a pass per step, with every step it writes becoming part of the text the next pass re-reads. See chain-of-thought and test-time compute.
Why do AI models make things up?
Because a false statement can be an extremely likely next token. The model scores what fits the pattern of the text, not what is true, and it has no separate step that checks a claim against the world. That is the root of hallucinations, and it is why grounding a model in real records, as an agent does with your workspace, matters more than a stern instruction.
Does next-token prediction mean the model does not understand anything?
That argument is unsettled and worth holding loosely. What is not in dispute is the mechanism: prediction is all that runs. What surprised researchers is how much capability shows up when the same move is scaled far enough, from translation to code to multi-step planning, which is the subject of scaling laws.
Do I have to pick a model in Taskade?
No. The Auto setting routes each job to one of 15+ frontier models from OpenAI, Anthropic, Google, and open-weight providers, so you never pin a version that goes stale. If you want to control how much reasoning a task gets, Thinking Modes in Learn Taskade walks through the choice in plain language.