Foundations: The Theory Under Modern AI

Markov Chain

Q: What Is a Markov Chain?

Definition: A Markov chain is a model of a system that moves between a set of states, where the probability of the next state depends only on the current state and not on how you got there. That "no memory of the path" rule is called the Markov property. TL;DR: Markov chains model anything that moves between states with fixed probabilities: customer lifecycles, page visits, machine status, ticket queues. Andrey Markov invented them in 1913 to win an argument about Russian poetry. Google later modelled the entire web as one. Build an AI app free → Picture a support ticket. At any moment it sits in exactly one state: New, In Progress, Waiting on Customer, or Closed. From each state there are fixed probabilities of where it goes next. The Markov property says something specific and strong: to predict the next move, you need only know where the ticket is now. Its full history is irrelevant. A ticket that reached "Waiting on Customer" after four reopenings behaves identically to one that arrived there directly. stateDiagram-v2 [*] --> New New --> InProgress: 0.8 New --> Closed: 0.2 InProgress --> Waiting: 0.5 InProgress --> Closed: 0.4 InProgress --> InProgress: 0.1 Waiting --> InProgress: 0.7 Waiting --> Closed: 0.3 Closed --> [*] That assumption is often wrong in detail and useful anyway. It is wrong because history usually carries some signal. It is useful because it makes the system solvable with arithmetic instead of simulation.

Q: What is a transition matrix?

A grid of probabilities where the entry in row i, column j is the chance of moving from state i to state j. Every row sums to 1. Raising the matrix to the nth power gives the n-step transition probabilities.

10 min read

On this page (20)

Definition: A Markov chain is a model of a system that moves between a set of states, where the probability of the next state depends only on the current state and not on how you got there. That "no memory of the path" rule is called the Markov property.

TL;DR: Markov chains model anything that moves between states with fixed probabilities: customer lifecycles, page visits, machine status, ticket queues. Andrey Markov invented them in 1913 to win an argument about Russian poetry. Google later modelled the entire web as one. Build an AI app free →

What Is a Markov Chain?

Picture a support ticket. At any moment it sits in exactly one state: New, In Progress, Waiting on Customer, or Closed. From each state there are fixed probabilities of where it goes next.

The Markov property says something specific and strong: to predict the next move, you need only know where the ticket is now. Its full history is irrelevant. A ticket that reached "Waiting on Customer" after four reopenings behaves identically to one that arrived there directly.

That assumption is often wrong in detail and useful anyway. It is wrong because history usually carries some signal. It is useful because it makes the system solvable with arithmetic instead of simulation.

Reading a Transition Matrix

The whole chain fits in a grid. Rows are where you are, columns are where you go, and every row sums to 1.

                    TO
              New   InProg  Wait   Closed
            +-----------------------------
   New      | 0.00   0.80   0.00   0.20    -> sums to 1.00
F  InProg   | 0.00   0.10   0.50   0.40    -> sums to 1.00
R  Wait     | 0.00   0.70   0.00   0.30    -> sums to 1.00
O  Closed   | 0.00   0.00   0.00   1.00    -> sums to 1.00
M           +-----------------------------
                                     ^
                          Closed only goes to Closed.
                          That is an ABSORBING state.

Two things to read off immediately. Every row must sum to 1, because the system has to go somewhere. And any state whose row has a 1 on the diagonal is absorbing: once entered, never left. Churn, cancellation, and permanent failure are all absorbing states.

To find where things stand after two steps, multiply the matrix by itself. After n steps, raise it to the nth power. That is the whole computational content of a Markov chain.

The Stationary Distribution

Run most chains long enough and the proportion of time spent in each state settles to a fixed set of numbers, regardless of where you started. That is the stationary distribution, and it is usually the number you actually want.

For a support queue, the stationary distribution tells you the steady-state share of tickets sitting in each state, which is what staffing decisions depend on. For a website, it tells you the long-run share of visits to each page, which is precisely what PageRank computes.

Question you have	Markov quantity that answers it
Where will this settle?	Stationary distribution
How long until it settles?	Mixing time
Will it ever get stuck?	Absorbing states
How long until it gets stuck?	Expected absorption time
Where will it be in 3 steps?	Transition matrix cubed

When Does a Chain Settle?

Not every chain converges. Two conditions guarantee it, and both have plain-English readings.

Condition	Formal meaning	Plain English	Failure looks like
Irreducible	Every state reachable from every other	No dead ends or isolated islands	A state you can enter but never leave
Aperiodic	No fixed-length cycle forces the timing	The chain does not tick in lockstep	Strict alternation, A to B to A to B forever

A chain that is both irreducible and aperiodic has exactly one stationary distribution and converges to it from any starting point. A chain with absorbing states is not irreducible, and it converges to sitting in the absorbing states instead. Both behaviors are useful; you just need to know which one you have.

The Feud That Invented It

Andrey Markov developed these chains around 1913, partly to settle a mathematical argument. A rival claimed the law of large numbers required independent events. Markov set out to prove that dependent events could obey it too, and to make the point concretely he analyzed the sequence of vowels and consonants in Pushkin's verse novel Eugene Onegin, counting 20,000 characters by hand.

He showed that a sequence with strong local dependence still exhibited stable statistical behavior. The first Markov chain in history was a model of Russian poetry, built to win an argument.

Are Large Language Models Just Markov Chains?

Technically yes, and the qualification matters more than the answer.

A language model with a fixed context window satisfies the Markov property, if you define the "state" as the entire context window. The next token depends only on the current state, which is exactly the definition.

The difference is that a classic chain stores its transitions in a table you can print. An LLM's state space is so large the table could never exist, so the transitions are computed by a transformer instead of looked up. Calling an LLM "just a Markov chain" is true in the way that calling a city "just some buildings" is true.

The Markov Family

Several related models extend the basic chain, and they are easy to mix up.

Model	You can see the state?	You can act?	Typical use
Markov chain (MC)	Yes	No	Forecasting, PageRank, queues
Hidden Markov model (HMM)	No, only observations	No	Speech recognition, tagging
Markov decision process (MDP)	Yes	Yes	Reinforcement learning
Partially observable MDP (POMDP)	No	Yes	Robotics, real-world agents

Most practical AI agent problems are honestly POMDPs: the agent cannot see the full state of the world and must act anyway. That is why agent memory and observation matter so much in practice.

Where Markov Chains Break

Three failure modes are worth knowing before you trust one.

History often matters. A customer who downgraded twice before is not equivalent to a first-time downgrader. When path matters, either enrich the state definition or drop the model.
Probabilities are not stable. Real transition rates drift with seasons, pricing, and product changes. A matrix fitted last year may describe a system that no longer exists.
Fitting needs data. Estimating an n by n matrix needs enough observed transitions in every cell. Rare transitions produce noisy estimates that look authoritative.

Build a Live State Tracker Without Writing Code

You do not need a matrix library to get most of this value. What you need is the transitions recorded as they happen, so the real distribution becomes visible instead of assumed.

Describe it to Taskade Genesis: "a board that tracks every customer through trial, active, at-risk, and churned, logs each transition with a timestamp, and shows the current mix." Taskade EVE builds it as living software over your projects, and an automation moves records between states on the triggers you define. The stationary distribution stops being a calculation and becomes a chart you look at.

Start building free →

PageRank: the web modelled as one enormous Markov chain
Monte Carlo Method: estimating chain behavior by sampling
Next-Token Prediction: the Markov property inside a language model
Context Window: what defines an LLM's "state"
Reinforcement Learning: Markov decision processes with actions
Information Theory: the other foundation under prediction
Agent Memory: what agents keep when the state is not fully observable

Frequently Asked Questions About Markov Chains

What is a Markov chain in simple terms?

A Markov chain is a system that moves between states where the next state depends only on the current one, not on the path taken to reach it. Ticket queues, customer lifecycles, and web browsing are all commonly modelled this way.

What does the Markov property mean?

It means the future is conditionally independent of the past, given the present. Knowing where the system is right now is enough to predict the next step, and the history adds nothing.

What is a transition matrix?

A grid of probabilities where the entry in row i, column j is the chance of moving from state i to state j. Every row sums to 1. Raising the matrix to the nth power gives the n-step transition probabilities.

What is a stationary distribution?

The long-run share of time a chain spends in each state, independent of where it started. For a chain that is irreducible and aperiodic it is unique, and it is usually the practical answer you want, such as the steady-state mix of a queue.

Are large language models Markov chains?

Technically yes, if you define the state as the whole context window. The difference is scale: a classic chain stores transitions in a printable table, while an LLM's state space is so large the transitions must be computed by a transformer instead.

What is an absorbing state?

A state that, once entered, is never left, shown as a 1 on the diagonal of the transition matrix. Churn, cancellation, and permanent failure are absorbing states, and the useful question becomes how long until absorption rather than where things settle.

Who invented Markov chains?

Andrey Markov, around 1913, partly to disprove a rival's claim that the law of large numbers required independent events. He demonstrated the point by hand-counting vowels and consonants across 20,000 characters of Pushkin's Eugene Onegin.

What is the difference between a Markov chain and a hidden Markov model?

In a Markov chain you observe the state directly. In a hidden Markov model the state is hidden and you observe only signals produced by it, so you must infer the state. Speech recognition is the classic hidden Markov application.