Definition: Entropy is the average amount of information per symbol produced by a source, measured in bits. Shannon defined it as the sum of p × -log2(p) over every possible symbol. It is the theoretical minimum average bits per symbol for lossless compression: a wall you can approach but never cross.
TL;DR: Entropy measures how surprising a source is on average. A predictable source has low entropy and compresses well; an unpredictable one has high entropy and barely compresses at all. Every AI training loss is chasing this floor. Build an AI app free →
You have an intuition for this already. A colleague who says the same three things in every meeting is low entropy, and you can summarize them in a sentence. A colleague whose input you genuinely cannot predict is high entropy, and you have to quote them in full.
What Is Entropy?
Entropy answers one question: on average, how many bits does it take to say what happened?
Take a source that emits four symbols. If all four are equally likely, every message is a genuine surprise and you need 2 bits each. If one symbol appears 90% of the time, most messages tell you almost nothing, and a good code can average well under 1 bit.
SOURCE A: uniform SOURCE B: skewed
A 25% B 25% C 25% D 25% A 90% B 6% C 2% D 2%
Best code: Best code:
A=00 B=01 C=10 D=11 A=0 B=10 C=110 D=111
Always 2 bits Usually 1 bit
Entropy = 2.00 bits Entropy = 0.61 bits
^ maximum uncertainty ^ nearly predictable
Both sources have four symbols. The difference is entirely in the shape of the distribution.
How the Shape of a Distribution Sets Entropy
Spread probability evenly and entropy is maximal. Concentrate it and entropy collapses. This table walks the full range over four symbols.
| Distribution | Probabilities | Entropy | Reading |
|---|---|---|---|
| Uniform | 25 / 25 / 25 / 25 | 2.00 bits | Maximum uncertainty |
| Mildly skewed | 50 / 25 / 13 / 12 | 1.75 bits | Typical real source |
| Heavily skewed | 90 / 6 / 2 / 2 | 0.61 bits | Nearly predictable |
| Degenerate | 100 / 0 / 0 / 0 | 0.00 bits | No information at all |
The bottom row is the important one. A source that always emits the same symbol carries zero information, because you never needed to be told.
For a single biased coin you can watch the entire curve at once:
Maximum at a fair coin, zero at either certainty. Entropy is a measure of genuine doubt, and doubt peaks when the outcomes are balanced.
The Source Coding Theorem
Shannon proved two things about entropy in 1948, together called the source coding theorem (or noiseless coding theorem):
- No lossless encoding can average fewer bits per symbol than the source's entropy.
- It is always possible to design an encoding that gets arbitrarily close to it.
That pairing is unusually strong. Entropy is not a heuristic or an estimate. It is a hard floor, and you can always walk right up to it.
This is why an already-compressed file barely shrinks when you zip it again. The first pass removed the redundancy; what remains is close to pure entropy, and there is nothing left to squeeze.
The Entropy of English
Shannon ran an experiment in 1951 to estimate the entropy of English using human predictors. He showed people partial sentences and asked them to guess the next letter, recording how many attempts each guess took. From those guess counts he bounded the entropy of English at roughly 0.6 to 1.3 bits per character.
That number is striking because plain ASCII spends 8 bits per character. English is therefore about 75% redundant. That redundancy is exactly why you can read text with missing letters, why autocomplete works, and why a language model can predict the next word at all.
| Encoding of English text | Bits per character | Comment |
|---|---|---|
| Naive ASCII | 8.00 | No modelling at all |
| Letter frequencies only | ~4.10 | E is common, Z is not |
| Word-level statistics | ~2.00 | Context starts helping |
| Shannon's human estimate | 0.60 to 1.30 | A brain full of context |
| Modern frontier LLM | below 1.00 | Inside Shannon's range, on a different corpus |
Every row down that table is a better model of the source. Entropy did not change. Our estimate of it got sharper.
The last two rows are not a head-to-head race. Anything under 1.00 still falls inside Shannon's 0.6 to 1.3 interval, and the two figures come from different corpora and different evaluation setups, so "below 1.00" does not by itself establish that a model out-predicts his human subjects.
Entropy vs Cross-Entropy
This is the distinction people trip over most. Entropy is a property of the source. Cross-entropy is a property of the source plus your model of it.
| Entropy | Cross-entropy | |
|---|---|---|
| Depends on | The true distribution only | True distribution and your model |
| Can you compute it? | Only if you know the truth | Yes, from your model's predictions |
| Can you lower it? | No, it is fixed | Yes, by improving the model |
| Where you see it | Theory, compression limits | Every AI training curve |
An AI training run cannot reduce entropy. It reduces the gap between its cross-entropy and the entropy floor. When a loss curve flattens, the model is running out of gap, which is KL divergence approaching zero.
Why Entropy Matters for AI Work
Three practical consequences follow directly from the definition.
- Context lowers effective entropy. Once a model knows your project, your customers, and your naming conventions, its distribution over the next token narrows. That is the mechanism behind retrieval-augmented generation and context engineering, stated in bits.
- Confident and wrong is a real failure mode. A model can have low entropy over a badly-shaped distribution. Low uncertainty is not the same as accuracy, which is why hallucinations can sound so assured.
- Temperature is an entropy dial. Raising temperature flattens the output distribution and raises entropy, which reads as creativity. Lowering it sharpens the distribution and reads as consistency.
Building the Low-Entropy Version of Your Own Work
Most operational reporting is high redundancy: the same rows, mostly unchanged, sent on a schedule. The useful version sends only the surprise.
Describe that to Taskade Genesis: "a weekly board showing only the projects whose status or owner changed since last week." Taskade EVE builds it as living software over your existing projects, and an automation delivers it without anyone assembling a report. You get the bits that carry information and none of the ones that do not.
Related Concepts
- Information Theory: the field entropy sits at the center of
- Cross-Entropy: entropy plus the cost of a wrong model
- KL Divergence: the gap between cross-entropy and the floor
- Perplexity: entropy expressed as a branching factor
- Compression and Intelligence: why the floor is an intelligence benchmark
- Temperature: the sampling control that raises or lowers output entropy
- Channel Capacity: entropy's counterpart on the transmission side
Frequently Asked Questions About Entropy
What is entropy in information theory?
Entropy is the average number of bits needed to describe one symbol from a source, computed as the sum of p × -log2(p) over all symbols. It measures average surprise, and it is the hard lower bound on lossless compression.
Is information entropy the same as thermodynamic entropy?
They share a mathematical form and are historically connected, but they measure different things. Thermodynamic entropy counts microscopic arrangements of a physical system; information entropy counts average uncertainty in a message source. Von Neumann reportedly suggested Shannon use the name partly because nobody would be able to argue about it.
What does high entropy mean?
High entropy means the source is unpredictable, so each symbol carries a lot of information and the data compresses poorly. Random noise has maximum entropy. Highly structured or repetitive data has low entropy and compresses well.
Can entropy be reduced?
Not for a fixed source. Entropy is a property of the true distribution. What you can reduce is the gap between your model's cross-entropy and that floor, which is exactly what training a model does.
What is the entropy of English?
Shannon's 1951 human-prediction experiment bounded it at roughly 0.6 to 1.3 bits per character, against 8 bits for naive ASCII. English is therefore about 75% redundant, which is why autocomplete and text prediction work.
Why can't you compress an already-zipped file?
Compression removes redundancy. Once it has been removed, the remaining data sits close to its entropy floor, and no further coding can shrink it meaningfully. A second pass often adds a few bytes of header for no gain.
How does entropy relate to AI temperature settings?
Temperature rescales a model's output distribution before sampling. High temperature flattens it, raising entropy and producing more varied text. Low temperature sharpens it, lowering entropy and producing more predictable text.
Do I need to calculate entropy to use AI tools?
No. Entropy explains why context improves answers and why some outputs feel repetitive, but you never compute it yourself. To build something, describe it to Taskade Genesis in plain English.
Further Reading
- Compression Is Intelligence: entropy, cross-entropy, and the compression duality
- Claude Shannon and the Invention of the Bit: where the definition came from
- Foundations: the rest of the theory under modern AI
- Information Theory: the parent concept