Definition: Information theory is the mathematics of measuring, compressing, and transmitting information. Claude Shannon founded it in a single 1948 paper, "A Mathematical Theory of Communication," which introduced the bit as the unit of information and proved hard limits on how far any message can be compressed and how fast it can be sent without errors.
TL;DR: Before 1948, "information" was a vague word. Shannon made it a measurable quantity with a unit (the bit), a floor (entropy), and a ceiling (channel capacity). Every large language model is trained by minimizing a Shannon quantity. Build an AI app free →
You use this idea daily without naming it. When you tell a colleague "the meeting moved" instead of reading them the full calendar invite, you are compressing: stripping the predictable parts and sending only what they could not have guessed. Shannon's contribution was proving exactly how far that stripping can go.
What Is Information Theory?
Information theory measures information as reduction in uncertainty. A message carries a lot of information when it tells you something you could not have predicted, and almost none when it confirms what you already expected. "The sun rose today" carries nearly zero information. "The server is down" carries a lot, because you did not expect it.
Shannon formalized this. The information content of an event with probability p is -log2(p) bits. Rare events carry many bits, common events carry few. That single formula is the seed of the entire field.
Event probability Information content
---------------- -------------------
p = 1/2 (coin) -> -log2(0.5) = 1 bit
p = 1/4 -> -log2(0.25) = 2 bits
p = 1/256 -> -log2(1/256) = 8 bits
p = 1 (certain)-> -log2(1) = 0 bits <- tells you nothing
The last line is the one that matters most. A guaranteed outcome carries zero information, no matter how many words you use to describe it.
Why Information Needed a Unit
Engineering fields stall until their central quantity gets a unit. Thermodynamics needed the joule, electricity needed the volt, and communication needed the bit. Before Shannon, telegraph and telephone engineers had rules of thumb about bandwidth and noise but no way to say how much a channel could actually carry, or whether a given code was efficient.
Shannon supplied the unit and the two theorems that bound it. His framing was deliberately blunt: the meaning of a message is irrelevant to the engineering problem. What matters is only how many possible messages there were and how likely each one was.
| Before 1948 | After 1948 |
|---|---|
| "This line is noisy" | Noise has a measurable capacity cost |
| "This code seems compact" | Codes are measured against the entropy floor |
| "Add more power to reduce errors" | Add redundancy instead, and errors go to zero |
| Analog signals with accumulating noise | Digital signals that can be perfectly regenerated |
The Two Theorems That Define the Field
Shannon proved two results in 1948 that still bound every system built since.
Source coding theorem (the compression floor). No lossless encoding can average fewer bits per symbol than the source's entropy. You can always get arbitrarily close, and you can never get past it.
Noisy-channel coding theorem (the transmission ceiling). Every channel has a capacity. Send below it and you can drive the error rate as close to zero as you like, using redundancy. Send above it and errors are unavoidable, no matter how clever the code.
The second theorem was the shocking one. Engineers had assumed noise meant you traded speed against accuracy forever. Shannon proved that below capacity you get both, which is why digital storage and transmission replaced analog everywhere.
The Core Quantities
Information theory is a small family of related measures. They are easy to confuse, so it helps to see them side by side.
| Quantity | What it measures | Where you meet it | Article |
|---|---|---|---|
| Self-information | Surprise of one specific event | Rare-event detection | This page |
| Entropy | Average surprise across a whole source | Compression limits | Entropy |
| Cross-entropy | Bits paid when your model is wrong | Every LLM training loss | Cross-Entropy |
| KL divergence | The waste alone, floor removed | Distillation, RLHF | KL Divergence |
| Perplexity | Cross-entropy as a branching factor | Model eval papers | Perplexity |
| Channel capacity | Maximum reliable throughput | Modems, Wi-Fi, storage | Channel Capacity |
Why Every Language Model Is a Shannon Machine
A large language model predicts a probability distribution over the next token. Training adjusts the model so that the tokens which actually appear get assigned high probability. The loss function that drives this is cross-entropy, measured in bits or nats.
That means a modern training run is doing exactly what Shannon described: building a better model of a source so that the source can be encoded in fewer bits. The connection is not an analogy. It is the same equation.
- Lower loss = better compression. A model with lower cross-entropy on your text could encode that text in fewer bits. See Compression and Intelligence.
- Perplexity is a Shannon number. A reported perplexity of 8 says the model is as uncertain as someone picking uniformly among 8 options.
- Context reduces entropy. Giving a model relevant context narrows its distribution, which is the practical reason retrieval and context engineering work at all.
Shannon's Other Inventions
The 1948 paper is the famous one, but Shannon's range was unusual. His 1937 master's thesis showed that Boolean algebra could describe electrical switching circuits, which is the origin of digital circuit design. During the war he worked on fire-control systems and on SIGSALY, the encrypted voice link between Roosevelt and Churchill, and took tea with Alan Turing at Bell Labs. He later built Theseus, a maze-solving mechanical mouse whose memory lived in relays under the floor, and wrote one of the first serious papers on programming a computer to play chess.
For the full story, see Claude Shannon and the Invention of the Bit.
Where Information Theory Shows Up in Your Work
You are already running an informal version of this whenever you decide what to write down and what to leave out. A weekly status update that repeats last week's is high-redundancy and low-information. A dashboard that only surfaces exceptions is a compressor: it strips the predictable and shows you the surprise.
That is a system you can build rather than maintain by hand. Describe it to Taskade Genesis: "a dashboard that shows only the accounts whose status changed this week, and who owns each one." Taskade EVE assembles it as living software reading from your connected projects, and a reliable automation posts the changed rows on a schedule. The predictable parts stay out of your way.
Related Concepts
- Entropy: the average-surprise measure at the center of the field
- Cross-Entropy: what an AI training loss number actually means
- Channel Capacity: the noisy-channel theorem and reliable systems
- Compression and Intelligence: why prediction and compression are one job
- Markov Chain: the other 20th-century idea inside next-token prediction
- Token: the symbol a language model actually measures entropy over
- Large Language Models: the systems trained on a Shannon objective
Frequently Asked Questions About Information Theory
What is information theory in simple terms?
Information theory is the math of measuring how much a message tells you. It defines information as reduction in uncertainty, measures it in bits, and proves how far data can be compressed and how fast it can be sent over a noisy line without errors.
Who invented information theory?
Claude Shannon, a mathematician and engineer at Bell Labs, founded it with his 1948 paper "A Mathematical Theory of Communication." The field appeared essentially complete in that one paper, which is unusual in the history of science.
What is a bit in information theory?
A bit is the amount of information in a single yes-or-no answer where both outcomes are equally likely. Formally, an event with probability p carries -log2(p) bits, so a fair coin flip carries exactly 1 bit and a 1-in-256 event carries 8.
Does information theory care what a message means?
No, and Shannon said so explicitly. The engineering problem is reproducing a message at another point, not understanding it. Meaning is irrelevant to how many bits the message costs, which is precisely what made the theory general enough to cover text, audio, images, and DNA.
How is information theory used in AI?
Every language model is trained by minimizing cross-entropy, a Shannon quantity, and evaluated with perplexity, which is cross-entropy exponentiated. Lower loss literally means the model could compress the training text into fewer bits.
What is the difference between entropy and information?
Information is the surprise of one specific event. Entropy is the average information across every event a source can produce. A single rare word carries high information; the entropy of English is the average across all words.
Why did Shannon's noisy-channel theorem matter so much?
It proved that noise sets a speed limit, not an accuracy limit. Below a channel's capacity you can make errors arbitrarily rare by adding redundancy. That result is why digital replaced analog: a digital signal can be perfectly regenerated, while an analog one degrades at every hop.
Do I need information theory to use AI tools?
No. It explains behavior you already notice, such as why giving a model more relevant context improves its answers. To actually build something, describe it to Taskade Genesis in plain English and it assembles the working app for you.
Further Reading
- Claude Shannon and the Invention of the Bit: the full biography and timeline
- Compression Is Intelligence: the modern reading of Shannon's theorems
- History of Computing: where Shannon's thesis sits in the wider story
- Foundations: the rest of the theory under modern AI