Foundations: The Theory Under Modern AI

Cross-Entropy

Q: What Is Cross-Entropy?

Definition: Cross-entropy is the average number of bits you spend encoding data from one distribution when you are using a code built for a different distribution. In machine learning it is the standard loss function: it measures how many bits a model wastes because its predictions do not match reality. TL;DR: Cross-entropy is the bill you pay for being wrong. It equals entropy (the unavoidable floor) plus KL divergence (your model's error). Every LLM training curve is cross-entropy falling toward a floor it can never reach. Build an AI app free → Suppose a source emits symbols with true probabilities P, but you built your encoding around your beliefs Q. You assigned short codes to the symbols you thought were common and long codes to the ones you thought were rare. If your beliefs were wrong, you will spend more bits than necessary. Cross-entropy is exactly that average cost: sum of p × -log2(q). Note which distribution appears where. The frequency comes from reality (p), because reality decides how often you pay. The price comes from your model (q), because your model decided the code lengths. TRUTH (p) YOUR MODEL (q) CODE LENGTH COST ---------- -------------- ----------- ---- A 50% A 25% 2 bits 0.50 x 2 = 1.00 B 25% B 50% 1 bit 0.25 x 1 = 0.25 C 13% C 13% 3 bits 0.13 x 3 = 0.39 D 12% D 12% 3 bits 0.12 x 3 = 0.36 ----------- Cross-entropy = 2.00 bits Entropy = 1.75 bits Wasted (KL) = 0.25 bits You swapped the codes for A and B. A is twice as common as B, so that swap costs you a quarter of a bit on every symbol, forever.

8 min read

On this page (18)

Definition: Cross-entropy is the average number of bits you spend encoding data from one distribution when you are using a code built for a different distribution. In machine learning it is the standard loss function: it measures how many bits a model wastes because its predictions do not match reality.

TL;DR: Cross-entropy is the bill you pay for being wrong. It equals entropy (the unavoidable floor) plus KL divergence (your model's error). Every LLM training curve is cross-entropy falling toward a floor it can never reach. Build an AI app free →

What Is Cross-Entropy?

Suppose a source emits symbols with true probabilities P, but you built your encoding around your beliefs Q. You assigned short codes to the symbols you thought were common and long codes to the ones you thought were rare. If your beliefs were wrong, you will spend more bits than necessary.

Cross-entropy is exactly that average cost: sum of p × -log2(q).

Note which distribution appears where. The frequency comes from reality (p), because reality decides how often you pay. The price comes from your model (q), because your model decided the code lengths.

   TRUTH (p)        YOUR MODEL (q)      CODE LENGTH     COST
   ----------       --------------      -----------     ----
   A  50%           A  25%              2 bits          0.50 x 2 = 1.00
   B  25%           B  50%              1 bit           0.25 x 1 = 0.25
   C  13%           C  13%              3 bits          0.13 x 3 = 0.39
   D  12%           D  12%              3 bits          0.12 x 3 = 0.36
                                                        -----------
                                        Cross-entropy =  2.00 bits
                                        Entropy       =  1.75 bits
                                        Wasted (KL)   =  0.25 bits

You swapped the codes for A and B. A is twice as common as B, so that swap costs you a quarter of a bit on every symbol, forever.

The Decomposition That Explains Everything

Cross-entropy always splits into two parts, and keeping them separate resolves most confusion about training curves.

Cross-entropy = Entropy + KL divergence

Training a model cannot touch the entropy term. It only ever shrinks the KL term. That is why every loss curve flattens: the model is running out of waste to remove, and what remains is the floor set by the data itself.

Cross-Entropy vs KL Divergence vs Log Loss vs Perplexity

These four terms describe closely related quantities and get used interchangeably in ways that confuse people. Here is the whole family.

Term	Formula	What it measures	Where you see it
Entropy	`sum p × -log2(p)`	Irreducible floor	Theory, compression limits
Cross-entropy	`sum p × -log2(q)`	Total bits paid	Training loss, LLM pre-training
KL divergence	`H(P,Q) - H(P)`	Waste from a wrong model	Distillation, regularization, RLHF
Log loss	Same as cross-entropy	Total bits paid	Binary classification, statistics
Perplexity	`2^(cross-entropy)`	Effective branching factor	Language model evaluation

Log loss is not a different thing. It is the same quantity under a name inherited from the statistics literature rather than the information theory one.

The common question is why frameworks minimize cross-entropy rather than KL divergence, given that KL is the quantity you philosophically care about. The answer is that they differ only by the entropy of the data, which is a constant you cannot change. Minimizing either yields the identical optimal model, and cross-entropy is the one you can compute without knowing the true distribution.

Why Training Loss Is a Compression Rate

This is the part that reframes what a model is. A model that assigns high probability to the text that actually appears can encode that text in few bits. So cross-entropy loss, measured in bits per token, is a compression rate.

A frontier model reporting under 1 bit per character on English text is claiming it could compress that text better than any conventional compressor, and somewhere at or inside the range Shannon's 1951 human predictors reached. That last part is not a clean win: his 0.6 to 1.3 bits per character overlaps anything under 1, and the two experiments used different corpora and different setups, so the numbers are not directly comparable. See Compression and Intelligence for how far that claim actually goes.

Bits vs Nats

Loss is often reported in nats rather than bits, because frameworks use natural log by default. The conversion is a constant factor.

Unit	Log base	Convert to the other
Bit (shannon)	log2	`nats × 1.4427`
Nat	ln (base e)	`bits × 0.6931`

A loss of 2.0 nats is about 2.885 bits. Nothing conceptual changes; only the ruler does.

What This Means When You Read Model Claims

Cross-entropy is the number underneath most benchmark talk, so a few cautions transfer directly.

Lower loss on a different dataset is not comparable. Cross-entropy depends on the data's own entropy. A model with lower loss on simpler text is not better.
Loss can fall while usefulness does not. The model may be getting better at predicting text that nobody needs predicted. This is the gap between benchmark and product.
Tokenization changes the number. Bits per token and bits per character are different denominators. Compare per-character when comparing across tokenizers.

Lowering Uncertainty on Your Own Work

The practical version of all this is simple: a model with your context has less to guess about. Cross-entropy on your specific documents drops when the model can see the project, the customer, and the history instead of predicting from generic priors.

That is what a workspace does. When you build with Taskade Genesis, your projects are the context, your agents read from them, and your automations act on the result. Describe the system in plain English and Taskade EVE assembles it as living software. You never set a loss function. You just stop making the model guess.

Start building free →

Entropy: the floor cross-entropy is measured against
KL Divergence: the waste term, isolated
Perplexity: cross-entropy converted to a branching factor
Information Theory: the parent field
Compression and Intelligence: the duality this loss implies
Model Training: the process that minimizes this number
Evals: why loss alone does not tell you if a model is useful

Frequently Asked Questions About Cross-Entropy

What is cross-entropy in machine learning?

Cross-entropy is the loss function that measures how many bits a model wastes because its predicted probabilities do not match reality. It equals the data's entropy plus the model's KL divergence from the truth.

What is the difference between cross-entropy and KL divergence?

Cross-entropy is the total bits paid. KL divergence is cross-entropy minus entropy, which isolates only the waste caused by a wrong model. They differ by a constant, so minimizing either produces the same optimal model.

Is log loss the same as cross-entropy?

Yes. Log loss is the same quantity under a name inherited from statistics. Binary cross-entropy and binary log loss are also the same thing.

Why is cross-entropy used instead of accuracy?

Accuracy only counts whether the top prediction was right. Cross-entropy scores the entire probability distribution, so a model that is right but unsure is penalized, and one that is confidently wrong is penalized heavily. That gradient signal is what makes training work.

What is a good cross-entropy loss value?

There is no universal target, because the floor depends on the data's own entropy. What matters is the trend during training and comparison against the same dataset and tokenizer. Loss numbers from different datasets are not comparable.

Why is loss sometimes reported in nats instead of bits?

Frameworks default to natural log, which produces nats. One nat is about 1.4427 bits. The choice of log base changes the unit, not the meaning.

Does lower cross-entropy mean better compression?

Yes, literally. Cross-entropy in bits per symbol is the average code length a model implies, so a model with lower loss on some text could encode that text in fewer bits. See Compression and Intelligence.

Do I need to understand cross-entropy to use AI?

No. It explains why models improve with better context and why loss curves flatten. To build something useful, describe it to Taskade Genesis and it assembles the working app.