Foundations: The Theory Under Modern AI
On this page (8)

Foundations: The Theory Under Modern AI

Foundations covers the mathematics and machine designs that modern AI inherited rather than invented. A large language model in 2026 is trained by minimizing cross-entropy, a quantity Claude Shannon defined in 1948. It samples the next token from a conditional distribution, which is the Markov property Andrey Markov described in 1913. It runs on a machine that stores its program in the same memory as its data, which is John von Neumann's 1945 design.

TL;DR: Modern AI is built from four old ideas: information has a measurable unit (the bit), uncertainty has a floor (entropy), the future can depend only on the present (Markov chains), and a machine can describe itself (self-replication). Everything else is engineering on top. Build an AI app free →

KL Divergence

KL divergence measures the bits wasted by using a wrong model. Learn why it is asymmetric, how it differs from cross-entropy, and where it appears in distillation and RLHF.

Markov Chain

A Markov chain moves between states where the next step depends only on the current one. Learn transition matrices, stationary distributions, and how LLMs relate to Markov models.

Monte Carlo Method

The Monte Carlo method estimates answers by random sampling instead of solving equations. Learn how it works, where the error goes, and how it powers AI search and simulation.

PageRank

PageRank modelled the entire web as one Markov chain and ranked pages by where a random surfer lands. Learn the algorithm, the damping factor, and what it means for AI search.

Perplexity

Perplexity converts a language model's cross-entropy into an effective number of choices. Learn how to read it, why lower is better, and why it is not comparable across tokenizers.

Self-Replicating Code

A quine prints its own source. Learn how self-reference works without infinite regress, von Neumann's universal constructor, the Trusting Trust attack, and what it means for AI.

Turing Completeness

A Turing-complete system can compute anything computable. Learn what it takes, why it is surprisingly cheap, the halting problem, and why capability is not the same as usefulness.

Von Neumann Architecture

The von Neumann architecture stores program and data in one memory. Learn why that made software possible, what the von Neumann bottleneck is, and how GPUs work around it.

Why a Foundations Category Exists

Most AI explainers start at the transformer and work forward. That leaves a gap: you can read about attention, tokens, and context windows without ever learning what a model's loss number means, why "perplexity 8" is a sentence about branching, or why a model that predicts well is doing the same job as a file compressor.

This category fills that gap. Each article connects one old idea to the exact place it surfaces in a system you use today.

The Four Threads

Thread Founding idea Where it shows up in 2026 Start here
Measurement Information has a unit, and uncertainty has a floor Training loss, evals, compression Information Theory
Prediction The next state depends only on the current one Next-token sampling, PageRank, churn models Markov Chain
Computation A simple enough machine can compute anything Turing-complete tools, agent loops Turing Completeness
Replication A machine can carry a description of itself Quines, code-writing agents, Workspace DNA Self-Replicating Code

Articles in This Category

Measurement (information theory)

  • Information Theory. Shannon's 1948 paper, the bit, and why every model is a communication channel
  • Entropy. The average surprise in a source, and the compression floor nobody can beat
  • Cross-Entropy. What a training loss number actually measures, in bits
  • Perplexity. Cross-entropy converted into "how many options is the model choosing between"
  • KL Divergence. The waste from using a wrong model, and where it shows up in distillation and RLHF
  • Channel Capacity. Shannon's noisy-channel theorem, and reliable systems from unreliable parts
  • Compression and Intelligence. Why predicting well and compressing well are the same job

Prediction (stochastic processes)

  • Markov Chain. Memoryless state machines, transition matrices, and where every chain settles
  • Monte Carlo Method. Estimating an answer by sampling instead of solving
  • PageRank. Google modelled the entire web as one Markov chain

Computation and replication

Key Reading

How These Ideas Reach Your Work

You do not need any of this theory to build something. But it explains the behavior you already notice. A model that "sounds confident and is wrong" is one with low entropy over a bad distribution. An agent that loops forever is missing the bounded state machine an ontology would have given it. A benchmark score that keeps improving while the product does not is cross-entropy falling on the wrong distribution.

When you build an app with Taskade Genesis, the theory is already handled. You describe the system you want, Taskade EVE assembles it as living software, and your projects supply the context that lowers the model's uncertainty on your specific work. That last part is the practical version of this entire category: a model with your context has less to guess about.

Start building free →

← Back to all topics