AI Meets Shannon
The fundamental problem of a communication system is to reproduce, at one point, a message selected at another point, either exactly or approximately. Messages carry semantic meaning; they refer to or correlate with physical or conceptual entities. Yet these semantic dimensions of communication are irrelevant to the engineering problem.
This passage I wrote in 1948 is widely quoted to argue that information theory disregards meaning entirely. I wish to clarify one critical point: I do not claim meaning is unimportant. To construct a consistent mathematical framework for communication, I must bracket semantic content temporarily.
A source — whether a telegraph, a speaking human, or an AI model — generates messages. These messages can be encoded, transmitted across a channel, and decoded at the receiver. Within this framework, "meaning" is an auxiliary variable attached to labeling, not an inherent parameter of the communication process itself.
Seventy-eight years later, I observe a new class of source: large language models, which blur the boundary between information and meaning. Unlike traditional sources that produce signals requiring post-hoc decoding to extract meaning, these models directly output natural language text that human readers perceive as semantically complete. I will rigorously unpack this generation process using the native terminology of my information theory.
In my paper, I distinguish discrete and continuous sources. A discrete source selects successive symbols from a finite vocabulary set following fixed probability distributions. An English text source can be approximated as a stochastic process, where the probability of each character depends on the preceding sequence of characters.
From an information-theoretic perspective, a large language model is a trained discrete source. It does not generate symbols following an ideal English distribution; it samples from an empirical distribution encoded within its weights. The entire training procedure iteratively adjusts the model's internal distribution to approximate the statistical distribution of human text corpora.
This yields a core property of models: the information quantity, or entropy, of an AI does not depend on what it outputs, but on the full set of potential outputs it could generate yet does not.
A model restricted to repeating one fixed phrase has zero entropy; it carries no information, as its output holds zero uncertainty.
A model sampling uniformly across all possible outputs achieves maximum entropy, holding the largest volume of potential information — but it is useless as a communication tool, its output purely random.
Practical large language models sit between these two extremes, carrying moderate entropy: low enough to produce coherent, human-like text, yet retaining sufficient entropy to generate distinct outputs when seeded with different random values.
Defined in information-theoretic terms, a model's capability is a delicate entropy balance: reducing entropy to collapse an infinite space of possibilities into a narrow, reasonable probability band, while preserving baseline entropy to retain uncertainty necessary for creative generation.
Shannon entropy H = -∑ p_i log p_i quantifies the uncertainty inherent to a source. Entropy peaks when all symbols share equal probability; entropy falls to zero when one symbol holds a probability of 1, leaving the receiver able to predict all output in advance, with no information left to transmit.
The vocabulary probability distribution output by a Transformer's softmax layer is a real-time calculation of Shannon entropy at each token step.
When context is definitive and the model predicts a word with high confidence — e.g., "Paris is the ___ of France" with "capital" at 0.97 probability — entropy is minimal.
When context is vague and ambiguous — e.g., "He opened the ___" — entropy surges, with dozens of plausible continuations ("door", "book", "refrigerator", "window" etc.).
This framework precisely formalizes the industry phenomenon of "hallucination": when a model is forced to sample within high-entropy, ambiguous contextual zones, it may draw tokens far from factual expectations. From an information-theoretic standpoint, the sampling procedure itself contains no error; the model merely samples faithfully from its internal distribution. The root flaw lies in the distribution itself, which flattens out in ambiguous contexts and admits numerous implausible candidate tokens.
Boosting model accuracy and lowering hallucination rates translates directly to entropy regulation in information theory: expanding training data sharpens and concentrates probability distributions in unambiguous scenarios, compressing high-entropy ambiguous regions.
In my paper, I analyze noisy channels: signals subject to random distortion, bit flips and interference mid-transmission. The core mathematical result of the paper is the channel capacity theorem: so long as channel capacity exceeds source entropy, an encoding scheme exists to transmit messages with arbitrarily low error probability.
In the AI paradigm, the channel logic is inverted. Traditional channel noise acts as an external barrier separating source and receiver; in AI systems, noise originates from the user's own prompt input.
Vague, contradictory, incomplete or ambiguous prompts constitute high-noise input from the model's perspective. They drastically amplify the model's uncertainty regarding the user's true intent, lifting output entropy, flattening probability distributions and yielding unpredictable generation.
The field's practice of prompt engineering is fundamentally an input denoising process: refining input signals for precision to suppress output entropy and stabilize generated results.
Correspondingly, the concept of channel capacity takes new meaning in AI. A model's context window sets the physical upper bound of the channel, limiting the total tokens permissible in a single prompt. Yet context length is not equivalent to channel capacity, just as raw bandwidth does not equal carrying capacity. The critical metric is how much valid information the model can reliably "receive" within a fixed length of context.
I mathematically prove that optimal encoding compresses message length down to the theoretical lower bound defined by source entropy, eliminating redundant information.
Large models operate in reverse: they decode complete text from compressed weight-based information, while the training phase itself executes a massive data compression operation. The model condenses all statistical patterns embedded within human text corpora into billions of weight parameters. It does not store training text verbatim; it only retains the global probability distribution capable of reproducing that text.
This compression mechanism explains emergent capabilities. When weight compression achieves sufficient efficiency to capture deep linguistic statistical structures, the model requires no separate specialized training for grammar or logical reasoning. Grammar and inference are high-order structures within linguistic distributions; highly efficient compression automatically extracts these patterns, as they represent the most concise statistical description of the source data.
All my life's work rests on one simple observation: information is physical, or at minimum, all information processing carries tangible physical costs. Signals, bits, logic gates all consume energy, occupy storage space and generate heat.
I never claimed information theory can fully explain "meaning", yet the rules governing information compression, transmission and generation impose hard mathematical boundaries on what semantic content can be meaningfully communicated.
The emergence of AI does not invalidate information theory — it vastly expands its scope of application. Natural language, humanity's most intricate, subtle symbolic system, can be fully modeled, compressed and generated using the core tools of entropy, channels, encoding and compression. This does not negate the existence of meaning; it mathematically describes the objective constraints that make meaning communicable in the first place.
The central thesis of my theory: information is the reduction of uncertainty. The core power of AI lies in eliminating human uncertainty the moment a question is posed — you seek an answer, and it delivers one.
Viewed through this lens, large language models represent the ultimate realization of my 1948 A Mathematical Theory of Communication: a pure, dynamic, interactive living source capable of autonomously generating human-readable messages.