AI Meets Godel
The advancement of mathematics has pushed toward greater rigor, leading to the widespread formalization of most mathematical branches, such that proofs can be constructed upon a small set of axioms. This paper proves that large language models, regarded as a formal system, possess fundamental inherent limitations. These limitations share a formally parallel structure with the restrictions I proved for formal arithmetic systems in 1931.
Before diving into technical details, I outline the core argument. A language model can be interpreted as a formal system M: its symbol set is vocabulary V; its well-formed formulas are finite sequences over V; its inference rule is the forward pass within the Transformer architecture — given preceding context, compute the probability distribution of the next token.
Unlike classical formal systems, M does not "prove" formulas; it samples formulas from its distribution. This practical difference, though empirically significant, does not alter the logical structure of my argument.
Definition 1. Let M be a trained Transformer language model with parameter set θ and vocabulary V.
Definition 2. A generation run is an inference of M: given input sequence x ∈ V*, M outputs y ∈ V*, where each token satisfies y_i ∼ P(·|x, y_previous; θ).
Definition 3. We say M "can generate" a statement s, written M ⊢ s, if and only if there exists an input x and sampling path along which s is produced with non-zero probability.
Remark: M ⊢ s does not require M to output s on every sampling trial. It only demands s carries non-zero weight within the output distribution. This weaker condition suffices for all subsequent reasoning.
Theorem 1. The set of all statements M is capable of generating is countably infinite.
Proof. Every output of M is a finite sequence over a finite vocabulary V. The collection of all finite sequences V* is countable. The full corpus of outputs produced across infinitely many generation runs is a countable union of finite sets, hence countable. ∎
Theorem 2. There exist uncountably infinitely many statements outside the expressive range of M.
Proof. By Theorem 1, model outputs are countable; the set of all meaningful natural-language statements forms at minimum a continuum (uncountably infinite). The two sets differ in cardinality, so the result follows immediately. ∎
This fact alone does not constitute a restrictive argument against M — any finite human being can only utter countably many sentences as well. Yet it reveals a key intuition: M's seemingly boundless knowledge is merely a countably infinite collection, a quantity far beyond human intuitive comprehension.
In my 1931 paper, I constructed a formal sentence equivalent to "This sentence cannot be proved within the system." Following the same spirit, I artificially construct a family of statements describing the model's own capacities.
Definition 4. Let G(M) denote the sentence: "M cannot correctly infer any of its own parameters."
Theorem 3. If M is consistent (i.e., M never generates logically contradictory pairs of statements), then G(M) is undecidable within M: M cannot consistently assert G(M) to be true, nor consistently assert G(M) to be false.
Proof Sketch. Suppose M ⊢ G(M), meaning M generates "I cannot infer my own parameters."
"Inferring its own parameters" refers to a concrete operation: M's forward pass only serves text generation; its computation flow contains no backward routine to inspect the concrete values of its weight parameters θ.
M generates this statement not from direct self-inspection of its weights, but from statistical patterns of human claims about "AI self-limitation" present in training data. Two cases arise:
If M can access its parameters via external tool calls, G(M) is false, and M produces a false proposition, breaking consistency.
If M cannot access its own parameters (the standard case for mainstream models), M still outputs G(M) based purely on statistical induction rather than direct factual observation, rendering the claim epistemically unstable.
Core conclusion: M cannot generate true statements about its own capacities grounded in direct self-observation. All self-referential claims are derived from statistical aggregation of human writing about AI. The same self-referential sentence can be swapped into outputs of a different model without altering its truth value. This is the operational meaning of undecidability in the AI context. ∎
I acknowledge this argument carries far less strictness than my original 1931 proof, yet it establishes a clear structural symmetry.
The classical Incompleteness Theorem: Any recursively axiomatizable, consistent formal system containing basic arithmetic contains a true proposition that can neither be proved nor disproved within the system.
The AI variant of incompleteness, as I term it: Any system whose "knowledge" derives solely from statistical patterns in training data contains true propositions about its own capacities that can neither be reliably inferred from training statistics nor directly verified by the system itself.
Both forms of incompleteness share one core mechanism: self-reference. A system cannot fully describe itself from within its own bounds, unless it ascends to a stronger meta-system outside itself.
Is an AI required to "Gödelize" itself, elevating to a higher-order meta-system, to overcome this incompleteness? I leave this question open. Nonetheless, I must state clearly: any path toward AGI that relies merely on scaling autoregressive training will forever remain confined to the category of systems unable to fully see themselves.