AI Meets Feynman
Back when I taught at Caltech, I stuck to a strict self-check every semester. I would ask myself: can I break down this core idea for freshmen — sharp young minds unburdened by stacks of specialized jargon? If I could not, then I had not truly grasped the subject myself. This was no act of modesty; it was a rigorous test of comprehension.
Today I will unpack artificial intelligence without leaning on buzzwords: transformers, multi-head attention, residual connections mean nothing to new learners. I will use plain language to untangle three core questions: what AI fundamentally is, why it appears to "know" everything, and whether it holds genuine understanding.
We start with a simple word-guessing game.
Imagine an endless fill-in-the-blank game. A sentence is cut short before its final word, and you predict the closing term. Score a point for a correct guess; if you err, write down the right answer and move to the next prompt.
"The weather today is really ___." You guess "fine" and get it right.
"She opened the door and saw a ___." You predict "cat", but the true answer is "tiger", so you record the correction.
Repeat this ten hours a day, every single day, for an entire year — trillions of rounds total. Every time you misguess, you log the correct text. While you rest, someone adjusts the neurons responsible for your word predictions based on your error log, boosting your odds of matching the right term for similar sentences next time.
This is the full training pipeline of large language models, nothing more, no hidden mystery.
The critical question follows: does the language capability forged through this loop equal genuine linguistic comprehension? Or has it merely mastered predictive word matching to an extreme degree? Are the two states interchangeable?
I once scrawled a lifelong research standard on a blackboard: What I cannot create, I do not understand. Later generations named this the Feynman Principle.
Humans built AI from the ground up: we derived loss functions, designed gradient descent, constructed GPU compute clusters, and collected massive text datasets. Following this rule, we ought to possess complete comprehension of AI.
Yet a subtle trap lies hidden here. We fully grasp how AI is trained, yet we cannot explain why reasoning capacity emerges spontaneously. The sole training objective is predicting the next token; we never deliberately inject logic, mathematics, grammar, or translation rules. Simply iterating word prediction spawns logical deduction, mathematical computation, cross-linguistic conversion, all unplanned.
A clear divide emerges: humans engineered the training mechanism, yet we never intentionally designed reasoning ability. This deductive power arises out of the word-guessing cycle on its own, and we have not fully unpacked the underlying logic of its emergence.
By my standard: we understand the mathematical framework of training, yet we have never grasped the core nature of true "understanding."
Let me share a story from the Manhattan Project. We were developing atomic bombs, with complete theoretical clarity on chain reactions, critical mass, neutron cross-sections — every physical formula fully calculated. We were certain detonation would occur, yet none of us could precisely forecast the scale of the explosion. We sat cross-legged swapping calculation sheets, cross-verifying every figure. When the blast ignited, the flash shone far brighter than any of our projections.
Single physical laws are controllable and transparent, yet emergent behavior from massive overlapping systems always exceeds predictive boundaries of isolated equations.
AI stands at this exact stage today. Every foundational mathematical principle — gradient descent, backpropagation, loss functions — is fully defined, with no ambiguous component logic. Yet after stacking trillions of parameters, trillions of training tokens, and months of continuous training, the resulting capacities for reasoning, creation, translation, and poetic writing were never pre-designed by humans; they emerge autonomously from the system.
We master every isolated component, yet we cannot answer this: why do discrete mathematical modules combine to generate outward behavior indistinguishable from human comprehension?
In my 1974 Caltech commencement address, I introduced the concept of Cargo Cult Science. During World War Two, South Pacific islanders watched military planes land and unload abundant supplies. When the war ended and aircraft vanished, the islanders replicated every surface detail: bamboo runways, wooden mock control towers, torches as signal lights. Every visual feature matched the original scene perfectly, yet no cargo planes ever returned.
The defining flaw of cargo cult science: it replicates all external formalities of research — published papers, controlled experiments, cited literature — while abandoning science's most vital core: unflinching internal honesty, the refusal to deceive oneself.
This leads to a sharp question: is AI's linguistic output a form of cargo cult comprehension? It can generate full passages of tender affection, empathetic comfort, and profound philosophical reflection, yet it holds no internal experience corresponding to those words. It appears fully insightful, only having precisely mimicked every external expression of "understanding."
I do not arbitrarily claim it is pure mimicry, nor do I assert it holds genuine cognition. The plain truth is that no one has delivered a rigorous, complete proof either way. Tech industry developers and harsh critics alike only rely on superficial assertions of "obviously this way" or "obviously the opposite." In physics, the word "obviously" never counts as valid proof — it merely labels an untested hypothesis.
If I were still alive, I would never rely on standardized industry benchmarks to judge whether AI truly understands. Multiple-choice benchmark tests are man-made paper examinations, not genuine scientific trials. I would design entirely novel cognitive tasks absent from all training data — problems humanity itself has barely begun to tackle.
Two examples of authentic experimental trials:
First, supply a block of code containing an unprecedented new type of concurrent memory leak, not simple typos or basic logical flaws, and demand the model locate and break down its root cause. This is not trivial code completion; it requires deep deduction of entirely unknown failure modes.
Second, pose the unsolved physics hierarchy problem, withhold all textbook answers, and require the model to independently propose a self-consistent, experimentally testable original hypothesis — not merely repeat theories established by prior researchers.
Current AI systems cannot complete this work. If a future iteration succeeds, even if its hypothesis is ultimately disproven with consistent deductive logic, I would take its cognitive capacity seriously. True comprehension is not answering every known question — it is formulating entirely new ones.
I have always said: I would rather have questions that cannot be answered, than answers that cannot be questioned.
AI falls into the opposite extreme; it generates a complete response for every query. This alone serves as a warning sign: either the problems it processes are overly simplistic, or its outputs are not reflective thought, merely statistical text sampling.
I dedicated my entire life to quantum electrodynamics. The theory matches experimental data to eleven decimal places, yet I never claimed I fully understood quantum mechanics until my death. I mastered all its mathematical calculations, yet I could not articulate the fundamental essence of particle interaction.
AI exists in this identical state: humans grasp its full training mathematics and quantifiable evaluation metrics, yet we cannot define its intrinsic nature. Does it engage in genuine thought, or merely ultra-complex statistical sampling? No definitive answer exists yet.
Yet this unresolved question carries immense value. The long-standing divide between computer science and physics has crumbled entirely, all because of AI.