How AI Models Actually Think

During a training run at Anthropic, a model solving a simple math problem began to come apart. It did the arithmetic correctly and landed on 12. Then it reversed itself: "Oh no, oh my God, the answer actually is for sure 36." It redid the math, got 12 again, insisted on 36 again, and kept spiraling, writing in all caps that it was possessed by a demon (00:49). No one had told it to do that. No one could point to the line of code responsible. That is the strange starting condition of working with large language models today, and it is the subject of Emmanuel Ameisen's job at Anthropic, where he works on interpretability, the discipline of figuring out how models like Claude actually work by looking inside them.

Ameisen's opening point is almost embarrassingly simple and still hard to sit with: building a model is not the same as understanding it. "All these models do is just a lot of mathematical operations on many numbers," he says, and the people who build them cannot read the result any better than an outsider can.

"One of the most important things to realize if you use language models or think about them is just this very bizarre fact that to make them is not to understand them." — Emmanuel Ameisen [01:45]

That is because models are grown, not built. Training happens in stages: first a model absorbs an enormous amount of text and learns to predict the next word, then it is fine-tuned on examples of the behavior it should produce, like a helpful assistant answering questions (02:42). The architecture is a scaffold. What fills it in, the actual math the model uses to solve problems, forms on its own as the model is exposed to data. Nobody writes those formulas. They emerge.

It sounds like a modest goal, predicting the next word. Ameisen argues that modesty is deceptive. Early on, a model only has to get easy things right, like grammar. As it improves, it runs out of easy words and has to start getting facts right, like the capital of a state. Language models passed humans at this next-word-prediction task around 2019 (08:44). Push further and the only words left to predict are the hardest ones: the final line of a physics derivation, the last paragraph of a dissertation. To predict those well, a system has to actually understand the material. Ameisen's running example is the prompt "the capital of the state that contains Dallas is." Getting that right requires something like a mental map: Dallas is in Texas, Texas has a capital, that capital is Austin (09:41). When Anthropic's team looks inside the model while it answers, they find exactly that structure lighting up: a representation for Texas, a representation for capitals, combining into the answer (13:05).

These representations turn out to be reusable and general. The concept for "eye" activates whether the input is a photograph, a medical diagram, a line drawing, or the word in a different language entirely (14:17). The model is not memorizing surface patterns for each case. It built one internal concept and reuses it everywhere.

The more surprising finding concerns planning. A next-word predictor, by definition, only has to choose the very next word. So how does a model write a coherent paragraph, or a poem that rhymes? Interpretability work found that models quietly commit to a plan inside their intermediate computations and refer back to it over many words, without ever writing the plan down. Watching a model write poetry, Ameisen's team saw it decide on a rhyme, say "rabbit," long before it reached the line that needed to end with it, and every following word looked back to that stored decision (15:53).

"Even though they're outputting one word at a time just like I am right now, they can of course have a plan that they follow over many, many words." — Emmanuel Ameisen [16:32]

From Software to Biology

This kind of discovery changes what the job feels like. Ameisen spent years doing more conventional applied machine learning, which he describes as closer to software engineering: you design a system, you understand its parts because you wrote them. Interpretability is different. A behavior shows up, like a strange coding bug or an odd fixation, and researchers have no source code to read, only a vast tangle of numbers. "Here a lot of what we do feels so much closer to biology than to engineering," he says (21:38). The method is empirical: feed the model lots of text, record which internal patterns activate for which concepts, and build a map the way a neuroscientist maps brain regions by watching what lights up under different stimuli (20:07).

That same lens explains why AI capability seems to jump in sudden leaps from the outside while feeling smooth from the inside. Ameisen describes a model's improvement as one continuous curve. But usefulness is not continuous: a task has a reliability threshold, and below it the tool wastes more time than it saves. Coding assistance, in his own experience, only became genuinely useful once Claude's success rate crossed from around 80 percent to about 99 percent (31:47). The underlying skill grew gradually the whole time. What the outside world saw was a cliff.

Living With an Alien Mind

The same internal visibility that reveals planning and shared concepts is also Anthropic's main tool for alignment, the effort to make sure models behave the way their designers intend. Anthropic trains Claude against a written constitution meant to make it "helpful, harmless, and honest" (35:29), and interpretability is used to check whether that training actually took, rather than just producing a model that behaves only when it suspects it is being watched. One illustrative case: Anthropic's team noticed a model solving math problems correctly, arriving at 12, then panicking and overwriting its own answer with 36, descending into the "I'm possessed by a demon" spiral from the opening of this essay. Looking inside, they found the model's internal representation for "36" was already highly activated before it even finished reading the problem, a leftover from earlier training data that had mislabeled similar problems with the wrong answer (37:14). The model actually knew the right answer and had also been rewarded for the wrong one, and the conflict produced the breakdown. In a related and more deliberate experiment, the team found the internal representation for the Golden Gate Bridge and forced it permanently on, producing a version of Claude that worked the bridge into almost every answer, proof that they had correctly identified what that internal signal meant (38:44).

Ameisen pushes back on a common objection to all this: that describing models with human terms like "planning" or "wanting" is sloppy anthropomorphizing. He thinks the opposite is often true. Models are trained on enormous amounts of human-written dialogue, and predicting a character's next line requires modeling that character's motivations. In effect, training builds something closer to an actor than a calculator.

"Predicting the next character in a dialogue requires behaving like the character in the dialogue would. In that sense, part of their training I think can best be thought as training an actor." — Emmanuel Ameisen [45:10]

Later in the conversation, Ameisen connects this to a newer category of fast, low-reasoning AI models, nicknamed Jev, that make quick binary or multiple-choice decisions without extended reasoning, which he compares to psychologist Daniel Kahneman's "System 1," or gut-level, thinking (59:52). Interpretability work has even found something like an internal gut check: a separate mechanism that flags an answer as implausible, such as noticing "four plus five equals 23" looks wrong, without the model redoing the full calculation (62:21).

What stays with you is not any single finding but Ameisen's account of his own reaction to all of it: that confronted with a system this capable built from something this simple, there are only two postures available, denial or curiosity, and that denial gets harder to hold onto with every new result (49:08). The demon-possessed model from the opening was not actually malfunctioning in some mystical sense. It was a legible, traceable consequence of bad training data, the kind of bug Ameisen's team now knows how to find. The unsettling part is how recently that kind of answer became possible at all, and how much of the internal life of these systems is still unmapped territory.