Goodfire's Neurons as an Interpretable Basis

In 2020, the engineer Noam Shazeer published a five-page paper proposing a new design for the inner layer of a transformer, the architecture behind every modern language model. The new design worked better. He didn't know why. "We offer no explanation as to why these architectures seem to work," he wrote. "We attribute their success, as all else, to divine benevolence." Six years later, that component, called a gated MLP, sits inside nearly every frontier model, and the field's working assumption has been that its inner workings are close to unknowable: millions of neurons, each one meaning several different things depending on context, wired together in patterns too tangled to read directly. You need a helper model, the thinking went, a sparse autoencoder trained to repackage the mess into something legible.

Kurt Tigges, a member of technical staff at the interpretability startup Goodfire, thinks that assumption was too pessimistic. Working with Nick Cammarata, he co-authored a new paper, "Neurons as an Interpretable Basis," that argues individual neurons in large language models are more directly readable than the field believed (07:36). Not perfectly readable. Not without effort. But readable enough that the helper model may often be unnecessary.

"The individual neurons in MLPs are more interpretable than the field has thought to this point." — Kurt Tigges [07:36]

Inside the gate

To see why this is surprising, it helps to look at what a modern neuron actually does. A multilayer perceptron, or MLP, is the part of a transformer that processes information after attention has decided which words to relate to which. In the gated version Tigges studies, each neuron gets two separate signals from the layer before it: an "up" signal and a "gate" signal, computed by two entirely different sets of weights (15:18). The gate decides when the neuron should fire; the up projection decides how strongly. They're multiplied together, which lets a neuron act like a small logic circuit: fire only when both a context and a specific detail are present, the way a neuron might stay quiet during a recipe discussion generally but spike specifically at the word "garlic" (16:20).

The model Tigges's team studied in depth, Qwen 3.6 35B-A3B, has roughly 5 million such neurons spread across 40 layers, divided into 256 experts per layer of 512 neurons each, with only 8 experts active on any given token (01:09). Because the gating function (called SiLU) allows negative values through, each neuron effectively carries two meanings, one for its positive activation and one for its negative, which means the 5 million neurons behave more like 10 million distinct signals (13:39).

What survives when you delete almost everything

The paper's central experiment is almost gleefully destructive. Tigges's team ranked every neuron's activation strength and then ablated, meaning deleted or zeroed out, roughly 90% of them, keeping only the strongest spikes (02:46). The result, on most benchmarks, was barely a dent in performance. On some ArcAGI puzzle sets it actually improved, apparently because a model with fewer competing signals reaches its answer faster and with more confidence, instead of "distracting itself" with marginal activity (37:28). Reasoning tasks did suffer at first, with the model looping and second-guessing itself, but a small amount of additional training, about 100 million tokens, roughly a single day on a few GPUs, restored most of the lost performance, and in some cases pushed scores five to fifteen points above the original (96:54). Tigges's read is that the sparsified model hadn't lost its knowledge; it had lost its calibration about how confident to be with 90% of its capacity gone (98:10).

That finding matters because it suggests the field's intuition about where meaning lives in a network may be wrong in an important way. If only the strongest 10% of neuron activations are doing most of the work, then the object worth interpreting is smaller and more tractable than previously assumed.

An agent that reads five million neurons

Scale is still the obstacle: five million neurons is too many for a human to read by hand. Tigges's team built an AI agent they call the neuron scientist. Given a cluster of text passages where one neuron fires, the agent writes a paragraph-length hypothesis about what unifies them, then tests it: it writes synthetic sentences, runs counterfactual edits, and checks whether the neuron behaves as predicted (42:47). A separate, independent model then reads only the written description, never the agent's reasoning, and scores each token in a new passage for how strongly the neuron should fire there (51:28). Those predicted values are patched into the real model in place of the neuron, and researchers measure how much of the damage from deleting that neuron gets undone. The answer: about 74% of the resulting KL divergence, a standard measure of how much a model's token-by-token predictions have been disturbed, gets recovered this way (51:02). That's a meaningful validation, because a vague or wrong description would not restore nearly that much behavior.

"The neuron scientist's job is to come up with a hypothesis that unifies these different examples where the neuron becomes active and then predicts where it'll activate in the future." — Kurt Tigges [42:47]

What the agent often finds complicates the simple idea of monosemanticity, the notion that a neuron represents exactly one concept. Tigges describes a neuron that fires on the diet, habitat, and egg-laying sites of insects, three subtopics that may or may not count as "one concept" (57:39). He describes another that fires on clothing, exercise equipment, dishware, and tablecloths, a seeming jumble until you realize the unifying idea might be "things you'd find at a garage sale," a pattern the model could represent without ever announcing it to a human observer (58:48). The useful question, in Tigges's framing, isn't whether a neuron is strictly mono- or polysemantic. It's whether its full range of meanings can be listed and understood at all.

This leads to a reframing the paper leans on hard: thinking of neurons less as nouns, static labels for concepts, and more as verbs, signals that direct what happens downstream. Tigges describes neurons tied to reasoning that push a model to be more or less confident, to pause and recheck, or to abandon one approach for another, functioning as control-flow gates rather than passive detectors (64:44).

That reframing connects to a question that's long puzzled people watching these models: where do the facts live? Tigges's answer is that no single neuron holds a fact. A model with 5 million neurons plainly knows more than 5 million facts, and the way out of that puzzle is that facts live in the connections between neurons, not inside them, since a modest number of neurons wired together can represent an enormous number of possible combinations (81:53).

"A neuron is not a fact, but the fact is in the connections between neurons." — Kurt Tigges [81:53]

The same logic extends to the experts in Qwen's mixture-of-experts design, a setup where only a handful of specialized sub-networks activate per token. The older view held that experts mix many unrelated topics together in ways resistant to analysis. Tigges's team instead found experts concentrated on one to three narrow topics each, such as algebraic notation, units of time, or constitutional law (85:53). The router, it turns out, isn't picking among chaos; it's picking among genuine specialists.

None of this amounts to a finished map of how these models think. Tigges is candid that the residual stream, the running total of information passed between layers, remains a far messier object than the MLP neurons, still entangled in ways sparse autoencoders were originally built to address (79:28). And a related paper on transcoders found that their learned features mostly just reconstruct individual neurons or small combinations of them, which argues for studying neurons directly rather than through another translation layer (78:37). The host, Nathan Labenz, frames the honest summary near the top of the episode: the field had assumed helper models were required, and now there's real reason to think neurons can be studied directly at scale, but whether that understanding can be assembled into a full account of model behavior is still an open question.

"Interpretability remains unsolved." — host Nathan Labenz [05:22]

That caution matters, because the stakes attached to this work are not academic. If future architectures could be nudged toward sparser, more localized representations without paying much of a performance cost, that would change the calculus around how much compute companies need to spend on safety and monitoring, a tradeoff Nvidia's Jensen Huang has publicly estimated at 80 to 90% of total resources (87:20). Tigges's data suggests that tax might be smaller than feared, because these models may already be more legible than anyone thought to check.