AI's Black Box May Be Tidier Than Anyone Thought
The Quick Version
- A researcher at the AI lab Goodfire says the inside of today's AI models is far tidier than experts believed.
- In an early audit of one open model, he estimates that about 90% of its "neurons" have one main job.
- He argues that the most famous examples of messy neurons were simply misread.
- It is an essay, not a peer reviewed study. The detailed technical paper is still to come, and Goodfire sells tools for looking inside AI.
- Top researchers are split. One calls it a possible turning point, while another calls it useful but not a big change.
For years, people have called AI a black box. We can see what goes in and what comes out, but the middle has looked like a hopeless tangle. A new essay argues that the tangle was partly in our heads.
On October 15, Nick Cammarata, a researcher at the interpretability lab Goodfire, published a long essay with a bold title: "Neural Networks Are Already Clean." His claim is that much of what happens inside modern AI can be read directly, if you look closely enough and use better tools.
If he is right, that is good news for anyone who has to trust an AI system. Seeing how a model reaches an answer means we could check its work instead of taking it on faith. But the claim is early, and it has not yet been tested by outsiders.
What a Neuron Actually Is
An AI model is built from millions or billions of tiny parts called neurons. Despite the name, they are not brain cells. Each one is a small piece of math that turns up when it notices a certain pattern and stays quiet otherwise.
Picture a giant office. Every worker reads the same memo and decides how loudly to speak up. If each worker has one clear job, you can understand the whole office by learning who does what.
The worry for the past decade was that many of these workers seemed to juggle jobs that had nothing to do with each other. Researchers call such neurons "polysemantic," which simply means "many meanings."
The leading explanation was called superposition. It said that when a model runs short of neurons, it crams unrelated ideas into the same one because they rarely clash. If that were true, you could not read neurons directly. You would first need a second AI tool to pull the ideas apart, and much of the field went that way.
Cammarata helped shape that view. He co-wrote "Zoom In," a well known 2020 paper with Chris Olah that first described superposition in a real network. Now he is arguing against his own earlier conclusion.
The debate in brief. Early examples of "tangled" neurons shaped a decade of research. Goodfire's essay says those examples were misread.
The Cat That Was Really a V
The most famous tangled neuron sits inside InceptionV1, an early image recognition model. It seemed to respond to two unrelated things: cat faces and the fronts of cars.
Cammarata looked again and says it is really a detector for pointed V shapes. Fox snouts and flames switch it on, while round shapes such as the letter O push it the other way. Cats and cars both happen to have V shapes, so the neuron was doing one job all along.
Same neuron, two readings. What looked like a cat and car mix is, by the essay's account, one job: spotting pointed V shapes.
He says the main example of superposition, a car signal that seemed to hide among dog neurons, was also misread. The model was simply grouping cars and dogs by the way they face the camera.
He also blames much of the old mess on dated design and weak tools. When he trained a modern model of the same size on the same images, he reports that 99 to 100% of the neurons in its final layer were easy to understand.
So why do some neurons still seem to do several things? Cammarata calls the pattern "recursive generalization." A neuron grows like a tree branch that sprouts side shoots. One neuron that spotted the letters "iy" grew to spot "diy," and then "do it yourself," which has no "iy" at all. The extra jobs are related, so they reveal how the model thinks rather than hide it.
The Numbers, With an Asterisk
The headline figures come from language models, the kind behind chatbots. In early audits of Qwen3-32B, an open model from Alibaba, Goodfire estimates that about 90% of neuron "poles," meaning each side of a neuron's range, have one main job, often with small related offshoots. About 7% have two related jobs, and only 2 to 4% are not yet understood.
Goodfire's early, unreviewed estimate for Qwen3-32B. The essay says the exact split depends on where you draw the line for "one thing."
The company says it sees similar early results in far larger models, up to trillions of parameters, such as Kimi K3. For contrast, a 2023 OpenAI project that Cammarata started found only about 1.7% of neurons readable, though that used older tools and an older model.
Now the asterisk. Cammarata says the exact numbers shift depending on where you draw lines, such as what counts as "one thing." AI agents did much of the bulk work of studying neurons. And the technical paper with the full details has not been released, so outside researchers cannot yet check the work.
Not Everyone Is Convinced
Roon, a well known OpenAI employee who posts under that name, wrote that the work "may be remembered as a major moment in scientific history." He also says he is a close friend of Cammarata.
Neel Nanda, who leads interpretability research at Google DeepMind, was far more measured. "This is interesting work, but I don't think it's a significant update to the big picture of interpretability," he wrote. He agrees that studying neurons directly could help with practical questions, such as why a model did something sketchy, and that it is much easier to apply to the biggest models.
But Nanda still believes superposition is real. He thinks AI models hold far more facts than they have neurons, so some of that knowledge is likely packed into faint signals the essay does not fully address. He also doubts that lessons from small image models carry over, and he thinks the essay defines "one job" too loosely. Still, he praised Cammarata for revisiting his own past work with modern tools.
Keep in mind, too, that Goodfire sells interpretability software. That does not make the findings wrong, but it does give the company a stake in good news about reading AI.
Why It Matters Beyond the Lab
The promise and the gaps. Every point on the right needs an answer before the points on the left can be relied on.
Today, most AI users are simply asked to trust it. If neurons really are this readable, that could change. Safety teams could look for warning signs inside a model, and auditors could ask companies to show their evidence rather than just describe it.
That is the hopeful case. The cautious case is that reading individual parts is not the same as understanding the whole. Knowing what each worker in a giant office does is a big step, but it does not tell you everything about how the office makes decisions.
What to Watch
- The technical paper. Goodfire's detailed write up is the real test, and should show how the neurons were labeled and checked.
- Independent checks. Other labs can test the claims on open models such as Qwen, so look for confirmation or pushback.
- The faint signals. Nanda's main doubt is about weak activity in neurons. Clear evidence either way would settle much of the debate.
- Real tools. Watch whether this approach shows up in safety audits and products, not just research essays.
For now, the fair reading is simple. A respected researcher who helped build the "AI is tangled" view now says it was mostly a misreading. That is worth taking seriously, and it is also worth waiting for the proof.
Sources
- Goodfire (Nick Cammarata), Neural Networks Are Already Clean, October 15, 2026
- Nick Cammarata on X, launch post for the essay, October 15, 2026
- Neel Nanda on X, response to the essay, October 15, 2026
- Roon on X, quote post on the essay, October 15, 2026
- The Decoder, Is AI interpretability close to a breakthrough? A new essay has researchers divided, October 16, 2026
- LessWrong, Linkpost: Neural Networks Are Already Clean, October 15, 2026
- Hacker News, Neural networks are clean discussion thread, October 2026
- Distill (Olah, Cammarata and others), Zoom In: An Introduction to Circuits, March 10, 2020
- OpenAI (Bills, Cammarata and others), Language Models Can Explain Neurons in Language Models, May 9, 2023