OpenAI Now Watches Its AI's 'Brain Activity' as It Writes
Starting with GPT-5.6, OpenAI runs small classifiers on its top models' internal activations for every reply, and can stop an answer mid-sentence.
OpenAI is now watching what its most powerful models "think" as they write, not just what they say. Starting with GPT-5.6, every assistant reply from its most capable models passes through small classifiers that read the model's internal activity token by token, and can stop the answer partway through. It is one of the clearest signs yet that interpretability research has left the lab and become part of everyday AI safety.
OpenAI described the system in a 2 November post on its alignment research blog by Alex Makelov, Xiaojun Xu, Sundaram Ananthanarayanan, Ethan Mcdonald and Tom Dupre la Tour. The company says the classifiers, which it calls probes, run on both internal and external traffic. Pivot News, explainx.ai and Frontier Now have since picked up the story.
How the probes work
A language model processes text through layers of numbers called activations. OpenAI describes them as the model's "neural patterns." A probe is a lightweight model that reads those activations, usually from a layer near the middle of the network, and decides whether the behavior looks misaligned or breaks a policy. It can also flag a user with harmful intent.
The probes run directly on the inference engine and give a score for every token the model produces. When a score crosses a threshold, OpenAI stops streaming the reply to the user and sends the message to a reasoning model for a final judgement. Tool calls are blocked until they are marked safe.
That is the key difference from the text classifiers OpenAI already uses. A text filter has to wait for words to exist before it can judge them. A probe sees the warning signs inside the model as the answer forms.
OpenAI trains the probes on up to 1 million messages, drawn from synthetic and de-identified conversations, with labels written by its most capable reasoning models acting as judges.
What OpenAI claims, and what it admits
The performance claims come from OpenAI alone and have not been independently tested. The company says probes held up better against the jailbreaks it tested than its text-based classifiers, including much larger ones. It says they are cheaper to run, easier to make "fail-closed," and get more accurate as the monitored model gets more capable. OpenAI also cautions that the figures in its post come from internal experiments and are "not necessarily representative of deployed classifiers."
The limits are spelled out plainly. Probes are trained with labeled examples, so they can only catch "known and measurable" kinds of bad behavior. A probe trained on one model does not work on another, so each new base model needs fresh probes, and accuracy can slip as a model's weights change. OpenAI also notes robustness is highest against attacks that were not designed to beat the probe itself.
Why it matters
For years, interpretability meant researchers studying a model's insides after the fact. OpenAI is now using those insides as a live tripwire. OpenAI calls probes one layer of "defense in depth," next to text filters, model review and human response, not a replacement for them.
That raises a question every builder should be asking. If the best signal of what an AI is about to do lives inside the model, who outside the company gets to check it?
Photo: "EEG Recording Cap" by Chris Hope, via Wikimedia Commons, CC BY 2.0. Graded, extended and overlaid by PSI.
Sources
- Training activation classifiers for safety and alignment (OpenAI Alignment blog)
- OpenAI deploys activation classifiers to cut off unsafe GPT-5.6 replies mid-stream (Pivot News)
- OpenAI Activation Probes: Token-by-Token Safety Monitor (explainx.ai)
- OpenAI deploys activation probes on GPT-5.6 (Frontier Now)