The favourite recipe for shrinking reasoning models has a hidden flaw
Shanghai AI Laboratory says the most popular way to distill small reasoning models quietly teaches them to lean on information they will never see in production — and its fix recovers the two to three benchmark points the standard method leaves on the table.
What happened
In a paper posted to arXiv on August 3 and featured on Hugging Face's Daily Papers, researchers at Shanghai AI Laboratory identify what they call the "privilege illusion" in on-policy self-distillation — the technique many teams now use to compress reasoning ability into small models. During training, the teacher model conditions on reference answers the student will never see at inference time. The student quietly learns to lean on that privileged information, and the crutch disappears in deployment.
The numbers make the case. The standard method's advantage collapses with scale — from just over five benchmark points on a 1.7-billion-parameter student to essentially nothing at 32 billion. The proposed fix, Dual-Anchored Policy Distillation, closes the information gap so teacher and student always see the same context, and holds a two-to-three point edge across the whole size range on reasoning and coding benchmarks.
Why it matters
Distillation is the workhorse behind nearly every capable small model shipping today — the models running on phones, laptop chips, and single consumer graphics cards are almost all students of much larger teachers. If the standard recipe silently trains your student to depend on information it won't have in production, your evaluation numbers during training are lying to you about deployment quality. That is exactly the failure mode edge teams can least afford, because on-device models rarely get a second chance after they ship.
There is a reliability angle too. The paper reports that students trained the new way make far fewer confidently wrong claims — roughly a seventy percent reduction — the kind of output that erodes trust in small models faster than raw benchmark gaps do.
Key details
The flaw is an information-asymmetry problem, not a capacity problem: the teacher sees reference answers during training that the student lacks at inference, and the student optimises against that mismatch. The fix anchors training so both models work from matched context, applied in both directions. Results so far cover one model family (Qwen3 students from 1.7 to 32 billion parameters), no code or checkpoints have been released yet, and community traction is early — the paper appeared within the last day.
Bottom line
If your small-model pipeline uses on-policy self-distillation, audit it for train-versus-inference information mismatch this week — especially if your student is eight billion parameters or larger, where the paper says most of your distillation gain may be illusory. This is a training-recipe change, not a new model, so it is cheap to trial against your current run once code lands.
Sources: arXiv 2608.01735, Hugging Face paper page, AI Weekly coverage.