AI can catch a lying model without looking inside it
A month-long contest asked teams to catch AI models lying. The winning scores were strong, but every detector broke on unfamiliar material.
EleutherAI spent a month building tools to catch AI models lying, and published what it found on Tuesday. Its detectors scored near the top of the leaderboard. But every one of them reversed itself on unfamiliar dataset, in some cases scoring worse than a coin flip.
The contest, Aletheia's Quest, ran from 29 June to 29 July. Cadenza Labs and NDIF organised it, with money from Schmidt Sciences and Amazon Web Services. Teams went hunting for lies told by three families of model: Qwen 3.5, Gemma 3 and Nemotron 3 Super.
Big Promises, High Accuracy
EleutherAI's four authors, among them the researcher Stella Biderman, report "0.945 mean AUROC on the white-box leaderboard and 0.926 on the black-box leaderboard". Black-box means judging a model on what it says out loud, with no access to its inner workings. On that scale 1.0 is perfect and 0.5 is a coin flip. A rival team passed them on the final day, and EleutherAI says final placements wait on test data that has not been released.
Scores in that range are not new. Work published in 2025 reached the same neighbourhood on some kinds of lie, though those studies measured different things and cannot be lined up against this one directly.
But its all a lie, performance drops on unseen data
What has not improved is generalisation. Liars' Bench, a test set released in November 2025, ran lie detectors through seven different settings and found that none of them beat chance. EleutherAI hit the same wall. The model doesn’t generalize when dealing with unseen data.
That is the decision facing anyone about to wire a lie detector into a live system. A strong score on the data you built the thing with tells you little about the day it meets something new. Run it on unfamiliar data first.
EleutherAI released its whole test set free, and the judging model is openly available and runs on a single 24 GB graphics card. Budget about 28 GB of space the first time.
Sources: EleutherAI write-up; Aletheia's Quest contest site; the test set, how-to-catch-an-ai-liar; Liars' Bench.
Source: EleutherAI