Judgment is not a benchmark

What a scoring rule removes, and why

A benchmark is a set of questions with a rule for grading them. Someone chose the questions. Someone fixed what counts as a correct answer. Everyone agreed, in advance, that the resulting number would be treated as evidence of a capacity. The instrument is excellent, and this publication depends on it: without benchmarks the ledger would be a list of impressions, and the entries for Professional exams and IMO mathematics could not carry a date.

But look at what the instrument removes, deliberately, in order to work. Nobody sitting the olympiad decides which problems are worth solving that year. Nobody sitting a bar examination will be sued if the client follows the advice and loses the house. The examination separates the capacity from the standing, on purpose, so that the capacity can be measured cleanly. That separation is the whole method. It is also the reason a score, however high, cannot tell you whether judgment has moved: judgment is precisely what was removed in order to obtain the number.

The instrument has its own troubles at the moment, and they are worth naming before leaning on it. Evaluations designed to last for years now saturate in months. The 2026 AI Index reports up to 42 percent invalid questions on some widely used tests, and public leaderboards shaped by private retries and selective disclosure.² The same report catches the shape of current systems in one line: they can win a gold medal at the olympiad and still fail to tell the time reliably.³ None of this is an argument against measuring. It is an argument about what a measurement certifies, which is always narrower than the headline it produces.

The three places a score cannot look

Judgment is exercised in three moments, and a scoring rule is blind to all three by construction.

Before the question. Somebody decides that this is the thing worth asking, and that the cost of asking it is worth paying. At the olympiad a problem committee does it. In Philip Tetlock’s forecasting tournaments the questions were written by political scientists and filtered to fall between roughly 10 and 90 percent likelihood, because questions that are nearly certain or nearly impossible reveal nothing about the forecaster. Tetlock knew what his own instrument left out and said so: foresight is one element of good judgment, “but there are others, including some that cannot be counted and run through a scientist’s algorithms.”⁴ The people who choose which questions get asked are doing most of the work, and none of it appears in anyone’s score.

At the threshold. A probability is not a decision. In the days before the Abbottabad raid, the estimates that reached the president ranged from 30 to 95 percent that bin Laden was in the compound. He said “fifty-fifty” and ordered it anyway.⁵ The number was not what settled the matter. What settled it was a judgment about what the two errors cost: raiding an empty house, and letting the possibility pass. That judgment cannot be extracted from the estimate, and its absence is not a defect of the estimate. Tetlock’s own scoring depends on the same move: a Brier score means something only once someone has fixed, in advance, how much worse a miss is than a false alarm. Pricing the two errors is a choice of ends rather than a computation, which is why Choosing ends sits in the held column of the ledger.

After the fact. When the unlikely branch arrives, someone has to say whether the decision was bad or merely unlucky. Annie Duke calls the confusion of the two resulting, and it is how people ordinarily judge one another: a forecaster who says 30 percent is declared wrong when the 30 percent happens.⁶ Assisted decisions inherit the problem twice over. The tempting moves are to blame the model when the improbable occurs, and to hide behind it when it does: the model said seventy percent. Both are ways of not answering. Reconstructing what was knowable at the time, and attributing the outcome honestly, is a judgment about judgment, and no scoreboard performs it.

Why the confusion is comfortable

The confusion between capacity and standing survives because it suits everyone at once.

It suits a vendor, who can say a system “makes decisions” when the system produces recommendations that a person signs. It suits an organisation, which can remove the person and keep the signature line. And it spares the rest of us a tiring question: what a person is actually for, once the answers are cheap.

There is an honest answer, and it is not flattering. A person is for being answerable. That is the position which cannot be delegated without disappearing altogether, and it is why the held column of the ledger looks the way it does. It contains no talents. Nothing in it claims that people write better, reason faster or see further. It contains positions: Responsibility, Office, consent, witness, the choice of ends. Joseph Weizenbaum drew the line in 1976 and it has not moved since: deciding is a computation, choosing engages values, and “there is all the difference between deciding and choosing.”⁷

Julia Galef has the sharper image for what a system produces when it produces a claim. She distinguishes the press secretary, who makes claims, from the board, which makes bets.⁸ A model is structurally a press secretary: fluent, calibrated on request, holding nothing. Ask it for a probability and you get a number. Ask a person for the same number and you can ask a second question, which is what they would say if it were their money, their licence, their name on the file. The second question is the one that costs something to answer.

The arrangement to avoid

Between the two clean positions, keeping a person for standing or accepting the machine’s output on its merits, there is a middle one that collects the drawbacks of both. The person is kept “in the loop” as a formality: present enough to be blamed, absent enough to have any real say.

This is not hypothetical. Madeleine Clare Elish named the failure mode after studying automation accidents: the moral crumple zone, in which “responsibility for an action may be misattributed to a human actor who had limited control over the behavior of an automated or autonomous system,” protecting the integrity of the technical system at the expense of the nearest operator.⁹ Lisanne Bainbridge had explained the mechanism four decades earlier: the designer leaves the operator exactly the tasks that could not be automated, then removes the daily practice that would keep them able to perform those tasks.¹⁰

The arrangement has now been measured. In a randomised study of diagnostic reasoning on written cases, the model alone scored 92 percent on the rubric, physicians with their usual resources 74, and physicians given the model 76 (Medical diagnosis).¹¹ Read the third number twice. The machine did not fail. The pairing did: the doctors kept their authority and gained an assistant, and the combination performed barely better than the doctors alone and far worse than the machine alone. Whatever that arrangement was, it was not each party doing what it does best.

The law has begun to describe what a genuine position requires, which makes a useful check on any process. The European regulation on high-risk systems asks that oversight be exercised by people who understand the system’s capacities and limitations, who remain aware of automation bias, who can disregard the output, and who can stop the machine.¹² That is the difference between holding a position and holding the liability.

What follows in practice

Where a process asks for a capacity, measure it. Let the benchmark do its work, and let the ledger record honestly when a line moves, with a date and a source. There is nothing dignified in pretending that a task has not been ceded.

Where a process asks for standing, keep a person and give them the means: the time to disagree, the information to disagree with, and the authority to stop the thing. A signature obtained from someone who could not have refused is not oversight. It is paperwork.

And watch the entry for Long-horizon autonomy, which is contested and will be the hardest test of the distinction drawn here. A system that carries a task for hours or days will be asked to carry the standing as well, because interrupting something that has been working unattended since Tuesday feels absurd. The ledger will say when the capacity moves, and the measurement is improving: the doubling is now counted in months rather than years.¹³ It will not say that the standing moved with it, because a benchmark cannot certify that, and never could. It is built by removing precisely the thing that would have to be certified: who asked, what an error costs, and who will answer on the morning the improbable arrives.

There is a line in the held column for Moral judgment, and it may not stay there forever. If it moves, it will move with a date and a source, and not with a sentence written the week after a gold medal.

Notes

1. An advanced Gemini Deep Think model scored 35 of 42 at IMO 2025, gold-medal level, graded by the olympiad’s coordinators (Google DeepMind, July 21, 2025). OpenAI reported the same score for an experimental model days earlier, graded by former medallists rather than by the IMO. Five human contestants scored a perfect 42 that year. ↩

2. Stanford HAI, AI Index 2026: “Evaluations intended to be challenging for years are saturated in months”; reliability and gaming concerns “with error rates up to 42% on widely used evaluations.” On leaderboard distortion, Singh et al., “The Leaderboard Illusion,” April 29, 2025, documents private variants tested before release and gains of up to 112 percent from overfitting to an arena. ↩

3. Stanford HAI, AI Index 2026: “AI models can win a gold medal at the International Mathematical Olympiad but still can’t reliably tell time, illustrating what researchers call jagged intelligence.” ↩

4. Philip E. Tetlock and Dan Gardner, Superforecasting (2015), chapter 12. On question selection in the Good Judgment Project tournaments, see the same chapter, and Tetlock, Expert Political Judgment (2005), chapter 1, on the clairvoyance test. ↩

5. Superforecasting, chapter 3, on the estimates presented before the Abbottabad raid and the president’s “fifty-fifty.” Tetlock reads the phrase either as a failure of aggregation or as an executive shorthand for a threshold already crossed. ↩

6. Annie Duke, Thinking in Bets (2018), chapter 1, on resulting. On judging a decision by what was knowable when it was made, see Howard Marks, The Most Important Thing (2011), chapter 16. ↩

7. Joseph Weizenbaum, Computer Power and Human Reason (1976), p. 259. ↩

8. Julia Galef, The Scout Mindset (2021), chapter 6: “The press secretary makes claims; the board makes bets.” ↩

9. Madeleine Clare Elish, “Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction,” Engaging Science, Technology, and Society, 2019. ↩

10. Lisanne Bainbridge, “Ironies of Automation,” Automatica 19(6), 1983. ↩

11. Goh et al., JAMA Network Open, October 28, 2024: 50 physicians, written cases; the model alone at 92 percent on a diagnostic reasoning rubric, physicians with conventional resources at 74, physicians with the model at 76. ↩

12. Regulation (EU) 2024/1689, Article 14, on human oversight of high-risk systems: understanding capacities and limitations, remaining aware of automation bias, being able to disregard the output and to interrupt the system. ↩

13. METR, Time Horizon 1.1, January 29, 2026: the length of software task a model completes half the time has doubled roughly every seven months since 2019 and about every four months since 2024, with wide confidence intervals and a task set that is mostly software. ↩