The Ledger · Entry 175 · Science
Reviewing a scientific paper
ContestedSince 2025Criterion performanceLast reviewed 2026-08-18
Event
For AAAI-26, one clearly labelled machine review is generated for each of the 22,977 papers that reached full review, in less than a day and for under a dollar a paper, alongside the human reviews.
The pilot was explicit about what it withheld. The machine reviews carried no score and no recommendation, no human reviewer was replaced, and the organisers stated that no human decision was being replaced by a machine decision. Read carefully, that is the interesting part. A review written for every submission at a conference receiving more than thirty thousand papers is a volume no committee can supply, and the pressure that makes it attractive is the same pressure that has made human review hard to obtain. What a reviewer does that the pilot did not ask a machine to do is stake a name on a recommendation and answer for it to the authors and to a chair. EMNLP 2026 is running an experiment of its own. The line is open, and the thing to watch is the workload rather than the prose.