JudgementBench: how good are LLMs at judging when AI output is good?

Modern AI gives us a discrimination problem: it’s easy to get tons of ideas and bits of written content, and hard to know which have any merit. For research automation to work, you need some way to validate when pieces of work are good.

The easy case is when you have precise validation — e.g. checking a mathematical proof, or looking at experimental results. But conceptual and macrostrategy research lacks this kind of solid grounding. It still seems possible for humans to distinguish good work from bad, but it’s more taste-based, and judgements are correspondingly noisier.

Even approximate validation could be an important part of research automation. We ask: what could this look like? How can we measure it? Which AI systems are good at it? JudgementBench is our prototype benchmark assessing this.

How it works

The benchmark consists of several macrostrategy-relevant rating tasks.

We constructed each task in the benchmark in two phases:

  1. We chose a brainstorming prompt (e.g. “Ways democracy may predictably change with advanced AI”), and had AI systems generate candidate answers.
  2. We have human researchers grade all of the answers; with discussion in cases of disagreement.

Then for the actual benchmark, the task is to grade the answers. A model is evaluated by how well its scores correlate with the human gold scores. Before correlating we smoothly compress the range of low scores on both sides, because we’re less interested in gradations of judgement about how bad bad ideas are. A score of 1 would be perfect agreement with the humans, and 0 no relationship at all.

Of course even the best human judgement isn’t a perfect ground truth about which answers are actually best. Based on tests where a random half of a seminar group is scored against the other half, we suspect that the noise ceiling is in the vicinity of 0.75.

Moreover, the underlying task here is judging AI output quality. This is a moving target, as it changes with the strength of the underlying AI systems used for generation. Our experiments so far don’t have large enough datasets per-generator to test transfer, so we are presenting results aggregated across different models generating. But we are conscious that what strong performance looks like may change with time; and might already look different for different generating models.

Results

Bars are 90% confidence intervals for a model’s score relative to the other models, over the underlying distribution of tasks. A task that is hard for every model is not uncertainty about which model is better, so each task’s average across the other models is subtracted before the interval is taken; what remains is mostly task-to-task variation in a model’s relative standing, with run-to-run noise a small share. The preferred scaffold is the same object under all views.

When ensembling, we present the items-to-be-judged in different random orders each time. The “plain ask” just gave the judges the generating task description and asks them which items are best. “Minor prompt engineering” mostly gives some generic (not task-specific) advice about what might make for good answers; but in some cases it also corrected some questionable wording choices in the original generating task descriptions, which had been introduced by coding agents. Since questionable-wording-choices-introduced-by-AI-agents are a feature of the world today, we decided it made sense to report these results as well.

This initial version of JudgementBench is somewhat noisy; fine-grained rankings should be taken with a pinch of salt. We do however think our results support reasonably high credence in the following:

  • Recent progress on judging macrostrategy-relevant ideas has been very incremental compared to progress on easier-to-verify tasks. For example, the gap between Opus 4.6 and Fable 5.1 is well within noise, and Opus 5.5 scores lower than 4.6. Suggestively, though not demonstrated at this sample size, the line of models from Opus 4.7 to 5.5 all score below 4.5 and 4.6 on the task; the individual differences sit at the edge of the error bars, but the pattern is consistent across the line.
  • The benchmark is not yet saturated by unscaffolded models: we’ve shown that it’s possible to do significantly better than frontier chain-of-thought alone.
  • The gains available at the frontier go beyond subtle or subjective improvements in taste: the best-performing models still occasionally make quasi-objective errors. Astra will sometimes miss or underweigh critical flaws in the ideas, leading it to occasionally prefer severely flawed proposals over perfectly reasonable ones.
  • Large frontier models (Fable and Astra) seem to gain less than older models from ensembling.

A note on publication. For now we don’t intend to publish the benchmark, because we’d prefer to retain the ability to discriminate on new models without worrying that the benchmark has been trained against.