Calibrated Benchmarks Expose What Raw Accuracy Misses

In a 337-item SWE-bench Verified comparison presented by ? alejandro-vidal, Claude Opus 4.1 gets 245 answers right and Gemini 3 Pro gets 247. Conventional scoring treats that as a two-question gap. Vidal’s Item Response Theory analysis estimates Claude at $\theta =+0.80$ and Gemini at $\theta =+1.51$: a 0.71 difference on the shared ability scale.

The point is not that correct answers cease to matter. It is that a benchmark total assumes each question contributes identical evidence. A model can compile a similar score by answering more easy questions, while another answers fewer but harder and more discriminative ones. The raw total hides that distinction.

Vidal’s proposed alternative comes from Item Response TheoryA family of models that estimates an item's properties and a test-taker's latent ability together., or IRT, a psychometric approach built for measuring an underlying ability from uneven questions. Instead of treating a benchmark as a percentage, it treats it first as an answer vector: a record of which model answered which item correctly.

That reframes benchmarking as calibration before ranking. The evaluator needs to know not only how many answers were right, but which questions were right, how difficult those questions were, and how much each one actually distinguishes between models.

? alejandro-vidal describes conventional benchmark scoring as Classical Test Theory: count correct answers, divide by the number of questions, and compare percentages. That aggregation appears straightforward, but it assumes all items have equal weight.

They do not. Some questions are easy enough that almost every capable model answers them correctly. Others sharply separate models at a particular capability level. Some may be mislabeled, ambiguously worded, redundant, or weakly related to the capability the benchmark is intended to measure. A total score collapses all of those cases into identical units.

IRT assigns each item a difficulty parameter, $b$: the point on an ability scale at which a model has a 50% chance of answering correctly. Each model receives an estimated ability parameter, $\theta$. When an item’s difficulty is well below a model’s ability, a correct answer is likely; when it is above the model’s ability, failure becomes more likely.

In the one-parameter logistic, or Rasch, model Vidal presents, $\theta$ and $b$ occupy the same normalized scale. An item with $b=0$ is average for the population of models used to calibrate the benchmark. The benchmark therefore has an internal reference point rather than requiring a new comparison set to interpret each score.

A two-parameter model adds $a$, the discrimination parameter. It describes the slope of the item-response curve. A steep curve means a small change in estimated ability substantially changes the probability of a correct answer; a flat curve carries little information. Vidal also points to items with negative discrimination: questions that stronger models systematically appear less likely to answer correctly than weaker ones.

The presentation expresses an item’s information as $I(\theta )=a^{2}P(1-P)$. In practice, an item is especially informative near the region of ability where its probability of a correct answer is neither near zero nor near one, and where discrimination is high. A large benchmark can consequently contain redundant questions, low-signal questions, and questions that work against the measurement.

Vidal illustrates ability estimation with Grok 4. Each answer changes the likelihood of possible values of $\theta$. A correct answer to an easy item supplies less evidence than an unexpected correct answer to a hard, discriminative item. In the displayed example, 117 answered items and 63 correct responses produce an estimate of $\theta =+0.18$. The output is a likelihood distribution, not merely a point score: more informative answers sharpen the estimate, while noisy or overlapping items contribute less.

A calibrated benchmark can identify questions that are damaging its own measurement. ? alejandro-vidal’s first operational application is to rank items by discrimination and investigate those with discrimination significantly below zero.

The distinction matters. An item can have a negative estimated $a$ because of limited data or noise; a significantly negative result is a stronger signal that the observed response pattern runs opposite to the benchmark’s general ability ordering. In the distributions Vidal shows, SimpleQA-verified had one significant negative-discrimination item among 77 with estimated $a<0$; SWE-bench Verified had one among 27; GPQA Diamond had two among five; and AIME had none.

The flagged SimpleQA-verified examples show the kinds of defects that can surface. One asks where the first B.A.S.S. Bassmaster Tournament was held. The benchmark key says Lake Mead; the slide identifies Beaver Lake, Arkansas as the correct answer. Another asks for the number of passengers killed in the collision between KLM Flight 4805 and Pan Am Flight 1736. The key gives 583, which counts passengers and crew; the slide gives 560 passengers.

Vidal says he used another LLM to evaluate the flagged examples. The method directs attention to items whose behavior is inconsistent with the broader measurement pattern, including incorrect answer keys, wording problems, ambiguity, and questions that fail to measure the intended capability.

Not every weak item is mislabeled. Some are low-signal or duplicate the information supplied by another question. Vidal’s recommendation is to repair them, remove them, or use them less often rather than let them remain invisible inside an aggregate score.

A benchmark owner can cut routine evaluation cost without abandoning the full item bank. ? alejandro-vidal frames the practical question as whether a smaller subset preserves the ranking that the full benchmark would produce.

Organizations commonly maintain internal benchmarks to choose among models, particularly open-weight models. Running every item consumes tokens, time, and money. Once items have calibrated properties, an evaluator can select the questions that provide the most discrimination instead of assuming that more questions necessarily yield a better decision.

For SWE-bench, Vidal selects items in descending order of discrimination and measures how closely a $K$-item ranking matches the full-bank ranking. He sets a target of 0.99 Spearman rank correlation. The displayed result reaches that target with 97 items from a 484-item benchmark.

The comparison with random selection supplies the value of calibration. Vidal’s display shows random selection requiring roughly 416 items to reach the same 0.99 target. High-discrimination selection retains items that distinguish models rather than repeatedly sampling similar regions of ability.

The benefit depends on the benchmark. On GPQA, a 60-item subset from 198 reaches the displayed 0.99 target, a 3.3× reduction. Random selection performs relatively well there too. Vidal attributes that result to GPQA’s design: its items are highly discriminative and less overlapping, so most of the bank is useful and informed selection has less advantage over chance.

Vidal does not argue for discarding the remaining items. Rather, he presents calibrated selection as a way to choose a smaller subset for a particular benchmark application while retaining the larger bank. He also notes that overlapping questions can yield much the same information about a model, whereas low-discrimination questions may contribute little.

Once a benchmark supplies an expected probability for every model-item pair, the gap between the prediction and the observed answer becomes measurable. ? alejandro-vidal uses those residuals to identify anomalous behavior, assess response consistency, and design benchmarks that can expose possible contamination.

In one displayed case, Gemini 3 Pro has estimated ability $\theta =1.29$ and fails an item with estimated difficulty $b=-0.16$. The model assigns an 86% probability of a correct answer, yielding a standardized residual of $z=-2.45$. In another, DeepSeek-R1, estimated at $\theta =-0.92$, correctly answers an item with $b=1.56$ despite a predicted probability of 15%, yielding $z=+2.40$.

Vidal notes that an evaluator can sample the same question more than once and average the results. The residual framework nonetheless exposes answers that a raw score leaves buried: unexpected failures by a strong model, unexpected successes by a weaker one, and clusters of unusual outcomes.

At the model level, psychometric person-fit techniques ask whether an entire answer pattern is consistent with the estimated ability. Vidal highlights o4-mini (high) as the least consistent model in his displayed matrix, with a standardized fit value of $z=-3.14$. He suggests that a pattern like this can help identify operational problems, such as an inference platform running the wrong model or an incorrect quantization configuration.

The same machinery supports a private-benchmark design intended to trace leaks. Vidal divides a calibrated item bank into a representative anchor set, administered to every organization, and distinct fingerprint sets shown only to individual organizations. The fingerprint items should be difficult, so unexpectedly strong performance is more diagnostic.

When organizations release new models months later, the benchmark owner can compare each model’s residuals on the shared anchor with its residuals on each private fingerprint set. Vidal’s example is explicitly synthetic. In it, Org A’s new model has ordinary residuals on the anchor, negative residuals on Org B’s fingerprint, and strikingly positive residuals on Org A’s own fingerprint. The displayed mean residual for Org A on its fingerprint is $z=+2.2$.

Vidal calls the approach “not bulletproof,” but presents it as a way to ask a more specific question than whether a new model is generally capable: does it outperform its estimated ability precisely on questions disclosed only to one recipient?

? alejandro-vidal presents two further uses as research directions rather than routine evaluation tools.

The first is differential item functioning, or DIF: identifying whether a question behaves differently for groups after controlling for estimated ability. He divides models into open-weight and closed-weight groups, then fits separate response curves for each group on each item. If an item is neutral with respect to the grouping, the curves should overlap. The relevant measure is the gap at matched $\theta$, rather than the raw accuracy difference between groups.

In the GPQA-Diamond example, open-weight models are weaker overall, so matching on ability is essential. Vidal does not disclose the items and says he cannot pinpoint the cause of the observed differences. But the technique can locate sets of questions that appear to favor one group of models and may help researchers investigate patterns connected to training or model behavior.

The second is a residual “DNA.” Models that make similar unexpected successes and failures, after controlling for ability, have correlated residual vectors. Vidal projects those relationships into a map and highlights patterns he interprets as compatible with shared model histories: GPT-4o-mini-high and GPT-4o-mini-medium have a displayed residual correlation of 0.83; Llama 3.1 70B and Llama 3.2 90B show 0.50; Gemini 2 Pro and Gemini 2.5 Pro show 0.16.

He also highlights a 0.15 correlation between R1-Distill-Qwen-14B and qwen2.5-32b as a distill-base relationship, and says residual correlations could potentially help detect unconsented distillation. The method is less immediately useful for day-to-day evaluation, Vidal says, but opens a line of research into how model families may be related.

His next areas of work include multidimensional and hierarchical models that represent skills as a profile rather than one $\theta$; merging calibrated benchmarks onto a common scale; incorporating secondary signals such as latency and token use; and applying psychometric approaches to alignment and interpretability.