The number your LLM judge reports is not the quality of the model it grades. It's that quality passed through an imperfect classifier, and the classifier tilts the result in a predictable direction. A run of 2026 papers now quantifies the tilt: raw judge scores are biased rather than merely noisy, the bias survives on frontier judges, and the standard fixes people reach for recover less than teams assume. If you gate releases, route traffic, or report model quality off a judge, the score you trust is measuring the judge as much as the model.
Key takeaways
- A judge model is a classifier with sensitivity and specificity below 1, so raw pass rates are systematically biased, not just noisy (James Fiedler, arXiv:2605.06939).
- Swapping one judge for another moves verdicts on identical answers. MiniMax-M2.7 flips 14.7% of its own verdicts simply by reversing which answer is shown first (Yang et al., arXiv:2607.08535).
- Juries of similar judges barely help. Measured error correlation reached 0.944 to 0.972, so adding judges of the same family buys almost no independent signal (Yang et al.).
- High agreement between judges is not accuracy. Judges can agree with each other while diverging from ground truth, a reliability-without-validity failure (Norman et al., arXiv:2606.19544).
- The Rogan-Gladen correction can recover an unbiased score from a judge of known quality, but sharing one calibration across two models being compared can push the comparison in the wrong direction with high apparent confidence (Fiedler).
What does it mean to call the judge a biased instrument?
It means the judge is a measuring device with two error rates, and both of them shift your headline number. Treat the judge as a binary classifier that labels each response pass or fail. Its sensitivity is the rate at which it correctly passes a genuinely good answer. Its specificity is the rate at which it correctly fails a genuinely bad one. Neither reaches 1 in practice.
When both sit below 1, the observed pass rate is a mix of true passes the judge caught and true fails it waved through, minus the true passes it wrongly rejected. James Fiedler's Bias and Uncertainty in LLM-as-a-Judge Estimation states the consequence directly: "the naive estimator, i.e., raw judge outputs, is systematically biased." The bias has a sign and a size set by the judge's error rates, so it does not average out as you collect more samples. More data tightens the confidence interval around the wrong center.
That reframes what a judge score is. It's an estimate produced by an instrument whose calibration you rarely measure, reported as if it were the quantity itself. A senior engineer would treat any uncalibrated judge number the way they treat a sensor with unknown drift.
Why does swapping the judge change the score?
Because the measurement is a property of the judge, not of the responses, changing the judge changes the reading even when nothing about the candidate model moves. Zongyou Yang, Yinghan Hou, and Xiaokun Yang name this the "evaluator-replacement ambiguity" in When the Judge Changes, So Does the Measurement, and they argue it's a validity problem rather than an implementation detail.
Their audit runs two axes: a parameter axis over Qwen3 from 1.7B to 32B, and a release axis over MiniMax M2 through M2.7 API versions, across four datasets including LLMBar's 419 adversarial pairs and 1,997 Chatbot Arena samples. Scaling up a judge does not reliably scale up its agreement with ground truth. Qwen3 improves from 0.463 to 0.617 accuracy on LLMBar going 1.7B to 4B, yet adjacent MiniMax releases differ by at most 0.022. Upgrading to a newer judge you assume is better can leave the measurement flat, or move it for reasons unrelated to quality.
The operational implication is uncomfortable. A leaderboard, a regression gate, or a quality trend line is only comparable across time if the judge stays frozen. Change the judge model, change its version, or change its prompt, and last quarter's numbers no longer sit on the same scale as this quarter's.
How much do position and verbosity bias move a verdict?
Enough to flip real decisions, and the effect persists on strong judges. Yang et al. report that "MiniMax-M2.7 still changes 14.7% of verdicts under A/B reversal," meaning nearly one verdict in seven depends on which answer the judge happened to see first. Position sensitivity is the best documented of these artifacts, and verbosity, where the longer answer wins on presence rather than correctness, runs right alongside it.
Sadman Kabir Soumik's Judging the Judges tests the standard countermeasures head on and finds position swapping, evaluating each pair in both orders and averaging, delivers meaningful improvement. The catch is that it doubles judge calls, and it corrects position bias specifically while leaving verbosity and style preferences in place. Soumik's broader result is that "individual mitigation strategies have limited effectiveness," so stacking one trick rarely closes the gap.
The honest read is that these biases are structural to how a language model scores text, not bugs you patch once. You measure them per judge, report them, and design around the residual you cannot remove.
Can a jury of judges fix it?
Only if the jurors make independent errors, and members of the same model family do not. Yang et al. estimated the error correlation inside homogeneous juries at 0.944 to 0.972. At that correlation the jurors behave like copies of one another, so majority voting mostly re-confirms a shared mistake. Their conclusion is blunt: jury size expansion is "nearly useless" when errors correlate that tightly.
This is where Soumik's finding that panels perform best needs a precise reading. Panels help when the panel is genuinely diverse, meaning judges from different model families that fail on different inputs, and the gain arrives with a multiplied inference bill. A three-judge panel of one vendor's models at three sizes is not the diversity that buys error cancellation. It's three correlated votes at triple the cost.
Structured debate between judges looked more promising in the audit, shifting accuracy by as much as 0.317 in some settings, but Yang et al. flag that the debate runs lacked the audit trail to tell genuine deliberation from parser failures. A large swing you cannot explain is not yet a result you can ship.
How do you correct a biased judge score?
You measure the judge's sensitivity and specificity on a labeled slice, then invert the classifier's error to recover the true rate. Fiedler applies the Rogan-Gladen estimator, a correction borrowed from epidemiology, where you back out true prevalence from a test's observed positives once you know its false-positive and false-negative rates. Given an observed judge pass rate and known sensitivity and specificity, the corrected estimate is (observed rate + specificity minus 1) divided by (sensitivity + specificity minus 1). When the judge is perfect, both error terms vanish and the correction returns the raw number. When it is not, the correction pulls the estimate back toward the truth.
The correction is no free pass, and Fiedler is careful about where it breaks. Its reliability "depends critically on judge quality and, for model comparisons, on calibration stability." The sharpest failure mode is sharing one calibration across two models you are comparing. That shortcut "can introduce severe bias, including cases where the comparison estimate points in the wrong direction with high apparent confidence," a sign reversal Fiedler reproduces on MMLU-Pro data. A confident, corrected, and backwards answer beats an uncorrected one at nothing, because it launders the error through a formula.
The takeaway for practitioners: calibrate the judge on a labeled sample that matches your task distribution, correct the raw rate, and calibrate each compared model separately rather than reusing one calibration for both.
Why does high judge agreement not mean the score is right?
Because reliability and validity are different properties, and judges buy the first cheaply while failing the second. Justin Norman, Michael Rivera, and D. Alex Hughes make the split explicit in Reliability without Validity: reliability is consistency of judgments, validity is whether those judgments track the thing you care about. Their large-scale finding is that "LLM-as-a-judge models exhibit high agreement but low validity."
Consistency is seductive because it looks like rigor. Two judges that agree 90% of the time feel trustworthy, and a judge that returns the same verdict on reruns feels stable. Neither tells you the verdict is correct. Judges share training data, share stylistic preferences, and share blind spots, so they converge on the same wrong answer with impressive consistency.
The practical guard is to never accept inter-judge agreement as evidence of accuracy. Agreement is a necessary condition, not a sufficient one. Validity has to be earned against human labels or a trusted external reference on your own data, the same discipline behind measuring agent reliability beyond a single pass@1 number.
Which bias-mitigation strategies are worth the cost?
Rank them by the error each one actually removes against the compute it adds, because none removes everything and several remove little. The table below summarizes what the 2026 evidence supports, drawn from Soumik and Yang et al.
| Strategy | What it targets | Evidence-backed effect | Added cost |
|---|---|---|---|
| Position swapping (both orders, averaged) | Position bias | Meaningful reduction in order sensitivity (Soumik) | 2x judge calls |
| Calibration correction (Rogan-Gladen) | Systematic pass-rate bias | Recovers an unbiased rate if judge quality is known; can flip sign under shared calibration (Fiedler) | Labeled calibration set |
| Diverse panel (different families) | Idiosyncratic per-judge error | Best single result when jurors are uncorrelated (Soumik) | N judges, N-times cost |
| Homogeneous jury (same family, sizes) | Random judge noise | Nearly useless at error correlation 0.944 to 0.972 (Yang et al.) | N-times cost, little gain |
| Structured debate | Shallow verdicts | Large but unexplained swings up to 0.317; not yet auditable (Yang et al.) | High, multi-turn |
The pattern is that the cheap structural fixes, order swapping and calibration, earn their keep, while the expensive ensemble moves only pay off under real diversity. Spending three times the tokens on three correlated judges is the common way teams buy confidence without buying accuracy.
What should you require before trusting a judge result?
Require the judge to report the conditions under which its number holds, the way you would demand error bars from any instrument. Yang et al. propose a minimal audit trail, and it converts cleanly into a reporting checklist:
- What slice was judged, so the number ties to a distribution rather than floating free.
- How biased the judge was under simple perturbations, reported as the verdict-flip rate under answer-order reversal.
- How correlated repeated or panel judgments were, reported as an error-correlation estimate alongside jury size.
- Whether protocol-level decisions were auditable, including parser success rates, fallback rules, and raw intermediate verdicts for any debate step.
Their standard is worth quoting as a bar to hold vendors and internal tooling to: "an LLM-as-judge result should not be accepted as reliable unless it reports what slice was judged, how biased the judge was under simple perturbations, how correlated repeated judgments were, and whether protocol-level decisions were auditable." A judge score without those four is a point estimate from an uncalibrated sensor.
What to watch as judge-based evaluation matures
Three shifts are worth tracking. First, judge freezing becomes a versioning discipline: teams that treat the judge model, its version, and its prompt as a pinned dependency keep comparable trend lines, and teams that silently upgrade the judge break their own history. Second, calibration sets become standing artifacts, small labeled slices maintained per task so corrections stay valid as the underlying model and data drift. Third, the reporting bar rises, and a judge number shipped without a flip rate and a slice definition starts to read as unfinished work.
The deeper point is that automated evaluation is measurement, and measurement earns trust through calibration and disclosed error, not through the fluency of the grader. The same evaluation-gap discipline that applies to closing the enterprise agent evaluation gap applies to the judges themselves. If you already run automated evals with a tool like Promptfoo, the next increment is to calibrate and audit the judge inside that harness rather than trusting its raw verdict.
Sources
- Bias and Uncertainty in LLM-as-a-Judge Estimation, James Fiedler, 2026.
- When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability, Zongyou Yang, Yinghan Hou, and Xiaokun Yang, 2026.
- Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models, Justin D. Norman, Michael U. Rivera, and D. Alex Hughes, 2026.
- Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines, Sadman Kabir Soumik, 2026.