LLM Judge Calibrator

Your judge says 94 percent of production responses pass. Paste its confusion matrix from your human-labeled calibration set and find out what the measuring instrument is actually doing.

Start here: judge error is not independent of the system you are judging

Every classical prevalence correction on this page, and the attenuation identity further down, assumes the judge's error rate does not depend on which system produced the output. Published work on LLM judges says that assumption fails, and it fails in the direction that flatters whatever produced the text.

Dated external reference - fetched 2026-08-11 - not computed by this page

Self-Preference Bias in Rubric-Based Evaluation of Large Language Models, arXiv:2604.06996v2, submitted 22 Jul 2026. From the abstract:

among rubrics where generators fail, judges can be more than 50% more likely to incorrectly mark them as satisfied when the output is their own
On HealthBench, a medical chat benchmark with subjective rubrics, we observe that SPB skews model scores by up to 10 points, a potentially decisive margin when ranking frontier models.

Second reference, Quantifying and Mitigating Self-Preference Bias of LLM Judges, arXiv:2604.22891, submitted 24 Apr 2026. From the abstract:

Empirical analysis across 20 mainstream LLMs reveals that advanced capabilities are often uncorrelated, or even negatively correlated, with low SPB.

Links: arxiv.org/abs/2604.06996v2 and arxiv.org/abs/2604.22891. Quoted text above is copied verbatim from those abstract pages on the date stamped. Nothing in this box is produced by the arithmetic below, and a later revision of either paper does not change any number this page computes.

The first link is pinned to v2 on purpose, and the pin is load-bearing. Submission history as checked on 2026-08-11: v1 8 Apr 2026, v2 22 Jul 2026, v3 3 Aug 2026. The first quoted sentence appears verbatim in v2 and v3; v1 reads up to 50% rather than more than 50%, so an unversioned citation of that number would have been ambiguous. The unversioned page arxiv.org/abs/2604.06996 currently serves v3.

So the headline estimator here is prediction-powered inference (PPI), which needs no assumption at all about how the judge errs. Rogan-Gladen, the classical screening-test correction, stays on screen next to it with its independence assumption printed and its breaking points demonstrated. The gap between the two numbers is the point of the tool.

Inputs

Two things go in. First, the four counts from comparing your judge against human labels on a calibration set. Second, the judge's verdicts on the run you actually want a number for. Nothing you type leaves the page.

1. Calibration set: judge verdict against human label

Rows are what the judge said. Columns are what the human said. "Pass" means the response was acceptable. Counts, not percentages.

The judge agreed and was right.
The judge waved through a bad response. This is the count that inflates your pass rate.
The judge flunked a good response.
The judge caught it.
2. How was the calibration set chosen?

This is a hard gate, not a checkbox. PPI is valid because the labeled subset is a random draw from the same pool the judge scores. Hand-picking interesting cases destroys that and the tool will refuse rather than print a number it cannot stand behind.

3. The run you want the number for

The unlabeled items the judge scored but no human reviewed. This is the pool the calibration set should have been drawn from.

Size of the production run, excluding the calibration items.
The raw number behind the pass rate you have been quoting.
4. Optional: a second system, same judge

Fill this in only if you want the between-system gap panel. Leave the rate blank to skip it.

Label only. Defaults to "System A".
Label only. Defaults to "System B".
Scored by the same judge, same rubric.
Used only to compare against the width of the corrected interval.

Nothing computed yet

Fill in the four calibration counts, choose how the calibration set was built, and give the size and pass count of the unlabeled run. Or press Load sample above for a realistic 200-example calibration set against a 5,000-item production run, where the judge reports 94 percent and the corrected answer is roughly ten points lower.

Everything runs in this tab. There is no backend, no API key, and no external dependency.

Refusal states and edge cases you can load

Each of these loads a real set of counts that breaks one of the estimators. Watching a tool refuse is more informative than watching it always answer.

The math, written out

Prediction-powered inference (the headline)

With n labeled pairs and N unlabeled items, Y the human label in {0,1} and f the judge's verdict in {0,1}:

theta_PP = mean_unlabeled( f )  -  mean_labeled( f - Y )

           = U_pass/N  -  (FP - FN)/n

s2_d  = sample variance of (f - Y) over the n labeled pairs
      = (n/(n-1)) * ( (FP+FN)/n  -  ((FP-FN)/n)^2 )

s2_f  = sample variance of f over the N unlabeled items
      = (N/(N-1)) * p_app * (1 - p_app)

95% CI = theta_PP  +/-  1.96 * sqrt( s2_d/n  +  s2_f/N )
Dated external reference - fetched 2026-08-11 - not computed by this page

Source: Prediction-Powered Inference, Angelopoulos, Bates, Fannjiang, Jordan and Zrnic, arXiv:2301.09633. From the abstract:

The framework yields simple algorithms for computing provably valid confidence intervals for quantities such as means, quantiles, and linear and logistic regression coefficients, without making any assumptions on the machine-learning algorithm that supplies the predictions.

The mean-estimation setup in that paper assumes the labeled pairs and the unlabeled points are independent and identically distributed draws from a common distribution. That is the sampling condition the gate above enforces, and it is the only condition this estimator needs. The reference implementation the authors maintain describes PPI as, verbatim:

Prediction-powered inference (PPI) is a framework for statistically rigorous scientific discovery using machine learning.

Links: arxiv.org/abs/2301.09633, github.com/aangelopoulos/ppi_py. The interval above is the plain normal-approximation PPI interval for a mean; this page does not implement the power-tuned PPI++ variant.

Rogan-Gladen (shown beside it, not as the answer)

sens = TP / (TP + FN)          spec = TN / (TN + FP)
J    = sens + spec - 1         (the Youden index)

theta_RG = ( p_app + spec - 1 ) / J

Var(theta_RG), by the delta method:

  Var(p_app) = p_app(1-p_app)/N
  Var(sens)  = sens(1-sens)/(TP+FN)
  Var(spec)  = spec(1-spec)/(TN+FP)

  Var(theta_RG) ~= [ Var(p_app)
                     + theta_RG^2       * Var(sens)
                     + (1-theta_RG)^2   * Var(spec) ] / J^2

Those partial derivatives are d/dp = 1/J, d/dsens = -theta/J and d/dspec = (1-theta)/J. Propagating the sensitivity and specificity uncertainty is the whole point: a binomial interval on the apparent rate alone is the number that makes a 200-example calibration set look like a measuring instrument accurate to a tenth of a point.

Dated external reference - fetched 2026-08-11 - not computed by this page

The corrected estimate and its asymptotic standard error come from Rogan W J and Gladen B, Estimating prevalence from the results of a screening test, American Journal of Epidemiology, 1978 (PubMed 623091). The out-of-range behaviour is documented in the R epiR package reference for epi.prev, which states, verbatim:

If apparent prevalence is less than (1 - diagnostic test specificity) the Rogan Gladen estimate of true prevalence will be less than zero (Speybroeck et al. 2012).
If the apparent prevalence is greater than the diagnostic test sensitivity the Rogan Gladen estimate of true prevalence will be greater than one.

Link: search.r-project.org/CRAN/refmans/epiR/html/epi.prev.html. The third failure mode, sens + spec <= 1, is not a citation, it is division: at exactly 1 the denominator is zero, and below 1 the denominator is negative so the correction runs backwards.

Cohen's kappa

p_o = (TP + TN) / n

p_e = [ (TP+FN)(TP+FP) + (FP+TN)(FN+TN) ] / n^2      <- from the OBSERVED marginals

kappa = (p_o - p_e) / (1 - p_e)

Expected agreement is computed from the marginals you entered, never hardcoded to 0.5. That is the entire difference between kappa and a decorative number. A judge that says pass to everything, on a set that is 85 percent pass, scores 85 percent raw agreement and kappa of exactly 0.00. Load that case from the buttons above and watch it happen.

The attenuation identity

If the judge's two error rates are equal, call that common rate e:
    e = 1 - sens = 1 - spec

then for a system whose true pass rate is theta, the judge reports

    p_app = e + theta * (1 - 2e)

so a true gap G between two systems is reported as G * (1 - 2e),
and 1 - 2e is exactly sens + spec - 1, the same J that Rogan-Gladen divides by.

This page reports measured_gap / (1 - 2e) as a ceiling on what the identity licenses, not as a correction to apply to your numbers. Under independent symmetric error, judge noise can only shrink a real gap, so the implied true gap is the most you could claim. The floor is zero, because error that is correlated with which system produced the output can manufacture a gap from nothing - which is precisely what the dated references at the top of this page report happening. The page also prints the two error rates separately, so you can see whether the symmetry precondition holds on your own data before you look at the number.

What this tool is not

Sources

Dated external reference - all links fetched 2026-08-11

Every quoted sentence on this page was copied verbatim from the linked page on 2026-08-11. Paraphrases carry no quote marks. No number produced by the calculator above depends on any of these pages remaining unchanged.