Eval Significance Calculator

Paired testing for LLM eval runs. No key, no upload, no backend.

Paste two eval runs, or just the counts, and find out whether that 3-point score move is a real improvement or sampling noise.

Eval runs share the same cases, so the correct test is a paired one on the cases that actually changed. This page runs the exact paired test, shows the effect size at least as prominently as the p-value, measures your harness against its own A/A noise floor, and refuses to print a verdict when the data cannot support one.

Or load one at a time: Load sample loads 1 with 3 as the floor, which is the whole argument in one click. Synthetic All samples are made-up numbers. No model, vendor, or benchmark is named or implied.

Inputs

Paired 2x2 table

Baseline failed, candidate passed.

Baseline passed, candidate failed.

b and c are exactly the two counts eval-regression-viewer already reports as new passes and new failures, so its output pastes straight in. Leave a and d blank if you only have b, c and n: the paired test still runs, but the pass-rate comparison and the score-based interval need all four.

Settings

Variants tried: how many prompt or config variants you tested before picking this one. Above 1, the uncorrected p-value is optimistically biased and a corrected value is shown next to it. Refuse below: the minimum discordant-pair count (b + c) at which this tool will print a verdict at all. Default 10, which is a convention rather than a derived constant; edit it. The exact test is valid at any count, but below roughly this many changed cases it has almost no power, so a verdict in either direction would mislead.

A/A noise floor (optional)

Run the same configuration twice on the same cases and paste those discordant counts here. There is no default and no built-in figure: the floor is always your own measurement of your own harness.

Power planning

Leave the target blank to plan for the effect you actually observed.

Result

Nothing to test yet

Enter the total number of cases plus the two discordant counts, or hit Load sample to see the argument in one click.

If you already ran eval-regression-viewer, its new passes and new failures numbers are the only two inputs this page really needs.

Where this fits

eval-regression-viewer

The upstream tool. It diffs two runs case by case and reports new failures and new passes: those two counts are exactly the b and c this page needs. That repo answers what changed. This one answers whether it means anything.

claude-eval-harness

Actually run the two configurations, bring your own key, and produce the pair of run files you feed in here.

prompt-diff

See exactly what changed between the two prompts you are comparing, before you argue about whether the change did anything.

evaluating-llm-output-reference

The conceptual grounding: what to measure and why, before you reach for a test statistic.

llm-eval-datasets-starter

Where to get cases in the first place, so your eval set is not twelve hand-picked prompts.

rag-evaluation-rubrics

Rubrics for the scored mode, so the numeric score you paste in means something consistent.

temperature-top-p-playground

The sampling theory behind why two runs of the same configuration disagree at all.

Dated external notes

Vendor documentation quoted below was fetched and confirmed on 2026-08-07. It motivates the A/A floor feature; it is not used in any calculation on this page, and nothing here goes stale in a way that could make the arithmetic wrong.

An A/A noise floor used to be optional because you could pin sampling to a fixed setting and call the run deterministic. On current models that lever is going away, which makes run-to-run variance something you measure rather than switch off.

"Setting temperature, top_p, or top_k to any non-default value on Claude Opus 4.7 or later models, including Claude Opus 5, returns a 400 error." Anthropic, Claude model migration guide. platform.claude.com/docs/en/about-claude/models/migration-guide
"Deprecated parameters: The sampling parameters temperature, top_p and top_k are now deprecated." Google, Gemini API release notes, entry dated July 21, 2026. That entry names only gemini-3.6-flash and gemini-3.5-flash-lite. ai.google.dev/gemini-api/docs/changelog
"temperature, top_p, and top_k are deprecated and ignored. In future model generations, supplying these parameters returns an HTTP 400 error." Google, Latest Gemini model guide, sampling parameter deprecation section. ai.google.dev/gemini-api/docs/latest-model

What is deliberately not claimed here. The Anthropic sentence quoted above covers Opus 4.7 and later; it says non-default values, so omitting the field is a different case from sending it. Anthropic's Claude Fable 5 page does not mention sampling parameters at all, so no rule is asserted for it, and no rule is inferred for one model from a sibling model. No claim is made about any vendor other than the two quoted above, and the Google guide quoted last states that it applies to Gemini 3.6 Flash, Gemini 3.5 Flash-Lite and future Gemini releases.