Eval peeking planner
You ran an eval, checked it at 100 cases, checked again at 200, and stopped the moment it went green. That is not a 5 percent test any more. This page measures how much worse it got by simulating it, then computes the sample size and the group-sequential boundaries that make peeking legitimate. Every number below is computed in your browser when you press the button. Nothing is looked up from a table and nothing leaves your machine.
Independent project. Not affiliated with, endorsed by, or connected to any statistics vendor, publisher, or model provider. Method notes and the one known discrepancy are at the bottom.
1. What peeking costs: live Monte Carlo
The setup is a paired binary eval: the same cases scored by two model versions, so every case is either concordant (both right or both wrong) or discordant (one right, one wrong). Under the true null the two versions are equally good, so each discordant case falls either way with probability one half. The simulation draws that process, applies a two-sided test after each look, and counts how often it ever crosses the line.
Control: the same simulation, testing only at the final look
This is the load-bearing check. If the test itself were miscalibrated, the inflation above would prove nothing. Testing once, at the end, on identical data must return the nominal alpha. Each rate carries a 95 percent Monte Carlo interval; the verdict asks whether that interval covers the nominal alpha.
Robustness: does the answer depend on the discordance rate?
Discordance is the parameter most teams cannot state for their own eval. If the inflation moved with it, this whole page would need a number you do not have. It does not move.
2. Boundaries that make peeking legitimate
If you are going to look K times, spend the alpha on purpose. A group-sequential design fixes a critical value for each look so the overall false-positive rate lands back on alpha. Two classical shapes: Pocock uses the same critical value at every look, O'Brien-Fleming makes the early looks almost unreachable and gives the final look back nearly its full alpha.
The constants are not tabulated in this page. They are recovered here by recursive Simpson quadrature over the continuation region: integrate the joint density of the running sum, look by look, truncated to the interval where the trial has not yet stopped, then bisect on the constant until the exit probability equals alpha. The published column is there so you can see the recovery, not so the page can recite it.
The "Published" column is the classical tabulated value, quoted to the three decimals those tables are normally printed at. No edition or table number is cited for it, because none could be confirmed against a file you could check; treat that column as a cross-check the page does not ask you to trust, and the computed column as the claim.
A design that is not in any table
The same solver, run for whatever K, alpha, power and shape you ask for. No published column here because there is nothing on file to compare against, so the page does not pretend otherwise.
3. How many paired cases you actually need
A paired sample size is meaningless without a discordance rate. The number of pairs you need to detect a fixed improvement is driven almost entirely by how often the two versions disagree, not by the accuracy level itself. Two teams chasing the same 3-point move can differ by a factor of eight in required N. Any quoted paired N that does not state its discordance rate has buried the assumption; this page refuses to print one without it.
Reverse: what discordance does a quoted N imply?
Someone hands you a paired N with no discordance rate attached. Solve for the assumption that was buried in it, then decide whether it resembles your eval.
Known discrepancies and honest limits
The continuity correction is miscalibrated here, so this page does not use it by default
McNemar's test is often quoted with Yates' continuity correction. Run the simulation above and read the two columns side by side at the smallest look count: the uncorrected statistic lands on the nominal alpha, and the corrected one lands visibly below it. Measured on the last run of this page: nothing measured yet, press Run simulation above. A test that comes in under the alpha you asked for is not safe, it is a different test with a smaller alpha and less power, and it understates the peeking penalty for the same reason. The column is kept in the output so you can see the gap rather than take the claim on trust. No figure here is transcribed; the sentence above is filled in by the run.
One boundary result does not match the published value
The quadrature recovers seven of the eight boundary constants and three of the four inflation factors to the precision the published tables are printed at. The exception is the Pocock inflation factor at five looks: this page computes 1.2284 against a published 1.207. The two O'Brien-Fleming factors and the two-look Pocock factor agree. That single disagreement is unresolved. It is stated here rather than smoothed over, because a page whose whole argument is "compute it, do not recite it" does not get to hide the one place where computing and reciting disagree. Press the button and check the number yourself; if you have a resolution, the repository is linked below.
What the simulation does and does not model
- Looks are equally spaced in cases. Unequal spacing changes the boundaries; the quadrature here assumes equal information increments.
- The paired simulation assumes the discordance rate is stable across the run. If your eval set is ordered by difficulty, early looks are not exchangeable with later ones.
- There is no stopping for futility, and no effect under the alternative in the peeking simulation. It measures the false-positive rate under a true null only.
- Connor's formula is a normal approximation to the paired binomial. At very small N or very small discordance it will be optimistic; the mid-p exact column in the simulation is the honest check on that.
- Monte Carlo estimates carry error. Every rate in the control table is printed with its interval, and the interval shrinks as the square root of the trial count.
Method
- Normal distribution. The error function is evaluated from its Maclaurin series below 2 and from the standard continued fraction above it, both driven to double precision. Quantiles come from bisection on that CDF.
- Binomial sampling. Inversion when the mean count is below 30, transformed rejection above it. The Stirling tail corrections the rejection step needs are computed at load time from exact factorials rather than pasted in as constants.
- Randomness. A seeded 32-bit counter-based generator, so a given seed reproduces a given table exactly. The seed is an input.
- Boundaries. Recursive numerical integration of the continuation-region density, composite Simpson with M nodes per look, then bisection on the shape constant. The inflation factor comes from a second bisection, on the drift that achieves the requested power under the recovered boundaries.
- Threading. The work runs in a Web Worker built from a Blob so the page stays responsive. Where workers are unavailable the same code runs on the main thread in timed chunks. The status line tells you which path ran.
Related tools
This is the pre-run half. Once the eval has actually run:
- eval-significance-calculator for testing a finished comparison
- eval-regression-viewer for seeing which cases moved
- llm-judge-calibrator for checking the judge before you trust the scores