Embedding Similarity Calibrator

Embedding Similarity Calibrator

Your retrieval threshold is 0.8. Against what? Paste your own vectors. This page samples unrelated pairs, computes the empirical cosine distribution, and draws it against the exact closed-form null for random unit vectors in the same dimension. Everything runs in this browser.

Headline finding

The theoretical null and your corpus disagree about the same threshold by a factor of four

Calibrate your own vectors

Nothing leaves the browser. No network call is made by this page at any point.

Input formats

Refused with a message rather than a silent failure: empty input, whitespace only, a line that is not a vector, more than 5,000 vectors, a dimension above 8,192, more than 6,000,000 total components, and more than 24,000,000 characters of text.

The preflight gate

Four conditions block calibration. They are not warnings printed next to a result, because a calibration computed on top of any of them is a number with no meaning attached.

A dimension split
Two or more vector lengths in one input. Half a migration. The page prints the histogram of lengths with counts and refuses to pool them.
Zero vectors
An all-zero vector, usually an empty chunk that made it through ingest. Cosine against it is zero divided by zero. Nothing downstream of that is a number.
Norms outside tolerance
Cosine is scale invariant, so an unnormalized vector does not corrupt cosine. A dot-product index is not scale invariant: there, norm is popularity, and the longest vector wins queries it does not match. The check is tolerance based, not an equality test, for the reason in the next paragraph.
A ranked list sorted as if a distance were a similarity
Only checked when you supply a query vector. The page computes the Spearman rank correlation between the row order you pasted and cosine to the query. A correctly ordered result list runs best match first, so cosine falls as you go down the rows and the correlation is close to -1. A correlation close to +1 is the inverted case: the rows climb in cosine, which is what you get when a distance is sorted descending as if a larger value meant a better match. The row being called the best match is the worst one, and every top-k in the system is upside down.

Why the norm check has a tolerance. A unit vector stored in float32 does not have norm 1. The rounding is real and the accumulation is real, and a check written as norm == 1 reports every vector in a healthy index as broken. This page measures both effects live in your browser rather than asserting a constant, and prints what it found in the hygiene panel. The tolerance used is 1e-3 on the absolute deviation, which passes float32 and float16 storage and still catches a vector that was never normalized.

Why duplicates are reported in two classes. Bit-identical vectors and vectors at cosine 1 minus 1e-6 are different bugs. Bit-identical means the same text was embedded twice: a re-run that did not clear the collection, an ingest loop that double-appends, a retry without an idempotency key. Cosine near 1 but not identical means two different strings landed in nearly the same place: boilerplate headers, a shared footer, a template with one field changed. One is a pipeline bug, the other is a chunking decision. Merging them into one count hides whichever is smaller.

Anisotropy, and why the reranking panel is not the headline

Subtracting the corpus mean vector before scoring is the best known trick in this area, and it is the one thing on this page that is allowed to do nothing.

The size of the effect is set by one measurable quantity: the norm of the mean of your unit-normalized vectors. Write each vector as v = mu + r, where mu is the corpus mean and r is what is left. Cosine between two vectors carries a term in |mu|^2 that is shared by every pair in the corpus. That shared term is what mean removal deletes. When |mu| is small the term is small, the deletion changes almost nothing, and a demo built around it produces a table where nothing moved.

So the mean-vector norm is reported as the primary anisotropy number, the reranking panel sits behind it, and the expected outcome is printed before the rerank runs: a mean-vector norm at or above 0.5 should move rankings, and below 0.5 it should not. Stating it first is the point. A rerank that changes nothing on a low-anisotropy embedder is a correct measurement of that embedder, and without the expectation printed in advance it reads as a broken feature.

The rerank is measured over held-out queries drawn from your own corpus, up to 25 of them, and the verdict rule is fixed before the run: rankings moved when the mean Kendall tau between the before and after orderings falls below 0.90, which is more than one document pair in twenty swapping order. Mean top-10 overlap is reported next to it because that is the number a retrieval system actually pays for, and the two disagree in an interesting way: a low-anisotropy corpus can still shuffle the bottom of a top-10 list, because near-ties at rank 9 and 10 flip under any perturbation at all.

The assumption this rests on

Every unrelated-pair number on this page rests on one assumption, and it is an assumption rather than a fact: a randomly drawn pair of your vectors is unrelated.

In a broad corpus that is close enough to true to be useful. In a topically narrow corpus it is wrong in a specific and dangerous direction. If your index is one product's documentation, one legal domain, or one company's support history, then two random chunks really are related, their cosine really is high, and the distribution this page calls "unrelated pairs" is shifted right. Every threshold derived from it comes out too high, and it comes out too high with a tight confidence interval attached, which is the worst way to be wrong. The page cannot detect this for you, because detecting it requires labels it does not have.

What this refuses to compute

One softening, stated plainly: when the only blocking condition is norms outside tolerance, the page offers to normalize copies in memory and re-run. The result then carries a banner saying so for as long as it is on screen, and the vectors in your index are still the ones you pasted.

The null distribution, derived

Two independent vectors drawn uniformly from the unit sphere in d dimensions have a cosine whose density is exactly

f(x) = (1 - x^2)^((d-3)/2) / B(1/2, (d-1)/2)     for x in (-1, 1)

Equivalently (1 + X)/2 is Beta((d-1)/2, (d-1)/2). That form gives the two numbers this page leans on without any approximation. A Beta(a, a) variate has variance 1 / (4(2a + 1)); substituting a = (d-1)/2 makes 2a + 1 = d, so the variance is 1/(4d), and X = 2Y - 1 multiplies it by 4. So the null has mean 0 and standard deviation exactly 1 over the square root of d. The page also runs a seeded Monte Carlo at your dimension on every calibration and prints the measured standard deviation next to the predicted one with a standard error, so the identity is checked in front of you rather than asserted.

The tail probability is the regularized incomplete beta function: P(X >= t) = I_((1-t)/2)((d-1)/2, (d-1)/2). At d=1536 and t=0.8 that number is far below the smallest positive double, so evaluating the density directly returns zero and any ratio built on it returns NaN. It is computed here in log space throughout: a Lanczos log-gamma for the beta normalizer, a Lentz continued fraction for the incomplete beta, and the result reported as a base-10 exponent. The implementation is checked against three cases with elementary closed forms, and the page prints that check with the errors when you calibrate: at d=2 the tail is arccos(t)/pi, at d=3 the cosine is uniform on (-1, 1) so the tail is (1-t)/2, and at d=4 the tail is (1/pi)(pi/2 - t sqrt(1-t^2) - arcsin t).