Temperature and top_p Playground

The honest explainer

Why temperature 0 is still not fully deterministic in practice

In exact arithmetic, greedy decoding is deterministic: the highest logit wins, every time. The nondeterminism you actually see does not come from the maths. It comes from how the maths is executed.

• Floating point addition is not associative. (a + b) + c can differ from a + (b + c) in the last bits. A GPU sums thousands of terms in whatever order the work happened to be split into, and that split depends on the shape of the batch, the tile size, and which kernel the library picked.
• Your request is usually batched with other people's requests. A different batch shape means a different kernel and a different reduction order, so your logits change in the low bits. If the top two candidates are within that noise, argmax flips, and the whole rest of the sequence goes somewhere else.
• Mixture-of-experts routing has per-batch capacity. When an expert fills up, tokens are dropped or rerouted. Which tokens fill it depends on the other requests in the same batch, so your output can depend on other people's traffic.
• Kernels are not always deterministic. Atomic accumulation, split-K reductions, and autotuning that picks a different kernel on a different run or a different GPU model all change the low bits.
• Provider-side factors. A fleet usually has more than one GPU generation in it. A model alias can be repointed to a new snapshot without the name changing. Speculative decoding and prompt caching change the shapes the prefill and decode kernels see, even when the arithmetic is meant to be equivalent.
• Ties have to be broken somehow. If two tokens hold exactly the same top logit, argmax picks one, usually the lowest index. That is deterministic for a given implementation but arbitrary, and a one-bit change can flip which one counts as highest. You can reproduce that on this page with the two-way-tie preset at temperature 0.

top_k and top_p interact, and the order is not standard

top_k is a rank cut and top_p is a mass cut. top_k keeps exactly k tokens no matter how the probability is spread. top_p keeps as few as one token on a confident step and dozens or hundreds on a flat one. Load the long-tail preset and move one knob at a time with the other one out of the way: at k = 5 you get exactly 5 candidates no matter what p does, while with top_k raised to 17 or above, p = 0.90 keeps 14 of the 17 tokens.

Run them together and the tighter filter wins at each step. That is why a (temperature, top_k, top_p) triple that behaves well on one stack can behave differently on another.

Order matters because the second filter usually reads a renormalized distribution. This tool applies temperature, then top_k, then top_p, and it measures p against the distribution renormalized over the top_k survivors. That is what happens in the common implementation where top_k masks the losing logits to negative infinity and top_p then reads the softmax of what is left. The visible consequence: after a tight k, the same p keeps more tokens than it would have on the full distribution, because the surviving mass was rescaled up to 1. Stage 3 on this page shows the exact cumulative sums p is compared against.

Providers differ and most hosted APIs do not document the internal order. In Hugging Face transformers, the default generation pipeline builds a list of logits processors and applies temperature scaling before top_k and top_p. In llama.cpp the sampler chain is explicit and user-configurable, so the order is whatever you set it to, and stacks in that family often add min_p, typical sampling, or repetition penalties into the same chain. Assume the order is different somewhere, and do not port a knob triple between providers expecting identical behavior.

The parameter surfaces themselves also differ, and they change over time:

• OpenAI's chat completions API exposes temperature (0 to 2) and top_p, has no top_k, and the documentation recommends changing one of the two rather than both. The reasoning models are the exception: on the o-series and GPT-5 the sampling parameters are fixed, and a request that sets a non-default temperature is rejected.
• Anthropic's Messages API historically exposed temperature (0 to 1), top_p and top_k with the same "alter one, not both" guidance. On newer Claude models, starting with Opus 4.7 and including Sonnet 5, those three parameters were removed: a request that sets them returns a 400, and the guidance is to steer behavior with the prompt instead. Older models such as Sonnet 4.6 still accept them.
• Google's Gemini API exposed temperature, topP and topK together for years. Google deprecated all three in July 2026; on the newest Flash models they are accepted and then silently ignored, and Google's documentation says future models will reject them outright.

Three providers, three different answers, and two of them changed in the last year. The lesson is not "memorise the table", it is "check the current docs for the provider you are actually calling, because both the ranges and the availability move". Everything in this list was true when the page was written and is exactly the kind of claim that goes stale.

Why "raise temperature for creativity" is a crude description

What temperature actually does: it divides every logit by T before the softmax. That is a monotone transform, so it never changes the ranking of tokens. It only changes the ratios between them. As T goes to 0 the distribution collapses onto the argmax. As T grows it flattens toward uniform over the whole vocabulary, and it never reaches exactly uniform for any finite T.

• It cannot add an idea that was not already in the distribution. All it does is move probability mass down the existing ranking. In a language model the tail is mostly typos, broken syntax, wrong facts, and tokens from the wrong language, not good ideas.
• The effect compounds. Sampling is autoregressive. One unlikely token early drags the rest of the sequence into a region the model handles worse, and the model will then continue that mistake confidently.
• Truncation changes what raising T does. With top_p held fixed, raising T flattens the distribution, which widens the nucleus, so you get a flatter choice over a larger candidate set at the same time. The knobs are not independent. You can watch that happen here: fix top_p and drag temperature, and the stage 3 survivor count moves.
• top_k = 1 is greedy regardless of temperature. Because temperature is monotone, it never reorders anything, so the rank-1 token is the same at every T. Try the top_k = 1 chip and then drag temperature: nothing downstream changes.
• Structured output wants low temperature. For code, JSON, and tool calls the "creative" part of the tail is syntax errors and invented field names.
• What people usually want is diversity across samples, not a flatter distribution at every single step. Drawing several completions, varying the prompt, or asking explicitly for a different angle tends to get that with far fewer broken outputs than pushing T past 1.

What this page is, and what it is not

Every number here is computed in the page from the logits in the editor above, using the same steps a real sampler uses: subtract the max for numerical stability, divide by temperature, exponentiate, normalize, truncate by rank, truncate by mass, renormalize. There are no libraries and no network requests.

The logits are invented and editable. They are chosen to make each lesson visible on a vocabulary of a handful of tokens instead of a hundred thousand. Nothing on this page reports a measurement from any specific model, and no benchmark figures are quoted. The sampling demo uses a seeded pseudo-random generator, so it reproduces exactly and is not a source of hidden randomness.

Real vocabularies run from roughly 30,000 tokens to a little over 250,000 depending on the tokenizer (Llama 2 uses 32,000, Llama 3 uses 128,256, Gemma uses 256,000), so a real top_p nucleus on an uncertain step can contain hundreds of tokens where this page shows a handful. The shape of the behavior is the same. The scale is not.