Metastable Failure Explorer

The dependency sees lambda_eff, the total attempt rate. Each attempt fails with probability p(lambda_eff), the M/M/c chance that a request waits longer than the client timeout. A request that fails is retried, so lambda_eff = lambda * sum over i of p(lambda_eff) to the i. Every root of that equation is an equilibrium. A root is stable when the slope of the right-hand side is below 1 there, which is what makes the collapsed root something the system stays in.

Delay does not appear anywhere in the fixed-point equation above, so a backoff control cannot move an equilibrium. It moves timing. This is a separate fluid simulation: retries are scheduled into future time bins by the backoff policy, and the timeout rate is re-evaluated from the instantaneous attempt rate in each bin. Same model, same equilibria, different path through them.

Erlang C written the textbook way needs a^c / c!. In float64, 171! is not finite and a^c stops being finite well before that, so the textbook form returns a non-finite value on any fleet worth modelling. This page computes the same quantity in log space, using lgamma for the factorial term, so the ratio stays finite. These four lines are computed by this page, in your browser, from the model above.

• Your production system. Every verdict on this page is a property of the numbers in the Model box. Nothing here is fitted to, or predictive of, a real service.
• A heavier tail than M/M/c. Service times here are exponential, and the waiting-time tail is P(W > T) = C(c,a) * exp(-(c*mu - lambda) * T), which is exact for that model. Real service-time distributions have heavier tails, so a real dependency starts producing timeouts at a lower utilization than this model does. The bracketing structure survives; the specific tipping utilization does not.
• Queue memory in the transient. The transient engine is a fluid model: it evaluates the steady-state timeout rate at each bin's instantaneous attempt rate. It carries no backlog between bins, so it under-reports how long a real queue stays hot after a spike. Crossing the unstable equilibrium for one bin does not latch here; a real queue can latch on less.
• Kingman's approximation. Kingman gives a mean wait. The event that generates a retry is a timeout, which is a tail event, so this page uses the exact Erlang C waiting-time tail for the stated model instead of a mean-wait approximation with a timeout bolted on.
• Correlated or contagious failure. One client population, one dependency. No failover, no cache stampede, no cross-shard contagion, all of which are separate sustaining effects.
• Per-attempt delay tables. If you want the delay schedule itself, attempt by attempt, with jitter strategies and worst-case total wait, that is retry-backoff-calculator. It models no amplification at all, which is exactly the gap this page fills.

• Nathan Bronson, Abutalib Aghayev, Aleksey Charapko, Timothy Zhu. Metastable Failures in Distributed Systems. HotOS 2021. doi:10.1145/3458336.3465286. The paper defines a metastable failure as a bad state that persists after the trigger is removed, sustained by a feedback effect that often involves work amplification, and names request retries as one of the most common of those sustaining mechanisms. Its section 2.1 worked example supplies the case-study fixture on this page.
• Google, Site Reliability Engineering, chapter 21, Handling Overload, section Deciding to Retry. sre.google/sre-book/handling-overload. Source of the two retry-budget numbers this page uses as a preset: a per-request budget of up to three attempts, and a per-client budget that only retries while retries stay under a 10 percent share of requests, which the chapter states holds request growth to roughly 1.1x instead of the roughly 3x a bare three-attempt cap allows.
• Google, Site Reliability Engineering, chapter 22, Addressing Cascading Failures, section Retries. sre.google/sre-book/addressing-cascading-failures. Source of the guidance that retries should always use randomized exponential backoff, that retries per request should be limited, and that an overloaded backend should return a distinct status so callers back off instead of retrying. Randomized backoff and a retry budget are separate recommendations there; this page treats them as separate controls for that reason.
• The Erlang C formula and the M/M/c waiting-time tail are standard queueing theory. This page implements them directly and checks its log-space implementation against an independent Erlang B recursion in the numerics table above.

• retry-backoff-calculator - the per-attempt delay table for one client: attempts, base delay, cap, jitter strategy, worst-case total wait. It answers how long a single request waits. This page answers what the whole client population does to the dependency.
• llm-timeout-budget - the same timeout-versus-tail question for a single request path across every layer that can time it out.
• idempotency-and-safe-retries-reference - whether a request is safe to retry at all, which is the question that comes before how often.
• background-jobs-and-queues-basics-reference - queues, workers and dead-letter handling in plain language.
• incident-response-runbooks - what to do at 3am once you are already in the bad state.