LLM VRAM Calculator

Work out how much GPU memory a local model actually needs. Weights and KV cache are arithmetic; overhead is an estimate, and this page tells you which is which.

Inputs

Model
Presets fill parameter count and architecture. Every field stays editable.
Total parameters, not active parameters. For a mixture-of-experts model every expert still has to sit in memory.
Serving
Quantizing the cache is a real lever. In llama.cpp it is -ctk and -ctv; a quantized V cache requires flash attention, and llama.cpp refuses to start without it. Quality cost is small at 8 bits and gets real below that.
Architecture (this is what the KV cache depends on)
From the model's config.json: num_hidden_layers, num_attention_heads, num_key_value_heads, head_dim (or hidden_size / num_attention_heads when head_dim is absent). Key/value heads below attention heads means grouped-query attention, and it is the single biggest factor in KV cache size.
Overhead assumptions (estimates you can change)
These two are the guessed part of the estimate. The percentages are a rule of thumb applied to weights plus KV, not a measurement, and they are the reason the result is a range instead of a number.
Your GPU
Everything on this page is in GiB (1024 cubed bytes), which is what nvidia-smi reports. A GeForce card advertised as 24 GB reports 24576 MiB, so the number carries straight over. Datacenter parts quoted in decimal GB, like the H200 at 141 GB, report a little less than that in GiB, so read nvidia-smi and type the real figure in.

Estimate

Nothing to calculate yet.

Pick a model preset, or press Load sample to see an 8B model at 64k context on a 12 GiB card: it does not fit, and the page shows exactly what to change.

How much of this is real arithmetic

Read this before you trust a number off this page.

Near-exact: weights and KV cache

Both follow directly from numbers you supplied. Weights are parameters times bits per weight. KV cache is 2 (one key, one value) x layers x key/value heads x head dimension x context x batch x bytes per element. If your parameter count, bits per weight and architecture fields are right, these two are right to within a rounding error.

The caveats that remain: whole-file bits per weight for the mixed GGUF classes (anything ending in _K_M or _K_S) is an average, and the exact value moves a few percent between models. Bits per weight is exact for FP16, FP8, INT8, Q8_0 and Q4_0, where the block layout fixes it.

Estimated: activation, workspace, runtime overhead

This is a rule of thumb, not a measurement, and this page never presents it as one. What actually lands in that bucket: attention and MLP scratch buffers, the logits buffer, CUDA graph memory, the allocator's fragmentation, and the CUDA context itself, which costs a few hundred MB per process before a single weight is loaded. It scales with how many tokens are processed at once far more than with context length.

That is why the result is a range. If you need a real number, load the model and read nvidia-smi.

Your inference engine changes the answer
  • llama.cpp / ollama: allocates the KV cache for the context you ask for, up front. Supports quantized K and V caches and partial GPU offload by layer.
  • vLLM: preallocates a paged KV pool sized by gpu_memory_utilization, which currently defaults to 0.92, so it will look like it is using nearly the whole card no matter how small your model is. Check the default for your version, it has moved. Paged attention removes most of the waste from padding sequences to the max length.
  • transformers: the least memory-frugal path. Cache is FP16 by default and there is no paging.
  • ExLlamaV2 and similar: offer quantized cache modes, so the KV row on this page can drop a long way.

Flash attention removes the O(context squared) attention score matrix from memory, which matters most at long context and large batch. Paged attention changes how the KV pool is allocated, not how big one sequence's cache is.

Architectures this formula does not describe
  • Multi-head latent attention (the DeepSeek V2/V3 family) compresses the cache into a latent vector. This page would overestimate it badly.
  • Sliding-window or interleaved local attention (Mistral's window, Gemma 2's alternating layers) caps the cache per window instead of growing with full context, so real usage is lower than shown once context passes the window.
  • State-space and hybrid models (Mamba and friends) carry a fixed-size state instead of a growing cache.
  • Mixture of experts: the KV math holds, but weights must count every expert, not just the active ones.