Tools Blog

KV Cache Calculator

This calculator estimates how much GPU memory you need to serve a large language model. It accounts for model weights, the KV cache (the part most people under-budget), activation/runtime overhead, and the nominal memory capacity of common current and announced accelerators. GPU presets compare memory only; they do not validate framework, precision, interconnect, or sharding support, and Preview capacities can change.

It’s intended to pair with the KV Cache blog post . All math runs in your browser — nothing is sent to a server. Paste a Hugging Face model id (e.g. Qwen/Qwen2.5-7B-Instruct) to auto-fill the architecture from the model’s config.json — only metadata is fetched, never weights — or pick a curated preset, then tweak any field in Advanced options to override.

Mixed-precision models are handled generally, not per-model: any repo’s quantization_config is classified by its quant_method (the same field the Transformers library dispatches on), covering openai/gpt-oss-120b-style MXFP4 experts, compressed-tensors checkpoints like unsloth/Kimi-K3 (MXFP4) and moonshotai/Kimi-K2.5 (INT4 + group scales, declared only inside text_config), DeepSeek-style FP8, and a cautious generic fallback for methods the tool doesn’t fully model. Weights are split into quantized and full-precision shares (group-scale overhead included), shown both as resident VRAM and checkpoint size on disk — these differ when a runtime dequantizes on load, and the Quantized format (in-memory) option lets you model exactly that (native MXFP4, MXFP8, or dequantized BF16).

Composite architectures are sized per layer group: layer_types drives sliding-vs-full attention counts (gpt-oss caches 128-token windows on half its layers and full context on the rest), and MLA + linear-attention hybrids like Kimi-K3 bill growing KV only for the full-attention layers while adding the O(1) per-sequence recurrent state (KDA/DeltaNet/Mamba, derived from config geometry) for the rest. For exact numbers, hit Deep inspect next to the Load button after loading a repo — it reads each shard’s safetensors header over HTTP Range requests (slower, one small request per shard) for per-tensor dtypes and true payload bytes, still never weights.

Load fetches only the model's config.json and metadata (a few KB) — never weights. Deep inspect additionally reads each shard's safetensors header via HTTP Range requests (one small request per shard, KBs–MBs of JSON, still never weights) for exact per-tensor dtypes, the quantized split, and the true payload size — it takes noticeably longer on many-shard repos. Gated repos return 401; try a public mirror like unsloth/<model>.
Prompt + generated tokens per sequence (i.e. context length). Sliding-window attention only caches a window of these.
Applies to the non-quantized share when "Quantized params" > 0 (see Advanced).
× GPUs
Memory-only presets: advertised per-GPU capacity is treated as GiB, and GPU count assumes an ideal pooled total. Runtime, precision, sharding, and interconnect support are not validated; usable VRAM can be lower. Preview capacities may change.
Advanced options ↺ reset all overrides
Runtime workspace, activations, and allocator slack.
Driver, framework, and multi-GPU communication buffers.
Tokens shared across all sequences (counted once instead of per-sequence).
96% — higher = less fragmentation waste.
Architecture (override / custom model) Edits here become "manual overrides" that survive switching presets. A "↺ reset" link will appear next to any field you've changed.
Share of total params stored in the quantized format below (e.g. MXFP4 MoE experts). 0 = fully uniform precision.
How the quantized share sits in VRAM. Pick "Dequantized → BF16" when the runtime/GPU can't keep the packed format resident (e.g. MXFP4 experts upcast on pre-Hopper GPUs) — the disk size stays checkpoint-native. Native support depends on the accelerator and runtime; the GPU preset does not validate compatibility.
Of the total layers, how many keep a sequence-growing KV cache.
Layers holding O(1) recurrent state instead of a growing KV cache (KDA, DeltaNet, Mamba). Auto-derived from the config when possible.
Fixed per-sequence state (recurrent S-matrix + conv). Derived from config geometry at compute dtype; engines that keep FP32 state need ~2×.
Memory used
Weights KV cache Activations + framework
Model weights (resident in VRAM)
Checkpoint size on disk
KV cache (per token)
KV cache (total)
Activations + runtime overhead
Total VRAM required
Available VRAM
Headroom
Show formulas