TechCompare LogoTechCompare
Showing centimeters. Press to switch to inches.Showing inches. Press to switch to centimeters.
Back

LLM VRAM Calculator

Estimate the GPU memory needed to run or fine-tune a large language model, or see which models fit the memory you have.

B
Architecture

Changing one of these switches the preset to Custom.

Runtime

Estimated VRAM required

16.0 GB

9.65B params at Q4_K_M, 262,144 token context, batch 1, inference.

Weights
5.9 GB
KV cache
8.6 GB
Overhead
1.5 GB

Estimate accuracy: Weights within ~3% of most GGUF files from Q4_K_M up, ~7% for Q3_K_M and Q2_K. KV cache within ~5% for standard GQA models, ~10% for MLA or hybrid state estimates. Real VRAM may vary with framework (vLLM vs llama.cpp vs Transformers), Flash Attention, and driver overhead.

KV cache exceeds model weights: Consider lowering the context length to save on VRAM. Contexts between 8K and 64K are generally more typical for local setups.

Hybrid linear attention applied: 24 of 32 layers use fixed recurrent state, so only 8 full-attention layers contribute to the context-growing KV cache. The fixed state adds about 0.06 GB at batch 1 and does not grow with context length.

Memory by context length

What makes up the total at each context length, with everything else as set above.

16.0 GB at 256K tokens context, your setting

Weights
5.9 GB
Fixed state
0.1 GB
KV cache
8.6 GB
Runtime
1.5 GB

Context length in tokens. Each step to the right doubles it.

The model numbers match its config at Qwen/Qwen3.5-9B @ c202236. Each point is still an estimate, like the total above.

Weights stay the same at every length. The KV cache grows with the context. Fixed state belongs to the linear attention layers, so it stays flat. Runtime is the allowance the calculator adds for buffers, so it grows a little with the cache. Fixed state and runtime together are the overhead in the result above.

Hardware that fits

Unified memory machines show a model budget, not their installed memory. We hold back 8 GB for the system, or use the vendor's GPU limit when that's lower. It's a planning estimate, not a hard limit.

RTX 5060 Ti 16GB
Consumer
16 GB
100% used
Apple M5 32GB
Unified
~24 GB
67% of estimate
RTX 3090
Consumer
24 GB
67% used
A100 40GB
Datacenter
40 GB
40% used

Starting from your hardware instead? See every model that fits in 16 GB of VRAM, the smallest size this setup fits.

How to use this tool

  1. Pick a model

    Select a model from the dropdown (Llama 3.1, Qwen 2.5, Mistral, Mixtral, Gemma, DeepSeek, Phi-3, and more). The calculator loads that model's parameter count, layer count, and hidden size automatically.

  2. Choose a quantization format

    Pick a quantization level: FP16 (full precision, ~2 bytes/param), Q8_0 (~1 byte/param), Q5_K_M, or Q4_K_M (~0.61 bytes/param). Lower precision shrinks the weight memory at a small quality cost.

  3. Set context length and batch size

    Enter the context length in tokens (8K, 32K, 128K, etc.) and the batch size. The KV cache grows linearly with both, so long contexts can dominate the total VRAM at low quantization.

  4. Read the VRAM estimate

    The calculator returns total VRAM with a breakdown of weights, KV cache, and runtime overhead. It also lists which GPUs and unified-memory machines have enough memory to run that configuration.

  5. See how memory grows with context

    Under the result, the Memory by context length chart runs the same math at many context lengths. Switch it to compare weight formats or batch sizes, or open the numbers as a table. Past a model's maximum context, the chart marks the part the model can't run.

  6. Or start from the memory you have

    Switch to Find what fits, then pick your GPU or type how much memory you can use, and set the context length. The finder sizes every model preset with the same math and lists the ones that fit, newer and larger models first. Open any of them to see its full breakdown in the calculator.

About this tool

The LLM VRAM Calculator estimates how much GPU memory you need to run or fine-tune a large language model. Pick a model from the dropdown (Llama 3.1, Qwen 2.5, Mistral, Mixtral, Gemma, DeepSeek, Phi-3, and more), choose a quantization format, and set the context length and batch size.

You get total VRAM with a breakdown of weights, KV cache, and runtime overhead, plus the GPUs and unified-memory machines that have enough memory. Already have the hardware? Switch to Find what fits to go the other way: pick a GPU or a memory amount and see every model that fits in it.

Use it before downloading a model to confirm it fits, when shopping for hardware to run a specific size class, or when deciding whether quantization (Q4_K_M, Q5_K_M, Q8_0) buys you enough headroom to run a tier larger. The math follows the usual split of weights, KV cache, and overhead, so treat the result as a planning estimate and leave some headroom for your runtime.

Formula

Total VRAM = weights + KV cache + activation overhead.

  • Weights = params × bytes_per_param (FP16 = 2, Q8 ≈ 1, Q4_K_M ≈ 0.61)
  • KV cache = 2 × layers × KV_heads × head_dim × context × batch × KV_dtype_bytes, for standard full-context attention

Models without separate KV-head metadata fall back to hidden_size, while sliding-window and cache-compression presets reduce the result when applicable.

For training, use an architecture and batch-size specific budget. A common mixed-precision AdamW estimate is about 18 bytes per parameter before activations, compared with 2 bytes per parameter for FP16 inference. This page's training result is a planning estimate, not a full training-memory model.

When to use it

Use it to choose a model that fits your existing GPU (Find what fits lists them for you), plan a multi-GPU build for a specific model, or compare quantization options before downloading multi-GB GGUF files. For common memory sizes, a page lists every model that fits at Q4_K_M and Q8_0, with 8K and 32K context. There's one for 8 GB, 12 GB, 16 GB, 24 GB, 32 GB, 48 GB and 96 GB of VRAM, and for 64 GB, 128 GB, 256 GB and 512 GB of unified memory, Macs included.

Pair with the Power Cost Estimator to factor running cost over the lifetime of an inference rig, and the RAM Latency Calculator if you're considering CPU offload as a fallback.

Quantization explained

Modern quantization (Q4_K_M, Q5_K_M, etc.) doesn't lower every weight to the same precision.

K-quants store most weights at the target bit width, plus a scale for each small block of weights. The _M mixes also keep some tensors at a higher precision: Q4_K_M stores about half the attention V and FFN down tensors, and the output layer, as Q6_K. That's why Q4_K_M averages about 0.61 bytes per parameter, not the half a byte that 4 bits would suggest.

The trade-off is good: you keep most of the memory savings of pure 4-bit, with quality close to Q8_0 or FP16.

What we don't model

The calculator gives you a static memory ceiling. It doesn't capture transient memory spikes during long generations, batch scheduling overhead in serving engines, or the small reduction from techniques like prefix caching.

The hardware list keeps installed Apple unified memory separate from an estimated model budget based on an 8 GB reserve. That budget is a planning heuristic, not an Apple-defined VRAM limit.

Leave additional headroom for your actual macOS, apps, runtime, context, and batch workload to avoid out-of-memory errors.

Popular models and configurations

Pre-computed VRAM estimates for common model and quantization combinations.

Which LLMs fit in 16 GB of VRAM
Every model at Q4_K_M and Q8_0, with 8K and 32K context
View details
Which LLMs fit in 24 GB of VRAM
The RTX 3090 and 4090, from 8K to 32K context
View details
Which LLMs fit in 128 GB of unified memory
An M5 Max Mac, DGX Spark or Ryzen AI Max+ 395
View details

Guides that use this tool

Longer walkthroughs that put this tool next to the others a reader needs.

LLM VRAM for local AI hobbyists: which models fit your GPU
Local AI Hobbyist
View details

Frequently asked questions

How much VRAM do I need to run Llama 3.1 70B locally?
At Q4_K_M quantization with 8K context, Llama 3.1 70B needs about 50 GB of VRAM. That's a little more than a single 48 GB pro card (RTX 6000 Ada) or two 24 GB consumer GPUs hold, so those need a shorter context: 4K with a Q8 KV cache comes to 47.8 GB. A Mac with 64 GB+ unified memory holds the 8K version. FP16 needs roughly 157 GB, far beyond any consumer card.
Which models fit on my GPU?
Open the model finder and pick your card, or type the memory you can use. It sizes every model preset at your context length with the same math as the calculator and lists the models that fit, with how much memory each one leaves free. The list starts with the best picks, weighing each model's size against how long ago it came out, since newer models often do as much with fewer parameters. You can sort by size or release date instead. Models that only just fit are marked tight, since real use can run a little over the estimate. A model whose numbers couldn't be checked against its published config is still listed, but never counted as a fit. The finder checks one GPU or one unified memory pool. It doesn't plan multi-GPU splits or offload to system RAM.
What's the difference between Q4_K_M, Q5_K_M, and Q8_0?
These are GGUF quantization formats with different precision-vs-size trade-offs. Q8_0 stores weights in 8 bits (~1 byte/param) for near-FP16 quality. Q5_K_M and Q4_K_M use 5 and 4 bits respectively with mixed precision blocks. Q4_K_M is a common choice: 0.61 bytes per parameter against 2 for FP16, with little quality loss for most uses.
Why does context length affect VRAM so much?
The KV cache stores key and value vectors for every token in context, growing linearly with context length. Doubling from 16K to 32K context doubles the KV cache memory. For long-context models (128K+) the cache often exceeds the weight memory, especially at low quantization.
What is KV cache and can I reduce it?
The KV cache stores intermediate attention values to avoid recomputing them on every new token. You can reduce it by lowering context length (most effective), switching from FP16 to Q8 KV cache (halves memory), or using attention variants like sliding-window attention if your model supports it.
How much extra VRAM does training or fine-tuning need?
Plan on about 18 bytes per parameter for full mixed-precision fine-tuning with AdamW, before activations. It's not a fixed 4x rule: that budget covers FP16 and FP32 weights, FP32 gradients, and two FP32 Adam states. Against a 2-byte FP16 inference weight, that is about 9x for persistent tensors, and peak training VRAM can be higher because activations and temporary buffers depend on batch and context. LoRA reduces this to inference plus adapter and optimizer memory, while QLoRA quantizes the base model on top of LoRA for further savings.
Can I split a model across multiple GPUs?
Yes. Tensor parallelism splits each layer across GPUs (best with NVLink), and pipeline parallelism splits whole layers between GPUs (works without NVLink). Most inference engines (vLLM, llama.cpp, exllama) handle this automatically. Multi-GPU adds 5-15% memory overhead for communication buffers.
Does Apple unified memory count as VRAM?
Effectively yes for inference, but it is shared system memory rather than dedicated VRAM. Apple Silicon uses a single memory pool shared between CPU and GPU, so a Mac Studio with 256 GB of unified memory can hold a model that needs 150 GB of 'VRAM' when the rest of the system has enough headroom. Throughput is lower than discrete GPUs (especially for prompt processing), and actual available memory varies with macOS, other apps, and the inference runtime.
Why doesn't my model fit even though the math says it should?
Inference engines reserve memory for activations during generation, batch buffers, CUDA kernels, and system overhead. Always leave 10-20% headroom on the GPU. If you're at the limit, lower the context length or batch size, switch to Q8 KV cache, or drop one quantization step.

Related tools

LLM API Pricing Calculator

Compare API costs across major models (OpenAI, Anthropic, Google) with prompt caching.

Use tool

LLM Token Counter

Count tokens in any prompt for GPT, Claude, Gemini, and Llama with exact OpenAI tokenization.

Use tool

Power Cost Estimator

Estimate annual electricity costs for your PC, Server, or TV.

Use tool

RAM Latency Calculator

Convert DDR3/DDR4/DDR5 timings (CL, tRCD, tRP, tRAS) into true latency in nanoseconds.

Use tool

Use this tool on your own site

Drop the LLM VRAM Calculator into your blog, docs, or CMS with one iframe snippet. Free, no signup. See all embeddable tools