Estimated VRAM required

16.0 GB

9.65B params at Q4_K_M, 262,144 token context, batch 1, inference.

Weights
5.9 GB
KV cache
8.6 GB
Overhead
1.5 GB

Estimate accuracy: Weights within ~3% of most GGUF files from Q4_K_M up, ~7% for Q3_K_M and Q2_K. KV cache within ~5% for standard GQA models, ~10% for MLA or hybrid state estimates. Real VRAM may vary with framework (vLLM vs llama.cpp vs Transformers), Flash Attention, and driver overhead.

KV cache exceeds model weights: Consider lowering the context length to save on VRAM. Contexts between 8K and 64K are generally more typical for local setups.

Hybrid linear attention applied: 24 of 32 layers use fixed recurrent state, so only 8 full-attention layers contribute to the context-growing KV cache. The fixed state adds about 0.06 GB at batch 1 and does not grow with context length.

Hardware that fits

Unified memory machines show a model budget, not their installed memory. We hold back 8 GB for the system, or use the vendor's GPU limit when that's lower. It's a planning estimate, not a hard limit.

RTX 5060 Ti 16GB
Consumer
16 GB
100% used
Apple M5 32GB
Unified
~24 GB
67% of estimate
RTX 3090
Consumer
24 GB
67% used
A100 40GB
Datacenter
40 GB
40% used
Interactive tool by TechComparePowered by TechCompare