Estimated VRAM required
16.0 GB
9.65B params at Q4_K_M, 262,144 token context, batch 1, inference.
Estimate accuracy: Weights within ~3% of most GGUF files from Q4_K_M up, ~7% for Q3_K_M and Q2_K. KV cache within ~5% for standard GQA models, ~10% for MLA or hybrid state estimates. Real VRAM may vary with framework (vLLM vs llama.cpp vs Transformers), Flash Attention, and driver overhead.
KV cache exceeds model weights: Consider lowering the context length to save on VRAM. Contexts between 8K and 64K are generally more typical for local setups.
Hybrid linear attention applied: 24 of 32 layers use fixed recurrent state, so only 8 full-attention layers contribute to the context-growing KV cache. The fixed state adds about 0.06 GB at batch 1 and does not grow with context length.
Unified memory machines show a model budget, not their installed memory. We hold back 8 GB for the system, or use the vendor's GPU limit when that's lower. It's a planning estimate, not a hard limit.