How much VRAM does Mistral Medium 3.5 128B need at Q4_K_M? Mistral's open weights model

By TechCompare · Updated · How we check

Mistral Medium 3.5 needs ~190 GB at Q4 and 256K context, which takes 3 cards of 80 GB. At 128K it needs ~138 GB, which fits a single 141 GB H200. Dense means every parameter stays resident, with no expert offload.

Calculator

Estimated VRAM required

190 GB

128B params at Q4_K_M, 262,144 token context, batch 1, inference.

Weights
78.1 GB
KV cache
94.5 GB
Overhead
17.3 GB
Doesn't fit in 180 GB of VRAM at Q4_K_M with 256K of context. Lower the context to 128K (138 GB) to fit a single NVIDIA B200. Or keep all 256K of context with Q3_K_M (173 GB). Quants below Q4_K_M lose noticeable quality.

Estimate accuracy: Weights within ~3% of most GGUF files from Q4_K_M up, ~7% for Q3_K_M and Q2_K. KV cache within ~5% for standard GQA models, ~10% for MLA or hybrid state estimates. Real VRAM may vary with framework (vLLM vs llama.cpp vs Transformers), Flash Attention, and driver overhead.

Hardware that fits

Unified memory machines show a model budget, not their installed memory. We hold back 8 GB for the system, or use the vendor's GPU limit when that's lower. It's a planning estimate, not a hard limit.

MI300X
Datacenter
192 GB
99% used
Apple M5 Ultra 256GB
Unified
~248 GB
77% of estimate
NVIDIA B300
Datacenter
288 GB
66% used

Multi-GPU setups

2× RTX PRO 6000 96GB
Pro · 95.5 GB per card
192 GB
100% used

Pooled cards show up once a model outgrows the biggest single card of that kind. Each card past the first keeps about 1 GB for its own buffers, and that's counted here. Your inference engine has to support the split, and speed also depends on how the cards connect.

Just barely too small

NVIDIA B200
Datacenter
180 GB
short by 9.9 GB

Mistral Medium 3.5 128B at Q4_K_M needs about 190 GB of VRAM at its native 256K context, so a single 192 GB MI300X or 3 cards of 80 GB run it fully resident. Mistral's July 2026 open-weights release is the first downloadable model in the Medium line, dense at 128B with no expert routing.

How this is calculated

Weights at Q4_K_M are 78.1 GB. The KV cache with 8 KV heads at head_dim 128 across 88 layers is 94.5 GB at the full 256K window, plus 17.3 GB of overhead.

Being dense, every parameter must stay resident, so there's no expert offload to fall back on. Capping context to 128K halves the cache to 48 GB and the total to ~138 GB, which fits a single 141 GB H200.

Verdict

Choose Medium 3.5 for simple serving. A dense architecture has no expert weights to move to system RAM, and throughput is predictable. Versus Qwen3.5 122B, Medium 3.5 carries a larger cache at 256K (95 GB against 7 GB) but offers Mistral's agentic tuning and European hosting. A single MI300X 192GB or an H200 pair handles it at the full window.

More Mistral scenarios

Kimi K3.1 Pro at Q4_K_M
A 2.78T expert pool with 24 latent attention layers and fixed state in the other 69.
View details
Mistral Large 4 at Q4_K_M
A 1T expert pool with 63 latent cache owners and 44 layers that also store an index key.
View details
Muse Spark Open at Q4_K_M
Only 16 of its 62 layers grow with context, and the other 46 hold a 2,048 token window.
View details

Frequently asked questions

Is Mistral Medium 3.5 open weights?
Yes, the 3.5 release finally made the Medium line downloadable, under Mistral's standard open license. Earlier Medium versions were API-only.
Mistral Medium 3.5 or Qwen3.5 122B?
Medium 3.5 needs about 190 GB at Q4 and Qwen3.5 122B about 90 GB at the same 256K context. Qwen3.5 122B is MoE, and its expert weights can move to system RAM when VRAM runs short, at a real speed cost. Medium 3.5 is dense and simpler to serve with predictable per-token latency, and the Mistral stack integrates more cleanly with European hosting stacks. Benchmark both on your workload, since they're close enough that ecosystem decides.