How much VRAM does Mistral Medium 3.5 128B need at Q4_K_M? Mistral's open weights model
By TechCompare · Updated · How we check
Mistral Medium 3.5 needs ~190 GB at Q4 and 256K context, which takes 3 cards of 80 GB. At 128K it needs ~138 GB, which fits a single 141 GB H200. Dense means every parameter stays resident, with no expert offload.
Calculator
Estimated VRAM required
190 GB
128B params at Q4_K_M, 262,144 token context, batch 1, inference.
Estimate accuracy: Weights within ~3% of most GGUF files from Q4_K_M up, ~7% for Q3_K_M and Q2_K. KV cache within ~5% for standard GQA models, ~10% for MLA or hybrid state estimates. Real VRAM may vary with framework (vLLM vs llama.cpp vs Transformers), Flash Attention, and driver overhead.
Hardware that fits
Unified memory machines show a model budget, not their installed memory. We hold back 8 GB for the system, or use the vendor's GPU limit when that's lower. It's a planning estimate, not a hard limit.
Multi-GPU setups
Pooled cards show up once a model outgrows the biggest single card of that kind. Each card past the first keeps about 1 GB for its own buffers, and that's counted here. Your inference engine has to support the split, and speed also depends on how the cards connect.
Just barely too small
Mistral Medium 3.5 128B at Q4_K_M needs about 190 GB of VRAM at its native 256K context, so a single 192 GB MI300X or 3 cards of 80 GB run it fully resident. Mistral's July 2026 open-weights release is the first downloadable model in the Medium line, dense at 128B with no expert routing.
How this is calculated
Weights at Q4_K_M are 78.1 GB. The KV cache with 8 KV heads at head_dim 128 across 88 layers is 94.5 GB at the full 256K window, plus 17.3 GB of overhead.
Being dense, every parameter must stay resident, so there's no expert offload to fall back on. Capping context to 128K halves the cache to 48 GB and the total to ~138 GB, which fits a single 141 GB H200.
Verdict
Choose Medium 3.5 for simple serving. A dense architecture has no expert weights to move to system RAM, and throughput is predictable. Versus Qwen3.5 122B, Medium 3.5 carries a larger cache at 256K (95 GB against 7 GB) but offers Mistral's agentic tuning and European hosting. A single MI300X 192GB or an H200 pair handles it at the full window.
More Mistral scenarios
Related guides
Frequently asked questions
Is Mistral Medium 3.5 open weights?
Mistral Medium 3.5 or Qwen3.5 122B?
Related tools
RAM Latency Calculator
Convert DDR3/DDR4/DDR5 timings (CL, tRCD, tRP, tRAS) into true latency in nanoseconds.
Use toolPower Cost Estimator
Estimate annual electricity costs for your PC, Server, or TV.
Use toolData Transfer Calculator
Calculate file transfer time or the speed you need over USB, Wi-Fi or Ethernet, with latency.
Use toolMemory and Storage Throughput Visualizer
Visualize the large speed gap between CPU cache, RAM, and storage.
Use tool