How much VRAM does Mistral Medium 3.5 128B need at Q4_K_M? Mistral's open flagship
Mistral Medium 3.5 is a two-card model at Q4: ~150 GB at 256K context, or ~114 GB at 128K on a single 141 GB H200. Dense means no offload cheats, but also no per-token routing penalties.
Mistral Medium 3.5 128B at Q4_K_M needs about 150 GB of VRAM at its native 256K context, so a single 192 GB MI300X or two 80 GB cards run it fully resident. Mistral's July 2026 open-weights release finally gives the Medium line a downloadable model, dense at 128B with no expert routing.
By TechCompare · Updated
Calculator
Estimated VRAM required
150 GB
128B params at Q4_K_M, 262,144 token context, batch 1, inference.
Estimate accuracy: Weights within ~2%. KV cache within ~5% for standard GQA models, ~10% for MLA or hybrid state estimates. Real VRAM may vary with framework (vLLM vs llama.cpp vs Transformers), Flash Attention, and driver overhead.
Hardware that fits
Apple entries show installed unified memory separately from a model budget based on an 8 GB reserve. That budget is a planning heuristic, not a fixed hardware limit.
Just barely too small
How this is calculated
Weights at Q4_K_M are 71.7 GB. The KV cache with 8 KV heads at head_dim 128 across 60 layers is 64.4 GB at the full 256K window, plus 13.6 GB of overhead. Being dense, there's no expert-offload trick - every parameter must stay resident, so the floor is set at 256K context by the cache. Capping context to 128K halves the cache to 32 GB and the total to ~114 GB, which fits a single 141 GB H200.
Verdict
The 150 GB total at 256K comes from 71.7 GB of Q4 weights plus 64.4 GB of FP16 KV cache plus 13.6 GB of overhead. Dense architecture means there are no expert weights to move to system RAM, but inference is also simpler, with predictable throughput. Versus Qwen3.5 122B at the same footprint class, Medium 3.5 trades a slightly larger cache at 256K for Mistral's agentic tuning and European data-residency story. A single MI300X 192GB or H200 pair handles it at full window.
More Mistral scenarios
Related guides
Frequently asked questions
Is Mistral Medium 3.5 open weights?
Mistral Medium 3.5 or Qwen3.5 122B?
Related tools
RAM Latency Calculator
Convert DDR3/DDR4/DDR5 timings (CL, tRCD, tRP, tRAS) into true latency in nanoseconds.
Use tool ➜Power Cost Estimator
Estimate annual electricity costs for your PC, Server, or TV.
Use tool ➜Data Transfer Calculator
Estimate transfer times for files over USB, WiFi, Ethernet, and more.
Use tool ➜Memory and Storage Throughput Visualizer
Visualize the massive speed difference between CPU cache, RAM, and storage.
Use tool ➜