How much VRAM does Mixtral 8x7B need at Q4_K_M? MoE memory math

How this is calculated

MoE is a compute optimization, not a memory one. The 8 expert FFN blocks plus the shared attention give Mixtral roughly 47B total parameters that must all be loaded in VRAM. At Q4_K_M that's about 26 GB of weights, plus ~4.3 GB of KV cache at 32K context. The benefit is throughput - inference runs at the speed of a 12B dense model, so you get 70B-class quality at 12B-class generation speed if it fits.

Verdict

Mixtral is the case study for MoE memory reality: only 2 of 8 experts run per token, but the router can hit any expert on any forward pass, so all 47B of parameters must stay resident. At Q4_K_M that's 26 GB of weights plus 4.3 GB of KV cache at 32K context, totaling about 34 GB. A 32 GB card fits it cleanly, a 24 GB card needs Q8 KV and a context cap to clear the gate, otherwise the same gen-rate math applies. The reward is throughput that matches a 12-13B dense model because only 2 experts compute per token, so you get 47B-class quality at 13B-class generation speed. That's the structural MoE win that justifies the memory cost.