How much VRAM does MiniMax-M3 428B (MoE) need at Q4_K_M? MiniMax 1M block-sparse

How this is calculated

The 428B parameter pool requires 240 GB of weights at Q4_K_M. The 60-layer architecture consists of 3 dense layers plus 57 MoE layers, using group-query attention with 4 KV heads at head_dim 128. Block-sparse MSA is an attention compute optimization that attends to the top-16 blocks of 128 tokens, not a storage compression, so the key-value cache is stored full-resolution. At the full 1M context the FP16 key-value cache uses a significant 129 GB. Activation and software overhead adds about 37 GB. In active-only mode the resident weights shrink to 13 GB, but the 129 GB cache stays, which drops total usage to 156 GB.

Verdict

The 405 GB resident total splits as 240 GB of Q4 weights, 129 GB of FP16 KV cache at the full 1M context (4 KV heads at head_dim 128 across 57 MoE layers plus 3 dense layers), and 37 GB of overhead. Hardware-wise resident mode needs six H100 80GB cards or three H200 141GB cards with NVLink. Active-only loading shrinks the weights to about 13 GB, but the cache stays at 129 GB and drives the total to roughly 156 GB, which fits dual 80 GB cards or a 192 GB unified-memory workstation. Block-sparse MSA is the architectural note: it speeds attention compute by attending only to top-16 128-token blocks, but the cache is stored at full sequence resolution so future queries can attend anywhere, which is why capping context rather than active-offload is the actual footprint lever. At 256K the active-only total drops to about 60 GB.