Mistral Large 4 VRAM requirements at Q4 and 1M context

How this is calculated

The model has 72 layers of indexed attention with hidden size 6,144. Its feed forward blocks use 128 routed experts with 4 chosen per token. What makes the cache unusual is how the layers share it. Nine layers reuse the latent cache of layer 35, so only 63 layers own a latent of 576 values (512 latent plus 64 rope). On top of that, 44 layers have their own indexer and also store a 128 value index key. Per token the cache therefore holds 83,840 bytes in FP16, which adds up to 87.9 GB across 1,048,576 tokens. The cache is 12% of the whole estimate, so it matters but doesn't dominate. Cutting the context to 32K shrinks the cache to 2.7 GB and the total to 619 GB. A supported 8-bit cache would bring the full context total to about 664 GB, though that needs backend support and isn't a feature of every engine. The estimate follows the Transformers implementation. It leaves out vision processing for image input, workspace for prompt processing, and any draft model for speculative decoding. The model can write up to 262,144 tokens in one reply, and generated tokens fill the cache like prompt tokens do, so size for prompt plus output.

Verdict

Eight cards of 80 GB give 640 GB, which doesn't hold the full context estimate. The total takes 9 cards of 80 GB, or 4 cards of 192 GB, before any sharding overhead. At 32K context the total of 619 GB falls under that single node pool on paper, but it leaves very little room for uneven shards or prefill workspace, so test it before counting on it. An eight card node of 192 GB cards offers 1536 GB, which clears the full context estimate on paper with room for the uneven parts. Serving a few users at once changes the math, because each sequence needs its own cache while the weights are shared. A middle context of 128K comes to 628 GB for one sequence, and many deployments never need the full million. Mistral also publishes an NVFP4 checkpoint aimed at Blackwell GPUs, and GGUF conversions come from ggml-org and unsloth. Compare their actual file sizes with the 560 GB used here, because real quantized files mix precisions. The weights use the Mistral Ecosystem Open-Weight License, so read it before you deploy commercially.