How much VRAM does Nemotron 3 Ultra 550B A55B (MoE) need at Q4_K_M? NVIDIA 1M Mamba-2

How this is calculated

The 550B parameter pool requires 308 GB of weights at Q4_K_M. The architecture is a hybrid: 12 attention layers, 48 Mamba-2 state-space layers, and 48 MoE expert blocks. Only the 12 attention layers contribute to the key-value cache, which is just 13 GB at the full 1M context with 2 KV heads at head_dim 128. The Mamba-2 layers maintain a fixed recurrent state instead of growing a cache. Activation and software overhead adds about 32 GB. In active-only mode the resident weights shrink to 31 GB, which drops total usage to 48 GB. The tiny cache means active-only is remarkably efficient for a 550B model.

Verdict

The 353 GB resident total splits as 308 GB of Q4 weights, 13 GB of FP16 KV cache at the full 1M context (only 12 attention layers at 2 KV heads and head_dim 128 contribute, since 48 of 108 transformer blocks are Mamba-2 state-space layers with zero cache), and 32 GB of overhead. Hardware-wise resident mode needs five H100 80GB cards or three H200 141GB cards with NVLink. Active-only loading shrinks the weights to about 31 GB while the 13 GB cache stays, dropping the total to roughly 48 GB, which fits a single 80 GB pro card with room for batch. The Mamba-2 layers are the payoff: where a pure-attention 108-layer model would need 100 GB-plus of cache at 1M context, the 48 Mamba layers maintain a fixed recurrent state instead, which is why Ultra is the most single-card-friendly 500B-class MoE on the list.