How much VRAM does Nemotron 3.5 Lightning 30B need at Q4_K_M? NVIDIA's single-card 1M model

By TechCompare · Updated · How we check

Nemotron 3.5 Lightning is a 29 GB resident model at the full 1M context, so a 48 GB pro card runs it with all experts loaded. Shorter contexts shrink the cache and the total.

Calculator

Estimated VRAM required

28.4 GB

31.6B params at Q4_K_M, 1,048,576 token context, batch 1, inference.

Weights
19.3 GB
KV cache
6.5 GB
Overhead
2.7 GB

Estimate accuracy: Weights within ~3% of most GGUF files from Q4_K_M up, ~7% for Q3_K_M and Q2_K. KV cache within ~5% for standard GQA models, ~10% for MLA or hybrid state estimates. Real VRAM may vary with framework (vLLM vs llama.cpp vs Transformers), Flash Attention, and driver overhead.

Hardware that fits

Unified memory machines show a model budget, not their installed memory. We hold back 8 GB for the system, or use the vendor's GPU limit when that's lower. It's a planning estimate, not a hard limit.

RTX 5090
Consumer
32 GB
89% used
A100 40GB
Datacenter
40 GB
71% used
RTX PRO 5000 48GB
Pro
48 GB
60% used
Apple M5 Pro 64GB
Unified
~56 GB
51% of estimate

Nemotron 3.5 Lightning 30B at Q4_K_M needs 29 GB of VRAM with all experts resident at the full 1M context. NVIDIA's hybrid Mamba-2 plus transformer design keeps the cache small, so a 1M-context model fits a single 48 GB card. It uses 3B active parameters per token, but that doesn't shrink what has to stay loaded, because different tokens pick different experts.

How this is calculated

Only 6 attention layers carry a key-value cache. 23 of the other blocks are Mamba-2 state-space layers with a fixed-size recurrent state, 0.05 GB at batch one. Weights at Q4_K_M are 19.3 GB, the KV cache with 2 KV heads at head_dim 128 across 6 attention layers is 6.5 GB at 1M, and overhead adds 2.7 GB.

Verdict

Choose Lightning when you want a 1M window on one card: it totals 29 GB resident, so a single 48 GB card handles it outright. Nemotron 3 Ultra 550B, the other hybrid, needs 384 GB at the same context.

More Nemotron scenarios

Kimi K3.1 Pro at Q4_K_M
A 2.78T expert pool with 24 latent attention layers and fixed state in the other 69.
View details
Mistral Large 4 at Q4_K_M
A 1T expert pool with 63 latent cache owners and 44 layers that also store an index key.
View details
Muse Spark Open at Q4_K_M
Only 16 of its 62 layers grow with context, and the other 46 hold a 2,048 token window.
View details

Frequently asked questions

How can a 1M-context model fit in 29 GB?
Most of the transformer blocks are Mamba-2 state-space layers, which keep a fixed-size recurrent state instead of growing a key-value cache. Only 6 attention layers accumulate KV. That keeps the cache to 6.5 GB at 1M tokens, next to 19.3 GB of Q4 weights and a few GB of overhead.
Nemotron 3.5 Lightning or gpt-oss 20B for a local agent?
Lightning for long-context agentic work: its 1M native window costs about 29 GB resident. gpt-oss 20B for maximum compatibility and the cleanest tooling, but its window is 128K and it needs about 17.0 GB. Even at 1M, Lightning's cache stays at 6.5 GB, so most of its budget goes to weights.