Which LLMs Fit in 96 GB of VRAM?

Two groups stand out. The first is the 120B mixture of experts models: gpt-oss 120B needs 80.9 GB, Nemotron 3 Super 120B 80.8 GB and Qwen3.5 122B 82.3 GB. They run only 5B to 12B parameters per token, so they're fast for their size.

The second is Mistral Medium 3.5 128B, a dense model that ranks first at 89.2 GB. It's tight, and every token reads all 128B parameters, so it runs far slower than the MoE models.

At 32K, Mistral Medium 3.5 128B grows from 89.2 GB to 98.9 GB and no longer fits. gpt-oss 120B only goes from 80.9 GB to 81.9 GB. gpt-oss uses sliding window attention on half its layers, and the Nemotron models keep a cache on only a few layers, so long context costs them little.

That matters on a card this size, since it often runs coding tools and agents that send long prompts. For 64K or 128K, open the finder from the card below the table.

Llama 3.3 70B fits at Q8_0, with 84.6 GB at 8K and 93.5 GB at 32K, which is tight. Qwen 2.5 72B fits too, at 95.8 GB with 32K.

The 120B MoE models don't fit at Q8_0. Nemotron 3 Super 120B needs 140.2 GB there. So the choice on 96 GB is often a 120B MoE model at Q4_K_M or a 70B dense model at Q8_0. The MoE models are newer and write tokens faster.

Nvidia sells the RTX PRO 6000 Blackwell in workstation, Max-Q and server versions, all with 96 GB. They run the same list. The Max-Q version runs at a lower power limit and gives up some speed for it.

Two 48 GB cards or four 24 GB cards also add up to 96 GB. Runners can split a model across them, but each card keeps some memory for its own buffers, so treat the tight rows as a miss on a split setup. A 128 GB machine like the DGX Spark or an M5 Max Mac has more memory for models, but much less bandwidth, so the same model runs slower there.