How much VRAM does DeepSeek V3 671B need at Q4_K_M with full 128K context?

How this is calculated

DeepSeek V3 uses Multi-head Latent Attention (MLA), which is the main reason 128K is even runnable. MLA compresses the KV cache to roughly 576 bytes per token instead of the standard 2 x layers x hidden x bytes that a vanilla transformer pays. The KV cache figure shown above will be much higher than reality - in practice DeepSeek V3 at 128K context consumes roughly 2.3 GB of KV cache. Weights still dominate at 376 GB regardless. The inference economics are unusual: per-token compute matches a ~37B dense model thanks to sparse expert activation, but the memory footprint matches a 671B dense model.

Verdict

The architecture is the headline: MLA compresses per-token KV to a 576-byte latent, which at the 128K context keeps the cache at roughly 2.3 GB instead of the 200 GB-plus a vanilla transformer with 61 layers at hidden 7168 would need. The Q4 weights still dominate at 376 GB and overhead adds about 20 GB, totaling roughly 416-420 GB resident. Hardware-wise that's 6x H100 80GB, 4x H200 141GB, or 3x MI300X 192GB all with NVLink for tensor parallelism. Active-only loading shrinks the weights to about 21 GB by routing cold experts through system RAM, but the per-token PCIe bandwidth penalty makes throughput low. The structural MoE insight is that per-token compute matches a 37B dense model while memory matches a 671B dense model, which is why hosted per-token pricing makes self-hosting the niche case rather than the default.