How much VRAM does Inkling 975B (MoE) need at Q4_K_M? Thinking Machines 1M hybrid
Inkling 975B at Q4_K_M is a 653 GB resident deployment for 1M context, or about 77 GB with active-expert offload.
Inkling 975B at Q4_K_M with native 1M context needs about 653 GB of VRAM with all experts resident. Thinking Machines' flagship MoE activates 41B parameters per token from a 975B total pool. The hybrid global and sliding-window attention keeps the key-value cache under 48 GB at the full 1M context. Active-expert offload drops the resident footprint to roughly 77 GB.
By TechCompare · Updated
Calculator
Estimated VRAM required
653 GB
975B params at Q4_K_M, 1,048,576 token context, batch 1, inference.
Estimate accuracy: Weights within ~2%. KV cache within ~5% for standard GQA models, ~10% for MLA or hybrid state estimates. Real VRAM may vary with framework (vLLM vs llama.cpp vs Transformers), Flash Attention, and driver overhead.
Sliding-window attention applied: This model caps 5 of every 6 layers at a 512-token window. KV cache estimate is 83% smaller than naive full-attention math at this context length.
Hardware that fits
Apple entries show installed unified memory separately from a model budget based on an 8 GB reserve. That budget is a planning heuristic, not a fixed hardware limit.
No single GPU in our catalog has enough memory. Multi-GPU or CPU offload required.
How this is calculated
The 975B parameter pool requires 546 GB of weights at Q4_K_M. The 66-layer architecture uses group-query attention with variable KV heads: 8 in the 11 global layers, 16 in the 55 sliding-window layers. The sliding window caps 55 of 66 layers at a 512-token window with only 11 layers doing full global attention. The result is a 47 GB key-value cache at the full 1M context. Activation and software overhead adds about 59 GB. In active-only mode the resident weights shrink to 23 GB, which drops total usage to 77 GB.
Verdict
The 653 GB resident total breaks into 546 GB of Q4 weights, 47 GB of FP16 KV cache at the full 1M context (variable KV heads: 8 in 11 global layers, 16 in 55 sliding-window layers capped at a 512-token window), and 59 GB of overhead. Hardware-wise resident mode needs nine H100 80GB cards or five H200 141GB cards with NVLink. Active-only loading shrinks the weights to about 23 GB by routing cold experts through system RAM, while the 47 GB cache stays, dropping the total to roughly 77 GB. That fits a single 80 GB pro card with headroom for batch, or a 96 GB unified-memory workstation. The 512-token sliding window on 55 of 66 layers is what keeps the cache tractable at full context, which is why active-only is viable here at all. At 256K the active-only total drops to about 54 GB with even more batch headroom.
More Inkling scenarios
Related guides
Frequently asked questions
Why does Inkling use variable KV heads across layers?
Can I run Inkling on a single GPU?
Can I run Inkling 975B on a single GPU?
When does the hosted API beat self-hosting for a 975B-class MoE?
Related tools
RAM Latency Calculator
Convert DDR3/DDR4/DDR5 timings (CL, tRCD, tRP, tRAS) into true latency in nanoseconds.
Use tool ➜Power Cost Estimator
Estimate annual electricity costs for your PC, Server, or TV.
Use tool ➜Data Transfer Calculator
Estimate transfer times for files over USB, WiFi, Ethernet, and more.
Use tool ➜Memory and Storage Throughput Visualizer
Visualize the massive speed difference between CPU cache, RAM, and storage.
Use tool ➜