TechCompare LogoTechCompare

How much VRAM does Inkling 975B (MoE) need at Q4_K_M? Thinking Machines 1M hybrid

By TechCompare · Updated · How we check

Inkling 975B at Q4_K_M is a 653 GB resident deployment for 1M context, and 546 GB of that is weights.

Calculator

Estimated VRAM required

653 GB

975B params at Q4_K_M, 1,048,576 token context, batch 1, inference.

Weights
546 GB
KV cache
47.4 GB
Overhead
59.3 GB

Estimate accuracy: Weights within ~2%. KV cache within ~5% for standard GQA models, ~10% for MLA or hybrid state estimates. Real VRAM may vary with framework (vLLM vs llama.cpp vs Transformers), Flash Attention, and driver overhead.

Sliding-window attention applied: This model caps 5 of every 6 layers at a 512-token window. KV cache estimate is 83% smaller than naive full-attention math at this context length.

Hardware that fits

Apple entries show installed unified memory separately from a model budget based on an 8 GB reserve. That budget is a planning heuristic, not a fixed hardware limit.

No single GPU in our catalog has enough memory. Multi-GPU or CPU offload required.

Same 975B @ Q4_K_M, 1,048,576 context scenario, pre-loaded. Change the quant, shrink the context, or try a different preset.
Edit in the full tool

Inkling 975B at Q4_K_M with native 1M context needs about 653 GB of VRAM with all experts resident. Thinking Machines' MoE model activates 41B parameters per token from a 975B total pool. The hybrid global and sliding-window attention keeps the key-value cache to 47 GB at the full 1M context. Those 41B active params don't shrink what has to stay loaded, because different tokens pick different experts.

How this is calculated

The 975B parameter pool requires 546 GB of weights at Q4_K_M. The 66-layer architecture uses group-query attention with variable KV heads: 8 in the 11 global layers, 16 in the 55 sliding-window layers.

The sliding window caps 55 of 66 layers at a 512-token window with only 11 layers doing full global attention. The result is a 47 GB key-value cache at the full 1M context. Activation and software overhead adds about 59 GB.

Verdict

Resident mode needs 9 cards of 80 GB or 5 H200 cards of 141 GB with NVLink. The 512-token sliding window on 55 of 66 layers is what keeps the cache at 47 GB at full context, and at 256K it falls to about 12 GB.

More Inkling scenarios

Kimi K3.1 Pro at Q4_K_M
A 2.78T expert pool with 24 latent attention layers and fixed state in the other 69.
View details
Mistral Large 4 at Q4_K_M
A 1T expert pool with 63 latent cache owners and 44 layers that also store an index key.
View details
Muse Spark Open at Q4_K_M
Only 16 of its 62 layers grow with context, and the other 46 hold a 2,048 token window.
View details

Related guides

inkling pricing
Per-token cost for the inkling family this VRAM entry serves.
View guide
BPE tokenization explained
How tokenizers chunk text into the units VRAM serves.
View guide

Frequently asked questions

Why does Inkling use variable KV heads across layers?
The 11 global-attention layers use 8 KV heads while the 55 sliding-window layers use 16 KV heads. The sliding-window layers are capped at a 512-token window, so their extra KV heads cost little at long context. The global layers dominate the cache at full context, so the effective count used in sizing is 8.
Can I run Inkling on a single GPU?
No. Resident mode needs 653 GB across 9 cards of 80 GB. The 41B active count doesn't shrink that, because different tokens pick different experts. Moving expert weights to system RAM cuts VRAM, but the host then has to hold most of the 546 GB of weights and every token gets slower.
Can I run Inkling 975B on a single GPU?
No. The 975B pool at Q4 is roughly 546 GB of weights, exceeding every consumer GPU and most datacenter GPUs. For most users, the hosted API from Thinking Machines Lab or OpenRouter partners is the practical option.
When does the hosted API beat self-hosting for a 975B-class MoE?
Almost always, until your monthly volume reaches the multi-hundred-million-token range with strict data residency requirements. The self-hosting bill at this scale is a small cluster, ops staffing, and utilization risk, while the API is a line-item you can dial to zero. The self-host argument is strongest for burst-proof capacity planning and air-gapped data, not for everyday cost.

Related tools

RAM Latency Calculator

Convert DDR3/DDR4/DDR5 timings (CL, tRCD, tRP, tRAS) into true latency in nanoseconds.

Use tool

Power Cost Estimator

Estimate annual electricity costs for your PC, Server, or TV.

Use tool

Data Transfer Calculator

Estimate transfer times for files over USB, WiFi, Ethernet, and more.

Use tool

Memory and Storage Throughput Visualizer

Visualize the large speed gap between CPU cache, RAM, and storage.

Use tool