K2 Horizon 375B A23B VRAM: about 396 GB at Q4 and 512K context

By TechCompare · Updated · How we check

K2 Horizon 375B A23B needs about 396 GB with Q4_K_M weights, FP16 cache, and 524,288 tokens. At 32K context, the estimate is about 261 GB.

Calculator

Estimated VRAM required

396 GB

375B params at Q4_K_M, 524,288 token context, batch 1, inference.

Weights
229 GB
KV cache
131 GB
Overhead
36.0 GB

Estimate accuracy: Planning estimate. Rounded model counts, tensor formats, cache packing, and workspace can change actual allocation. Confirm the exact checkpoint and serving engine with a load test.

K2 Horizon uses 61 full-context GQA layers with 8 KV heads. Its 23B active count describes computation, while all-resident weights include the full 375B pool. Quantization choices are planning scenarios and require a compatible checkpoint and backend.

Hardware that fits

Unified memory machines show a model budget, not their installed memory. We hold back 8 GB for the system, or use the vendor's GPU limit when that's lower. It's a planning estimate, not a hard limit.

Apple M5 Ultra 512GB
Unified
~504 GB
79% of estimate

Multi-GPU setups

8× RTX PRO 5000 72GB
Pro · 50.4 GB per card
576 GB
70% used
2× NVIDIA B300
Datacenter · 199 GB per card
576 GB
69% used

Pooled cards show up once a model outgrows the biggest single card of that kind. Each card past the first keeps about 1 GB for its own buffers, and that's counted here. Your inference engine has to support the split, and speed also depends on how the cards connect.

K2 Horizon 375B A23B is IFM's MoE model with roughly 375B stored parameters and 23B active per token. Its native 524,288-token context makes the cache 33% of the full-context budget. At Q4_K_M, weights take about 229 GB and the FP16 attention cache takes another 131 GB. A 10% runtime allowance brings the total to 395.73 GB for one sequence. The active count doesn't replace the stored expert pool.

How this is calculated

The released configuration specifies 61 layers, hidden size 6,144, 48 query heads, eight KV heads, and a 128-value head dimension. Grouped-query attention shares cached keys and values across query heads, so memory uses the eight KV heads. This preset has no sliding-window or shared-cache reduction.

The formula is two tensors times 61 layers times eight KV heads times 128 values times context times bytes per value. At 524,288 tokens in FP16 it yields 130.9966 GB. Add 229 GB for approximate Q4 weights and 35.98 GB of overhead to reach 395.73 GB, or about 368.55 GiB.

At 32,768 tokens the cache falls to 8.19 GB and the total to 260.64 GB. These are inference budgets using rounded published parameter counts. The Q4 factor doesn't establish that a compatible converted checkpoint exists, and the runtime allowance doesn't explicitly simulate prompt activations or communication buffers.

Verdict

Choose context and concurrency before selecting the memory pool. Moving from 32K to the full 512K window adds about 135 GB to this modeled total, enough to change a multi-GPU plan.

Five nominal 80 GB GPUs exceed 396 GB in aggregate, but that says nothing about whether a five-way split is supported or balanced. Prefer a validated parallelism layout with measured per-device headroom. For many shorter requests, a smaller context cap may be more useful than reserving the entire native window for every sequence.

More K2 Horizon scenarios

Kimi K3.1 Pro at Q4_K_M
A 2.78T expert pool with 24 latent attention layers and fixed state in the other 69.
View details
Mistral Large 4 at Q4_K_M
A 1T expert pool with 63 latent cache owners and 44 layers that also store an index key.
View details
Muse Spark Open at Q4_K_M
Only 16 of its 62 layers grow with context, and the other 46 hold a 2,048 token window.
View details

Frequently asked questions

Does A23B mean I need memory for only 23B parameters?
A23B is the active parameter count per token. The full model still stores about 375B parameters, and routing can select different experts for different tokens. All-resident inference therefore budgets the complete pool. Offloading can reduce the GPU-resident portion, but the rest needs host memory or another storage tier and a serving strategy that handles the resulting data movement.
Why is the cache so large despite GQA?
Eight KV heads are fewer than the 48 query heads, which already shrinks the cache to a sixth of what caching every query head would take. There are still 61 layers and up to 524,288 token positions per sequence. Multiplying those dimensions makes the full FP16 cache about 131 GB. GQA reduces cache width, but it doesn't stop the cache from growing with context.
What happens if I select Q8 cache?
The calculator uses one byte per logical cache value rather than two, so this cache budget falls from about 131 GB to 65.50 GB at full context. With the same Q4 weights and 10% allowance, the total becomes about 323.68 GB. Treat that as a memory scenario until your serving engine confirms compatible cache quantization and its actual scale, metadata, and alignment overhead.
How should I budget concurrent requests?
Weights can be shared across requests, but each sequence has its own cached token positions. With two full 512K sequences, the modeled FP16 cache is about 262 GB rather than 131 GB. Real continuous-batching systems allocate cache pages according to their scheduler, so set context and concurrency limits together. Also test prompt-processing peaks, which can exceed a steady decode allocation.
Is a GPU with enough aggregate memory guaranteed to work?
No. You need support for the model's architecture and tensor formats, a valid parallelism split, and enough memory on each participating device. KV cache and other tensors may be replicated rather than evenly divided, depending on the engine. Use the linked publisher configuration to check dimensions, then validate the exact runtime setup. The calculator reports a storage budget and doesn't predict tokens per second.