How much VRAM does Qwen 3.8 27B need at Q4_K_M? Dense 1M context mid-size

By TechCompare · Updated · How we check

Qwen 3.8 27B at Q4 needs ~29 GB at 128K context or ~95 GB at the full 1M window with YaRN. The KV cache, not the 17 GB of weights, sets the hardware. A 48 GB card handles it at 128K, and that's how most people should run it.

Calculator

Estimated VRAM required

94.2 GB

27.5B params at Q4_K_M, 1,048,576 token context, batch 1, inference.

Weights
16.8 GB
KV cache
68.8 GB
Overhead
8.8 GB

Estimate accuracy: Weights within ~3% of most GGUF files from Q4_K_M up, ~7% for Q3_K_M and Q2_K. KV cache within ~5% for standard GQA models, ~10% for MLA or hybrid state estimates. Real VRAM may vary with framework (vLLM vs llama.cpp vs Transformers), Flash Attention, and driver overhead.

KV cache exceeds model weights: Consider lowering the context length to save on VRAM. Contexts between 8K and 64K are generally more typical for local setups.

Hybrid linear attention applied: 48 of 64 layers use fixed recurrent state, so only 16 full-attention layers contribute to the context-growing KV cache. The fixed state adds about 0.16 GB at batch 1 and does not grow with context length.

Hardware that fits

Unified memory machines show a model budget, not their installed memory. We hold back 8 GB for the system, or use the vendor's GPU limit when that's lower. It's a planning estimate, not a hard limit.

RTX PRO 6000 96GB
Pro
96 GB
99% used
AMD Ryzen AI Max+ 395 128GB
Unified
~112 GB
85% of estimate
Apple M5 Max 128GB
Unified
~120 GB
79% of estimate

Multi-GPU setups

4× RTX 5090
Consumer · 24.3 GB per card
128 GB
76% used

Pooled cards show up once a model outgrows the biggest single card of that kind. Each card past the first keeps about 1 GB for its own buffers, and that's counted here. Your inference engine has to support the split, and speed also depends on how the cards connect.

Qwen 3.8 27B at Q4_K_M needs about 95 GB of VRAM if you stretch it to a full 1M context with YaRN, past its 262,144 token native window, but the weights are only 17 GB and most of the rest is key-value cache. Alibaba's dense 27B is the practical local-deployable member of the Qwen 3.8 family.

How this is calculated

Weights at Q4_K_M are 16.8 GB. Only 16 of the 64 layers use full attention and keep a KV cache, with 4 KV heads at head_dim 256. That cache is 69 GB at the full 1M context and 8.6 GB at 128K, and overhead adds 9 GB at the long window. The other 48 are Gated DeltaNet layers that keep a fixed 0.16 GB state.

At a 1M window, most of this mid-size dense model's VRAM goes to the KV cache. At a 128K working context the total is about 29 GB and the model fits on a 48 GB card.

Verdict

Run it at 128K, where the total is 29 GB (16.8 GB of Q4 weights, 8.6 GB of FP16 KV cache and 2.7 GB of overhead). That fits a 48 GB RTX 6000 Ada or two 24 GB consumer cards. The full 1M window takes the cache to 69 GB and the total to 95 GB, which needs datacenter GPUs.

Most workloads stay under 128K, and RAG often works better than raw long context for retrieval tasks.

More Qwen scenarios

Kimi K3.1 Pro at Q4_K_M
A 2.78T expert pool with 24 latent attention layers and fixed state in the other 69.
View details
Mistral Large 4 at Q4_K_M
A 1T expert pool with 63 latent cache owners and 44 layers that also store an index key.
View details
Muse Spark Open at Q4_K_M
Only 16 of its 62 layers grow with context, and the other 46 hold a 2,048 token window.
View details

Frequently asked questions

Why does a 27B model need 95 GB at 1M context?
The KV cache scales with context length times layers times KV width independently of weight size. At 1M tokens across the 16 full attention layers, with 4 KV heads at head_dim 256, the cache is 69 GB, over 4x the size of the Q4 weights. At 128K the same math yields 8.6 GB and the model fits mid-range pro cards.
Qwen 3.8 27B or Qwen3.5 27B?
Qwen 3.8 27B if you want the documented extension to 1M and the newer training data. Qwen3.5 27B has nearly identical VRAM math (the same 64 layer hybrid layout) at the same 256K native context. For RAG and standard agentic workloads at 128K or below, they're equivalent on hardware.