Local AI Hobbyist

LLM VRAM for local AI hobbyists: which models fit your GPU

Last updated 2026-11-11

Work out the total for your model, quant and context, then leave a gigabyte or two free. Qwen3 14B at Q4_K_M and an 8K context comes to about 11.4 GB.

Estimated VRAM required

11.4 GB

14.77B params at Q4_K_M, 8,192 token context, batch 1, inference.

Weights

9.1 GB

KV cache

1.4 GB

Overhead

1.1 GB

Estimate accuracy: Weights within ~3% of most GGUF files from Q4_K_M up, ~7% for Q3_K_M and Q2_K. KV cache within ~5% for standard GQA models, ~10% for MLA or hybrid state estimates. Real VRAM may vary with framework (vLLM vs llama.cpp vs Transformers), Flash Attention, and driver overhead.

Hardware that fits

Unified memory machines show a model budget, not their installed memory. We hold back 8 GB for the system, or use the vendor's GPU limit when that's lower. It's a planning estimate, not a hard limit.

RTX 5070

Consumer

12 GB

95% used

RTX 5060 Ti 16GB

Consumer

16 GB

72% used

Apple M5 32GB

Unified

~24 GB

48% of estimate

A100 40GB

Datacenter

40 GB

29% used

Qwen3 14B at Q4_K_M with an 8K context, pre-loaded. Swap the model, change the quant or stretch the context to see when it stops fitting.

Edit in the full tool

Running a model on your own GPU comes down to one question: does it fit in video memory? Qwen3 14B at Q4_K_M, a 4-bit quant, with an 8K token context needs about 11.4 GB. Of that, 9.1 GB is the model weights and 1.4 GB is the KV cache, which holds your conversation. A small runtime allowance covers the rest.

Why this matters for you

Three settings decide the size: the model, the quant and the context length. The quant sets how many bytes each parameter takes. Q4_K_M uses about 0.61 bytes per parameter, Q8_0 uses about 1.06 and FP16 uses 2. The context length sets the KV cache, which grows with every token the model keeps in view. The numbers on this page come from the same estimate as the LLM VRAM Calculator, so you can swap in your own model and see where it lands. It's a planning figure, not a promise. Runtimes add their own buffers, so leave some room.

Match the model to your card

Start from the memory on your card and work backward. At Q4_K_M and an 8K context, Llama 3.1 8B needs 6.6 GB, Qwen3 14B needs 11.4 GB, Qwen3 32B needs 23.9 GB and Llama 3.3 70B needs 50.0 GB. Each step up asks for a different class of card.

An 8 GB card can run the 8B model with its 6.6 GB total, though a long context or a bigger quant will push it past the limit. A 12 GB card has room for Qwen3 14B for everyday chats. A 24 GB card fits Qwen3 32B at an 8K context, but not at 32K, where the total reaches 31.0 GB.

Context length and the KV cache

Every token in the conversation adds to the KV cache. Qwen3 14B at Q4_K_M needs 11.4 GB at an 8K context and 15.9 GB at 32K. The weights don't change with context, so the whole gap is the KV cache. Long documents, code bases and chat histories all draw on it.

Many runtimes can also store the KV cache at 8 bits instead of 16. For Qwen3 14B at 32K, that takes the KV cache from 5.4 GB to 2.7 GB, and the total from 15.9 GB to 12.9 GB. It can cost a little accuracy on long inputs, so try it before you rely on it.

Q4_K_M, Q8_0 or FP16

Take Llama 3.1 8B at an 8K context. Q4_K_M needs 6.6 GB, Q8_0 needs 10.6 GB and FP16 needs 18.8 GB. Q4_K_M usually keeps most of a model's quality, which makes it the standard starting point for a hobby setup. Q8_0 sits very close to the original when you have the memory to spare. FP16 is for testing and fine tuning, not for everyday chat.

Mixture of experts models still need every weight

A mixture of experts model like gpt-oss 20B only runs part of its parameters on each token. That helps speed, but it doesn't help memory, because every expert stays loaded. The calculator puts gpt-oss 20B at 13.7 GB at Q4_K_M and an 8K context. That fits a 16 GB card.

Leave headroom

Your desktop, browser and other apps use video memory too. A card that is nearly full at load time can run out when the context grows or a second program starts. As a rule of thumb, leave a gigabyte or two free. If the model doesn't fit, you have four levers: a smaller quant, a shorter context, an 8-bit KV cache or a smaller model. Offloading layers to system RAM works too, but generation slows down a lot.

Verdict

Start with Q4_K_M and an 8K context, check the total against your card, and only then raise the quant or the context. A model that fits with room to spare runs better than one squeezed in to the last gigabyte. For the cost side of a hosted model, the LLM Pricing Calculator and the Token Counter cover it.

Open the LLM VRAM Calculator

Frequently asked questions

How much VRAM do I need to run a 70B model?

Llama 3.3 70B at Q4_K_M with an 8K context comes to about 50.0 GB in the calculator. That's more than two 24 GB cards hold together, so people shorten the context to fit two cards, add a third card, or offload part of it to system RAM, which slows generation. At 32K the total grows to 58.8 GB.

How much memory does a longer context need?

Llama 3.1 8B at Q4_K_M needs 6.6 GB at an 8K context and 10.1 GB at 32K. The KV cache is the part that changes, from 1.1 GB to 4.3 GB.

Is the VRAM estimate exact?

No. It's built from the model's layer count, hidden size and KV heads, plus a runtime allowance. Real use varies with the runtime, the GPU driver and settings like batch size. Treat the number as a planning figure and keep a margin.

Will more VRAM make a model run faster?

Not by itself. VRAM decides whether the model fits. Generation speed depends mostly on memory bandwidth, and a model that fits fully on the GPU runs much faster than one that spills into system RAM.

LLMs that fit in 16 GB of VRAM

Every model that fits a 16 GB card, with the closest misses.

LLMs that fit in 24 GB of VRAM

Every model that fits an RTX 3090 or 4090, with the closest misses.

Llama 3.1 8B at Q4

The 8B model at a 4-bit quant, a common fit for smaller cards.

gpt-oss 20B at Q4

A mixture of experts model that still keeps every weight in memory.

Llama 3.3 70B at Q4

What the 70B class needs and why it often spans two cards.

Llama 3.1 8B with a 128K context

How far the KV cache grows when you push the context to the limit.