TechCompare LogoTechCompare

How much VRAM does Llama 3.1 8B need to fine-tune at FP16? Adam optimizer math

Full fine-tuning at FP16 is the worst-case memory configuration. Use it only if you must. LoRA reduces memory to roughly inference + a few GB, and QLoRA reduces it further by quantizing the base model weights. For 99% of fine-tuning use cases, QLoRA on a single 24 GB GPU produces results indistinguishable from full fine-tuning.

Full fine-tuning Llama 3.1 8B at FP16 with native 128K context needs about 167.8 GB of VRAM, demonstrating the massive scaling of Adam and activations at high context.

By TechCompare · Updated

Total VRAM required
168 GB
Llama 3.1 8B at FP16
Weights
144 GB
Includes Adam optimizer states
KV cache
17.2 GB
128K tokens, FP16 KV

Calculator

Estimated VRAM required

41.1 GB

8B params at FP16, 131,072 token context, batch 1, training (Adam).

Weights
17.3 GB
KV cache
17.2 GB
Overhead
6.6 GB
Doesn't fit on a 32 GB consumer GPU at FP16. Q2_K (25.0 GB) is the smallest quant that fits a single RTX 5090.

Estimate accuracy: Weights within ~2%. KV cache within ~5% for standard GQA models, ~10% for MLA (DeepSeek). Real VRAM may vary with framework (vLLM vs llama.cpp vs Transformers), Flash Attention, and driver overhead.

LoRA fine-tune sizing: Forward weights at FP16, only ~1% of params get optimizer state (FP32 master + grad + AdamW m + v). Real LoRA peak depends on rank and target modules; this is the typical r=16 ceiling.

Hardware that fits

Apple M3 Max 64GB
Unified
48 GB
86% used
RTX 6000 Ada
Pro
48 GB
86% used
A100 80GB
Datacenter
80 GB
51% used

Just barely too small

A100 40GB
Datacenter
40 GB
short by 1.1 GB
Same 8B @ FP16, 131,072 context scenario, pre-loaded. Change the quant, shrink the context, or try a different preset.
Edit in the full tool

How this is calculated

Training weights + gradients + Adam optimizer buffers take 144 GB. The 128K KV cache takes 17.2 GB, and activation overhead adds roughly 6.6 GB, totaling 167.8 GB.

Verdict

The 167.8 GB total comes from 144 GB of training weights + gradients + Adam optimizer states (which at FP16 plus an FP32 master plus the two FP32 Adam buffers is roughly 4x the inference weight cost), 17.2 GB of FP16 KV cache at the full 128K context, and 6.6 GB of activation overhead. A single 24 GB consumer card cannot host this configuration, which is the entire point: Adam's two FP32 buffers per parameter triple-or-quadruple per-parameter memory, then activations saved for backprop add another chunk. LoRA drops the optimizer and gradient memory to a few GB by training only adapter weights, and QLoRA goes further by quantizing the base model to 4-bit before training the adapters, fitting a single 24 GB card comfortably with results that match full fine-tuning for most use cases.

More Training scenarios

DeepSeek V4 Pro 1.6T (MoE) at Q4_K_M
DeepSeek V4 Pro 1.6T at Q4_K_M with the full 1M-token context needs about 1012 GB of VRAM with every expert resident - this is the real shape of the model and the number to plan a deployment against.
View details ➜
Llama 4 Scout (17B/109B) at Q4_K_M
Llama 4 Scout at Q4_K_M with its native 10M context needs about 2231 GB of VRAM with all 109B params resident - that's the number you size hardware against.
View details ➜
gpt-oss 20B (MoE) at Q4_K_M
gpt-oss 20B at Q4_K_M with native 128K context needs about 19.4 GB of VRAM with all experts resident, dropping to roughly 9.3 GB with active-only weight loading.
View details ➜

Related guides

Token cost calculator
How per-token counts turn into API costs.
View guide ➜

Frequently asked questions

Why does training need so much more memory than inference?
Adam's optimizer states (two FP32 buffers per parameter) plus gradients triple or quadruple the per-parameter cost vs inference. Activations needed for backprop add another large chunk.
Can I fine-tune Llama 3.1 8B on a single 24 GB GPU?
Not with full fine-tuning at native context. Use LoRA (low-rank adapters) or QLoRA, both of which fit comfortably on 24 GB and produce comparable results for most use cases.
Why does training need so much more memory than inference?
Training holds activations for backprop, gradients, optimizer states (Adam = 2× model weights), and often a mixed-precision master copy. For an 8B model that's: 16 GB weights (FP16 + FP32 master + grads), 32 GB Adam states. Roughly 4-6× the inference footprint. QLoRA reduces this dramatically by training only adapter parameters in 4-bit base.
What does QLoRA actually store during fine-tuning?
The frozen base model at 4-bit plus a small set of trainable adapter matrices at full precision, with gradients and optimizer states applying only to the adapters. For an 8B model that turns roughly 48 GB of full fine-tuning memory into something that runs on a single 24 GB card. Output quality on standard fine-tuning tasks is statistically indistinguishable from full-parameter training.

Related tools

RAM Latency Calculator

Convert DDR3/DDR4/DDR5 timings (CL, tRCD, tRP, tRAS) into true latency in nanoseconds.

Use tool ➜

Power Cost Estimator

Estimate annual electricity costs for your PC, Server, or TV.

Use tool ➜

Data Transfer Calculator

Estimate transfer times for files over USB, WiFi, Ethernet, and more.

Use tool ➜

Memory and Storage Latency Visualizer

Visualize the massive speed difference between CPU cache, RAM, and storage.

Use tool ➜