TechCompare LogoTechCompare
Showing centimeters. Press to switch to inches.Showing inches. Press to switch to centimeters.

AI

Mixture of experts (MoE)

A model design that splits each layer into many expert blocks and sends every token to only a few of them. Only the chosen experts do work, so a MoE model computes like a smaller model, but all the experts still have to sit in memory.

Total and active parameters

A dense model runs every parameter for every token. A mixture of experts (MoE) model splits some layers into several expert networks, and a small router picks a few of them for each token. The rest sit idle for that token.

Mixtral 8x7B is the textbook case. Each layer has 8 feedforward experts and the router picks 2 per token, so each token has access to 47B parameters but only uses 13B of them. It isn't eight full 7B models: only the feedforward blocks repeat in each layer. DeepSeek-V3 goes much further, with 671B parameters in total and 37B activated per token.

Memory follows the total, speed follows the active count

The router can send any token to any expert, so all of them have to be loaded. Each Mixtral token only runs 13B parameters, yet the model still needs 28.7 GB of weights at Q4_K_M, against 8.0 GB for a dense 13B model. DeepSeek-V3's Q4_K_M weights come to 410 GB, which eight 80 GB cards (640 GB together) can hold.

The work per token follows the active count instead. Each DeepSeek-V3 token runs through 37B parameters, so the compute resembles a 37B dense model, not a 671B one. That's why MoE models can generate text quickly for their total size, as long as the whole set of weights fits somewhere.

Reading the names

  • 8x7B (Mixtral): the experts per layer and a rough size for each. The real total is 47B, not eight full copies.
  • A3B and A22B: the number after the A is the active count in billions. In the calculator's model list, Qwen3 30B A3B has 30B total and 3B active, and Qwen3 235B A22B has 235B total and 22B active.
  • 17B/109B (Llama 4 Scout): active and total together, 17B active out of 109B.

When a model gives one number, check whether it's the total or the active count before you size a GPU. Memory needs the total, and speed follows the active number.

Work it out yourselfOpen the LLM VRAM Calculator

Read next

DeepSeek V3 671B at Q4_K_M gpt-oss 20B at Q4_K_M Llama 4 Scout (17B active, 109B total)

Related terms

Parameter countThe number of learned weights in a model, written as 8B, 70B or 405B for billions. It sets how big the model file is and how much memory it needs, but it doesn't tell you how good the model is.VRAMThe memory on a graphics card, next to the GPU. To run a large language model on a GPU, its weights, its KV cache and some runtime overhead all have to fit in VRAM, so VRAM decides which models and context lengths a card can handle at full speed.QuantizationStoring a model's weights with fewer bits per number, like 4 or 8 bits instead of 16. A quantized model takes far less VRAM and usually generates tokens faster, in exchange for a small drop in quality.KV cacheThe attention keys and values a model stores for every token it has already read, so it doesn't recompute them for each new token. It grows with the length of the conversation and can take more memory than the model weights at long context.

Sources

Back to the glossary