How Much VRAM Do You Need for an LLM? Formulas and Tables

Updated 2026-10-26 · 11 min read

Calculate GPU memory for LLM inference from the model's parameter count and config: weights, KV cache and overhead, with estimate tables for 8B, 14B, 32B and 70B models.

How much VRAM do you need for an LLM? Less guesswork than most forum answers suggest. GPU memory for inference has three parts you can calculate: the model weights, the KV cache for the tokens in flight, and a runtime overhead. This guide gives you the formulas, a worked example from a real model config, and estimate tables for 8B, 14B, 32B and 70B models.

Short answer: VRAM ≈ weights + KV cache + overhead. Weights take about 2 GB per billion parameters at FP16, 1 GB at 8-bit, and roughly 0.6 GB at 4-bit (GGUF Q4KM). So a 7–8B model at 4-bit runs on an 8 GB card with a modest context, a 32B model at 4-bit needs about 24 GB, and a 70B model at 4-bit needs about 48 GB. Long contexts and many concurrent users add KV cache on top, and that part can exceed the weights.

How much VRAM an LLM needs: the formula

VRAM_total ≈ weights + kv_cache + overhead

weights  = parameters × bytes_per_parameter
kv_cache = 2 × layers × kv_heads × head_dim × context_tokens × bytes_per_value × batch
overhead ≈ 1–2 GB per GPU as a starting point (more with large batches)

Everything below uses decimal gigabytes (1 GB = 10^9 bytes). GPU vendors label memory in binary units, so a "24 GB" card holds 24 GiB, about 25.8 decimal GB. The difference gives you a little headroom on paper; spend it on safety margin, not on a longer context.

Weights: parameters times bytes per parameter

The parameter count is in the model name or, more precisely, on the model card or in the safetensors metadata on Hugging Face. Llama 3.1 8B has 8.03 billion parameters, Llama 3.3 70B has 70.55 billion. Multiply by the storage size of one parameter:

| Precision | Bits per weight | GB per billion parameters | Typical engine | |---|---|---|---| | FP32 | 32 | 4.0 | Training, rarely for inference | | FP16 / BF16 | 16 | 2.0 | vLLM, SGLang, Transformers | | FP8 / INT8 | 8 | 1.0 | vLLM, SGLang on supported GPUs | | GGUF Q80 | 8.5 | 1.06 | llama.cpp, Ollama | | GGUF Q6K | about 6.6 | about 0.82 | llama.cpp, Ollama | | GGUF Q5KM | about 5.7 | about 0.71 | llama.cpp, Ollama | | GGUF Q4KM | about 4.8–4.9 | about 0.61 | llama.cpp, Ollama | | AWQ / GPTQ 4-bit | 4 plus scales | 0.5 plus overhead | vLLM, SGLang |

Why the GGUF values are not round numbers: GGUF quantizes weights in blocks and stores a scale per block. Q80 stores 32 weights in 34 bytes (32 one-byte values plus a 16-bit scale), which is exactly 8.5 bits per weight. The K-quants use super-blocks with extra scales, and the "M" variants keep some sensitive tensors at a higher precision, so Q4KM is a mix of 4-bit and 6-bit tensors. The effective values in the table come from dividing published GGUF file sizes by parameter count: the Q4KM file of Llama 3.1 8B in bartowski's GGUF repository is 4.92 GB, which works out to about 4.9 bits per weight, and the 70B Q4K_M file is 42.5 GB, about 4.8.

AWQ and GPTQ checkpoints are 4-bit for the linear layers, but usually keep the embedding table and output layer in 16-bit. For models with large vocabularies (Llama 3 and Qwen 2.5 both have vocabularies above 128,000 tokens) that adds a noticeable amount, so check the actual checkpoint size instead of assuming 0.5 GB per billion.

For mixture-of-experts models, use the total parameter count for memory. Only a few experts are active per token, which makes them fast, but all experts must be loaded.

KV cache: the part people forget

During generation, every layer stores a key and a value vector for each token in the context so it does not have to recompute them. That is the KV cache:

kv_bytes = 2 × num_hidden_layers × num_key_value_heads × head_dim
             × context_tokens × bytes_per_value × batch

• 2 is one key plus one value.
• **numkeyvalue_heads**, not the number of attention heads. Most current models use grouped-query attention (GQA), where several query heads share one KV head. This is the biggest lever in the formula.
• **head_dim** is usually hiddensize / numattentionheads; some configs state it directly as headdim.
• **bytespervalue** is 2 for an FP16/BF16 cache, 1 for an FP8 or Q8_0 cache.
• batch is the number of sequences in memory at the same time, which for a server means concurrent requests.

Worked example: Llama 3.1 8B

The model's published config.json contains:

{
  "hidden_size": 4096,
  "num_attention_heads": 32,
  "num_key_value_heads": 8,
  "num_hidden_layers": 32,
  "head_dim": 128,
  "max_position_embeddings": 131072,
  "torch_dtype": "bfloat16"
}

(The file has more keys; these are the ones that matter here. head_dim checks out: 4096 / 32 = 128.)

KV cache per token with a 16-bit cache:

2 × 32 layers × 8 kv_heads × 128 head_dim × 2 bytes = 131,072 bytes = 128 KiB per token

From there:

| Scenario | Tokens in cache | KV cache | |---|---|---| | One chat, 8,192-token context | 8,192 | 1.07 GB | | One chat, 32,768-token context | 32,768 | 4.29 GB | | One chat, full 131,072-token context | 131,072 | 17.2 GB | | 32 concurrent requests at 4,096 tokens each | 131,072 | 17.2 GB |

The full context window of an 8B model needs more memory than the 8B weights at 8-bit. The same is true for 32 users with moderate prompts. This is why a model that "fits" on a GPU can still fail to start in vLLM with its default maxmodellen, and why raising OLLAMANUMPARALLEL costs memory.

GQA matters here: Llama 3.1 8B has 32 attention heads but only 8 KV heads. If every head kept its own keys and values, the cache would be four times larger. A model with even fewer KV heads is cheaper still: Qwen2.5-7B has 28 layers and 4 KV heads with a head size of 128, so it needs 2 × 28 × 4 × 128 × 2 = 57,344 bytes (56 KiB) per token, less than half of Llama 3.1 8B.

The formula does not cover every architecture. Models with sliding-window attention, multi-head latent attention (DeepSeek-style MLA) or hybrid linear-attention layers store less per token; read the model card or the engine's startup log for those.

Shrinking the KV cache

• Lower the context limit to what your application needs: --max-model-len in vLLM, -c in llama.cpp, OLLAMACONTEXTLENGTH in Ollama.
• Quantize the cache to 8-bit: --kv-cache-dtype fp8 in vLLM, -ctk q80 -ctv q80 in llama.cpp, OLLAMAKVCACHETYPE=q80 in Ollama. This halves the KV term at a small quality cost.
• Cap concurrency: --max-num-seqs in vLLM, -np in llama.cpp, OLLAMANUMPARALLEL in Ollama.

Overhead

On top of weights and KV cache, each GPU needs memory for the CUDA context, temporary activation buffers during prefill, attention workspaces and, in vLLM, CUDA graphs. This depends on the engine, the batch size and the prompt chunk size, so there is no exact formula. As a starting point, plan 1–2 GB per GPU for single-user or small-batch inference and more for large batches. llama.cpp's automatic fitting leaves a 1,024 MiB margin per device by default, which is a sensible reference.

Engines also differ in how they claim memory. vLLM reserves a fixed share of the GPU at startup (--gpu-memory-utilization, 0.92 by default in current releases), loads the weights, and turns everything left into KV cache. So vLLM always looks like it uses most of the GPU; what matters is the KV cache capacity it reports at startup. llama.cpp and Ollama allocate what the configured context needs.

Estimate tables for common model sizes

These are calculated estimates, not measurements. Parameter counts and model configs are taken from the published Hugging Face checkpoints: Llama 3.1 8B (32 layers, 8 KV heads), Qwen2.5-14B (48 layers, 8 KV heads), Qwen2.5-32B (64 layers, 8 KV heads) and Llama 3.3 70B (80 layers, 8 KV heads), all with a head size of 128. Q4KM is calculated at 4.85 bits per weight.

Weights only

| Model | Parameters | FP16 / BF16 | 8-bit (FP8, INT8) | Q80 | Q4K_M | |---|---|---|---|---|---| | Llama 3.1 8B | 8.03 B | 16.1 GB | 8.0 GB | 8.5 GB | 4.9 GB | | Qwen2.5-14B | 14.77 B | 29.5 GB | 14.8 GB | 15.7 GB | 9.0 GB | | Qwen2.5-32B | 32.76 B | 65.5 GB | 32.8 GB | 34.8 GB | 19.9 GB | | Llama 3.3 70B | 70.55 B | 141.1 GB | 70.6 GB | 75.0 GB | 42.8 GB |

KV cache (16-bit, one sequence)

| Model | Per token | 8,192 tokens | 32,768 tokens | |---|---|---|---| | Llama 3.1 8B | 128 KiB | 1.07 GB | 4.29 GB | | Qwen2.5-14B | 192 KiB | 1.61 GB | 6.44 GB | | Qwen2.5-32B | 256 KiB | 2.15 GB | 8.59 GB | | Llama 3.3 70B | 320 KiB | 2.68 GB | 10.74 GB |

Total estimate: weights + KV cache + 1.5 GB overhead, one sequence

| Model | Context | FP16 / BF16 | 8-bit | Q4KM | |---|---|---|---|---| | Llama 3.1 8B | 8k | 18.6 GB | 10.6 GB | 7.4 GB | | Llama 3.1 8B | 32k | 21.9 GB | 13.8 GB | 10.7 GB | | Qwen2.5-14B | 8k | 32.7 GB | 17.9 GB | 12.1 GB | | Qwen2.5-14B | 32k | 37.5 GB | 22.7 GB | 16.9 GB | | Qwen2.5-32B | 8k | 69.2 GB | 36.4 GB | 23.5 GB | | Qwen2.5-32B | 32k | 75.6 GB | 42.9 GB | 30.0 GB | | Llama 3.3 70B | 8k | 145.3 GB | 74.7 GB | 47.0 GB | | Llama 3.3 70B | 32k | 153.3 GB | 82.8 GB | 55.0 GB |

How to read this against real cards:

• 8 GB: 7–8B at Q4KM with a short context. Tight.
• 12–16 GB: 7–8B at 8-bit, or 14B at Q4KM.
• 24 GB: 7–8B at FP16, 14B at 8-bit, or 32B at Q4KM with about 8k context. At 32k, the 32B model needs an 8-bit KV cache or partial offload.
• 48 GB (one card or two 24 GB cards): 70B at Q4KM with a short context, or 32B at 8-bit.
• 80 GB: 32B at FP16 with a short context, or 70B at FP8 with a limited context. 70B at FP16 needs two 80 GB GPUs.

For a multi-user server, add the KV cache per concurrent request. Qwen2.5-32B serving 16 users at 8k tokens each needs 16 × 2.15 = 34.4 GB of KV cache on top of the weights.

Multi-GPU and tensor parallelism

When a model does not fit on one GPU, split it:

• Tensor parallelism (vLLM --tensor-parallel-size 2, SGLang --tp 2) splits each layer's weight matrices across GPUs. Weights and KV cache divide roughly evenly, but each GPU still pays its own overhead. The attention head count must be divisible by the tensor-parallel size, which is why sizes of 2, 4 and 8 are standard. If the TP size exceeds the number of KV heads, KV heads are replicated instead of split, so the per-GPU KV saving stops there. Tensor parallelism exchanges data between GPUs on every layer, so it works best with NVLink or similar interconnects; over PCIe it still works, with more communication cost.
• Layer (pipeline) splitting assigns whole layers to each GPU. llama.cpp does this by default (--split-mode layer, with --tensor-split 3,1 for uneven cards), and Ollama spreads a model across GPUs automatically when it does not fit on one. It needs far less bandwidth between GPUs and suits mismatched consumer cards, but a single request only uses one GPU at a time.

Worked example: Llama 3.3 70B at Q4KM on two 24 GB cards. Weights 42.8 GB plus 8k KV cache 2.7 GB plus about 1.5 GB overhead per card gives roughly 48.5 GB against about 51.5 GB of real capacity. It fits, with little room for a longer context or a second user. An 8-bit KV cache roughly halves the KV term if you need more.

CPU offload

llama.cpp and Ollama can keep some layers in system RAM and run them on the CPU. In llama.cpp you set the number of GPU layers with -ngl; Ollama decides automatically, and ollama ps shows the split (for example "40%/60% CPU/GPU").

Offload lets a model run, but speed drops sharply, because decoding is bound by memory bandwidth: every generated token reads all the weights once. A useful upper bound:

max tokens per second ≈ memory bandwidth (GB/s) / bytes read per token (GB)

For a dense model, bytes read per token is roughly the weight size. When part of the model lives in system RAM, that part is limited by RAM bandwidth, which on desktop and most server platforms is far below GPU memory bandwidth. Check the bandwidth figures for your own CPU platform and GPU and run the numbers before you buy RAM instead of a GPU.

Mixture-of-experts models are the exception that makes offload attractive. Only the active experts are read per token, so llama.cpp's --n-cpu-moe N (or --cpu-moe for all layers) keeps expert weights in RAM and the attention layers and KV cache on the GPU. vLLM has --cpu-offload-gb, but it streams weights over PCIe on every forward pass and is a last resort.

Apple Silicon and unified memory

On Apple Silicon, CPU and GPU share one memory pool, so a Mac with 64 GB or more can run models that would need a multi-GPU PC. Three caveats:

• macOS does not let the GPU use all of it. By default the GPU's wired memory limit is a fraction of total RAM, roughly two-thirds to three-quarters depending on the machine. On recent macOS versions you can raise it with sudo sysctl iogpu.wiredlimitmb=; the setting resets on reboot, and you must leave enough for the OS and your other applications.
• Bandwidth varies a lot between chip tiers, and the bandwidth formula above applies directly. A large model fits in memory long before it runs fast.
• Docker on macOS cannot use the Apple GPU, so run llama.cpp or Ollama natively on the Mac.

An LLM GPU memory calculator in Python

def estimate_vram_gb(params_b, bits_per_weight, layers, kv_heads, head_dim,
                     context, batch=1, kv_bytes=2, overhead_gb=1.5):
    """Rough inference memory estimate in decimal GB."""
    weights = params_b * 1e9 * bits_per_weight / 8
    kv_cache = 2 * layers * kv_heads * head_dim * context * kv_bytes * batch
    return (weights + kv_cache) / 1e9 + overhead_gb


# Llama 3.1 8B, Q4_K_M, 8k context, one sequence
print(round(estimate_vram_gb(8.03, 4.85, 32, 8, 128, 8192), 1))  # 7.4

# Llama 3.3 70B, FP8 weights and FP8 KV cache, 16 users at 4k tokens
print(round(estimate_vram_gb(70.55, 8, 80, 8, 128, 4096, batch=16, kv_bytes=1), 1))  # 82.8

Take layers, kvheads and headdim from the model's config.json (numhiddenlayers, numkeyvalueheads, headdim or hiddensize / numattention_heads). Then confirm with the engine itself: vLLM logs the available KV cache at startup, and llama.cpp prints its buffer sizes when it loads a model.

Next steps

• Pick the engine that matches your memory budget: Ollama vs vLLM vs llama.cpp.
• Provision a GPU machine reproducibly: A GPU VM for LLM inference on AWS with Terraform.
• Run a model with a chat UI using the LLM chatbot stack, or browse the MLOps stacks and the full stack catalog.