How Much VRAM Do You Need? Model Size, Quantization, and Memory Explained

A model that fits in VRAM at load time can still crash at the first long prompt. Weights are only one of three memory bills. The other two are the KV cache and runtime overhead. Budget all three and you can size any GPU before you buy it.

Quick Takeaways

• Core formula: VRAM ≈ (parameters × bits per weight ÷ 8) + KV cache + 1–2 GB runtime overhead.
• Q4_K_M averages ~4.89 bits per weight. It cuts FP16 weight memory by about 70% with only a small quality loss.
• KV cache grows linearly with context. A 70B model with GQA costs about 0.31 MB per token in FP16.
• Fine-tuning needs far more memory than inference. QLoRA shrinks the gap by roughly an order of magnitude.

The VRAM Formula

Three things occupy GPU memory during inference:

1. Model weights. Fixed once you pick a quantization level.
2. KV cache. Grows with context length and batch size.
3. Overhead. CUDA context, compute buffers, and fragmentation. Budget 1–2 GB.

weights_GB  = params_in_billions × bits_per_weight ÷ 8
kv_cache_GB = 2 × layers × kv_heads × head_dim × context × bytes_per_element × batch ÷ 1e9
total_GB    ≈ weights_GB + kv_cache_GB + 1.5

The leading 2 counts both keys and values. The bytes_per_element term is 2 for FP16/BF16 and 1 for FP8 or INT8.

Quantization: Weights at Lower Precision

Quantization stores each weight in fewer bits. A 7B model drops from 28 GB in FP32 to 14 GB in FP16, and to roughly 4–5 GB at 4-bit.

GGUF K-Quant Levels

GGUF is the file format behind llama.cpp, Ollama, and LM Studio. K-quants group weights into 256-weight super-blocks with hierarchical scales. They also keep sensitive tensors at higher precision.

Quality holds up to about 4 bits per weight. Below that, perplexity rises sharply.

Bigger Model at Lower Precision Usually Wins

On fixed VRAM, a larger model at Q4_K_M typically beats a smaller model at Q8_0. A 70B at Q4_K_M outperforms an 8B at Q8_0 on most benchmarks. Spend memory on parameters first, then on precision.

GGUF vs. GPTQ vs. AWQ vs. FP8

The KV Cache: The Hidden Memory Cost

Each generated token stores a key and a value vector in every layer. That cache stops the model from recomputing attention over the whole history. It trades memory for speed.

Grouped-Query Attention (GQA) keeps this manageable. Llama 3 models use 8 KV heads against 64 query heads on the 70B. That is an 8× reduction versus classic multi-head attention.

Worked KV Cache Numbers (FP16, Batch 1)

Quantize the cache to FP8 or Q8_0 and every figure halves. Q4 cache types cut it by roughly 4× at some quality cost.

Concurrency multiplies the cache. Eight users at 4K context on a 70B model need about 10 GB of KV cache on top of the weights.

Calculate It Yourself

Paste this into any Python 3 environment:

def vram_gb(params_b, bpw, layers, kv_heads, head_dim,
            ctx, kv_bytes=2, batch=1, overhead_gb=1.5):
    weights = params_b * bpw / 8                                   # GB
    kv = 2 * layers * kv_heads * head_dim * ctx * kv_bytes * batch / 1e9
    return round(weights + kv + overhead_gb, 1)

# Llama 3.1 8B, Q4_K_M, 8K context
print(vram_gb(8.03, 4.89, 32, 8, 128, 8192))     # ~7.5 GB

# Llama 3.1 70B, Q4_K_M, 32K context, FP16 KV
print(vram_gb(70.6, 4.89, 80, 8, 128, 32768))    # ~55.4 GB

# Same 70B, FP8 KV cache
print(vram_gb(70.6, 4.89, 80, 8, 128, 32768, kv_bytes=1))  # ~50.0 GB

Pull layers, kv_heads, and head_dim from the model’s config.json on Hugging Face (num_hidden_layers, num_key_value_heads, head_dim or hidden_size ÷ num_attention_heads).

Cutting Memory in Practice

llama.cpp with a Quantized KV Cache

llama-server \
  -m Llama-3.1-8B-Instruct-Q4_K_M.gguf \
  -c 32768 \
  -ngl 99 \
  -fa on \
  --cache-type-k q8_0 --cache-type-v q8_0

• -c 32768 sets the context window.
• -ngl 99 offloads all layers to the GPU. Lower it to spill layers to system RAM.
• -fa on enables flash attention, which quantized V-cache requires.
• --cache-type-k/v q8_0 halves cache memory with minimal quality loss.

vLLM with FP8 KV Cache

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 32768 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.90

• --max-model-len caps the cache pool size. Set it to what you need, not the model maximum.
• --gpu-memory-utilization 0.90 lets vLLM claim 90% of VRAM for weights plus cache.
• --kv-cache-dtype fp8 halves cache memory versus BF16.

GPU Tiers for Local Inference

Prices below are rough 2026 street ranges and vary by market.

Bandwidth sets decode speed. VRAM sets what fits. Buy VRAM first, then bandwidth.

Apple Silicon changes the equation. CPU and GPU share one memory pool, so a 64 GB Mac can hold a 70B Q4 model that no single consumer GPU can. You trade tokens per second for capacity.

Fine-Tuning Memory: Where the Numbers Jump

Training stores weights, gradients, optimizer states, and activations. Full fine-tuning with mixed-precision Adam costs about 16 bytes per parameter before activations.

Estimates vary with sequence length, batch size, and rank. Treat them as planning ranges.

QLoRA freezes a 4-bit base model and trains small BF16 adapters. Optimizer memory shrinks because only the adapters, about 1% of parameters, carry Adam states.

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",          # NormalFloat4
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,     # quantize the scales too
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B", quantization_config=bnb, device_map="auto"
)
model = prepare_model_for_kbit_training(model)

lora = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora)

Pair this with gradient checkpointing and a paged optimizer such as paged_adamw_8bit to cut peak memory further. Unsloth and Axolotl wrap the same ideas with extra savings.

Real-World Sizing Scenarios

Scenario 1: Private coding assistant on one 24 GB card.
Target a 32B coder model at Q4_K_M. Weights take about 20 GB. At 8K context the cache adds about 2 GB, which leaves almost no margin. Use --cache-type-k q8_0 --cache-type-v q8_0 and cap context at 16K. If you need 32K+, move to a 32 GB card.

Scenario 2: Team RAG endpoint on one 80 GB GPU.
Serve a 70B model in AWQ or GPTQ 4-bit. Weights take about 40 GB. That leaves roughly 30 GB for cache after overhead. At 4K tokens per request in FP16, each request costs about 1.3 GB, so you can batch around 20 concurrent users. Use FP8 KV cache to double that.

Scenario 3: Laptop prototyping.
A 7–8B model at Q4_K_M needs about 7.5 GB with 8K context. It runs on a 16 GB laptop through CPU/GPU offload, or fully on an 8 GB GPU with a shorter window.

Scenario 4: Weekend fine-tune on a budget.
QLoRA on a 7–8B model fits a single 24 GB card with room for batch size 4 at 512 tokens. Rent an 80 GB GPU only when you step up to 70B.

Sizing Checklist

1. Pick the smallest model that passes your evaluation set.
2. Start at Q4_K_M and move up only if quality tests fail.
3. Set your real context length, not the advertised maximum.
4. Add KV cache for your batch size. Quantize it if you are short.
5. Add 1.5 GB overhead, then keep 10% headroom.
6. For fine-tuning, assume QLoRA unless you have a measured reason not to.

FAQ

How much VRAM do I need to run a 7B or 8B model?

About 6–8 GB at Q4_K_M with 8K context. The weights take roughly 4.9 GB. FP16 needs about 16–18 GB.

Can I run a 70B model on a single 24 GB GPU?

Not fully on the GPU. A Q4_K_M 70B needs about 43 GB for weights alone. Options are two 24 GB cards, a Mac with 64 GB+ unified memory, or partial CPU offload at much lower speed.

Does a longer context window use more VRAM?

Yes. KV cache grows linearly with context length. A 70B GQA model uses about 0.31 MB per token in FP16, so 32K tokens cost roughly 10.7 GB. FP8 or Q8 cache types halve that.

What is the best quantization level for local LLMs?

Q4_K_M is the standard default. Choose Q5_K_M or Q6_K when you have spare VRAM and want higher fidelity. Avoid Q2 and Q3 unless memory is the only constraint.