Calculating LLM VRAM...
Choose a model, quantization, context length, and GPU to calculate local LLM VRAM.
How the LLM VRAM Requirements Calculator Works
The LLM VRAM Requirements Calculator estimates whether a local AI model fits on your GPU by separating memory into three parts: quantized model weights, KV cache, and runtime overhead. Model weights are based on parameter count and quantization. KV cache is based on model layers, hidden dimension, context length, batch size, and concurrent users. Runtime overhead covers tools such as Ollama, llama.cpp, vLLM, and Transformers.
This is more useful than a simple parameter-count chart because a model that fits at 4096 tokens can fail at 32768 tokens. Serving two users can also need far more VRAM than a single desktop chat session, even with the same model file.
How Much VRAM Do You Need to Run Llama 70B?
Llama 70B at Q4 quantization usually needs roughly 35 to 40GB of VRAM after model weights, KV cache, and runtime overhead are included. A 24GB GPU like the RTX 4090 can run some 70B models with CPU offload, but tokens per second drop because part of the model is moved through system RAM. A 32GB GPU is a better starting point, while 48GB gives more comfortable room for longer context.
The exact number depends on the model architecture and runtime. KV cache precision, flash attention support, tensor parallelism, and context length all change real memory usage. The calculator uses FP16 KV cache as a conservative default because many local inference stacks still reserve cache memory at higher precision than the quantized weights.
Why Quantization Changes VRAM Requirements
FP16 stores each model parameter in 2 bytes. Q4 stores each parameter at about 0.5 bytes before metadata, which is why Q4 is the standard local LLM choice. Q5 and Q6 can preserve more quality, but they increase VRAM requirements. Q2 and Q3 help a large model fit on a smaller GPU, but output quality can degrade noticeably, especially for coding, reasoning, and instruction following.
KV Cache and Context Length Explained
The KV cache stores attention keys and values for tokens already processed. It grows with context length and with the number of active requests. That is why 128K context can consume more VRAM than users expect, even if the model weights are small enough. If you are only chatting locally, batch size 1 and one concurrent user are realistic. If you are serving an API, increase batch size and concurrent users to see the real memory pressure.
Best GPUs for Local LLMs in 2026
For small 7B to 14B models, 12GB to 16GB GPUs are practical. For 30B to 34B models, 24GB is much more comfortable. For 70B models at Q4, 32GB is the practical consumer starting point and 48GB is a smoother workstation target. Frontier-class models such as Llama 405B and DeepSeek 671B belong in multi-GPU or hosted inference setups.
Common Model VRAM Reference
| Model | Parameters | Q4 weights | 4K context estimate | 32K context estimate | Practical GPU tier |
|---|---|---|---|---|---|
| Llama 3.1 8B | 8B | 3.7GB | 5-6GB | 8-10GB | 8GB to 12GB |
| Qwen3 32B | 32B | 15.3GB | 18-20GB | 24-30GB | 24GB to 32GB |
| Llama 3.1 70B | 70B | 32.9GB | 35-40GB | 45GB+ | 48GB or offload |
| Qwen3 235B-A22B | 235B | 109GB+ | 115GB+ | 130GB+ | Multi-GPU |
| Llama 3.1 405B | 405B | 188GB+ | 195GB+ | 220GB+ | Multi-GPU |
Frequently Asked Questions
Can an RTX 4090 run Llama 70B?
An RTX 4090 has 24GB of VRAM, so Llama 70B at Q4 normally does not fit fully in VRAM with a normal KV cache. You can use CPU offload in llama.cpp or Ollama, but generation will be slower than a full-GPU fit.
Is the RTX 5090 good for local LLMs?
Yes, the RTX 5090's 32GB VRAM makes it much better for local LLMs than 16GB GPUs. It is a strong fit for 30B-class models and can run some 70B Q4 workloads with careful context settings, but 48GB is still more comfortable for large long-context use.
What quantization should I choose for local LLMs?
Q4 is the normal starting point because it balances memory use and output quality. Use Q5 or Q6 when you have spare VRAM and care about quality; use Q3 or Q2 only when fitting the model matters more than quality.
Why does vLLM need more VRAM than llama.cpp?
vLLM is optimized for serving and throughput, so it reserves more memory for scheduling, cache management, and concurrent requests. llama.cpp is usually leaner for single-user desktop inference.
Does system RAM help if GPU VRAM is too small?
System RAM can help through CPU offload, but it is much slower than keeping the model on the GPU. Offload is useful for experimentation, not for low-latency chat or serving.
How much VRAM do I need for Qwen 32B?
Qwen 32B at Q4 usually lands around the high teens at short context and can move into the 24GB to 32GB range with long context. A 24GB GPU is a good minimum; 32GB is more comfortable.
Related Calculators
VRAM Requirement Calculator
Compare AI, gaming, and video VRAM needs.
GPU Price-to-Performance Calculator
Find the best value GPU for your budget.
PC Build Budget Planner
Plan a full local AI workstation build.
PSU Wattage Calculator
Size power delivery for high-VRAM GPUs.
CPU & GPU Cooling TDP Calculator
Estimate cooling requirements for heavy GPU loads.
GPU FPS Estimator by Game
Use the same GPU for gaming performance planning.