Local LLMs on 8 GiB of VRAM
Start small, keep context under control, and see why a weight file fitting is only the first check.
Open guide →MEMORY → MODELS
01 / HARDWARE BUDGET 02 / HARDWARE BUDGET 03 / HARDWARE BUDGET
Which model fits your GPU budget?
Enter your GPU memory and the space used by other applications. Compare all six checkpoints below; select a checkpoint to inspect its memory breakdown.
Estimated GPU allocation
GiB
Qwen3-8B
Room in estimate
- Model weights
- 4.58 GiB
- KV cache
- 1.13 GiB
- Runtime allowance
- 1.50 GiB
- Usable GPU budget
- 11.00 GiB
Memory fit is an estimate, not a speed or compatibility guarantee. All sizes use GiB (2³⁰ bytes). A green status leaves at least 10% of usable memory free.
Qwen3-8B memory guide →
Publisher architecture configuration ↗
Compare model memory
All rows use your selected precision, cache and concurrency. Context is capped to each model's tool limit. The optional weight override affects only the selected-model breakdown above; comparison rows use estimated weights.
| Checkpoint | Context / sequence | Estimated allocation | Budget check | Configuration |
|---|---|---|---|---|
| Qwen2.5-7B-Instruct | 8,192 | 6.19 GiB | Room in estimate | Adjust |
| Qwen2.5-14B-Instruct | 8,192 | 11.21 GiB | Exceeds budget | Adjust |
| Qwen2.5-32B-Instruct | 8,192 | 21.66 GiB | Exceeds budget | Adjust |
| Qwen3-8B | 8,192 | 7.21 GiB | Room in estimate | Adjust |
| Qwen3-14B | 8,192 | 11.02 GiB | Exceeds budget | Adjust |
| DeepSeek-R1-Distill-Qwen-32B | 8,192 | 21.66 GiB | Exceeds budget | Adjust |
Read your hardware budget guide
Local LLMs on 12 GiB of VRAM
Compare 7B, 8B and 14B models, with room for the display, runtime and conversation cache.
Open guide →Local LLMs on 24 GiB of VRAM
Explore 14B and 32B trade-offs, including context growth and the limits of one memory pool.
Open guide →How the estimate works
Start with the stored weights, add the conversation cache, then allow space for the runtime. The display and other applications consume a separate reserve. The result assumes the whole model and its cache stay on one GPU.
Weights (GiB) ≈ parameters × assumed bits per weight ÷ 8 ÷ 2³⁰
KV cache (GiB) = 2 × layers × KV heads × head dimension × bytes per cached value × context tokens × parallel sequences ÷ 2³⁰
Total allocation = weights + KV cache + runtime allowance
Weight presets use 4.8, 5.7, 6.6 and 8.5 effective bits per parameter for Q4_K_M, Q5_K_M, Q6_K and Q8_0. These are planning assumptions, not exact file sizes. Tensor mixtures, unquantized tensors and container metadata vary by conversion. Use the advanced weight override when you have measured the checkpoint allocation.
The cache calculation models a full dense GQA cache at the selected capacity. FP16 uses 2 bytes per value. The Q8_0 and Q4_0 estimates include block overhead (34 and 18 bytes per 32 values). Backend padding and allocation strategies can increase usage. Your runtime must support the chosen cache type and kernels; smaller caches can affect output quality.
What the fit labels mean
Usable memory is GPU capacity minus the display/other-app reserve. “Room in estimate” leaves at least 10% of that usable budget unused. “Tight budget” means the estimate fits but has less headroom. “Exceeds budget” means this full-GPU configuration exceeds the selected allocation. Partial CPU offloading may still work, with different performance.
Check it on your machine
Use the exact quantized file, runtime version and context settings you plan to deploy. Load one sequence first, inspect the runtime's memory report and GPU monitor, and then test a conversation near the intended context length. Record peak use during both prompt processing and generation before increasing concurrency.
In Ollama, ollama ps reports the processor split and allocated context. A model partially offloaded to CPU can run even when it exceeds the GPU budget. Do not interpret successful loading alone as proof of full GPU residency.
Where this tool stops
This is inference planning for the six named dense text checkpoints. It does not estimate training, fine-tuning, image generation, MLA, sliding-window caches, prefix sharing, multi-GPU placement, or Apple unified-memory limits. No tokens-per-second claims are made. Context is bounded to the presets listed in the source notes; fitting memory does not validate model quality at that length.
Sources and review date
Publisher specifications reviewed October 31, 2026. Weight sizes and memory tables are InfoIsland calculations using the assumptions above; they are not measured benchmark results.
- Qwen2.5-7B-Instruct model card · architecture configuration
- Qwen2.5-14B-Instruct model card · architecture configuration
- Qwen2.5-32B-Instruct model card · architecture configuration
- Qwen3-8B model card · architecture configuration
- Qwen3-14B model card · architecture configuration
- DeepSeek-R1-Distill-Qwen-32B model card · architecture configuration
- llama.cpp server documentation: context, parallelism and cache types
- Ollama: context length and offloading