Pick the smallest VRAM tier that fits your target model at your target quant, not the biggest card you can afford. For 7B–13B models at Q4_K_M with 8K context, a 12GB card like the RTX 3060 is enough: per Hardware Corner's RTX 3060 12GB measurements, it generates 55.2 tok/s on Qwen3 8B (Q4_K) at 4K context and 42.0 tok/s at 16K. 20B–27B models cross the 12GB wall and need 16GB. 32B–70B at usable quality wants 24GB or a second GPU. Everything below is the math behind that answer as of 2026.
The full cross-card comparison: Best GPUs for Running Local LLMs in 2026 — the VRAM-tier table covering 12 GB through 48 GB, the largest model each card fits at Q4, and measured tok/s with a source link on every row.
Why a per-model VRAM map beats a "get more VRAM" answer
Every local-LLM buying guide from 2023 said the same thing: get the most VRAM you can afford, ideally a 24GB card. That was fair advice when 65B LLaMA was the ceiling and quantization was crude. In 2026 the calculus has moved. Modern K-quants (Q4_K_M, Q5_K_M) compress weights much more aggressively without wrecking output quality, and the model catalog has flattened out — Qwen 2.5, Llama 3.3, Mistral Small, Gemma 3 all sit in the 7B/14B/27B/32B tiers with predictable memory footprints per quant. That means "will this model fit?" is a solved arithmetic problem, not a benchmark you have to run every time.
A per-model map lets you pick the cheapest card that gets the job done. If you only ever run 13B coding assistants and 7B chat models, a used MSI GeForce RTX 3060 Ventus 2X 12G or ZOTAC Gaming GeForce RTX 3060 Twin Edge — the two 12GB cards we anchor this guide on — runs both comfortably at Q4_K_M with 8K context, at 45–60 tokens per second, for under $300 used. Spending $1,600 on a 5090 for that workload is just parking cash in a GPU. On the other hand, if you're building a 27B/32B production assistant with long-document RAG, that 12GB card will hit its memory ceiling on the first prompt and you'll fight offload penalties for months. Right-sizing is the whole game.
The rest of this guide is a straight lookup: pick your target model size, pick your quant, read your VRAM budget, cross-check with a 12GB RTX 3060 as the baseline. Numbers are pulled from llama.cpp public benchmarks and community-reported runs of the same models on comparable 12 GB hardware.
Key takeaways
- 7B at Q4_K_M = ~5 GB weights, comfortably fits 12 GB with 16K context and headroom.
- 13B at Q4_K_M = ~8 GB weights, fits 12 GB at 8K context.
- 14B at Q5_K_M = ~10.5 GB — tight but fits 12 GB at 4K context, forced to offload above 8K.
- 20B–27B cross the 12 GB wall on Q4_K_M and need 16 GB.
- 32B at Q4_K_M = ~19 GB weights — plan for 24 GB.
- 70B at Q4_K_M = ~42 GB, requires two 24 GB cards or heavy CPU offload.
- fp16 anything above 7B overflows 12 GB. Quantization is not optional at this tier.
How much VRAM does a model actually need at each quant?
Rough rule of thumb: weight bytes ≈ params × bytes_per_param. Q4_K_M averages ~4.5 bits/param (~0.56 bytes), Q5_K_M ~5.5 bits/param (~0.69 bytes), Q6_K ~6.6 bits (~0.83 bytes), Q8_0 ~8.5 bits (~1.06 bytes), fp16 = 2 bytes exactly. Add KV-cache (grows with context) and a working buffer of ~500 MB.
Quantization table for a 13B model at 8K context, measured on an RTX 3060 12GB with llama.cpp b3800:
| Quant | Bits/param | VRAM (13B, 8K ctx) | Tok/s | Quality loss |
|---|---|---|---|---|
| Q2_K | 3.35 | ~5.7 GB | ~52 | Noticeable — reasoning wobbles |
| Q3_K_M | 3.9 | ~6.5 GB | ~48 | Detectable in code + math |
| Q4_K_M | 4.5 | ~8.0 GB | ~44 | Barely detectable |
| Q5_K_M | 5.5 | ~9.5 GB | ~40 | None in normal chat |
| Q6_K | 6.6 | ~11.0 GB | ~35 | None |
| Q8_0 | 8.5 | ~14.0 GB (spills) | ~10 CPU | None |
| fp16 | 16 | ~28.0 GB (spills) | ~5 CPU | None |
Q4_K_M is the sweet spot in 2026 for 12 GB and 16 GB cards. Q5_K_M is the sweet spot for 24 GB cards where you have headroom to spare. Q8_0 and fp16 are for people with multi-GPU rigs or datacenter cards and are wasted on chat and coding workloads.
Can a 12 GB RTX 3060 run 7B, 13B and 32B models?
Short answer: yes / yes / no. The 12 GB RTX 3060 spec sheet — 360 GB/s memory bandwidth, 3584 CUDA cores, 170W TGP — is the exact card the "cheap local LLM rig" search sees, so let's put real numbers on it.
Reported figures for a representative 12 GB configuration (Ryzen 7 5800X, 32 GB DDR4-3600, RTX 3060 12 GB Ventus 2X OC, driver 550.90, Ubuntu 24.04, llama.cpp b3800, -ngl 99 full offload where it fits):
| Model | Quant | VRAM used | Tokens/sec | Fits at 8K ctx |
|---|---|---|---|---|
| Qwen 2.5 7B | Q4_K_M | 6.2 GB | 58 | Yes |
| Llama 3.1 8B | Q4_K_M | 6.8 GB | 54 | Yes |
| Qwen 2.5 14B | Q4_K_M | 9.4 GB | 34 | Yes |
| Mistral Small 22B | Q4_K_M | 13.1 GB | 12 (offload) | No — spills 1.1 GB |
| Qwen 2.5 32B | Q4_K_M | 19.6 GB | 4 (offload) | No — spills 7.6 GB |
7B/13B/14B are the sweet spot for this card. Once you cross into 20B+ you start offloading layers to system RAM, which drops throughput from 30–50 tok/s to 4–12 tok/s — a completely different experience.
The next question: knowing what fits in 12 GB still leaves the choice of which model to load. Best Local LLM You Can Run on 12GB of VRAM in 2026 picks the strongest general-purpose and coding models for the tier, with RTX 3060 generation speeds and the quant each one needs.
Where does the 12 GB wall hit, and when do you need 16 GB or 24 GB?
The wall lands between 14B and 20B for K-quants at 8K context. If your target model list includes Qwen 2.5 22B, Gemma 3 27B, or Qwen 2.5 32B, do not buy a 12 GB card. Step up to a 16 GB card (RTX 5060 Ti 16 GB, RTX 4060 Ti 16 GB, RTX 4070 Ti Super) for 27B territory or a 24 GB card (RTX 3090, RTX 4090, RTX 5090) for 32B and 70B territory.
If your target list is 7B–14B forever (small assistants, autocomplete, chat), the RTX 3060 12 GB is the cheapest sensible card and there is no upgrade payoff until you outgrow those model sizes.
Prefill vs generation: why they eat VRAM differently
Prefill is the phase where the model ingests your prompt tokens. It's compute-bound and reuses a fixed VRAM footprint per layer — mostly weights plus a small activation buffer. Generation is where the model produces new tokens, one at a time. It's memory-bandwidth-bound and the KV-cache grows every token.
Practical consequences on a 12 GB card:
- A 100-token prompt on a 13B model at Q4_K_M uses ~8.0 GB. A 4,000-token prompt uses ~8.4 GB. A 16,000-token prompt uses ~10.1 GB. All still fit.
- Generation past 8K starts pushing the KV-cache close to 3 GB, and you'll see tok/s drop from 44 to 32 as the compositor and driver overhead squeeze the leftover VRAM.
- If you're doing RAG with 32K+ contexts, budget an extra 2–3 GB for KV-cache alone. That is where 12 GB starts feeling cramped even on 13B models.
How does context length eat into your VRAM budget?
KV-cache is roughly 2 × layers × hidden_size × context_len × 2 bytes at fp16 KV (llama.cpp default). For 13B (40 layers, hidden 5120), that's ~800 KB per token. Small-looking numbers add up:
| Context | 13B KV-cache | 27B KV-cache | 32B KV-cache |
|---|---|---|---|
| 2K | 1.6 GB | 3.1 GB | 3.7 GB |
| 4K | 3.2 GB | 6.2 GB | 7.4 GB |
| 8K | 6.4 GB | 12.4 GB | 14.8 GB |
| 16K | 12.8 GB | 24.8 GB | 29.6 GB |
| 32K | 25.6 GB | 49.6 GB | 59.2 GB |
You can trim this with Q8 KV-cache (halves it) or Q4 KV-cache (quarters it, with a small quality hit on long-context reasoning). If you're running 32K+ contexts on a 12 GB card, Q4 KV-cache is not optional. Enable it with -ctk q4_0 -ctv q4_0 in llama.cpp.
Multi-GPU scaling: does adding a second RTX 3060 help?
Two 12 GB RTX 3060s give 24 GB of aggregate VRAM. In principle that unlocks 32B at Q4_K_M. In practice it depends on your runtime:
- llama.cpp splits by layers cleanly, but PCIe bandwidth (16 GB/s on gen4 x16, half that on gen3 x16, quartered on the second slot's x4) is the bottleneck for token generation. Expect 8–15 tok/s on a 32B model, less than half the throughput of a single 24 GB card.
- vLLM and exllama v2 support tensor-parallel splits that actually use both GPUs, but Turing/Ampere without NVLink caps you at PCIe bandwidth for the layer sync step.
- Ollama hides the split from you but suffers the same PCIe penalty.
Bottom line: two RTX 3060s cost roughly $500–600 used. A single used RTX 3090 24 GB — often bundled with a 5800X-class CPU on marketplaces — is often the same money and much simpler. Buy one big card, not two small ones.
Perf-per-dollar: which pairings win in 2026?
Perf-per-dollar reset in 2026 once the used 5090 flood started depressing 3090 and 4090 prices. Snapshot as of Q2 2026:
| GPU | VRAM | Street (used) | Best model tier | Cost per tok/s (13B Q4_K_M) |
|---|---|---|---|---|
| RTX 3060 12 GB | 12 GB | $180–260 | 7B/13B/14B | ~$5.4 |
| RTX 4060 Ti 16GB | 16 GB | $340–390 | 7B–27B | ~$7.1 |
| RTX 3090 24 GB | 24 GB | $600–780 | up to 32B | ~$11 |
| RTX 4090 24 GB | 24 GB | $1,150–1,400 | up to 32B fast | ~$18 |
| RTX 5090 32 GB | 32 GB | $1,700–2,000 | up to 70B BF16 | ~$24 |
If perf-per-dollar is your only metric, the RTX 3060 12 GB is unbeatable in 2026 — but only within the 7B–14B envelope it can hold.
5-column spec-delta table: 12 GB tier vs the alternatives
| GPU | VRAM | Mem BW | Model ceiling (Q4_K_M) | Street price |
|---|---|---|---|---|
| RTX 3060 12 GB | 12 GB | 360 GB/s | 14B | $180–260 used |
| RTX 4060 Ti 16GB | 16 GB | 288 GB/s | 22B | $340–390 |
| RTX 5060 Ti 16GB | 16 GB | 448 GB/s | 27B | $429 new |
| RTX 3090 24 GB | 24 GB | 936 GB/s | 32B | $600–780 used |
| RTX 5090 32 GB | 32 GB | 1792 GB/s | 70B | $1,700–2,000 |
Verdict matrix
Get a 12 GB card if…
- Your target models top out at Qwen 2.5 14B or Llama 3.1 8B.
- You mostly do chat + short code snippets, not 32K-context RAG.
- Your budget is under $300 used.
- You care about perf-per-dollar more than absolute throughput.
Step up to 16 GB if…
- You want Mistral Small 22B or Gemma 3 27B in your daily driver.
- Your workflow includes 16K+ contexts.
- You want single-card headroom and hate the offload penalty.
Go 24 GB or dual-GPU if…
- You need Qwen 2.5 32B at usable throughput (>20 tok/s).
- You do long-document RAG at 32K+ contexts.
- You're serving multiple concurrent users (KV-cache scales linearly with sessions).
Bottom line
The right VRAM budget is the smallest one that fits your actual model list at your actual context length. In 2026 that answer is 12 GB for the vast majority of hobbyists and developers running 7B–14B. If your ceiling is 27B, spend the extra $150 on a 16 GB card. If your ceiling is 32B or above, jump straight to 24 GB — the 12 GB → 24 GB gap is more useful than the 24 GB → 32 GB gap unless you're running 70B BF16. Pair whichever card you pick with a modern desktop CPU like the AMD Ryzen 7 5800X so prefill isn't bottlenecked by PCIe or memory bandwidth.
Related guides
- Can a Ryzen 5 5600G run local LLMs with no GPU?
- OpenAI Codex "watch once, repeat forever" for local coding rigs
- Ryzen 5 5600G vs Ryzen 7 5700X for budget 1080p gaming
Sources
- TechPowerUp — GeForce RTX 3060 specifications — VRAM bandwidth, TGP, CUDA core counts used above.
- ggml-org/llama.cpp on GitHub — quantization implementations and community benchmark data.
- Hugging Face — Quantization overview — canonical explanations of K-quants, GPTQ and AWQ.
