Which GPU for Which LLM in 2026: A Per-Model Hardware Guide

When you ask "which GPU do I need to run a specific LLM model locally," the answer starts with the model's quantized weight size plus its KV-cache overhead: aim for at least 20% VRAM headroom over the quantized file. As of 2026, a 12GB RTX 3060 comfortably handles 7B-13B models at q4-q5, stretches to 30B at q3 with reduced context, and cannot host a 70B without heavy CPU/system-RAM offload that collapses tok/s.

Why model-specific hardware matching beats generic "best GPU" advice

Every published "best GPU for LLMs" listicle picks a card first and then hopes the workload fits. In practice, the workload picks the card. A 12GB RTX 3060 is the perfect home for a q4 Mistral 7B or Qwen 14B running a coding sidecar; the same card wheezes and thrashes trying to hold a q4 Llama 70B whose weight file alone is 43GB. Meanwhile, an RTX 4090 with 24GB of VRAM is dramatic overkill for the 7B use case that dominates local-LLM adoption but genuinely necessary if you want fp16 for finetuning or long-context 32B inference.

This guide is the version we wished existed when we bought our first inference card: a per-model table, not a "top 5 GPUs" listicle. We benchmark the featured ZOTAC Gaming GeForce RTX 3060 12GB as the canonical entry point because it remains the cheapest way to get 12GB of GDDR6 on a modern architecture, then compare what each popular model class demands against it, an RTX 4090, and datacenter-class MI300X. If you already own an MSI GeForce RTX 3060 Ventus 2X 12G, the tables below tell you exactly which weights to download and which to skip. If you are shopping, they tell you when a $400 card is enough and when spending more actually pays back in tok/s.

Key Takeaways

• VRAM budget = quantized weight size + KV cache. Both scale with context length; long prompts silently double memory need.
• A 12GB RTX 3060 runs 7B-13B at q4-q5 with room to spare, 30B at aggressive q3 with short context, and simply cannot host 70B without punishing offload.
• Quantization from fp16 to q4 typically cuts VRAM by 3-4x with a small quality drop; q3 and q2 shave more but visibly degrade reasoning tasks.
• Renting cloud A100/H100 time beats owning past the 32B threshold if you use big models infrequently.
• CPU and system RAM matter only when you offload; a fast Ryzen 7 pairs well with the 3060 for partial-offload safety nets.

How much VRAM does each popular model class actually need?

The rough-cut math per the llama.cpp memory notes (source) is: VRAM ≈ quantized_weight_bytes + kv_cache_bytes + a few hundred MB for CUDA runtime and framework overhead. The KV cache term is 2 × num_layers × hidden_dim × context_length × 2 bytes (fp16 cache) — for a 7B model with 32 layers, 4096 hidden dim, and 4096 context, that is about 1GB.

That means a q4 Mistral 7B (~4.2GB weights) fits in 6GB total. A q4 Llama-3 8B (~5GB weights) fits in 7GB. A q4 Qwen 14B (~8.5GB weights) needs about 10GB, leaving thin margin on a 12GB card if you push context past 4k. A q4 Llama 70B (~43GB weights) needs 48GB before you count context, which is why you either buy a datacenter card or accept the offload tax.

Compatibility: model × GPU × max quant that fits

The table below reports the largest quant we can load with the model's default context window on each card, without CPU offload. "Yes*" means it loads but only with reduced context (2k instead of 8k+).

The "tight" cases assume 2k context. Push to 8k and you lose the fit. Also note the RTX 4090's stated 24GB is misleading for 70B: you can technically load q3 with 1k context, but the moment you extend beyond a few hundred tokens the KV cache pushes you back into offload.

Benchmark: tok/s per model on the featured RTX 3060 12GB

Numbers below are steady-state text generation on Windows 11 with the latest NVIDIA studio driver, running llama.cpp b3450 (May 2026) at temperature 0.7 and a 512-token output. Prompt is a 200-token instruction; no batch, single stream.

Rows marked * are already brushing the 12GB ceiling and will fail at 4k context. Below 25 tok/s the user experience for chat degrades noticeably; if you find yourself in that band regularly, either drop a quant tier or step up to a card with more VRAM.

Quantization matrix: what you actually give up

For a 7B and a 32B example, this is the honest tradeoff. Quality columns are our internal HumanEval + MMLU delta versus fp16 baseline, averaged across five prompts each.

Rule of thumb: stay at q4 or better for coding assistants, RAG, and any workflow where wrong answers cost money. Drop to q3 only when you would otherwise offload; drop to q2 only for creative-writing brainstorms where the quality hit is acceptable.

What can a 12GB card run today, and what is out of reach?

Comfortable, no compromise:

• Mistral 7B, Llama-3 8B, Phi-3 Mini/Small: q6-q8 with 8k+ context, 50+ tok/s
• Qwen2 14B, DeepSeek-Coder 16B: q4-q5, 30-40 tok/s, 4k context
• Coding assistants, RAG pipelines, JSON extraction — the sweet spot

Possible with tradeoffs:

• Codestral 22B, Yi-1.5 34B: q3-q4 at 2k context; usable but not comfortable
• GLM-5.2 32B: q3 at 2k context; see our full GLM-5.2 review on the RTX 3060 for the numbers
• Mixtral 8x7B (46B): only with heavy CPU offload; expect 3-6 tok/s

Out of reach without offload:

• Llama 70B / DeepSeek 67B: need 40GB+ VRAM even at q4
• Any 100B+ dense model: rent, don't buy

The rtx-3060-12gb-70b-llm-offload-quant-reality-2026 article walks through the offload math in detail if you insist on trying.

Context length impact on the per-model VRAM budget

Context length is the silent VRAM killer. For a 7B model, going from 2k to 8k context roughly triples KV-cache size (from ~400MB to ~1.6GB); for a 32B model with wider layers, the same jump easily costs 3-4GB. Per techpowerup's RTX 3060 spec sheet, you have 12,288MB of GDDR6 at 360GB/s bandwidth to work with — every gigabyte the cache eats is a gigabyte you cannot spend on weights.

Practical implications: if you routinely feed 8k+ prompts (long docs, chat with big history), pick your quant assuming 20-30% extra VRAM for the cache. If you are strict about short prompts, you can push a tier higher on the quant ladder.

Perf-per-dollar: owned RTX 3060 vs renting bigger cards per model

Assume $340 street price for a featured 3060 12GB, 3-year ownership, $0.12/kWh, 8 hours/day active. Amortized total cost per hour of active use lands near $0.09 including power. Cloud A100 80GB PCIe rents for $1.19/hr on major providers; H100 80GB is $2.49-3.10/hr.

• Mistral 7B daily use: owning wins by 12x. Never rent.
• Qwen 14B daily use: owning still wins by 10x.
• Llama 70B twice a week for 2 hours: renting wins outright. A100 hourly × 16 hrs/month ≈ $19/month; buying an RTX A6000 (48GB) at ~$4,500 amortizes to $125/month.
• Continuous Llama 70B serving: buying dual A6000s or a used A100 40GB starts winning after ~14 months.

If you run 7B/13B daily and occasionally reach for a 70B, own the RTX 3060 and rent the big one when needed. Do not buy a $4,500 card to run a 7B model that already fits in 8GB.

Common pitfalls and gotchas

Even with the right card, we see the same five foot-guns take out first-time local-LLM setups:

1. Forgetting the KV cache in your VRAM math. People download a q4 Qwen 14B (~8.5GB), see it fit on their 12GB card, then blow up when they try a 16k prompt. Always add ~25% headroom before you commit to a quant.
2. Running fp16 KV cache when q8 cache would do. Both llama.cpp and vLLM support quantized KV cache; enabling it can nearly halve context-cache VRAM at a tiny quality cost. On a 12GB card that is often the difference between fitting your model at 8k and not.
3. Windows resource-monitor phantoms. Task Manager's "GPU memory" widget lags. Trust nvidia-smi in a terminal — the difference between "the model actually loaded" and "the model is thrashing" is one glance at the right dashboard.
4. Ollama defaulting to short context. Ollama silently caps context at 2048 unless you set num_ctx in the model's Modelfile. Users then blame the model for "forgetting" earlier turns. Bump num_ctx and rebuild the pull to match the model's actual max.
5. Skimping on system RAM. Once you offload even one layer, system RAM speed becomes the tok/s ceiling. 16GB DDR4-2133 will halve your offload throughput vs 32GB DDR4-3600. If a 3060 is your GPU floor, dual-channel DDR4-3200 or better is the RAM floor.

When NOT to buy an RTX 3060 12GB

Skip the 3060 if any of these hold:

• You already have a 30-series 8GB card (3070/3070 Ti). The 3060 12GB is a small upgrade in raw tok/s and only wins on VRAM headroom. If you can find a used RTX 3090 24GB near the same price you would spend on new 3060, buy that instead — 24GB unlocks a completely different model tier.
• Your only use case is a coding assistant for Python/TypeScript with Codestral 7B or DeepSeek-Coder 7B. Any 8GB card handles those.
• You need fp16 for fine-tuning or LoRA training on 13B+ models. The 3060's 12GB is not enough; jump to a 4090 or 3090.
• You want to serve concurrent users. Batch decoding on a 3060 is severely bandwidth-limited; a single fast card cannot substitute for a proper deployment tier here.

Bottom line: the right starter GPU for most local-LLM newcomers

The RTX 3060 12GB is the correct answer for at least 80% of first-time local-LLM buyers. Its 12GB VRAM ceiling is the difference between "can I run a 14B coding assistant" and "no, I'm stuck with 7B forever" — the RTX 3060 8GB variant traps you below Qwen2 14B and every capable coder model of 2026.

Pair it with a fast NVMe like the Western Digital 1TB WD Blue SN550 so weight loads take seconds, not minutes, and a modern host CPU so the moment you inevitably experiment with partial offload you do not thrash on the memory bus. Do not upgrade to a 4090 just because you can — first prove you have a workload that actually needs its extra VRAM.

Related guides

• Which LLMs Fit a 12GB RTX 3060? Per-Model VRAM Cheat Sheet (2026)
• llama.cpp vs Ollama vs vLLM on a 12GB RTX 3060 (2026)
• Run Local LLMs on a Ryzen 5 5600G With No GPU (2026)
• GLM-5.2 Review: Can the Top Open-Weights LLM Run Locally?

Sources

• NVIDIA — GeForce RTX 3060 product page
• TechPowerUp — RTX 3060 spec sheet
• llama.cpp — VRAM math + quantization documentation