Per-Model Hardware Picker: Matching 7B-70B LLMs to Your GPU

Which GPU do you need to run a specific open LLM locally?

You need enough VRAM to hold the model's weights at your chosen quantization plus the KV cache for your target context length plus roughly 10% activation overhead. As a fast cheat sheet for 2026: a 7B model fits on any 8 GB card at q4, a 13B fits on a 12 GB RTX 3060 12GB at q4_K_M with 8K context, a 32B fits on a 24 GB card at q4_K_M or on the 3060 at q3 with aggressive offload, and a 70B needs 48 GB of VRAM or multi-GPU. Pair the card with at least an AMD Ryzen 5 5600G or Ryzen 7 5800X so CPU offload is fast when you spill.

Why model-specific hardware matching beats generic "best GPU" advice

The "best GPU for local LLMs" lists you find on most sites are written for someone who does not exist. They average over a hypothetical workload, hand-wave VRAM math, and pick whichever card the affiliate program pays the most on this week. The result is a recommendation that fits no real reader, because the question is not "which GPU is fastest" — it is "which GPU lets me run the model I actually want to run."

The right framing is to start from the model and work backward. If you know you want a 14B reasoning model at q4_K_M with an 8K context, the GPU choice collapses to a small set of options: the RTX 3060 12GB, the MSI RTX 3060 Ventus 2X 12G, the Gigabyte RTX 3060 Gaming OC 12G, or a step up to a 16 GB card if you have the budget. If you know you want a 32B model at q4_K_M, the 3060 is not enough and you need a 24 GB card. If you want 70B at q4, you are in multi-GPU or pro-card territory.

This piece walks the entire ladder from 7B to 70B and pins each rung to a concrete GPU recommendation and the exact quantization that makes it work. The numbers come from public llama.cpp community measurements, Hugging Face model cards, and TechPowerUp specs — no first-party benchmarking, just arithmetic and synthesis.

Key takeaways

• VRAM is the binding constraint for local LLM inference, not raw FLOPS.
• A 7B model at q4_K_M fits anywhere; the question is how much context you want.
• A 13B-14B model at q4_K_M is the sweet spot for a 12 GB card with 8K context.
• A 32B model at q4_K_M needs a 24 GB card; the 3060 can run it at q3 with offload pain.
• A 70B model at q4 wants 48 GB of VRAM and is the practical floor for multi-GPU or pro cards.
• Quantization is the lever — moving from fp16 to q4_K_M shrinks the model 4× with small quality loss.
• Context length eats VRAM via the KV cache; budget for it before you commit to a model.
• Multi-GPU helps for larger models but adds latency from cross-card communication.

How do you calculate VRAM for a given model size and quantization?

The arithmetic is straightforward. Start with the number of parameters, multiply by bytes-per-weight to get the weight footprint, add the KV cache for your context length, and add roughly 10-15% activation overhead. Bytes-per-weight depends on the quantization scheme.

KV cache for a typical decoder-only model is 2 × layers × hidden_size × context_length × dtype_bytes per request. On a 14B model that works out to roughly 200 MB per 1K of context with fp16 KV. Halve that with int8 KV cache (supported by llama.cpp). For 8K context, budget ~1.5 GB of KV. For 32K context, budget ~6 GB, which is enormous compared to the weight savings of more aggressive quantization. KV is the silent VRAM tax that surprises most people the first time.

What can an RTX 3060 12GB actually run, from 7B to 32B?

The RTX 3060 12GB is the most VRAM-per-dollar card in the consumer market in 2026 and the single most common choice for local-LLM builders. Here is what it can actually do, assuming a real driver, a real KV cache, and 800 MB of headroom for the display compositor.

• 7B at q4_K_M with 16K context — runs entirely on GPU, generation around 50 tok/s per public community measurements.
• 7B at q8_0 with 8K context — runs entirely on GPU, generation around 40 tok/s, near-fp16 quality.
• 13B-14B at q4_K_M with 8K context — fits with ~1 GB to spare, generation around 22 tok/s.
• 13B-14B at q5_K_M with 4K context — fits, generation around 20 tok/s, very small quality gain.
• 32B at q3_K_M with 4K context — fits with offloading a handful of layers, generation around 8 tok/s.
• 32B at q4_K_M with 4K context — does not fit fully; expect 20-30% of layers on CPU and a generation rate of 3-5 tok/s.
• 70B at any usable quant — does not fit; do not try.

For most readers, the 14B class at q4_K_M is the right target. It runs comfortably, leaves context headroom, and the quality is close enough to the fp16 ceiling that a typical chat user will not notice. The Gigabyte RTX 3060 Gaming OC 12G and MSI RTX 3060 Ventus 2X 12G are interchangeable with the Zotac for this workload — pick whichever is cheapest in stock.

When do you need to step up to 24 GB or go multi-GPU?

You step up to 24 GB when your target model is in the 27B to 32B range and you want q4_K_M with comfortable context, when you need fp16 KV cache for production-grade accuracy, when you want to run two models concurrently (chat + embeddings, for example), or when you intend to fine-tune. The realistic 24 GB consumer choices in 2026 are the RTX 3090 and the RTX 4090. A used 3090 is the price-performance king for VRAM capacity; the 4090 is faster on prefill and worth the premium if you batch.

You go multi-GPU when no single consumer card fits your model. Two RTX 3060 12GB cards give you 24 GB of total VRAM and llama.cpp can split layers across them, but the performance penalty for cross-card communication on a desktop PCIe topology is noticeable — expect roughly 70% of single-card speed, not 100%. Two 3090s give you 48 GB and run a 70B at q4_K_M comfortably; that is the realistic floor for serious local 70B work without going to data-center hardware. The AMD Ryzen 7 5800X on a B550 motherboard supports the PCIe topology for two cards and is the budget-friendly platform for a dual-GPU rig.

Spec table: 7B-70B VRAM at q4 / q5 / q8 / fp16

Add KV cache on top. The 12 GB line is roughly the upper bound on the RTX 3060 12GB; the 24 GB line is the 3090 / 4090; the 48 GB line is dual-3090 territory; the 80 GB line is H100-class hardware.

Benchmark table: tok/s across model sizes on RTX 3060 12GB vs CPU

Generation throughput on the RTX 3060 12GB at q4_K_M, all layers offloaded, single-user, llama.cpp with reasonable default flags. CPU numbers from a Ryzen 7 5800X with DDR4-3200. Both columns are public community measurements gathered from the llama.cpp issue tracker, normalized to a single-prompt steady-state.

The GPU speedup shrinks as the model gets bigger because more layers are spilling to CPU, where the Ryzen 7 5800X becomes the bottleneck. For everything that fits entirely on GPU, the speedup is consistent and worth the build cost. For models that do not fit, you are CPU-bound and the GPU is barely helping.

Quantization matrix: q2/q3/q4/q5/q6/q8/fp16 with VRAM, tok/s, quality loss

q4_K_M is the practical default for 12 GB cards. Going higher buys diminishing returns; going lower starts to bite on reasoning. The community has converged on this for good reasons.

Prefill vs generation and how context length eats your VRAM budget

Prefill is compute-bound. The RTX 3060 12GB does roughly 13 TFLOPS in fp16 per TechPowerUp's RTX 3060 spec page. A 14B model prefilling 8K of tokens runs in 3-5 seconds; a 70B equivalent would take a minute. The card's prefill performance is mediocre by 2026 standards, but it is fine for interactive chat.

Generation is memory-bandwidth-bound. The 3060's 360 GB/s GDDR6 bus means a 14B q4_K_M model generates at roughly 22 tok/s — fast enough to feel like a real assistant. Larger models that fit entirely on the card scale roughly linearly with bandwidth pressure; smaller models scale roughly linearly with FLOPS once memory is not the bottleneck.

Context length is the silent killer. KV cache for a 14B model at 32K context is roughly 6 GB on its own. That leaves only 4-5 GB for weights and activations, which is incompatible with a q4_K_M 14B. The math forces you to choose: long context or fp16 KV. On a 12 GB card, expect 8K to be the comfortable ceiling for 13B-14B at q4_K_M. If you need long context, drop to a 7B model or move to a 24 GB card.

Multi-GPU scaling: does a second 3060 help for 70B?

Sort of. Two RTX 3060 12GB cards give you 24 GB of total VRAM but the PCIe interconnect between them is far slower than the on-card HBM you would get on data-center hardware. llama.cpp tensor-parallel and pipeline-parallel implementations work, but at roughly 60-70% efficiency relative to a single 24 GB card. A 70B at q4_K_M is on the edge of feasibility with two 3060s — you can fit the weights, but the per-token latency suffers and you trade quality for size.

A better question is whether two 3060s beat one 3090 for the same total VRAM. Almost always no — the single 3090 wins on latency, on memory bandwidth, and on power efficiency. The case for dual 3060 is incremental upgrade economics: if you already own one 3060 and finding a second one cheap is easier than buying a 3090, that path can be reasonable. Otherwise, save up for a single bigger card.

Perf-per-dollar and perf-per-watt across the tiers

The RTX 3060 12GB at roughly $290 in 2026 delivers about 70 tok/s per dollar amortized over five years of ownership on a 7B q4_K_M workload — the best VRAM-per-dollar in the consumer market. The MSI RTX 3060 Ventus 2X 12G and Gigabyte RTX 3060 Gaming OC 12G variants land in the same band. The RTX 3090 used at roughly $700 doubles the VRAM and gives you fp16 access at the price of older silicon. The RTX 4090 at roughly $1,800 wins on prefill and on raw tok/s but is overkill for 12 GB-class workloads.

Perf-per-watt favors the smaller cards on small workloads. A 3060 pulling 170 W at the wall on a 14B q4_K_M workload is cheaper to run than a 4090 pulling 350 W on the same task, even though the 4090 finishes faster. Build with the AMD Ryzen 5 5600G for a quiet low-power node, or step to the AMD Ryzen 7 5800X if you need real CPU horsepower for offload.

Bottom line: the model-to-GPU cheat sheet

• 7B at q4_K_M → any 8 GB card. The 3060 has runway for 16K context.
• 13B-14B at q4_K_M → RTX 3060 12GB at 8K context.
• 27B-32B at q4_K_M → 24 GB card. A 3090 used is the best value.
• 32B at q3 on a 3060 → possible with offload, but slow and lossy.
• 70B at q4 → 48 GB total VRAM, dual 3090 or pro card.
• 70B at q5+ → data-center hardware or accept very slow inference.

Common pitfalls

1. Math without KV. Most "will it fit" calculators give you the weight footprint and forget the KV cache. Add it.
2. Trusting marketing tok/s. Vendor numbers are best-case batched throughput. Single-user steady-state is what you actually get.
3. Buying for fp16 you do not need. Most security and chat workloads do not benefit meaningfully from fp16 over q4_K_M.
4. Ignoring power. A 12 GB card on a 450 W PSU during a Stable Diffusion peak will crash. Size the PSU above peak.
5. Skipping the CPU. Offload paths bottleneck on memory bandwidth. A weak CPU torpedoes spilled-layer performance.

When NOT to buy a local-LLM GPU

If you only need a model occasionally, the cloud is cheaper. If you live in a region with very high electricity prices, sustained inference adds up. If your workload is dominated by frontier-only models, no consumer GPU will help. Buy local for the workload that actually justifies it.

Related guides

• GPT-5.5-Cyber vs Mythos and the local-runnable alternatives — security-tuned models on the same rig.
• Ollama vs LM Studio vs llama.cpp on the RTX 3060 — picking the runner.
• GLM-5.2 review — frontier-class open weights on the 3060.

Citations and sources

• Hugging Face — for model cards, weights, and quantization metadata referenced throughout.
• llama.cpp GitHub repository — for community-measured tok/s figures.
• TechPowerUp RTX 3060 spec page — for FLOPS, bandwidth, and VRAM data used in the arithmetic.

This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.