Which GPU Runs Which LLM? A Per-Model VRAM Compatibility Guide (2026)

A single 12 GB card like the ZOTAC RTX 3060 12GB runs 7B and 13B models fully in VRAM at strong tok/s, handles 32B models with partial CPU offload, and can serve GLM-5.2 at q4. A 70B llama-class model still needs 24 GB or a dual-GPU rig for interactive latency — a single 12 GB card offloads too many layers to system RAM and falls to low single-digit tok/s.

Why VRAM is the gate for local inference in 2026

Ask any local-LLM builder what killed their last build plan and the answer is almost never raw compute. It's memory. A GeForce RTX 3060 12 GB has 12,742 GFLOPs of FP32 throughput and 360 GB/s of memory bandwidth (see TechPowerUp's RTX 3060 spec sheet), and it will happily push 40+ tok/s on a q4_K_M 7B model. What it will not do is load a 70-billion-parameter model into VRAM, no matter how patient you are — the weights alone are ~40 GB at q4 before you add any KV cache.

That's why this guide is organized around VRAM tiers, not around FLOPs or generation numbers. A budget builder targeting the featured 12 GB RTX 3060 has a very different envelope than someone spec'ing a workstation around an RTX 3090 or A6000-class. Both cards can run "AI." Only one can run a 32B model without touching system RAM.

We wrote this for hobbyists and small-team builders who are choosing hardware for their next local-inference rig in 2026 and want a straight answer to which card runs which model. The math is universal — VRAM budget minus context minus overhead equals the model size you can hold — but the answers change with quantization scheme, backend, and how many tokens of context you actually plan to feed.

Key takeaways

• 7B models at q4 fit in ~6 GB. Any 8 GB card runs them, but 12 GB gives you room for 8K-16K context without pain.
• 13B models at q4 fit in ~8 GB. A 12 GB card is the sweet spot; 8 GB cards are borderline once you add context.
• 32B models at q4 (GLM-5.2, DeepSeek-Coder-33B, etc.) need ~19 GB. A 12 GB card runs them at ~5-9 tok/s with CPU offload; a 24 GB card runs them at 25-35 tok/s fully in VRAM.
• 70B llama-class models at q4 need ~40 GB. A single 24 GB card cannot hold them; you need a dual-24 setup, one 48 GB card, or heavy offload.
• Quantization below q4 (q3, q2) shrinks memory further but degrades output quality noticeably on reasoning workloads; keep at q4_K_M or above when quality matters.
• Context length eats VRAM linearly. Budget 1-3 GB for KV cache at 8K, more at 32K, especially on 32B+ models.

How much VRAM does each model class actually need?

The dominant term is parameter count times bytes-per-weight. Under llama.cpp's k-quants (the default in Ollama and LM Studio, see ggml-org/llama.cpp), the bytes-per-weight roughly work out to:

Add 1-3 GB for KV cache at 8K context (grows with context and with head count), plus ~500 MB of runtime overhead. The Hugging Face quantization overview explains why 4-bit quantization is currently the quality-cost sweet spot for consumer inference.

That gives you the working number: at q4_K_M with 8K context, a 7B needs ~5 GB, a 13B needs ~8-9 GB, a 32B needs ~18-20 GB, and a 70B needs ~38-42 GB.

Can a 12 GB RTX 3060 run a 32B model?

Yes — but not fully in VRAM. A 32B model at q4_K_M wants ~19 GB. Your 12 GB card holds roughly 60-65% of the layers on-GPU and pushes the rest to system RAM. In llama.cpp / Ollama syntax that's --n-gpu-layers 44 on a 65-layer model (numbers vary by architecture). Real-world throughput on the ZOTAC RTX 3060 12GB paired with an AMD Ryzen 7 5800X lands at 5-9 tok/s for GLM-5.2 q4_K_M with 4K context.

That is usable — not for streaming chat, but for background agentic tasks where a paragraph every 20 seconds is fine. Bump the CPU to a faster 8- or 12-core part and the offload piece speeds up because llama.cpp is memory-bandwidth bound on the CPU side; dual-channel DDR4-3600 or DDR5 helps more than clock speed.

If you want a 32B model at 25+ tok/s, you need 24 GB. Full stop.

Quantization matrix for GLM-5.2 and llama-class models

Here is the practical map for GLM-5.2 (32B parameters) running on a 12 GB card with CPU offload versus a 24 GB card fully in VRAM. Tok/s numbers assume 4K context, a modern desktop CPU, and llama.cpp's default k-quant kernels.

Two things stand out. First, dropping from q4 to q2 barely helps 12 GB cards because the bottleneck is offload bandwidth, not the on-GPU portion. Second, on a 24 GB card, q4_K_M has become the default for a reason: it's within a few percent of fp16 on standard benchmarks and roughly 4x faster.

Prefill vs generation: where VRAM pressure spikes

New builders often size for the model weights alone and get surprised when a 12 GB card OOMs at 16K context. Prefill (processing the prompt) uses proportionally more VRAM than generation because the full attention matrix is materialized. On a 32B model at q4 with 8K context, prefill can push 2-3 GB extra beyond steady-state generation; at 32K, the KV cache alone approaches 5 GB.

Practical rule: reserve 2-3 GB of VRAM headroom above your model weights if you plan to use 8K+ context. If your usual prompts are short (chat, coding assist under 4K tokens), you can push the offload closer to the ceiling.

Spec-delta table across VRAM tiers

Here's how the popular tiers compare in 2026, using street prices for representative parts. The ZOTAC, MSI, and GIGABYTE 12 GB RTX 3060 cards are near-identical on tok/s (same GPU); the differences are cooler, PCB, and warranty.

Bandwidth matters as much as capacity once you're fully in VRAM. That's why a 4090 (1008 GB/s) at 24 GB blows past a 3060 Ti (448 GB/s) at 8 GB even on 7B models that fit both.

Multi-GPU scaling: when two 3060s beat one bigger card

Two 12 GB RTX 3060 cards give you 24 GB aggregate for under $700 combined — cheaper than a single used 24 GB card. Tensor-parallel inference (vLLM, exllama) can hit 70-80% of a single-card throughput at 2x the capacity, so you effectively get a "24 GB card" at 55-70 tok/s on 13B models. The catches:

• PCIe lanes matter. You need x8/x8 or better; a single x16 card slot bifurcated to two x8 slots is fine on Ryzen 7 5800X and X570-class boards.
• Power draw scales linearly. Two 3060s at 170 W each want an 850 W PSU minimum.
• Not every backend supports it cleanly. llama.cpp's --split-mode row works but requires manual tuning; vLLM handles it out of the box on Linux.

When one 24 GB card wins: any workload where you need a 32B or 70B model, since tensor-parallel adds latency per token and single-card avoids the inter-GPU sync. Multi-GPU makes the most sense at the 13B-to-32B boundary where you want more speed than one 3060 can give but don't want to buy a 4090.

Perf-per-dollar and perf-per-watt

Cheap cards win on dollars-per-token when they fit the model. Expensive cards win on tokens-per-watt. On a 13B q4 workload:

The 3060 leads on capex-per-tok/s at the 13B tier; the 3090 leads on total value for 32B; the 4090 dominates on perf-per-watt for long-running jobs where electricity actually matters. For an always-on rig, the 4090 pays back its price over a couple of years vs a 3060 farm.

Verdict matrix

Get a 12 GB card if… your target workloads are 7B and 13B chat, coding assist, or small agentic tasks; you occasionally want to poke at a 32B model with tolerable latency; your budget is under $400 for the GPU. The MSI RTX 3060 Ventus and GIGABYTE 3060 Gaming OC are both solid picks; pick whichever fits your case.

Step up to a 24 GB card if… you want interactive 32B (GLM-5.2, DeepSeek-33B) at 25+ tok/s; you need long context (32K+) at 13B; you plan to fine-tune or train LoRAs on top of a 7B/13B base.

Stay on 8 GB if… you only run 7B models, you're happy with 4K context, and your build has no room for a larger PSU or cooler. It's serviceable, not exciting.

Bottom line: the cheapest card that runs your target model

• Llama-3-8B / Mistral-7B / Phi-3.5: 8 GB minimum, 12 GB comfortable.
• Llama-3-13B / Yi-13B / DeepSeek-Coder-6.7B at long context: 12 GB.
• GLM-5.2 32B / DeepSeek-Coder-33B / Qwen2.5-32B at interactive speed: 24 GB.
• Llama-3.1-70B / Qwen2.5-72B: 48 GB (single card) or 2×24 GB.

At $340-400 street, a 12 GB RTX 3060 is still the highest floor-to-value ratio in 2026 for local LLM builders on a budget. Pair it with a strong CPU like the Ryzen 7 5800X if you plan to lean on CPU offload for the occasional 32B model.

Related guides

• Ollama vs LM Studio on a 12 GB RTX 3060 — which runner fits your workflow
• GLM-5.2 vs Frontier Models on GDPval-AA — what open-weights buys you in 2026
• Build a Home AI Assistant on a Raspberry Pi 4 8GB — the very low end of the same math
• Ryzen 7 9800X3D Record-Low Price — pairing CPU with your inference build

Citations and sources

• TechPowerUp — GeForce RTX 3060 spec sheet
• Hugging Face — Transformers quantization overview
• ggml-org/llama.cpp on GitHub

_As of 2026, prices and street availability shift monthly on Amazon and eBay; sizing math on VRAM and bandwidth does not._