Per-Model GPU Requirements 2026: Which 7B-70B LLMs Actually Fit on 8GB, 12GB, and 24GB

You need roughly 6GB of VRAM to run a 7B model at q4, 10-12GB for a 13B at q4, 22-24GB for a 34B at q4, and 40-48GB for a 70B at q4 — before you add any context. As of 2026, that puts a 7B comfortably on an 8GB card, makes a 13B the sweet spot for a 12GB MSI GeForce RTX 3060 Ventus 2X 12G, and forces a 70B onto a 24GB card with CPU offload or two GPUs working together. Small 1B-3B models will even run on the CPU of a Raspberry Pi 4 Model B 8GB at a few tokens per second. Below is the per-model math, the quantization table, and the honest answer to which card you actually need.

The generic advice you keep seeing — "just buy more VRAM" — is technically correct and practically useless. VRAM is a budget, not a purchase, and every local-LLM setup spends that budget across three lines: model weights, KV cache for context, and a working overhead for the runtime (llama.cpp, vLLM, or Ollama each carry a small tax). The math changes by model architecture (Llama 3.1 8B vs Qwen 2.5 14B vs Mistral 24B all sit differently in memory), by quantization scheme (q4_K_M eats less than q4_0, and q5_K_M costs almost as much as q6), and by how long the conversations you actually run get. A 14B model that fits in 12GB at 4k context can spill to CPU at 32k context because KV cache grew faster than you did. That is why sites keep recommending an RTX 3060 12GB for local LLMs and readers keep finding it can "only" run a 7B — they are running with a 16k context window and no idea where their 12 gigabytes actually went. This guide gives you the per-model, per-quant, per-context numbers so you can pick a card once and stop guessing. The reference platforms below are what most 2026 first-timers actually buy: a 12GB Ampere card for real inference, a Pi 4 for edge routing, and a fast SATA SSD like the Crucial BX500 1TB SATA SSD to hold the GGUF library that grows faster than anyone plans for.

Key takeaways

• A 7B model at q4_K_M needs about 5-6GB of weights plus 1-2GB of KV cache, so it runs comfortably on 8GB and cleanly on 12GB with 8k+ context.
• A 13B at q4_K_M sits at about 8-10GB of weights; on a 12GB card it hits 30-45 tok/s with 4k context but forces trade-offs at 16k+.
• A 34B at q4_K_M lands near 20GB of weights, meaning a 24GB card is the honest minimum for interactive speeds.
• A 70B at q4_K_M wants 40-48GB total; on a 12GB card you are running 3-6 tok/s with heavy CPU offload, on 2×12GB you can push 8-12 tok/s, and on a single 24GB card with offload you land near 5-10 tok/s.
• KV cache scales roughly linearly with context length and layer count — plan 1-4GB of headroom on top of weights depending on how long your prompts get.
• Below q4, quality drops sharply on 7B models but stays usable on 34B+ models, so a bigger model at q3 often beats a smaller model at q8.

How do you calculate VRAM for a given model size and quant level?

The short formula: VRAM_weights_GB ≈ parameters_billion × bits_per_param / 8 × 1.08. The 1.08 tail accounts for embedding tables, layer norms, and framework overhead that never disappear no matter how aggressive your quantization gets. At fp16 (16 bits), a 7B model is 7 × 16 / 8 × 1.08 = 15.1GB. At q4 (roughly 4.5 effective bits after k-quant grouping), the same 7B drops to 7 × 4.5 / 8 × 1.08 = 4.25GB of weights, and you round up to 5-6GB in practice because the actual q4_K_M mix stores some layers at higher precision.

On top of weights, KV cache for a Llama-family model at fp16 KV is roughly 2 × layers × hidden_dim × context_length × bytes_per_param / (1024^3) gigabytes. For Llama 3.1 8B (32 layers, 4096 hidden) at 8k context that is about 2GB; for a 70B (80 layers, 8192 hidden) at 8k context that jumps to about 5GB. Modern runtimes offer q8 or q4 KV cache to cut that in half or quarter, at some coherence cost on long contexts. Hugging Face's own optimization docs walk through this in more detail; see the Hugging Face LLM optimization guide for the underlying tensor shapes.

Add 500MB-1GB for the runtime itself (llama.cpp is lightest, vLLM heaviest), and you have your VRAM budget. For a rough interactive sizing rule as of 2026: usable_model_GB ≈ (card_VRAM - 2) / 0.6 at q4 with an 8k window. A 12GB card gives you about 16-17GB of effective q4 model budget on paper, which is why a 13B fits and a 24B does not.

What fits on 8GB (RTX 3060-class entry and Raspberry Pi 4 CPU fallback)?

8GB is the current floor for GPU inference. On an 8GB card you can run:

• 1B-3B models at q4-q8: Phi-3-mini, Llama 3.2 3B, Qwen 2.5 3B — all comfortably at 60-120 tok/s with room for 16k+ context. These are your routing, classification, and voice-assistant models.
• 7B models at q4_K_M: Llama 3.1 8B (yes, technically 8B fits), Qwen 2.5 7B, Mistral 7B — 30-50 tok/s at 4k context. Push to 8k context and you are near the redline; 16k spills to CPU.
• 13B models at q3: Possible but painful. Coherence drops and speed halves versus q4.

The Raspberry Pi 4 8GB is a different animal. It has no GPU worth mentioning for LLMs, so everything runs on the ARM Cortex-A72 CPU. Realistic performance per the Raspberry Pi Foundation product page hardware profile: a Llama 3.2 1B at q4 hits 3-5 tok/s, a Phi-3-mini at q4 hits 2-3 tok/s, a 7B at q4 crawls at 0.5-1 tok/s. That is not a chat rig; it is an edge router. Use it to run a small model that classifies incoming requests and forwards the hard ones to a bigger GPU box on your LAN. Pair it with a fast SATA SSD like the Crucial BX500 to hold your model library — you will accumulate 200-500GB of GGUF files inside a year, and the microSD card will not survive the write cycles.

What fits on 12GB (RTX 3060 12GB sweet spot)?

The 12GB tier is where local LLMs stop being a science experiment. According to TechPowerUp's RTX 3060 spec page, the card ships with 12GB of GDDR6 on a 192-bit bus, which is the specific combination that makes 13B-14B models practical. On a 12GB card as of 2026:

• 7B/8B models at q4-q6: 40-70 tok/s at 8k-16k context. This is where you actually feel like you have a local ChatGPT.
• 13B-14B models at q4_K_M: 25-40 tok/s at 4k context, 15-25 tok/s at 16k. Qwen 2.5 14B and Llama 3.1 8B fine-tunes both sing here.
• 20B-24B models at q3-q4: Mistral Small 22B and Qwen 2.5 32B (heavily quantized) — 8-15 tok/s. Coherent but noticeably slower.
• 32B models at q3: Possible with careful KV cache quantization and short context; realistically 5-10 tok/s.

The 12GB card also opens up the CUDA ecosystem — flash-attention 2, bitsandbytes, and every fine-tuning tutorial online — which the AMD alternatives at the same price point still struggle with in 2026 for anything beyond stock inference. Two solid 12GB SKUs to pick between: the MSI Ventus and the ZOTAC Twin Edge are both single-fan-versus-dual-fan variants of the same silicon at similar street prices as of 2026.

What fits on 24GB and beyond, and when do you need it?

24GB is the tier where the compromises largely stop. A single 24GB card (RTX 3090, RTX 4090, or the newer 24GB RTX 5000-series options) as of 2026 runs:

• 34B models at q4_K_M: 20-35 tok/s at 8k context. Qwen 2.5 32B and Mixtral variants live here.
• 70B models at q3_K_M or q4 with offload: 5-12 tok/s. Fits weights alone at q3 with careful KV quant; q4 spills 8-15GB to CPU RAM.
• Mixture-of-experts models: Mixtral 8x7B (46B total, 12.9B active) at q4 sits around 26GB — needs 24GB + a few gigs of CPU offload, but active-param speed feels like a 13B.

You want 24GB when: (a) you are running 34B+ models for reasoning or code, (b) your context windows regularly exceed 32k, (c) you are fine-tuning even LoRAs on 13B+, or (d) you want to serve concurrent requests to more than one user. For a solo interactive user, 12GB is usually enough; the moment you add batching, agents, or long RAG contexts, 24GB earns its price.

Spec-delta table: VRAM tier vs model class vs quant vs context vs tok/s

Quantization matrix (q2/q3/q4/q5/q6/q8/fp16)

Using a Llama-family 13B model as the reference case, here is how each quantization level lands on a 12GB card as of 2026:

The honest read: q4_K_M is the default for a reason, q5_K_M is the upgrade if you have the headroom, and q6/q8 are wasted VRAM for interactive use unless you are benchmarking. Below q4, a 7B collapses fast; a 34B or 70B holds together at q3, which is why the "big model at low quant beats small model at high quant" rule holds.

Context-length impact: how KV cache scales VRAM as context grows

KV cache is the sneaky VRAM cost nobody plans for. Every token you add to your context — user prompt, system message, RAG retrieval, agent tool output — allocates a slice of cache on every layer of the model. For a Llama-family 13B at fp16 KV cache, going from 4k to 32k context adds roughly 3-4GB on top of weights. On a 12GB card that turns a comfortably-fitting model into one that spills.

Mitigations, in order of impact:

1. Quantize the KV cache. llama.cpp supports q8_0 KV (halves it) and q4_0 KV (quarters it). q8_0 KV is nearly free in quality; q4_0 KV is fine for chat but degrades on long-context reasoning.
2. Cap the context window. If you never actually use 32k, set the model to 8k. You reclaim 3-5GB on a 13B.
3. Use grouped-query attention (GQA) models. Llama 3.1, Qwen 2.5, and Mistral all use GQA, which cuts KV cache 4-8× versus older multi-head models like Llama 2. Do not use a Llama 2 base in 2026 if you care about long context.
4. Offload KV to CPU. llama.cpp's --no-kv-offload toggles this. Slower per token, but keeps you off the weights spill.

Multi-GPU scaling: when two 12GB cards beat one 24GB card

Two 12GB cards give you 24GB aggregate at often 60-70% the cost of a single 24GB card as of 2026 street prices. The catch is bandwidth: PCIe 4.0 x8 per card is fine for tensor parallelism at 13B-34B, but at 70B you feel the inter-GPU hop. Realistic split behavior:

• 13B-34B on 2x12GB: 15-25 tok/s at 8k context — cleaner than a single 12GB card at the same model.
• 70B on 2x12GB: 8-12 tok/s. Slower than a 48GB single card would be but genuinely usable.
• Fine-tuning: LoRA on 13B+ is where two cards shine; a single 12GB card runs out of room during backprop even when inference fits.

Where a single 24GB card wins: lower system power draw (one card at 250W vs two at 170W each), less heat in the case, simpler cooling, and no inter-GPU communication overhead. For anything sub-70B and single-user, one 24GB card is the calmer setup.

Perf-per-dollar: cost to host each model tier on current cards

As of 2026, rough perf-per-dollar bands for local LLM inference:

• 8GB entry: $200-280 street. Fine for a 7B or edge routing; hits a ceiling fast.
• 12GB RTX 3060: $270-330 street. Best dollars-per-usable-token in 2026 for 7B-14B work. This is why every local-LLM guide, including our 32B models on 12GB VRAM breakdown, keeps pointing here.
• 16GB RTX 4060 Ti / 5060 Ti: $450-550. Small step up in VRAM, big step up in bandwidth. Worth it if you want 14B at higher quant with headroom.
• 24GB RTX 3090 (used): $700-950. The best 24GB dollar in 2026 if you can find a clean one; the pro tier without pro pricing.
• 24GB RTX 4090 / 5090: $1600-2400. Fastest single-user local LLM box you can build without going to Threadripper + multi-GPU territory.

For how memory bandwidth (not just VRAM capacity) constrains real-world tok/s, see our memory bandwidth breakdown for local inference.

Verdict matrix: Buy 12GB if... / Buy 24GB if... / Stay on CPU if...

Buy 12GB if: you are a solo user running 7B-14B for chat, coding assist, or RAG; your context windows stay under 16k most of the time; you value the cheapest "real" local-LLM box that also plays 1440p games as a bonus.

Buy 24GB if: you want to run 34B+ models interactively, you fine-tune anything beyond a 3B LoRA, you serve concurrent requests, or your workflows regularly break 32k context. Buy 2x12GB instead only if you have a case, PSU, and slot count that supports it and you enjoy tuning tensor-parallel splits.

Stay on CPU if: you are running 1B-3B models for classification or routing on an edge box, your latency budget is generous (2-5 seconds per response), or you are prototyping with no interactive UX requirement. A Pi 4 + a fast SSD like the Crucial BX500 makes a fine tinkering rig.

Common pitfalls

1. Buying VRAM without checking memory bandwidth. Two cards with the same 12GB label but different bus widths (192-bit vs 128-bit) can differ 30-40% in real tok/s. Always cross-check the bandwidth spec, not just capacity.
2. Forgetting the runtime tax. vLLM at startup can hold 1.5-2GB of buffers before your model even loads. A 12GB card running vLLM effectively gives you a 10GB budget.
3. Assuming q4 is q4. q4_0, q4_1, q4_K_S, and q4_K_M all use different bit-mix strategies. q4_K_M is the community default in 2026; the others exist mostly for compatibility.
4. Ignoring flash-attention support. Non-Ampere-or-newer NVIDIA and most AMD cards fall back to a slower attention kernel, which halves throughput on long context.
5. Buying two mismatched GPUs. Tensor parallelism strongly prefers identical VRAM per rank. A 12GB + 24GB pair effectively behaves like 2x12GB for most model splits — you waste half the big card.

When NOT to buy a bigger GPU

If you already have a 12GB card and your actual workload is a 7B model at 4k context, more VRAM buys you nothing. The bottleneck is memory bandwidth on the model you already run. Upgrade the model (or move to a bigger one you can actually feed with your context) before you upgrade the card. Similarly, if your true need is calling a hosted API 20 times a day, a $200/year hosted plan beats a $900 GPU by a huge margin. Local LLMs win on privacy, latency, and unbounded token count — not on total cost of ownership at low usage volumes. Our long piece on when a local rig actually beats a $500M hosted bill walks through the crossover math.

Bottom line

As of 2026, the honest per-model VRAM answer is: 8GB for 7B, 12GB for 13B, 24GB for 34B, and 48GB (or offload) for 70B — all at q4_K_M, before you spend on context. If you are new to local inference, an RTX 3060 12GB and a decent SATA SSD to hold your model library is the highest-value first buy on the market. If you already have a 12GB card and hit its limits, jump straight to a 24GB used RTX 3090 rather than a 16GB middle step. And if you are prototyping on a Raspberry Pi 4, use it for what it is good at — small edge models — and let the desktop GPU do the heavy lifting.

Related guides

• Can 12GB really run 32B models on an RTX 3060?
• Memory bandwidth from HBM to GDDR6 for local inference
• When a local RTX 3060 beats a $500M Claude bill
• AMD Ryzen AI Max 395 Strix Halo 128GB mini PC for LLMs
• The AgentPerf agentic benchmark for local coding rigs

Sources

1. Hugging Face — LLM optimization guide
2. TechPowerUp — GeForce RTX 3060 12GB spec sheet
3. Raspberry Pi Foundation — Raspberry Pi 4 Model B product page
4. llama.cpp — GGUF quantization types reference
5. Tom's Hardware — GPU hierarchy for AI inference (2026)