As of October 2026, the best all-round local LLM for a 12GB GPU is still Google's Gemma 4 12B (released May 2026 under Apache 2.0). A September 30, 2026 guide names its QAT Q4_0 build the first pick for 12GB cards: the 6.50 GiB file used 7,816 MiB of VRAM at 16K context and generated at 90.0 tok/s on a 12GB RTX 3080 Ti, per Atomic Chat's reported llama.cpp setup. On the cheaper RTX 3060 12GB, its Unsloth UD-Q5_K_XL quant is an 8.6 GB file that runs fully in VRAM at 33.3 tok/s generation and 1,152 tok/s prefill, using about 9.3 GB of VRAM at 4K context, per a community llama.cpp benchmark. If you want more speed and longer context, Qwen3.5-9B at Q4_K_M (5.7 GB) generates at 49.9 tok/s on the same card. For agentic coding, Qwen3.6-35B-A3B also runs on a 12GB RTX 3060, but only by offloading its MoE experts to system RAM: an October 15, 2026 update reports 38.9 tok/s at UD-Q4_K_M with -ncmoe 24 and 9.8 GB of VRAM, holding about 38 tok/s through 8K of context. The largest dense model that now fits is Qwen3.8 27B (August 2026, Apache 2.0), whose IQ3_XXS file is 10.1 GB. The RTX 3060 12GB itself was listed at $459.99 by PassMark as of October 12, 2026 (price may vary).
New measurements since October 16, 2026
Two more 12GB cards now have published Qwen3 figures, which puts every current 12GB NVIDIA card on one test. Per Hardware Corner's RTX 4070 Super page, the RTX 4070 Super generates Qwen3 14B (Q4_K) at 45.5 tok/s at 4K context with the model fully in VRAM. Per its RTX 4070 Ti page, the RTX 4070 Ti generates 45.8 tok/s on the same test. Both sit between the RTX 4070 (42.5 tok/s) and the RTX 5070 (54.2 tok/s).
| Card (12GB) | Qwen3 8B, 4K | Qwen3 8B, 16K | Qwen3 8B, 32K | Qwen3 14B, 4K | Qwen3 14B, 16K |
|---|---|---|---|---|---|
| RTX 4070 Super | 75.4 tok/s | 56.2 tok/s | 42.2 tok/s | 45.5 tok/s | 37.2 tok/s |
| RTX 4070 Ti | 75.8 tok/s | 57.6 tok/s | 42.1 tok/s | 45.8 tok/s | 37.8 tok/s |
What it changes: neither card raises the 12GB model-size ceiling. Qwen3 14B at Q4_K is still the largest dense model either one runs fully in VRAM, so the choice between them and an RTX 3060 is speed per dollar, not which models fit. Hardware Corner lists the quant as GGUF Q4_K and does not name the runtime. Its pages are undated, so these figures carry the date they were retrieved, October 19, 2026.
What changed — October 16, 2026
- Faster MoE-offload figure for Qwen3.6-35B-A3B. InsiderLLM's guide, updated October 15, 2026, reports the Unsloth UD-Q4_K_M build (20.6 GiB) at 38.9 tok/s empty, 38.5 tok/s at 4K and 38.2 tok/s at 8K on an RTX 3060 12GB with
-ncmoe 24, using 9.8 GB of VRAM and 32 GB of DDR4 (llama.cpp b10088). The same page withdrew an earlier-ncmoe 20result because that setting does not load. - Qwen3.8 27B is a new dense option at 3-bit. The Qwen3.8-27B card lists 27B parameters, 262,144 tokens of native context and an Apache 2.0 license. ISTA-DASLab's GGUF card lists a 10.1 GB IQ3_XXS file scoring 84.57 on LiveCodeBench v6 against 85.71 for BF16. On an RTX 3060 12GB, an experimental llama.cpp fork with MTP decoding reports 26.36 tok/s on a Rust coding fixture and 18.34 tok/s on reasoning, with as little as 366 MiB of VRAM left free. That is not a stock llama.cpp result.
What changed — October 2026
- Gemma 4 12B QAT Q4_0 is now the low-VRAM way to run the top pick. Per Atomic Chat's September 30, 2026 guide, the 6.50 GiB QAT Q4_0 file used 7,816 MiB of VRAM at 16K context and generated at 90.0 tok/s on a 12GB RTX 3080 Ti (llama.cpp b10988). That card has far more bandwidth than an RTX 3060, so expect lower speeds on a 3060.
- Qwen3.5-9B at Q6_K fits with room for 64K context. The same guide reports the 6.95 GiB Q6_K file at 7,276 MiB of VRAM at 16K context and 93.6 tok/s generation on the RTX 3080 Ti.
- RTX 3060 12GB pricing has moved up. PassMark records a last price change to $459.99 on 2026-10-12.
What changed — September 2026 refresh
- New picks. The refresh replaces the Qwen 2.5 recommendations with Gemma 4 12B, measured at 33.3 tok/s on an RTX 3060, and Qwen3.5-9B, measured at 49.9 tok/s on the same card. For agentic coding it adds Qwen3.6-35B-A3B with MoE expert offload at 22.9 tok/s.
- Corrected: 14B generation speed. The old uncited 14–22 tok/s range for 14B models was too low. Hardware Corner measured Qwen3 14B (Q4_K) at 31.2 tok/s on an RTX 3060 at 4K context.
- Corrected: prefill speed. The old 200–500 tok/s prefill estimate is replaced with measured figures: 973 tok/s for Qwen3 14B on an RTX 3060 and 2,100 tok/s on an RTX 4070, both at 4K context.
- The quantization matrix shows bandwidth ceilings. Its uncited tok/s estimates are replaced with ceilings computed from the RTX 3060's 360 GB/s bandwidth and each quant's file size.
- Models that do not exist were removed. "Llama 3.3 8B" and "DeepSeek-Coder-V2-Lite 7B" were dropped, along with an unsupported single-digit tok/s claim. DeepSeek-Coder-V2-Lite is now listed correctly as a 16B MoE with a 10.36 GB Q4_K_M file.
What's changed as of September 2026
The first version of this guide recommended the 2024 Qwen 2.5 family. Three things have changed since then.
- New dense models that fit. Gemma 4 12B (256K context, Apache 2.0) and Qwen3.5-9B (262K context, Apache 2.0) both fit in 12GB at 4-5 bit, with room left for the KV cache.
- MoE expert offload. llama.cpp's
--n-cpu-moekeeps attention and shared weights on the GPU and moves only the expert weights to system RAM. That makes a 35B-total / 3B-active model interactive on a 12GB card. - Measured speeds are higher than the old estimates. Hardware Corner measured Qwen3 14B (Q4_K) at 31.2 tok/s on an RTX 3060 at 4K context. This article's old 14-22 tok/s range for 14B models was too pessimistic, so the tables below have been rebuilt from cited measurements.
The table lists models whose quant file plus KV cache fits in 12GB, plus one labelled MoE-offload row. File sizes are the Hugging Face GGUF sizes for the exact quant. "Ceiling" is bandwidth divided by the bytes read per token, the theoretical maximum decode speed. Every quoted figure sits below it.
| Model (quant, file size) | Card (bandwidth) | Decode ceiling | Measured generation | Notes |
|---|---|---|---|---|
| Gemma 4 12B, UD-Q5_K_XL, 8.61 GB | RTX 3060 12GB (360 GB/s) | ~42 tok/s | 33.3 tok/s | 4K ctx, flash attention, q8 KV, ~9.3 GB VRAM |
| Gemma 4 12B, UD-Q6_K_XL, 10.69 GB | RTX 3060 12GB (360 GB/s) | ~34 tok/s | 26.0 tok/s | ~11.3 GB VRAM; little room for context |
| Qwen3.5-9B, Q4_K_M, 5.68 GB | RTX 3060 12GB (360 GB/s) | ~63 tok/s | 49.9 tok/s | llama.cpp, 8K ctx, all layers on GPU |
| Qwen3 14B, Q4_K_M, 9.00 GB | RTX 4070 12GB (504 GB/s) | ~56 tok/s | 42.5 tok/s @ 4K, 32.7 @ 16K | Fully in VRAM up to 16K ctx |
| Qwen3 14B, Q4_K_M, 9.00 GB | RTX 5070 12GB (672 GB/s) | ~75 tok/s | 54.2 tok/s @ 4K, 40.6 @ 16K | Fastest 12GB card in the set |
| Qwen3.8 27B (dense), GSQ-RCO IQ3_XXS, 10.1 GB | RTX 3060 12GB (360 GB/s) | ~36 tok/s | 26.36 tok/s (Rust coding fixture) | Experimental llama.cpp fork with MTP, run on a "Heretic" variant of this quant; 366 MiB VRAM free at minimum |
| Qwen3.6-35B-A3B (MoE), UD-Q4_K_M, 20.6 GiB | RTX 3060 12GB (360 GB/s) + DDR4 | n/a (offload-bound) | 38.9 tok/s empty, 38.2 @ 8K | Does not fit in 12GB: -ncmoe 24, 9.8 GB VRAM, 32 GB DDR4-2133 |
| Qwen3.6-35B-A3B (MoE), UD-IQ3_XXS, 13.21 GB | RTX 3060 12GB (360 GB/s) + DDR4 | ~318 tok/s on active bytes only | 22.9 tok/s | Does not fit in 12GB: -ncmoe 25 puts 7.7 GB of experts in system RAM, 7.1 GB VRAM, 64K ctx |
For the MoE row, the ceiling uses active-parameter bytes: 13.21 GB × 3B/35B ≈ 1.13 GB per token, per the Qwen3.6-35B-A3B card ("35B in total and 3B activated"). The real limit is system RAM and PCIe for the offloaded experts. This is a 12GB-plus-system-RAM setup, not a 12GB one. The larger UD-Q4_K_M build is faster in InsiderLLM's October 2026 run (38.9 tok/s at -ncmoe 24) than the IQ3_XXS run at 22.9 tok/s, but the two used different context sizes, RAM and llama.cpp builds.
The 12GB tier is the 2026 mainstream local-LLM entry point
Twelve gigabytes of VRAM is where local LLMs became a daily-driver workflow. The cheapest CUDA card in that bracket is the GeForce RTX 3060 12GB, with 360 GB/s of memory bandwidth (techpowerup.com/gpu-specs/geforce-rtx-3060.c3682). Common budget-build choices are the MSI GeForce RTX 3060 Ventus 2X 12G and the ZOTAC Gaming GeForce RTX 3060 Twin Edge. They share the same GA106 silicon and 12GB GDDR6, and differ only in cooler design.
The RTX 4070 (504 GB/s) and RTX 5070 (672 GB/s) decode faster, roughly in line with their bandwidth, but hit the same wall on model size.
The reason is mechanical. A 12B-14B dense model at 4-bit is 7 to 9 GB on disk (Gemma 4 12B Q4_K_M is 7.12 GB, Qwen3 14B Q4_K_M is 9.00 GB). That leaves 3 to 5 GB for KV cache and overhead. 8B-9B models open up longer context and higher quants, and 24GB fits 27B-32B dense models. For single-user chat and code, used-market price per GB still favours 12GB. Quants for new releases appear quickly on the Ollama registry (github.com/ollama/ollama) and Hugging Face (huggingface.co/models).
Key takeaways
- A 12GB card holds 12B-14B dense models at 4-bit with 4K-16K context, or 8B-9B models at 4-6 bit with room for 32K or more.
- Best general-purpose pick as of September 2026: Gemma 4 12B at Q4_K_M or UD-Q5_K_XL. It measures 33.3 tok/s at Q5_K_XL on an RTX 3060.
- Best speed and long-context pick: Qwen3.5-9B at Q4_K_M, at 49.9 tok/s on an RTX 3060.
- Measured RTX 3060 decode for 14B at Q4 is about 31 tok/s at 4K context and 23 tok/s at 16K. 8B-class models run at 59.5-64.5 tok/s at Q4_K_M.
- An RTX 5070 runs the same 14B model at 54.2 tok/s, but fits nothing larger than the 3060.
- 20B+ dense models don't fit at 4-bit. Qwen3.8 27B fits only at about 3 bits (IQ3_XXS, 10.1 GB). MoE models can run with
--n-cpu-moeexpert offload and enough system RAM: Qwen3.6-35B-A3B reaches 38.9 tok/s on an RTX 3060.
Step 0: which model size actually fits 12GB at a usable quant?
The first question is which model fits with useful headroom. As a rough rule from published GGUF sizes, a q4_K_M quant takes about 0.6-0.65 GB per billion parameters, and q5_K_M about 0.75-0.8 GB. For example, Llama 3.1 8B Q4_K_M is 4.92 GB. Add 1-2 GB for the KV cache at modest context and 0.5-1 GB for runtime overhead.
That puts the practical ceilings on a 12GB card, as of 2026, at roughly:
| Model class | q4_K_M weights | Practical context | Headroom feel |
|---|---|---|---|
| 7B | ~4.4-4.7 GB | 16k+ tokens comfortably | Very relaxed |
| 8B-9B | ~5.0-5.7 GB | 16k-32k tokens | Relaxed |
| 12B (Gemma 4) | ~7.1 GB | 8k-16k tokens | Comfortable |
| 14B (Qwen3) | ~9.0 GB | 4k comfortably, 16k with care | Tight |
| 20B+ dense | 11.5+ GB | Spills; partial offload | Doesn't fit cleanly |
| 27B (Qwen3.8) at IQ3_XXS | 10.1 GB | Tight; KV cache must be small | Fits only at ~3-bit |
14B at q4_K_M is the upper bound for all-in-VRAM. One RTX 3060 owner found gpt-oss-20b (about 11.5 GB) didn't fit at all. Larger dense models force partial CPU offload, where layers run at system-RAM and PCIe speed instead of the card's 360 GB/s. For more on the quant/context trade-off on this card, see our companion piece on LLM quantization on a 12GB GPU.
What's the best general-purpose model for 12GB right now?
As of September 2026, the general-purpose recommendation for a 12GB card has moved from Qwen 2.5 to three newer options.
- Gemma 4 12B is the balanced pick. Google targets it at "consumer GPUs and workstations" (model card), and its sliding-window attention keeps the KV cache small. It measures 33.3 tok/s at UD-Q5_K_XL on an RTX 3060. The 7.12 GB Q4_K_M file leaves more room for context.
- Qwen3.5-9B is the speed and long-context pick. Its card claims parity with Qwen3 and better results than Qwen3-VL on reasoning, coding and agent benchmarks. It measures 49.9 tok/s at Q4_K_M on an RTX 3060. As a thinking model, it needs a generous output-token budget.
- Qwen3 14B is the largest dense model that fits. It runs at 31.2 tok/s at 4K on an RTX 3060, 42.5 on an RTX 4070 and 54.2 on an RTX 5070, but leaves the least room for context.
Llama 3.1 8B remains a fast fallback at 64.5 tok/s at Q4_K_M. Gemma 2 9B and Qwen 2.5 14B still work but have been superseded. Mistral Small (22B) only squeezes in at a 3-bit quant with trimmed context.
Which one is "best" depends on whether you prioritize per-answer quality or interactive feel:
- Balanced quality, speed and context: Gemma 4 12B at Q4_K_M or UD-Q5_K_XL.
- Speed and long context: Qwen3.5-9B at Q4_K_M, or Q6_K if VRAM allows.
- Maximum dense parameters, short context: Qwen3 14B at Q4_K_M.
These models are on the Ollama registry (github.com/ollama/ollama) and Hugging Face (huggingface.co/models). Gemma 4 12B and Qwen3.5-9B also accept image input, per their cards. For the multimodal angle, see our HiDream o1 1.5 local 12GB analysis.
What's the best coding model that fits in 12GB?
Coding is where the extra parameters pay off most clearly, and in 2026 it is also where MoE offload changes the picture.
For a fully resident coder, Qwen 2.5 Coder 14B at q4_K_M (8.99 GB) still runs well, at 35.8 tok/s on an RTX 3060 at 4K context. The same source measured Qwen 2.5 Coder 7B at 69.3 tok/s. Qwen3.5-9B is the newer general model that also covers coding, per its card's coding and agent claims, and it leaves far more room for long source files.
For agentic coding, where tool calls and long context matter more than raw speed, the 2026 option is Qwen3.6-35B-A3B with MoE expert offload. Only 3B of its 35B parameters are active per token. Moving 25 layers of experts to system RAM (-ncmoe 25) cuts VRAM use to 7.1 GB, and on an RTX 3060 with 31 GB of DDR4 it runs at 22.9 tok/s at a 64K allocated context. Without the flag, the 13.2 GB model fails with an out-of-memory error. It does not fit in 12GB on its own. A newer run of the larger UD-Q4_K_M build with -ncmoe 24 reports 38.9 tok/s, about 38 tok/s through 8K of context, at 9.8 GB of VRAM (updated October 15, 2026).
Qwen3.8 27B is the dense alternative for coding. Its ISTA-DASLab IQ3_XXS quant is 10.1 GB and scores 84.57 on LiveCodeBench v6 against 85.71 for BF16, per that card. On an RTX 3060, an experimental llama.cpp fork with MTP reports 26.36 tok/s on Rust and 24.01 tok/s on C++ coding fixtures. Its README says it is "an experimental fork/patch set, not an upstream llama.cpp release," and the file leaves little VRAM for anything else.
DeepSeek-Coder-V2-Lite (16B MoE), the older "almost-fits" alternative, has a 10.36 GB Q4_K_M file. That leaves under 2 GB for KV cache and overhead, so it is marginal, and Qwen3.6-35B-A3B has largely replaced it. The full per-model rundown lives in our per-LLM model hardware guide.
The next question: if a coding model is the reason for the build, the follow-up is whether a 16 GB card would run it meaningfully better than this one. RTX 3060 12GB vs RTX 4060 Ti 16GB for Qwen2.5-Coder 14B (2026) finds generation speed roughly level between the two, with the 4060 Ti's premium buying faster prompt processing and room for a higher quant.
Quantization matrix: q2 to fp16 across the 7B-14B classes
Quantization is the biggest lever on a 12GB card. The table combines approximate GGUF bytes per parameter with the RTX 3060's decode ceiling (360 GB/s divided by weight size) and quality notes from community threads. In two cited cases, measured decode landed at 78-88% of the ceiling: Qwen3 14B Q4_K_M (9.00 GB, ceiling ~40) measured 31.2 tok/s, and Llama 3.1 8B Q4_K_M (4.92 GB, ceiling ~73) measured 64.5 tok/s.
| Quant | Bytes/param | 7B VRAM | 8B VRAM | 13B VRAM | 14B VRAM | 7B ceiling (3060) | 14B ceiling (3060) | Quality vs fp16 |
|---|---|---|---|---|---|---|---|---|
| q2_K | ~0.30 | ~2.1 GB | ~2.4 GB | ~3.9 GB | ~4.2 GB | ~171 tok/s | ~86 tok/s | Noticeable loss |
| q3_K_M | ~0.40 | ~2.8 GB | ~3.2 GB | ~5.2 GB | ~5.6 GB | ~129 tok/s | ~64 tok/s | Modest loss |
| q4_K_M | ~0.62 | ~4.4 GB | ~5.0 GB | ~7.9 GB | ~8.5-9.0 GB | ~82 tok/s | ~40-42 tok/s | Slight loss |
| q5_K_M | ~0.78 | ~5.5 GB | ~6.2 GB | ~9.9 GB | ~10.7 GB | ~65 tok/s | ~34 tok/s (tight) | Near-fp16 |
| q6_K | ~0.92 | ~6.4 GB | ~7.3 GB | ~11.7 GB | spill | ~56 tok/s | n/a | Near-fp16 |
| q8_0 | ~1.10 | ~7.7 GB | ~8.8 GB | spill | spill | ~47 tok/s | n/a | Effectively fp16 |
| fp16 | ~2.00 | spill | spill | spill | spill | n/a | n/a | Reference |
q4_K_M is still the sweet spot, with q5 practical for 12B models, and q5-q6 comfortable for 7B-9B. At 13B-14B, q6_K fits but leaves no room for context. Treat anything below q4 as a fallback. For Gemma 4 12B, going from UD-Q5_K_XL to UD-Q6_K_XL drops decode from 33.3 to 26.0 tok/s and raises VRAM from ~9.3 to ~11.3 GB.
Spec table: candidate models for a 12GB card
The table below is the short list. License and context notes are summarized from each model's published card on Hugging Face (huggingface.co/models).
| Model | Params | Native context | License | VRAM at q4_K_M |
|---|---|---|---|---|
| Gemma 4 12B Instruct | 11.95B | 256k | Apache 2.0 | ~7.1 GB |
| Qwen3.5-9B | 9B | 262k | Apache 2.0 | ~5.7 GB |
| Qwen3 14B | 14B | 32k native | Apache 2.0 | ~9.0 GB |
| Qwen3 8B | 8B | 32k native | Apache 2.0 | ~5.0 GB |
| Llama 3.1 8B Instruct | 8B | 128k | Llama community | ~4.9 GB |
| Qwen 2.5 Coder 7B | 7B | 128k | Apache 2.0 | ~4.7 GB |
| Qwen 2.5 Coder 14B | 14B | 128k | Apache 2.0 | ~9.0 GB |
| Gemma 2 9B Instruct | 9B | 8k native | Gemma terms | ~5.6 GB |
| Mistral Small (22B) | 22B | 32k | Mistral Research | spill at q4 |
| DeepSeek-Coder-V2-Lite | 16B MoE | 128k | DeepSeek License | ~10.4 GB (marginal) |
| Qwen3.8 27B | 27B dense | 262k | Apache 2.0 | spill at q4; 10.1 GB at IQ3_XXS |
| Qwen3.6-35B-A3B | 35B MoE (3B active) | 262k | Apache 2.0 | needs expert offload |
Apache 2.0 now covers most of this list, including Gemma 4, which marks a change from Gemma 2's custom terms. That matters if you intend to ship a derivative product or fine-tune on proprietary data.
Benchmark table: tok/s on an RTX 3060 12GB
These rows add older reference models to the September 2026 table above. Each is a measured community or reviewer figure (llama.cpp, batch size 1, all layers in VRAM), not a first-party measurement. Figures vary with context, KV-cache type and runtime build.
| Model | Quant | Card | Context | Prefill tok/s | Generation tok/s | Source |
|---|---|---|---|---|---|---|
| Llama 3.1 8B Instruct | q4_K_M | RTX 3060 12GB | 8K | n/a | 64.5 | tyolab |
| Qwen 2.5 Coder 7B | q4_K_M | RTX 3060 12GB | 8K | n/a | 69.3 | tyolab |
| Qwen3 8B | Q4_K | RTX 3060 12GB | 4K | 1,697 | 55.2 | Hardware Corner |
| Qwen 2.5 Coder 14B | q4_K_M | RTX 3060 12GB | 4K | n/a | 35.8 | tyolab |
| Qwen3 14B | Q4_K | RTX 3060 12GB | 4K / 16K | 973 / 678 | 31.2 / 22.7 | Hardware Corner |
While everything stays resident, decode speed scales roughly inversely with weight-file size. Context also costs speed even when everything fits: Qwen3 14B on the RTX 3060 drops from 31.2 tok/s at 4K to 22.7 tok/s at 16K.
Context-length impact: how a long prompt eats your 12GB budget
The KV cache grows roughly linearly with context length, so a 14B model at 16k can need over a gigabyte of cache on top of its 9 GB of weights. In one set of RTX 3060 tests, 14B Q4_K_M models would not load at 8K context and needed 4K.
Two llama.cpp settings used in Jean Brito's writeup recover much of that headroom: -np 1 (one server slot, so no memory is reserved for parallel requests) and --cache-type-k q8_0 --cache-type-v q8_0 (an 8-bit KV cache instead of F16).
Rules of thumb from 2026 community reports:
- An 8B model at Q4 holds 32K of context on an RTX 3060, where Qwen3 8B still generates at 31.9 tok/s.
- A 12B model like Gemma 4 12B at Q4-Q5 is comfortable at 4K-16K. Its sliding-window attention layers (model card) keep the cache smaller than a conventional dense 12B's.
- A 14B model at q4_K_M runs at 16K with careful settings (22.7 tok/s on an RTX 3060) and is safest at 4K.
- A 13B-14B model at q5_K_M is realistic only at 2K-4K context.
For long source files or transcripts, step down in size and up in quant. Qwen3.5-9B at Q6_K with 32K context is usually more useful than Qwen3 14B at q4_K_M with 4K.
Prefill vs generation throughput on 12GB cards
The RTX 3060 delivers about 360 GB/s of bandwidth and 13 TFLOPS of FP32 compute (techpowerup.com/gpu-specs/geforce-rtx-3060.c3682), and inference leans on each differently.
Prefill, which processes the input prompt, is compute-bound. On an RTX 3060, Hardware Corner measured 1,697 tok/s for Qwen3 8B and 973 tok/s for Qwen3 14B at 4K context. The Ada-generation RTX 4070 roughly doubles that, at 2,100 tok/s for the same 14B model. Prefill is the "thinking before it starts typing" delay a user perceives.
Generation is bandwidth-bound for a single user, because each token streams the full weights from memory. That's why decode speed tracks model size, and why the 672 GB/s RTX 5070 decodes Qwen3 14B at 54.2 tok/s against the 3060's 31.2.
For builds, the CPU and NVMe matter mostly for model load, any spill, and MoE offload, where the CPU and system RAM do real work. A 6-core AMD Ryzen 5 5600G is enough for dense models that fit. Add 32 GB of fast RAM if you plan to run MoE offload. A fast NVMe such as the Western Digital 1TB WD Blue SN550 NVMe cuts cold-load times for 9-13 GB files.
Perf-per-dollar: the 12GB 3060 vs a 12GB 5070 vs a 16GB step-up
Hardware Corner puts the RTX 3060 12GB at an average market value of $230 and calls the RTX 5070 a poor LLM value at $650: faster, but the same 14B ceiling. A used RTX 4060 Ti 16GB is the usual step-up.
The case for staying at 12GB:
- You mostly run 8B-14B models, and the 14B-at-q4 ceiling is acceptable.
- You don't need long context on a 14B model.
- You're willing to use MoE expert offload for the occasional larger model.
The case for stepping up to 16GB:
- You want 14B at q5_K_M routinely, not q4_K_M.
- You want 20B-class models fully resident at q4_K_M. gpt-oss-20b at about 11.5 GB did not fit on a 12GB card.
- You run 16k+ context workflows on 13B-14B models.
The case for skipping straight to 24GB:
- You want 27B-32B dense models fully resident at q4_K_M.
- You're doing fine-tuning, not just inference.
On balance, 16GB is a small capability jump for a meaningful price jump, while 24GB is a category change. The 12GB RTX 3060 is still the right entry point for most readers.
Verdict matrix
| Get the RTX 3060 12GB if… | Step up if… |
|---|---|
| You want the cheapest CUDA path to 12GB | You routinely want 14B at q5_K_M or higher |
| You mostly run 8B-14B chat and coding | You want 20B-class dense models resident at q4 |
| You want a learner/tinkerer rig under $300 used | You want long-context 13B-14B without compromise |
| Single-user chat is your primary workload | You batch multiple concurrent requests |
| You can live with q4-q5 on 12B-14B | You want fine-tuning capability (24GB territory) |
Bottom line and recommended pick
Buy the cheapest dual-fan RTX 3060 12GB you can verify. The MSI GeForce RTX 3060 Ventus 2X 12G and the ZOTAC Gaming GeForce RTX 3060 Twin Edge are interchangeable for LLM throughput. Pair the card with the AMD Ryzen 5 5600G, 32 GB of system RAM if you want MoE offload, and a fast NVMe such as the Western Digital 1TB WD Blue SN550 NVMe, so model loading and any spill don't drag on the GPU.
Default model picks for that rig, as of September 2026:
- General-purpose: Gemma 4 12B at Q4_K_M or UD-Q5_K_XL (33.3 tok/s at Q5_K_XL). Use Qwen3.5-9B at Q4_K_M (49.9 tok/s) when you want responsiveness and long context.
- Coding: Qwen 2.5 Coder 14B at q4_K_M (35.8 tok/s) or Qwen3.5-9B for fully resident work. For agentic work, try Qwen3.6-35B-A3B with
--n-cpu-moeexpert offload (22.9 tok/s at IQ3_XXS; 38.9 tok/s at UD-Q4_K_M with-ncmoe 24). - Runtime: Ollama is the easiest, since it handles quant selection and VRAM fitting. Switch to llama.cpp's
llama-serverwhen you need-np 1, a quantized KV cache, or--n-cpu-moe.
Related guides
Citations and sources
- https://x.com/ItsmeAjayKV/status/2062542245719572577: Gemma 4 12B on an RTX 3060 12GB, llama.cpp (June 2026).
- https://www.tyolab.com/blog/2026/05/11-64gb-ram-12gb-vram-the-honest-local-llm-benchmark/: 13 models on an RTX 3060 12GB, llama.cpp (May 2026).
- https://jeanfbrito.github.io/posts/running-qwen3.6-35b-moe-rtx3060-ncmoe/: Qwen3.6-35B-A3B with
-ncmoeoffload on an RTX 3060 12GB. - https://www.hardware-corner.net/gpu-llm-benchmarks/rtx-3060-12gb/: RTX 3060 12GB Qwen3 8B/14B benchmarks and market value (March 2026).
- https://www.hardware-corner.net/gpu-llm-benchmarks/rtx-4070/: RTX 4070 12GB benchmarks and bandwidth.
- https://www.hardware-corner.net/gpu-llm-benchmarks/rtx-5070/: RTX 5070 12GB benchmarks, bandwidth and value.
- https://huggingface.co/google/gemma-4-12B-it: Gemma 4 12B model card.
- https://huggingface.co/unsloth/gemma-4-12b-it-GGUF: Gemma 4 12B GGUF file sizes.
- https://huggingface.co/Qwen/Qwen3.5-9B: Qwen3.5-9B model card.
- https://huggingface.co/unsloth/Qwen3.5-9B-GGUF: Qwen3.5-9B GGUF file sizes.
- https://huggingface.co/Qwen/Qwen3-14B-GGUF: Qwen3 14B GGUF file sizes.
- https://huggingface.co/Qwen/Qwen3.6-35B-A3B: Qwen3.6-35B-A3B model card (35B total, 3B active).
- https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF: Qwen3.6-35B-A3B GGUF file sizes.
- https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF: Llama 3.1 8B GGUF file sizes.
- https://huggingface.co/bartowski/DeepSeek-Coder-V2-Lite-Instruct-GGUF: DeepSeek-Coder-V2-Lite GGUF file sizes.
- https://www.techpowerup.com/gpu-specs/geforce-rtx-3060.c3682: RTX 3060 bandwidth and compute specs.
- https://huggingface.co/models: Community quants and model cards.
- https://github.com/ollama/ollama: Ollama registry and runtime docs.
- Best Local LLMs for 12GB VRAM in 2026 — Atomic Chat (September 30, 2026) — Gemma 4 12B QAT Q4_0: 6.50 GiB file, 7,816 MiB VRAM at 16K, 90.0 tok/s; Qwen3.5-9B Q6_K: 6.95 GiB, 7,276 MiB, 93.6 tok/s (RTX 3080 Ti 12GB, llama.cpp b10988).
- https://insiderllm.com/guides/best-way-run-qwen-3-6-35b-moe-locally/: Qwen3.6-35B-A3B UD-Q4_K_M with
-ncmoe 24on an RTX 3060 12GB, 38.9 tok/s, 9.8 GB VRAM (updated October 15, 2026). - https://huggingface.co/Qwen/Qwen3.8-27B: Qwen3.8 27B model card (27B dense, 262,144-token native context, Apache 2.0).
- https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF: Qwen3.8 27B GSQ-RCO GGUF file sizes and LiveCodeBench v6 scores.
- https://github.com/kadenball/qwen38-27b-rtx3060-dcfr: Qwen3.8 27B IQ3_XXS on an RTX 3060 12GB with an experimental llama.cpp fork and MTP (October 2026).
- PassMark — GeForce RTX 3060 12GB price performance comparison — RTX 3060 12GB price of $459.99 (last price change 2026-10-12).
This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.
- https://www.hardware-corner.net/gpu-llm-benchmarks/rtx-4070-super/: RTX 4070 Super 12GB Qwen3 8B/14B generation at 4K, 16K and 32K context (retrieved October 19, 2026).
- https://www.hardware-corner.net/gpu-llm-benchmarks/rtx-4070-ti/: RTX 4070 Ti 12GB Qwen3 8B/14B generation at 4K, 16K and 32K context (retrieved October 19, 2026).
