llama.cpp on a 12GB RTX 3060: llama-bench Setup and Measured tok/s

The short path to benchmarking llama.cpp on an MSI GeForce RTX 3060 Ventus 3X 12G: build with the CUDA backend, download a GGUF model, run llama-bench -m model.gguf -ngl 999 -p 512 -n 128 -fa 1, and read the two throughput numbers — prompt processing (prefill) and text generation. Prefill dominates first-token latency on long prompts; generation is what you feel during interactive chat.

Why benchmark before you tune

llama-bench is the reference tool for measuring llama.cpp throughput on a given hardware+model+quantization+flag combination. It exists precisely because "how fast does this run?" is a moving target — a driver update, a llama.cpp release, a flag change, or a model swap all move the number. Establishing a repeatable baseline lets you attribute changes correctly instead of chasing anecdotes.

On a 12 GB card like the RTX 3060, you want to know how much your n-gpu-layers choice matters, whether flash attention is helping, and how a Q5 quant compares to Q4 on your specific rig. llama-bench is the answer to all of those questions in seconds per run.

This piece walks through building llama.cpp with CUDA, running llama-bench correctly, reading its output, and comparing common models on the 3060.

Key takeaways

• Build llama.cpp with GGML_CUDA=ON and match the CUDA arch to your GPU (Ampere = sm_86 for the 3060).
• llama-bench reports two throughput numbers: prompt processing (pp) and text generation (tg), both in tok/s.
• Offload as many layers as VRAM allows — -ngl 999 requests all layers; the runtime clamps to what fits.
• Flash attention (-fa 1) generally improves throughput and reduces VRAM use on Ampere; benchmark to confirm on your build.
• Community-reported 3060 12 GB numbers on Qwen2.5-14B-Q4_K_M land at ~600–700 tok/s pp and ~35–45 tok/s tg.

What you need

• MSI RTX 3060 Ventus 3X 12G (or any 12 GB 3060)
• Any modern desktop CPU — an AMD Ryzen 7 5800X or Ryzen 5 5600G is fine for the benchmark itself
• CUDA Toolkit 12.4 or newer (nvcc)
• gcc/clang, cmake ≥ 3.14
• Git
• ~30 GB of free disk for models
• A working NVIDIA driver (verify with nvidia-smi)

Building llama.cpp with CUDA support

The build has stabilized around CMake in 2026. The canonical steps:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86
cmake --build build --config Release -j

CMake arch 86 is Ampere (RTX 30-series). If you're on a different GPU, adjust: 80 for A100, 89 for Ada (RTX 40-series), 120 for Blackwell (RTX 50-series).

Confirm the build:

./build/bin/llama-cli --help | head -5
./build/bin/llama-bench --help | head -5

Both binaries should exist and report their supported flags. On start of any inference command, llama.cpp logs whether it detected the GPU — check nvidia-smi from another terminal during a run to verify VRAM usage climbs. If it doesn't, you're on the CPU path and your numbers will be dramatically lower.

Downloading a model

Grab a GGUF model from Hugging Face. Common defaults for a 3060 12 GB rig:

• bartowski/Qwen2.5-Coder-7B-Instruct-GGUF — pick the Q6_K file
• Qwen2.5-Coder-14B-Instruct-GGUF — pick Q4_K_M
• Meta-Llama-3.1-8B-Instruct-GGUF — pick Q6_K

Use huggingface-cli download <repo> --include "<file>" to fetch, or curl the direct URL. GGUF files are single-file bundles of weights + tokenizer + metadata; no additional config is needed.

Running llama-bench

The reference command:

./build/bin/llama-bench -m models/qwen2.5-coder-14b-instruct-q4_k_m.gguf -ngl 999 -p 512 -n 128 -fa 1

Flag meanings:

• -m: path to the GGUF model
• -ngl 999: offload up to 999 layers to the GPU. In practice llama.cpp clamps to the actual layer count; setting the number high tells it to use every layer. On a 12 GB card running a 14B model, all layers fit.
• -p 512: prompt-processing test with a 512-token prompt
• -n 128: token-generation test producing 128 tokens
• -fa 1: enable flash attention

Expected output (abridged):

| model | size | params | backend | ngl | fa | test | t/s |
|--------------------------------|-----------:|-----------:|------------|----:|---:|-----:|------------------:|
| qwen2 14B Q4_K - Medium | 8.37 GiB | 14.77 B | CUDA | 999 | 1 | pp512 | 650.31 ± 3.42 |
| qwen2 14B Q4_K - Medium | 8.37 GiB | 14.77 B | CUDA | 999 | 1 | tg128 | 41.82 ± 0.15 |

The two rows are the two measurements:

• pp512: prompt processing at 512 tokens — how fast the model ingests the input. On the 3060 12 GB for Qwen2.5-Coder-14B Q4_K_M this lands around 600–700 tok/s.
• tg128: text generation for 128 tokens — how fast the model produces output. Around 35–45 tok/s for the same model.

The ± numbers are standard deviations across the default 5-run repeats. Wide standard deviations indicate thermal throttling, background load, or a driver issue.

Benchmark table: measured tok/s on the RTX 3060 12 GB

Community-reported numbers on llama.cpp b3900+ with CUDA on an MSI RTX 3060 12 GB paired with a 5800X and 32 GB DDR4-3200, all with flash attention on:

The pattern: smaller models are prefill-bound and hit ~1000+ tok/s pp with 55+ tok/s tg. The 14B tier at Q4 is the ceiling that fits comfortably on 12 GB, and its tg rate is right at the boundary between "interactive" and "slow but usable". Q5 on a 14B pushes VRAM close to the limit and starts to slow down as the runtime handles memory more carefully.

Quantization matrix: what fits on 12 GB, at what speed

Higher quants of 14B don't just eat VRAM — they slow generation because the runtime pays more per operation. Q4_K_M is the sweet spot on 14B for the 3060.

Prefill vs generation: interpreting the two llama-bench numbers

The two throughput numbers tell you different things:

• pp (prompt processing / prefill): GPU is compute-bound. High-parallelism forward pass over every input token. Determines time-to-first-token on any prompt that isn't cached.
• tg (text generation): GPU is memory-bandwidth-bound. One-token-at-a-time autoregressive loop. Determines the felt speed of streaming output during chat.

For a chat workload where prompts change often, both numbers matter. For a chat workload where the system prompt is stable and gets cached, only tg matters after the first turn.

Tuning: n-gpu-layers, batch size, flash attention

Three flags move the needle materially on the 3060:

• -ngl (n-gpu-layers): set to 999 for models that fit. Reduce only if the model+ctx exceeds 12 GB and you want partial CPU offload.
• -b (batch size): affects prefill throughput. Higher is generally faster up to a memory-bound ceiling; the default 512 is a fine starting point.
• -fa 1 (flash attention): reduces attention VRAM use and generally increases throughput on Ampere. Benchmark on/off to confirm on your build.

Advanced flags worth exploring once you're comfortable:

• -ctv q4_0 -ctk q4_0: quantize the KV cache to Q4, which halves KV VRAM and lets you push context length higher on tight budgets.
• -t <threads>: CPU thread count for the parts that stay on CPU. Match to your physical core count; hyperthreading rarely helps here.
• --split-mode row: for multi-GPU rigs (not relevant on a single 3060).

Common pitfalls when benchmarking

1. Accidentally CPU-bound. If -ngl is unset or too low, layers stay on the CPU and numbers drop 10–20×. Confirm nvidia-smi shows VRAM usage.
2. Cold cache runs. The first llama-bench run pays for model load and CUDA kernel compilation. Discard the first-run number or use llama-bench's built-in warmup.
3. Thermal throttling. A poorly-cooled 3060 under sustained runs drops 5–15% off headline numbers. Confirm GPU temp stays under 75°C during runs.
4. Background load. A video encode or browser tab in the background will show up as high variance in llama-bench's std-dev column. Close everything.
5. Wrong llama.cpp version. Numbers move meaningfully across b3600 → b3900 → current. Cite the version you're running when comparing.

The settings that maximize 3060 throughput

Reference config for a 3060 12 GB running Qwen2.5-Coder-14B Q4_K_M:

• -m qwen2.5-coder-14b-instruct-q4_k_m.gguf
• -ngl 999
• -fa 1
• -b 512
• Context length 4096–8192 in real use (llama-bench's -p sets prompt size for the test, not runtime ctx)

That gets you the ~650 tok/s prefill and ~42 tok/s generation numbers cited above. For a 7B model swap in Q6_K and expect ~1000 tok/s pp / 55 tok/s tg.

Any driver, CUDA, or llama.cpp version change should trigger a fresh benchmark run. Numbers move; measure them.

Related guides

• Kimi K3 Lands #5 on Coding Agents: What You Can Actually Run Local
• Local LLM Use Cases in 2026: What a 12GB Rig Actually Delivers
• Best CPU for the MSI RTX 3060 12GB: 5600G vs 5700X vs 5800X
• Run Qwen Locally: Apple Silicon vs a 12GB RTX 3060 Rig in 2026

Citations and sources

• llama.cpp — official repository (build instructions and llama-bench source)
• NVIDIA — CUDA Toolkit download and documentation
• TechPowerUp — GeForce RTX 3060 GPU database entry

This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.