llama.cpp Vulkan on the RTX 3060 12GB: Setup and Measured tok/s for 2026

llama.cpp on an RTX 3060 12GB delivers roughly 40-70 tokens per second on a 7B model at Q4_K_M with all layers offloaded, according to widely referenced community benchmark threads on the ggml-org repository. The Vulkan backend gets close to that with the added benefit of one build that also runs on AMD and Intel GPUs. The CUDA build is still a few percent faster on NVIDIA, but Vulkan is the right pick if you switch cards or need a portable rig.

Why llama.cpp is the default local runtime

llama.cpp is the runtime that made local LLM inference feasible on consumer hardware. It runs anywhere, requires no framework install, and its GGUF weight format has become the de facto exchange format across the local-AI ecosystem. Every major open-weight model release now ships GGUF quantizations within days of the base weights, and every notable local runtime — Ollama, LM Studio, KoboldCpp, text-generation-webui — either wraps llama.cpp directly or shares its GGUF format.

The Vulkan backend is the newer story. Historically llama.cpp built with either CPU-only paths or vendor-specific accelerators (CUDA for NVIDIA, ROCm for AMD, Metal for Apple). Vulkan added a portable GPU backend that runs on any card with a modern Vulkan driver — NVIDIA, AMD, Intel Arc, mobile GPUs — from a single build. That is useful when you rotate cards, deploy to a mixed fleet, or want to avoid dependency hell.

The GIGABYTE RTX 3060 Gaming OC and MSI RTX 3060 12GB Ventus 3X are the two most common 12GB Ampere cards for this workload, both list under a modern midrange Ada card, and both fit the same performance envelope. What follows is a setup path plus what to expect at the other end.

Key takeaways

• 7B Q4_K_M offloads fully into 12GB with room for a chat context.
• CUDA build: roughly 45-70 tok/s on 7B Q4 on Ampere.
• Vulkan build: roughly 35-60 tok/s on the same model — a few percent behind.
• 13B fits at Q4 with less headroom; 32B partial offload only.
• KV cache growth is the sneaky ceiling on long-context chat.

What does the Vulkan backend do differently from CUDA?

Both backends do the same work — matrix multiplies for the transformer layers, activation functions, KV cache management — but they target different low-level APIs.

The CUDA backend calls NVIDIA's cuBLAS and its own hand-tuned kernels through the CUDA runtime. It is the reference implementation on NVIDIA hardware and remains a few percent ahead in raw throughput on Ampere and Ada cards.

The Vulkan backend uses SPIR-V compute shaders through the Vulkan compute API. That layer is vendor-neutral: the same binary runs on NVIDIA, AMD, and Intel GPUs with only driver differences underneath. Historically it lagged CUDA meaningfully on NVIDIA hardware, but the gap has narrowed with each release, and on the RTX 3060 12GB the practical difference is small enough that portability wins for many builders.

Two practical trade-offs are worth knowing:

• Vulkan avoids the CUDA toolkit install. That is a real benefit on machines where CUDA versioning is a hassle or where you swap distributions.
• Vulkan is worse than CUDA at extremely small batch sizes on very short prompts. The overhead per compute submission is higher. It gets better at longer contexts.

Step-by-step: building llama.cpp with Vulkan for the RTX 3060

The full authoritative reference lives at the llama.cpp repository. The condensed path:

1. Install the Vulkan SDK. On Linux, use your distribution's vulkan-tools and libvulkan-dev. On Windows, install the LunarG Vulkan SDK. Confirm with vulkaninfo — the RTX 3060 12GB should list as GeForce RTX 3060 with a full feature set.
2. Clone and build llama.cpp. git clone https://github.com/ggerganov/llama.cpp && cd llama.cpp and then build with the Vulkan flag: cmake -B build -DGGML_VULKAN=ON && cmake --build build --config Release. On Windows, cmake --build build --config Release --parallel uses your available cores.
3. Grab a GGUF. Any recent 7B chat model from Hugging Face at Q4_K_M is a good first target. wget it into the llama.cpp models directory.
4. Run with all layers on the GPU. ./build/bin/llama-cli -m models/your-model.gguf -ngl 999 -p "Hello" — the -ngl 999 tells llama.cpp to offload as many layers as fit. On a 7B Q4 model with 12GB, all layers offload.

If everything is wired up, the console prints tokens per second at the end of generation. Cross-check that number against the community-published Vulkan bands below.

How many tokens per second per model size?

Public throughput bands for llama.cpp Vulkan on the RTX 3060 12GB. Numbers vary by clock, thermals, and llama.cpp release. All values are for Q4_K_M unless noted.

Two calibrations for reading the table:

• Tok/s is generation throughput, not end-to-end request latency. Prefill on a long prompt runs faster than generation on both backends.
• Full offload matters more than the backend choice. A 7B at Q4 fully on the GPU beats a 13B partially offloaded on either backend, by a wide margin.

Quantization matrix on the RTX 3060 12GB

Quantization sets VRAM footprint and impacts quality. For a 7B model, the standard set of quants against a 12GB card:

Q4_K_M is the standard recommendation because it balances quality retention and VRAM footprint. Q5 or Q6 are worth it only for high-precision code or math workloads where the marginal quality improvement matters and you have the headroom.

Prefill vs generation split on the RTX 3060

Two distinct phases matter:

• Prefill processes the input prompt in parallel. It is compute-bound and hits the tensor cores hard. On a warm 3060 12GB with a 7B Q4 model and Vulkan, prefill lands in the multi-hundred-tokens-per-second range for typical prompt lengths.
• Generation produces new tokens one at a time. It is memory-bandwidth-bound. On the same setup it lands in the 35-60 tok/s range.

The practical consequence: a long system prompt with a short reply feels almost instant. A short prompt with a long reply — which is what a real chat session looks like — runs at generation throughput. Optimize for the phase your workload actually spends time in.

Context-length impact: how KV cache eats your 12GB

Each token in the context window contributes to the KV cache footprint. On a 7B model at Q4, the KV cache adds roughly the following per 1K tokens of context (approximate, based on model architecture):

For a single 7B Q4 model the practical context ceiling is around 32K on a 12GB card. Push past that and you have to drop precision, reduce KV precision (llama.cpp supports quantized KV cache), or step down to a smaller model. Do not treat the model VRAM footprint at zero context as your budget — the KV cache is real and grows with the workload.

Storage: does a fast SSD help?

Model loading benefits from NVMe, but inference itself is not disk-bound. Once a GGUF file is in RAM or VRAM, disk speed no longer matters. A Samsung 970 EVO Plus 250GB NVMe is a good boot-plus-active-model drive; pair it with a larger SATA library if you swap between many multi-gigabyte models. Cold-start load time for a 4.5GB Q4 model is a few seconds on NVMe versus a few tens of seconds on a slow SATA drive.

Verdict matrix

Use the Vulkan build if...

• You want one binary that runs on NVIDIA and AMD.
• You are on a fresh distribution and CUDA install is friction.
• You do not want to reinstall CUDA when you upgrade drivers.
• You are willing to accept a small throughput trade-off.

Stick with the CUDA build if...

• You are permanently on NVIDIA hardware.
• You are optimizing for the last few percent of throughput.
• You already run CUDA for other workloads and the toolkit is installed.
• Extremely short prompts are your dominant workload shape.

Common pitfalls

1. Vulkan driver too old. llama.cpp requires Vulkan 1.2 or newer features. On Linux, use a recent mesa or NVIDIA driver; on Windows, keep the vendor driver current.
2. Forgetting -ngl 999. Without an offload flag, llama.cpp keeps the model on CPU. All-layer offload is the default recommendation for the 7B Q4 sweet spot on the 3060 12GB.
3. Testing with FP16 on a 7B model. FP16 does not fit with any usable context. Use Q4_K_M as your baseline.
4. Assuming CUDA is universally faster. The gap narrows for long contexts and vanishes for some very recent llama.cpp releases. Retest whenever you upgrade.
5. Ignoring thermal throttling. The MSI Ventus 3X has a triple-fan cooler; the GIGABYTE Gaming OC is dual-fan. Both are fine in a case with reasonable airflow, but a poorly-vented mini-tower will cost throughput.

When NOT to use Vulkan

• You are on Apple Silicon. Use the Metal backend.
• You are on AMD Instinct. ROCm HIP is the tuned path.
• You need every last percent on NVIDIA. CUDA still wins by a hair.

Related guides

• RTX 3060 12GB Local LLM Deepseek v4 GLM 2026
• vLLM Single RTX 3060 12GB Batched Serving
• Baidu Unlimited OCR Local RTX 3060 12GB
• 32k Context VRAM Budget RTX 3060 12GB

Bottom line

The RTX 3060 12GB with llama.cpp Vulkan is the cheapest reliable path to real local LLM inference in 2026. You get 35-60 tokens per second on the 7B tier, comfortable 8K to 16K context, and a build that will keep working if you swap cards in the future. Pair it with a Ryzen 7 5800X for fast prefill and cold starts, a Samsung 970 EVO Plus NVMe for quick model loads, and 32 GB of DDR4 for KV overflow headroom. The result is a rig that does everything most local-AI users actually do, at a total build cost most cloud teams burn in a month.

Citations and sources

• llama.cpp repository — canonical documentation for build flags, GGUF format, and community benchmarks.
• TechPowerUp — RTX 3060 GPU specs — Ampere card details.
• Phoronix llama.cpp coverage — Linux-side benchmark posts referenced above.
• NVIDIA GeForce RTX 3060 official page — vendor spec reference.

This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.