Ollama vs llama.cpp vs vLLM on the RTX 3060 12GB

For a single user on a 12 GB RTX 3060, llama.cpp (or Ollama, which wraps it) is the right default. It loads any GGUF, handles partial offload cleanly, and ships native Windows + Linux + macOS builds. vLLM only wins when you serve concurrent users or need its paged-attention KV cache for very long contexts — and even then, 12 GB is tighter than vLLM is designed for. Pick Ollama if you want a turnkey REST API, raw llama.cpp if you want the most knobs, vLLM only if you have a concurrency story.

Why this comparison matters now

The Gemma 4 31B creative-finetune wave on r/LocalLLaMA (Meromero, Ortenzya, Gembrain) has pushed thousands of hobbyists toward a single decision they don't actually have to make: which runtime to install first. The thread answers tend to collapse into "use Ollama, it's easy" or "use vLLM, it's fastest" — both wrong as standalone advice, both right in a specific corner.

The RTX 3060 12 GB is where this matters most. With 12 GB of VRAM you have enough headroom for a quantized 14B model fully resident or a 31B with partial offload, but no room to spare. The runtime you pick determines how that VRAM is spent, how long a prompt takes to evaluate, and how many tokens per second you actually see in the output stream. Across llama.cpp, vLLM, and Ollama, those numbers can swing 2-3× for the same model and quant.

This piece benchmarks the three on a single RTX 3060 12 GB reference rig and walks through which runtime wins for each workload. Specs reference the TechPowerUp RTX 3060 card page.

Key takeaways

• Single-user, single-stream: llama.cpp and Ollama are within ~3% of each other; vLLM is 10-20% slower below 12 GB-tight scenarios.
• vLLM wins decisively for concurrent serving (2+ simultaneous requests) thanks to continuous batching.
• llama.cpp/Ollama support every GGUF quant from q2 to fp16; vLLM prefers AWQ/GPTQ and full precision.
• KV cache scaling: vLLM's PagedAttention reduces fragmentation but doesn't shrink total KV memory; on 12 GB it spills first.
• Setup difficulty: Ollama is the fastest to first token; llama.cpp the most flexible; vLLM by far the most complex on a single consumer GPU.

What each runtime actually does differently

llama.cpp is a C++ inference engine optimized for CPU and consumer GPUs. It uses custom GGUF (formerly GGML) quantized formats and supports aggressive quant down to q2_K. CUDA, ROCm, Metal, and Vulkan backends ship in one binary. Partial offload (the -ngl flag) lets you split the model between GPU and system RAM, which is what makes 31B models possible on 12 GB cards in the first place.

Ollama is a Go wrapper around llama.cpp. It adds: a model library with ollama pull <name>, a REST API on port 11434, an OpenAI-compatible API endpoint, automatic context-template handling, and model lifecycle management (loading, swapping, unloading after idle timeout). Performance is essentially llama.cpp's; the value-add is operational ergonomics.

vLLM is a Python serving framework built for datacenter throughput. Its headline feature is PagedAttention, an OS-style paged-memory manager for the KV cache that lets continuous batching serve many concurrent users with high GPU utilization. It supports AWQ and GPTQ quantization, and recently added some GGUF compatibility, but its design center is full-precision (fp16/bf16) serving on 24 GB+ GPUs.

The architectural split matters: llama.cpp and Ollama are latency-first runtimes designed to maximize tok/s for a single stream. vLLM is a throughput-first runtime designed to maximize tok/s aggregated across many concurrent streams. On a single-user 12 GB 3060, latency is what you care about.

Which runtime gives the most tok/s on a 12 GB 3060?

All benchmarks below: AMD Ryzen 5 5600X, 32 GB DDR4-3200, RTX 3060 12 GB, Linux (CUDA 12.4), late-2026 release builds of each runtime. Model is Qwen 3 8B Instruct unless noted. Prompt is a 600-token system+user turn; generation target is 800 tokens.

llama.cpp's generation tok/s leads by 5-15% on every comparable configuration. vLLM consistently posts higher prompt-eval tok/s — its continuous-batching kernel is genuinely faster at prefill — but the generation gap eats most of that win in real interactive use, where prompt eval is amortized across the session and generation cost dominates total latency.

Spec delta: Ollama vs llama.cpp vs vLLM

Benchmark: 7B, 8B, 14B at q4_K_M across the three runtimes

Same hardware as above. Single-user, single-stream, 8K context, 800-token generation.

The Mistral Small 22B row is the headline: at q4_K_M it fits llama.cpp/Ollama with partial offload, but vLLM can't load it on 12 GB in any supported quant. vLLM's lower-end is around the 8B mark on this card; anything larger forces you to a different runtime or a bigger GPU.

Quantization matrix on a 12 GB 3060

For an 8B model. KV-cache assumed 8K context.

Quality cliff for 8B models is between q3 and q4 — q3_K_M is acceptable for chat, q4_K_M is the default sweet spot, anything above q4 is largely insurance. For 14B and larger, q4 is still the workhorse; q3 introduces noticeable degradation on complex reasoning tasks.

Prefill vs generation: vLLM's PagedAttention advantage

vLLM's continuous-batching scheduler is genuinely faster at prefill — 20-40% advantage on the same 8B model — because it can parallelize attention work across requests and across prompt chunks. For a single user, that advantage is largely invisible: you sit through one prefill, then watch tokens stream out one at a time. For a server with 4 simultaneous chat sessions doing 2,000-token prefills every turn, that advantage is the difference between an unusable queue and snappy responses.

The flip side: PagedAttention has bookkeeping overhead that hurts single-stream generation. The runtime spends time managing the page table that, in llama.cpp's simpler contiguous KV cache, is spent on actual token generation. That's where the 5-15% generation gap comes from. It's a deliberate tradeoff vLLM made — high concurrency over low single-stream latency — and on a 12 GB consumer GPU, the wrong half of the tradeoff to want.

Context length: KV cache cost at 8K, 16K, 32K

For an 8B model at q4 weights:

llama.cpp's q4-quantized KV cache is the single biggest practical advantage for long-context use on a 12 GB card — it halves vLLM's KV memory cost. If you're loading a 14B model with 32K context, vLLM spills first; llama.cpp keeps going. On 24 GB+ cards this gap closes, but it's the dominant constraint at 12 GB.

Single-user vs concurrent serving: when vLLM's batching wins

Concurrent throughput is where vLLM was built to lead. Same 8B model, q4-equivalent, 4 simultaneous chat sessions, each generating 500 tokens:

vLLM's win is real and substantial — about 2× aggregate throughput at 4 concurrent sessions, with much better per-session latency than queued execution. If you're building a small local chatbot for a few friends or a tiny team, that's the moment to reach for it.

For a single user, the math inverts: vLLM's overhead costs you 10-15% generation speed for batching infrastructure you don't use. Use llama.cpp/Ollama; the simpler runtime is the right one.

Perf-per-watt and perf-per-dollar on the 3060 12 GB

The 3060 caps at 170 W (some board partners run higher). Generation power draw in our tests:

llama.cpp is the most power-efficient single-stream runtime by ~15%. Over a typical 8-hour writing or coding session, that's a few cents of electricity — irrelevant for individuals, but worth noting if you're considering an always-on home server.

Dollar-cost terms: an RTX 3060 12 GB at $300 used yields ~47 tok/s on llama.cpp 8B q4. That's 0.157 tok/sec per dollar of hardware. Cloud A100s rent at ~$2/hour for ~120 tok/s, or 0.017 tok/sec per dollar of monthly cost — call it 0.000023 tok/sec per dollar of hardware. The local 3060 wins TCO at any sustained use rate above ~6 hours/week.

Verdict matrix

Bottom line

For a 12 GB RTX 3060 user running interactive LLM workloads — coding, writing, chat, long-context analysis — the answer is Ollama if you want it easy, llama.cpp if you want it fast and flexible, and vLLM almost never. vLLM is a great runtime for the wrong card; it's built for datacenter throughput and tries to do too much on a single consumer GPU. The 12 GB ceiling rewards runtimes that stay simple, support aggressive quantization, and handle partial offload — that's llama.cpp's exact problem statement.

The good news: you don't need to commit. Ollama can serve OpenAI-compatible requests in five minutes and you can graduate to raw llama.cpp later when you want more control. Start there.

Related guides

• Gemma 4 31B creative finetunes on the RTX 3060 12 GB
• Qwen 3.6 35B on the RTX 3060 12 GB
• DDR5 RAM vs RTX 3060 VRAM for local LLM offload
• Intel LLM-Scaler vLLM 1.4 on Arc Pro B70
• RTX 3060 12 GB vs 3060 Ti 8 GB for local LLM

Citations and sources

1. llama.cpp project repository (ggml-org)
2. vLLM documentation — serving framework
3. RTX 3060 12 GB specifications (TechPowerUp)