llama.cpp vs Ollama vs vLLM on a 12GB RTX 3060: Which Wins in 2026?

On a 12GB RTX 3060, the fastest single-user runtime for local LLMs is llama.cpp — either directly or through Ollama, which wraps it. vLLM's paged-KV batching is engineered for concurrent serving and doesn't help a single desktop user, so its wins collapse the moment the workload is one person chatting. Pick Ollama for setup speed, raw llama.cpp for fine control, and vLLM only if you're serving a small team over HTTP.

Desktop chat vs server serving — why the answer differs

The three runtimes were built for different goals, and comparing them without naming the goal produces bad advice.

• llama.cpp is a C/C++ inference engine targeting single-stream generation with a heavy focus on quantized model support (GGUF format), CPU offload, and cross-platform builds. It's the runtime that made 7B models runnable on a laptop, and its git history reads like a bandwidth-optimization diary.
• Ollama wraps llama.cpp with a small Go daemon, a model registry (ollama pull llama3.5), an HTTP API compatible with OpenAI's chat completions endpoint, and automatic GPU-layer allocation. It's the "sudo apt install llm" of the space.
• vLLM is a Python-native serving system built at UC Berkeley. Its key trick is PagedAttention — a KV-cache implementation that lets many concurrent requests share memory efficiently. It's what you deploy behind a small team's chat UI or an internal RAG system with multiple simultaneous users.

For the "one user, one 3060, one prompt at a time" workload that most people actually have, the batching wins vLLM was built for don't materialize. Read the vLLM performance blog and you'll see the throughput bars are always plotted at 8, 16, or 32 concurrent requests — because that's where PagedAttention shines. At batch size 1, llama.cpp is competitive on tok/s and dramatically simpler to run.

The rest of this piece walks the head-to-head on a representative 3060 build — MSI RTX 3060 Ventus 3X 12G, Ryzen 7 5800X, 64GB DDR4-3600, Samsung 970 EVO Plus 1TB NVMe — using numbers gathered from public community measurements on r/LocalLLaMA and the runtimes' own release notes.

Key takeaways

• Easiest to install: Ollama by a mile. curl install.sh | sh; ollama run llama3.5 gets you chatting.
• Fastest single-user: llama.cpp at tuned settings, ~5% ahead of Ollama at the same quant. Ollama's overhead is real but small.
• Best for concurrency: vLLM, but only at 4+ concurrent requests. Under that, it loses to llama.cpp.
• Widest quant support: llama.cpp + Ollama share GGUF. vLLM prefers safetensors with narrower quant options.
• Best OpenAI-API compat: Ollama and vLLM tie; llama.cpp needs a wrapper.
• Worst installation: vLLM. Python + CUDA + specific PyTorch = "hope the versions align."

Which is easiest to install and keep updated?

Ollama's installer is idempotent and version-pinned to a bundled llama.cpp commit. This is a real feature: the runtime you install today will still run the model you installed today, six months from now. Raw llama.cpp changes fast enough that a model file working on release N-3 might have a slightly different perf profile on release N.

vLLM's install is finicky. Its CUDA and PyTorch pinning are strict; a mismatched driver or kernel-headers version produces cryptic errors. On Windows, WSL2 is not optional — vLLM does not target native Windows CUDA.

Single-user tok/s on a 12GB RTX 3060

Community measurements on 7B-14B GGUF models at Q4_K_M, single stream, no concurrent requests:

Ollama sits ~5-8% behind raw llama.cpp — the daemon's overhead is small but non-zero. vLLM sits another 5-10% behind at batch size 1, exactly the outcome its architecture predicts.

At batch size 8 (simulated concurrent requests), vLLM leaps ahead:

That's the vLLM story: for 8 concurrent users, it beats Ollama by ~3x aggregate throughput. For one user, it loses by ~5%.

Does vLLM's batching help a single local user?

No. Batching wins throughput by amortizing the KV-cache prefill and attention compute across multiple sequences. If there's only one sequence, there's nothing to amortize. What you get instead is:

• Slightly higher per-token latency (paged-attention bookkeeping isn't free).
• Higher idle VRAM footprint — vLLM reserves the paged-KV pool up front.
• Setup complexity you didn't need.

vLLM has a place, and that place is a self-hosted chat endpoint used by more than one person. Below that, it's the wrong tool.

Feature-delta table

The quant story matters most on 12GB. llama.cpp's Q4_K_M and IQ2_XS quantizations of 30B-class models are what let a 3060 punch above its VRAM. vLLM's supported quant schemes are narrower and the AWQ/GPTQ paths require pre-conversion, which is a real friction point compared to ollama pull qwen3.6:8b-instruct-q4_K_M.

Context length and KV-cache on a 12GB budget

KV-cache footprint scales with n_layers × hidden_size × n_ctx × 2. For an 8B model at 8k context:

vLLM's paged allocator reserves a pool sized to expected total context across all requests. For one request, that's overhead. Ollama and llama.cpp allocate per-request; you get exactly what you use.

At 32k context on a 3060, the math starts hurting no matter the runtime, but llama.cpp's --n-ctx 32768 --flash-attn combo is the tighest configuration for making it work.

Perf-per-watt and idle-VRAM

With a model loaded but idle (waiting for a prompt), a 3060 sits at ~15-20W and 6-8GB of VRAM reserved, depending on runtime.

Under generation:

Nothing dramatic. All three saturate the memory subsystem before they saturate the compute pipeline, which is the standard 3060 story on modern LLM workloads.

Real-world scenarios and recommendations

Scenario 1: Solo developer, wants an OpenAI-compatible local endpoint for VS Code / Continue. Pick Ollama. ollama serve gives you http://localhost:11434/v1/chat/completions. Done in five minutes.

Scenario 2: Power user, want to tune every knob and squeeze the card. Pick raw llama.cpp. Explicit --n-gpu-layers, --n-ctx, and quant selection lets you nose an extra 5-8% out of the same hardware.

Scenario 3: Small team, want a shared internal chat with 3-8 concurrent users. Now vLLM makes sense. Its batching wins scale with concurrency, and its OpenAI-compatible server is production-grade. Put it behind a reverse proxy and don't look back.

Scenario 4: Windows-only environment, occasional user. Ollama's Windows build is the least-friction path. LM Studio is a legitimate alternative if you want a GUI.

Scenario 5: You want to build the exact runtime yourself. llama.cpp is your only real choice. Fork the repo, hack on it. The build times are seconds, the code is legible C++.

Common pitfalls

1. Choosing vLLM because it "sounds professional". Then getting stuck on Python + CUDA version pinning. If you're a single user, don't.
2. Running raw llama.cpp on Windows without -DGGML_CUDA=ON. You'll get CPU inference at 2 tok/s and blame the card.
3. Comparing Ollama's default context (2048) with llama.cpp's tuned 8192. Set context explicitly on both to compare like-for-like.
4. Downloading .safetensors weights for Ollama. Ollama wants GGUF. Convert once, use forever.
5. Assuming vLLM's throughput numbers scale linearly. They don't; you need 4-8 concurrent requests to see the paged-attention wins.

When NOT to pick each

• Don't pick raw llama.cpp if you need OpenAI-API compatibility. The wrappers exist but Ollama gives you the same for free.
• Don't pick Ollama if you need to hot-swap between eight different quant variants of the same model at low latency. Its model-registry model isn't designed for that.
• Don't pick vLLM unless you have real concurrency. Its overhead is real for single users.

Bottom line

For a 12GB RTX 3060 running local LLMs today: install Ollama, run Llama 3.5 8B or Qwen 3.6 14B at Q4_K_M, expose the OpenAI API to whatever client you use, done. That covers 80% of solo desktop use.

Reach for raw llama.cpp when you need the last 5-8% of throughput or want to run a bleeding-edge model that isn't in the Ollama registry yet. Reach for vLLM when you have concurrent users and are ready to accept the setup complexity for the batching wins.

The hardware baseline for all three is the same: an MSI RTX 3060 Ventus 3X 12G paired with a Ryzen 7 5800X or Ryzen 5 5600G, 64GB DDR4-3600, and a Samsung 970 EVO Plus NVMe for fast model loads.

Real-world workflow scenarios revisited

Three concrete workflows worth walking through in detail:

Continue.dev + Ollama for local code completion. Configure Ollama to serve Qwen 3.6 8B Q4_K_M at localhost:11434. Continue.dev's config points at the Ollama endpoint; VS Code lights up with local autocomplete. On a 3060 the tab-latency lands around 300-400ms — noticeably slower than a cloud API's 50-100ms but usable. The privacy story is what earns this workflow; there's no code leaving your machine.

Home Assistant voice pipeline with a local LLM. Home Assistant's Assist pipeline supports Ollama as an LLM backend. Serve a 7B model, wire the STT/TTS integrations, and voice commands go end-to-end local. On a 3060 with Whisper.cpp for STT + Piper for TTS + Ollama for LLM, round-trip latency lands around 2-3 seconds — acceptable for voice-first home automation.

Automated document summarization with vLLM. For a small team's shared internal tool that summarizes uploaded PDFs, vLLM's throughput wins matter. Set up an Ubuntu box with a 3060, Docker, and vLLM's OpenAI-compatible container. Point five colleagues at it. Aggregate throughput at 5-8 concurrent requests genuinely beats what Ollama offers when serialized.

Framework tradeoffs summarized

Related reading

• Run Qwen 3.6 27B Locally on a 12GB RTX 3060
• RTX 5090 AI Build Guide
• Best Mini PC for Local LLMs in 2026

Citations and sources

• llama.cpp — official GitHub repository
• Ollama — releases and install docs
• vLLM — official documentation
• r/LocalLLaMA — community measurements and hardware threads

This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.