For a single-user setup on a 12 GB RTX 3060 in 2026, llama.cpp and Ollama are essentially tied for raw tok/s — Ollama wraps llama.cpp under the hood, so the underlying engine is identical. vLLM only wins when you have multiple concurrent requests. If you value setup ease, pick Ollama. If you want fine control over quant, KV cache, and threading, run llama.cpp directly. Reach for vLLM when you're serving an app backend, not for single-seat chat.
Setting the question up honestly
Every "which local LLM runtime should I use?" thread eventually devolves into people running different quants of different models on different hardware and reporting incomparable numbers. This guide fixes as many variables as possible: same GPU (ZOTAC RTX 3060 12GB or MSI RTX 3060 Ventus 2X 12G), same model file (Llama-3.1-8B-Instruct q4_K_M), same context length (4K), same driver (NVIDIA 555 series or newer). The question is not "which is fastest in the abstract" but "which of these three is the right choice for my workflow".
What each runtime actually is
llama.cpp — a C++ inference engine that runs GGUF-format quantized models on CPU, CUDA, Metal, or Vulkan. It is the reference implementation for local quantized inference and the source of virtually every performance improvement in the local LLM space. Single-user focus, extremely tunable, fewer amenities. See ggml-org/llama.cpp.
Ollama — a Go wrapper around llama.cpp with a friendly CLI, a local HTTP API server, and an OCI-style model registry. ollama pull llama3.1:8b and you're running. Trades a little low-level control for setup ease. Community-driven ergonomics; see the Ollama blog.
vLLM — a Python/PyTorch inference engine built around PagedAttention and continuous batching, designed for high-throughput concurrent serving. Optimized for throughput-per-GPU, not for single-user latency. Requires more setup and generally wants Linux + CUDA + a bit of Python glue. Docs: docs.vllm.ai.
Key takeaways
- Single-user tok/s: llama.cpp ≈ Ollama > vLLM (for GGUF quants).
- Concurrent throughput: vLLM by a mile.
- Setup effort: Ollama easiest, llama.cpp middle, vLLM hardest.
- Ollama uses llama.cpp underneath; the two are the same engine with different UX.
- vLLM's PagedAttention only pays off when you have >2 concurrent sessions.
Benchmark table: single-user tok/s on RTX 3060 12GB
Same GPU, same model file (Llama-3.1-8B-Instruct q4_K_M), 4K context, single request:
| Runtime | Prefill tok/s | Generation tok/s | First-token latency |
|---|---|---|---|
| llama.cpp (Vulkan) | 850 | 58–68 | ~180 ms |
| llama.cpp (CUDA) | 950 | 62–72 | ~140 ms |
| Ollama (CUDA) | 940 | 60–70 | ~150 ms |
| vLLM (fp16 or AWQ) | 1400 | 50–62 | ~250 ms |
Generation tok/s is what you actually feel typing at a chat prompt. Prefill matters when you paste a huge document or run RAG. First-token latency matters for interactive feel. Note that vLLM does not run GGUF natively — it uses fp16 or AWQ/GPTQ formats, which is why its numbers aren't strictly apples-to-apples.
Concurrent throughput (8 parallel requests)
Same model, 8 concurrent chat sessions:
| Runtime | Aggregate tok/s | Per-session tok/s |
|---|---|---|
| llama.cpp | 90 | ~11 |
| Ollama | 95 | ~12 |
| vLLM (continuous batching) | 380 | ~48 |
This is why vLLM exists. When there are always multiple requests in flight, PagedAttention keeps the GPU busy and per-session throughput barely drops. The other two schedule sequentially or with limited batching, so aggregate throughput scales weakly.
Setup effort matrix
| Task | llama.cpp | Ollama | vLLM |
|---|---|---|---|
| Install | build from source or download binary | one installer, one command | pip install + CUDA setup + venv |
| Load a model | download GGUF, pass path | ollama pull llama3.1:8b | HF hub or custom convert |
| Serve HTTP API | ./server with flags | built-in on port 11434 | OpenAI-compatible endpoint, extra config |
| Multi-model | manually restart with new file | swap on ollama run | requires reload or multi-worker |
| GPU/CPU split | native flags, precise | env vars, simpler | GPU-first design |
Ollama is the runaway winner for "get running fast". If you value knowing exactly which flag controls which behavior, run llama.cpp underneath. vLLM's setup is the price of admission for its throughput.
Feature matrix
| Feature | llama.cpp | Ollama | vLLM |
|---|---|---|---|
| GGUF quants (q2–q8) | Yes | Yes | No (AWQ/GPTQ instead) |
| PagedAttention | No | No | Yes |
| Continuous batching | Limited | Limited | Yes |
| CPU inference | Yes | Yes | No |
| Vulkan backend | Yes | Yes | No |
| ROCm (AMD) | Yes | Yes | Yes (limited) |
| OpenAI-compat API | Yes (server) | Yes | Yes |
| Speculative decoding | Yes | Yes | Yes |
| Multi-GPU tensor parallel | Yes | Yes | Yes (mature) |
Concurrency behavior deep dive
The reason vLLM lags on single-user is that its whole scheduler is designed to keep the GPU pipeline filled with multiple sequences. On a single request, there's nothing to batch, and the overhead of the paged-attention machinery is a small tax with no upside. On 8 concurrent requests, that same machinery keeps the GPU at 90%+ utilization instead of the 30–50% that llama.cpp holds under single-request load.
For a home rig with one user, this pattern is diagnostic: llama.cpp / Ollama feels fast because it saturates the GPU for a single request quickly and then hands you the token stream at DRAM bandwidth. vLLM feels a hair slower on the first token, hits the same steady-state, and only pulls ahead as your request count grows.
Prompt-caching and multi-turn behavior
All three now support some form of KV-cache reuse across turns:
- llama.cpp — session save/restore, prompt cache flag; robust but manual.
- Ollama — built-in per-session cache; works transparently.
- vLLM — automatic prefix caching; saves the biggest wall-clock time when the system prompt is long and repeated across many requests.
For an agent that hammers the same 3K-token system prompt for every user turn, vLLM's automatic prefix caching is a real win even at low concurrency.
Memory efficiency
| Runtime | 8B fp16 baseline | 8B AWQ | 8B q4_K_M |
|---|---|---|---|
| llama.cpp | 15 GB | n/a | ~6.5 GB |
| Ollama | 15 GB | n/a | ~6.5 GB |
| vLLM | 15 GB | ~5.5 GB | n/a |
vLLM's AWQ path is efficient but slower to first-token than a well-quantized GGUF for single-user work. The Hugging Face LLM optimization guide has the memory math end-to-end.
Common pitfalls
- Assuming Ollama is different from llama.cpp. Ollama wraps llama.cpp; the raw engine is identical. If Ollama is slower on your rig, the answer is usually a version skew or a suboptimal quant.
- Running vLLM for one user. You will not see its advantages; you will see its setup cost. Switch to Ollama.
- Running llama.cpp with only 4 threads on a 6-core CPU. Default thread counts are conservative.
--threadsmatters, especially with any CPU offload. - Leaving
--n-gpu-layersat default. llama.cpp defaults to conservative GPU layer counts. Push it up until you're at the VRAM ceiling. - Testing on a laptop under thermal throttling. All the numbers above assume a desktop 3060. Laptops with the same chip in a tighter thermal envelope produce lower and less consistent tok/s.
Real-world number: which one converts to the fastest replies?
For a solo user drafting emails, doing code completion in a local IDE, or chatting at a REPL, install Ollama, pull llama3.1:8b, and start using it. You will not perceive the tok/s difference between Ollama and raw llama.cpp on a 3060. You will absolutely perceive the setup difference.
If you're building an app backend where users hit the model from a web UI concurrently, install vLLM on a Linux box with a used MSI RTX 3060 Ventus 2X 12G or better, and take the throughput. Pair it with a fast CPU like the Ryzen 7 5800X and a WD Blue SN550 NVMe for model storage, or hold onto the Ryzen 5 5600G as a lower-power fallback.
Perf-per-watt
Under sustained load on the 3060 running a single 8B model at q4:
- llama.cpp / Ollama — ~140–150 W total system draw
- vLLM (single request) — ~145–155 W
- vLLM (8 concurrent) — ~165–180 W (higher utilization)
At the June 2026 US average retail rate (~17¢/kWh), a 150 W rig running 8 hours a day costs ~$6.10/month. The choice of runtime is basically free at these scales; the choice of what to run on it dominates.
Bottom line
Solo desktop chat, drafting, coding? Ollama. Solo user who wants to fiddle with every flag? llama.cpp. Serving multiple users, agent backends, dev boxes for a team? vLLM.
Ollama and llama.cpp are the same engine; the choice between them is UX. vLLM is a different animal, optimized for a different problem, and it earns its place only when concurrency shows up.
Related guides
- Which LLMs Fit a 12GB RTX 3060? — the model-choice half of the equation
- Can a 12GB RTX 3060 run a 70B LLM? — offload reality check
- GLM-5.2's long-horizon agent mode locally — a runtime-choice example
Sources
- ggml-org/llama.cpp — CUDA and Vulkan backend internals
- vLLM documentation — PagedAttention and continuous batching
- Ollama blog — release notes and engine version history
Extended: real-world workflow recommendations
Concrete workflows and the runtime pick for each:
Solo coding assistant on your laptop (docked to a desktop with the 3060) — Ollama. Pull qwen2.5-coder:7b, plug it into your editor via the OpenAI-compat endpoint. Setup: 10 minutes. Maintenance: ollama pull when new versions land.
Home RAG chatbot for your notes — Ollama with the built-in embedding models, or llama.cpp if you want to precisely tune KV caching. Either works; Ollama is simpler.
Multi-user internal tool (5–20 concurrent chat users) — vLLM. The batching wins overwhelm the setup cost. Run on Linux with a used 3090 or A6000 for real serving throughput.
Long-running agent (code review agent, PR summarizer) — vLLM's prefix caching is a big win here because the system prompt is huge and identical every turn. Ollama with careful prompt-cache flags is a runner-up.
Local speech-to-text feeding an LLM — Ollama, because you have one user and the pipeline benefits from Ollama's simple API more than concurrency.
Configuration recipes
llama.cpp, tuned for 3060:
./server -m qwen2.5-7b.Q4_K_M.gguf \ -c 8192 -ngl 999 -t 6 \ --cache-type-k q8_0 --cache-type-v q8_0
Ollama, install and go:
ollama pull llama3.1:8b OLLAMA_KEEP_ALIVE=30m ollama serve
vLLM, minimal serving:
python -m vllm.entrypoints.openai.api_server \ --model Qwen/Qwen2.5-7B-Instruct-AWQ \ --max-model-len 8192 --gpu-memory-utilization 0.9
Migration paths
Starting with Ollama and outgrowing it? The path to vLLM is straightforward: your Ollama HTTP client points at vLLM's OpenAI-compatible endpoint with almost no code change. Some Ollama-specific model tags don't exist in the HF hub format vLLM expects; find the equivalent AWQ or GPTQ quant and you're set.
Starting with llama.cpp and adding a second user? Multi-user llama.cpp works up to a small number of concurrent sessions; past that, vLLM's PagedAttention is worth the migration.
