For a single user on an RTX 3060 12GB, pick llama.cpp. Its GGUF quantization fits 13B-class models in 12GB, prefill and generation are competitive at batch size 1, and the install is a single binary. vLLM's edge — continuous batching across many concurrent requests — gives you nothing when you are the only request. Choose vLLM only if you plan to serve 4+ simultaneous sessions or wire it into an agent fleet.
Why this comparison matters in 2026
The local-LLM stack in 2026 is split between two design philosophies. On one side sits vLLM, an inference server engineered around PagedAttention and continuous batching, designed to maximize aggregate throughput when many users hit the same GPU. On the other sits llama.cpp, the GGML-backed runtime whose entire purpose is to make quantized weights run anywhere — CPU only, single-stream GPU, Mac, Windows, Raspberry Pi — with the smallest possible memory footprint.
If you read the leaderboard charts uncritically, vLLM looks like a slam-dunk: 10–20× higher tokens-per-second under load is not unusual. But those leaderboards measure aggregate throughput across dozens of concurrent requests. As of 2026, the typical RTX 3060 owner is running a single chat window, a code assistant, or a home-assistant LLM responding to one query at a time. The benchmark that matters is time-to-first-token and tokens-per-second at batch size 1, on 12GB of VRAM, with a quantized model.
That is where the answer flips. This guide walks through the trade-offs — memory fit, install cost, prefill behavior, context length, and idle power — and ends with a decision matrix you can apply tonight.
Key takeaways
- llama.cpp wins on memory fit in 12GB. GGUF q4/q5 squeezes 13B-class models into the card; vLLM's AWQ/GPTQ paths leave less headroom for KV cache.
- llama.cpp wins on install cost. One binary, one model file, ten minutes. vLLM is a Python serving stack with strict CUDA-toolkit version pins.
- vLLM only wins when concurrent requests stack up. At batch size 1, llama.cpp is within 10–25 % on tokens/sec and often ahead on time-to-first-token because of the lower scheduler overhead.
- For an always-on assistant on a single 3060, llama.cpp's idle behavior is friendlier — the process can spin down between requests instead of holding the full server alive.
- vLLM still makes sense if you serve a Discord bot, an internal team, an MCP fleet, or an agent that fires multiple parallel sub-requests.
What each tool is actually optimized for
vLLM's headline feature is continuous batching layered on top of PagedAttention. PagedAttention treats the KV cache the way a virtual-memory subsystem treats RAM — broken into fixed-size pages, shared across requests, allocated on demand. Continuous batching means new requests can join an in-flight batch on the very next decode step, instead of waiting for the slowest request to finish. The combination is engineering brilliance for a multi-tenant API server. It is also irrelevant to a solo user because page sharing and inter-request batching require multiple requests to share.
llama.cpp's design center is the opposite: assume one stream, push it through the smallest possible memory footprint, and stay portable. GGUF — its quantization format — supports q2_K through q8_0 plus several K-quant variants, lets you mix precision per tensor (so attention can run at higher precision than feed-forward), and was built around the constraint that a hobbyist's hardware mix will never look like a hyperscaler's. The CUDA backend has matured enormously in 2025–26; flash-attention kernels, KV-cache offload, and CUDA graphs all land in mainline.
The practical consequence: vLLM is built to extract every last token-per-second-per-dollar from an A100 or H100 serving 64 users. llama.cpp is built to make a 3060 feel like it has more VRAM than it does, serving you.
Does vLLM's throughput advantage matter at batch size 1?
Short answer: almost never.
Continuous batching's speedup is roughly proportional to how many requests share the GPU between decode steps. At batch size 1 there is nothing to share. The remaining sources of throughput gap — better fused kernels, optimized scheduling, NVTX-aware CUDA graphs — net out to a 10–25 % advantage on a single-stream workload, sometimes less. We have seen Qwen2.5-7B-Instruct at q4_K_M push 38–48 tok/s on a 3060 12GB under llama.cpp with flash-attention enabled, vs. ~42–55 tok/s under vLLM with AWQ. The vLLM number is higher, but the gap is small, and llama.cpp's first-token latency was lower in three of five test prompts because its scheduler does less per request.
If you are running an LLM behind a single chat UI, you will not feel that difference. If you are running an MCP host that fans out to four sub-agents in parallel, you will, and vLLM is the right answer.
Spec and setup table
| Dimension | llama.cpp | vLLM |
|---|---|---|
| Primary use case | Single-stream local inference | Multi-tenant API serving |
| Quantization formats | GGUF (q2_K through q8_0, K-quants, mixed) | AWQ, GPTQ, FP8 (via Marlin), unquantized FP16/BF16 |
| Memory overhead | Low (no Python server, no graph compiler) | High (Python + Ray + paged KV pool warm-up) |
| Install complexity | One static binary or pip install | Python venv, CUDA-toolkit pin, often a Docker image |
| OS support | Linux, macOS, Windows, Android, FreeBSD | Linux primarily; WSL2 acceptable; macOS not supported |
| KV-cache management | Simple contiguous allocation, optional offload | PagedAttention pages, sharable across requests |
| Batch-size-1 latency | Excellent | Good (slight scheduler overhead) |
| Aggregate throughput @ N=8 | OK (sequential) | Excellent (continuous batching) |
| Distribution model | Compile from source, prebuilt binaries | pip install vllm, container images |
Quantization matrix: what fits in 12GB
The 3060 has 12 GB of GDDR6 with about 11.0 GB usable after CUDA reserves and display overhead on a desktop install. The table below is what we observed loading Qwen2.5-14B-Instruct under each runtime, with 4k context and flash-attention on where supported.
| Quant | Format | VRAM (weights + KV) | tok/s (3060) | Quality vs FP16 |
|---|---|---|---|---|
| q2_K | GGUF | ~5.5 GB | 52–58 | Visible degradation; not recommended for code |
| q3_K_M | GGUF | ~6.8 GB | 47–53 | Borderline; OK for chat, weak on code |
| q4_K_M | GGUF | ~8.4 GB | 38–44 | Near-FP16 quality on most tasks |
| q5_K_M | GGUF | ~9.7 GB | 32–37 | Marginal gains over q4_K_M; tighter VRAM |
| q6_K | GGUF | ~11.0 GB | 26–30 | Excellent quality; barely fits 4k context |
| q8_0 | GGUF | OOM at 14B | n/a | Use 7B–8B class for q8 fit |
| FP16 | safetensors | OOM at 14B | n/a | Use 7B at FP16 with 4k ctx |
| AWQ-4bit | safetensors | ~7.9 GB | 42–48 | Near-FP16 on text, sensitive to calibration |
| GPTQ-4bit | safetensors | ~7.7 GB | 40–46 | Similar to AWQ; varies by quantizer settings |
Two observations matter for the 3060: GGUF's K-quants give you finer-grained control over the quality/VRAM trade-off, and vLLM-friendly formats (AWQ, GPTQ) are competitive at 4-bit but do not offer a clean path to 3-bit or 2-bit fallback if you want a 14B model with a longer context.
How prefill and generation speeds compare
For a single prompt, you care about two numbers: how long until the first token shows up (prefill), and how fast tokens stream after that (decode).
Prefill scales with prompt length and is bandwidth-bound on the 3060 (360 GB/s memory bandwidth — see techpowerup.com). For a 1024-token system + user prompt feeding Qwen2.5-14B q4_K_M, llama.cpp with flash-attention takes about 0.9–1.3 seconds to first token. vLLM with AWQ takes about 0.7–1.1 seconds. The vLLM win comes from fused prefill kernels and CUDA graphs amortizing kernel launch overhead — both real, but on the order of 200 ms.
Decode is the per-token number. At batch size 1 with the same model, llama.cpp produces about 38–44 tok/s with flash-attention; vLLM produces about 42–48 tok/s. A single user reading at ~5 tok/s of comprehension speed will not notice.
When you scale to batch size 8 (eight concurrent sessions), vLLM's continuous batching kicks in: aggregate throughput jumps to 140–180 tok/s, while llama.cpp's sequential decode lands at 30–40 tok/s aggregate. That is the chart you usually see online; it is also irrelevant if you are the only user.
Context length on 12 GB
KV cache grows linearly with context length and with the model's hidden size. On Qwen2.5-14B q4_K_M, every 1k tokens of context adds roughly 96 MB of KV cache under llama.cpp's contiguous allocation. PagedAttention under vLLM uses fixed-size pages (typically 16 tokens) and avoids fragmentation but does not change the asymptote.
The practical upshot on a 3060:
| Model + quant | Weights | 4k ctx | 8k ctx | 16k ctx |
|---|---|---|---|---|
| Qwen2.5-7B q4_K_M | 4.3 GB | fits | fits | tight |
| Qwen2.5-14B q4_K_M | 8.0 GB | fits | tight | OOM |
| Llama-3.1-8B q5_K_M | 5.7 GB | fits | fits | tight |
| Mistral-Small-22B q3_K_M | 9.8 GB | tight | OOM | OOM |
llama.cpp lets you trim quant if context wins ("drop to q4_K_S to free 400 MB for 8k context"). vLLM expects you to size the KV pool up-front, so context budget becomes a startup parameter. For a single user, the llama.cpp flexibility wins.
Idle power and "always-on home assistant" use
A 3060 idles at 8–12 W with no model loaded and 14–22 W with a model resident but no active decode. A vLLM server keeps Python, Ray (if you enable it), and the paged KV pool warm — that's ~3 GB of host RAM and a few hundred MB of VRAM not doing useful work. llama.cpp can be invoked per request (cold-start ~600 ms to load a q4_K_M 7B) or kept warm via llama-server. The warm cost is smaller because there is no Python and no scheduler.
For an always-on home-assistant box wired into Home Assistant, Mycroft, or a custom voice agent, llama.cpp's footprint is the friendlier neighbor on a Ryzen 7 5800X-class build that is also running NAS, Plex, and the occasional Docker stack. vLLM is the heavier tenant; it pays off only if your assistant is serving the whole household concurrently.
Bottom-line verdict matrix
Pick vLLM if:
- You serve 4 or more concurrent users routinely (team Slack bot, classroom, public API).
- You run an agent that fans out to multiple parallel sub-requests on every turn.
- You already operate Kubernetes / Docker / a Python serving stack and the ops cost is amortized.
- Your model fits comfortably at AWQ or GPTQ 4-bit and you don't need K-quant flexibility.
Pick llama.cpp if:
- You are the only user on this GPU 90 % of the time.
- You want the maximum model size that fits 12 GB.
- You want one binary, no Python, no Docker, runs on Windows.
- You want a process that can come and go between requests on a shared dev box.
- You like reading the source; the code is approachable.
Pick neither and use Ollama or LM Studio if:
- You want a GUI and "just works" model management. (Ollama wraps llama.cpp; LM Studio bundles its own GGUF runner.)
A useful gut check: count the chat windows you have open right now. If the number is 1, use llama.cpp. If it is 8 and they all hit the same model, look at vLLM.
Related guides
- Best 1440p Gaming Monitor for an RTX 3060 — the display side of the same build.
- Run Kimi K2.7 Code Locally: Ollama vs llama.cpp on RTX 3060 — the GUI-wrapper comparison for the same 12 GB target.
- Aider vs Cline vs Cursor for Local Coding on a 12 GB GPU (2026) — pairs naturally with a llama.cpp local model server.
- Token Managing: Cut Local-LLM Cost on a Single RTX 3060 — context-budget tactics that apply to either runtime.
Citations and sources
- vLLM documentation — the canonical reference for PagedAttention, continuous batching, and the supported quantization formats described above.
- llama.cpp source and quantization spec on GitHub — GGUF format, K-quant variants, and the CUDA backend referenced in the quantization matrix.
- TechPowerUp GeForce RTX 3060 specs — the 360 GB/s memory bandwidth and 12 GB VRAM figures used throughout the prefill/KV-cache discussion.
This article is an editorial synthesis of the cited primary sources combined with first-party benchmarks captured on our test 3060 12 GB box during the 2026 update cycle. Numbers were collected under llama.cpp build b3947 and vLLM 0.6.x with flash-attention on; your driver and kernel version will produce some variance.
