Ollama vs vLLM vs llama.cpp on a 12GB GPU: Which Wins for One User?

The short answer: for a single user chatting with an 8B model at batch size 1 on a 12GB card, llama.cpp wins on flexibility and Ollama wins on ergonomics — both share the same underlying engine, and neither will feel meaningfully slower than the other on identical settings. vLLM is faster only when you have concurrent requests to batch. If you are one person typing into a chat window, migrating off Ollama for speed reasons is a category error; migrate for a specific feature (structured output, custom sampling, a runtime you can embed), not for tokens per second.

Every few weeks a new blog post fires up the "vLLM is 10x faster than Ollama" chart, and every few weeks the same 12GB-card owner asks in a Discord whether they should swap runtimes. The chart is real; the takeaway is not. vLLM's 10x number comes from throughput measured with dozens of concurrent requests packed into continuous batches — a workload that describes a serving cluster, not a desktop. At concurrency one, on a card that already has the model loaded, the runtime is the smallest variable in the equation. This piece maps out what actually differs under the hood, what changes on a 12GB card specifically, and where the "leave Ollama for vLLM" advice does and does not hold up. If you have one ZOTAC RTX 3060 Twin Edge 12GB or a MSI RTX 3060 Ventus 3X 12G and you talk to models by hand, you are the audience for this article.

Key takeaways

• At batch size 1, all three runtimes are memory-bandwidth-bound on the GPU; the throughput ceiling comes from the card, not the runtime.
• Ollama's inference is llama.cpp under a management layer — the throughput gap between them is usually a settings difference, not an engine difference.
• vLLM starts winning at concurrency 4-8 and dominates by concurrency 16 through PagedAttention and continuous batching.
• On a 12GB card, a 7-8B model at 4-bit runs comfortably in all three; a 14B needs aggressive quantization; a 32B does not fit at any useful speed.
• Host RAM matters more than most guides admit — partial offload runs the un-offloaded layers at CPU memory-bandwidth speed, and a slow CPU or single-channel RAM becomes the bottleneck.

Step 0 — what is your actual workload?

Before picking a runtime, describe the workload in one sentence. It falls into one of three buckets, and the winning runtime changes between them:

1. Single-user chat. One person, one browser tab, occasional prompts of 500-4000 tokens, occasional 2000-token replies. Concurrency is one, tail-latency does not exist because there is no queue. All three runtimes tie.
2. Agentic loop with parallel tool calls. An orchestration layer that fans out 4-16 concurrent requests per user action (a research agent, a multi-file code refactor, a batch scoring job). Concurrency is 4-16, tail-latency dominates. vLLM's continuous batching pays off here.
3. Editor autocomplete backend. Latency-sensitive short completions (100-300 tokens), sometimes with 8-32 concurrent developers hitting one endpoint. vLLM wins on throughput; llama.cpp wins on cold-start and portability.

The 12GB card matters because bucket 2 and bucket 3 both want more VRAM than that card comfortably gives. Continuous batching in vLLM keeps the KV-cache blocks packed and reuses freed blocks efficiently — but the packing still costs VRAM per active sequence, and on a 12GB card the practical concurrency ceiling with a real-sized model is single digits.

What does each runtime actually do differently under the hood?

llama.cpp

llama.cpp is a C++ inference engine with hand-tuned CPU kernels, CUDA kernels, ROCm kernels, Metal kernels, and Vulkan kernels. Its native model format is GGUF — a container that carries the weights, the tokenizer, and enough metadata that a runtime can load it with no separate config. Its calling card is n_gpu_layers: you tell it how many transformer layers to offload to GPU and it runs the rest on CPU. That partial-offload path is why llama.cpp is the runtime you can actually use for a model that does not fit in your card.

At batch size 1, llama.cpp's throughput on a fully offloaded model is competitive with anything else on the same hardware. On partial offload, it is often the only game in town — vLLM does not support offloading transformer layers to CPU in the same way. The trade is the standard C++ trade: it is stable, portable, and terse, but the surface area of flags is huge and the defaults are conservative.

Ollama

Ollama is a Go-based service that wraps llama.cpp with three things llama.cpp does not ship: a model registry with named pulls (ollama pull llama3.1:8b), an OpenAI-compatible HTTP API on port 11434, and a lifecycle manager that loads and unloads models based on request patterns. The inference core is the same llama.cpp engine, and the throughput at identical n_gpu_layers and n_ctx settings will be within a few percent of a bare llama.cpp binary. Ollama makes model management easier, at the cost of some visibility into the actual llama.cpp parameters — the Modelfile exposes many but not all knobs.

The "Ollama is slower than llama.cpp" claim usually traces to (a) a difference in n_gpu_layers (Ollama picks a conservative default), (b) a difference in n_ctx (Ollama's default is smaller than most llama.cpp examples), (c) flash-attention or KV-cache quantization flags that are exposed differently between the two, or (d) keep-alive model unloading swapping the model back to disk between benchmark runs. Fix those and the numbers converge.

vLLM

vLLM is a Python-based inference server built around two ideas: PagedAttention, a memory manager that treats the KV-cache as fixed-size blocks and allocates them on demand, and continuous batching, a scheduler that packs new incoming requests into the same forward pass as in-flight requests instead of waiting for a batch to fill. It runs on full-precision weights, AWQ, GPTQ, and (recently) a subset of GGUF, but its comfort zone is 16-bit or 4-bit weights held entirely in VRAM. There is no partial-offload path in the llama.cpp sense — if the model plus KV-cache does not fit in the card, vLLM either fails to launch or you shrink max_model_len until it does.

vLLM's throughput advantage is real and measured. At high concurrency, the same GPU can serve 5-10x more tokens per second on vLLM than on Ollama, because vLLM is doing forward passes on packed batches while Ollama is doing them one at a time. At concurrency one, the packing has nothing to pack, and the runtime overhead is a wash. Its own documentation is upfront about this — the design target is serving, not single-user chat.

How much VRAM does each runtime really need for an 8B and a 14B model?

The following table assumes 4-bit weights, flash-attention enabled where available, and a modest 4096-token context. All figures are approximate and vary by tokenizer, quantization scheme, and KV-cache dtype.

The 14B row is where the 12GB ceiling becomes real. llama.cpp fits it with headroom by trimming context or offloading a few layers to CPU. Ollama fits it if the driver frees enough of the ~1GB Windows/desktop compositor overhead. vLLM does not fit it at 4-bit with a 4K context — you either drop to max_model_len=2048, use a smaller AWQ variant, or route the workload to a bigger card.

Which runtime is fastest at batch size 1?

The honest answer: within measurement noise on identical models, identical quantizations, and identical context lengths. The numbers below are approximate representative measurements on an RTX 3060 12GB, Llama 3.1 8B q4_K_M, 2048-token prompt, 512-token generation, single request. Public runs vary by driver version, CUDA version, and flash-attention configuration; treat these as directional.

The middle two rows are the honest apples-to-apples comparison. The bottom row is the one people cite when they insist Ollama is slow — it is a defaults problem, not an engine problem. Fixing flash-attention, n_ctx, and n_gpu_layers closes the gap.

Quantization matrix on a 12GB card

The choice of quantization drives both VRAM footprint and quality loss more than the choice of runtime does. This table assumes an 8B model on a 12GB card.

For an 8B model on a 12GB card, q4_K_M is the pragmatic default. If you are running the model for creative writing or code, step up to q5_K_M or q6_K — the throughput cost is small and the quality delta is real. q2_K and q3_K_M exist for cases where you need to fit a larger model class in the same VRAM budget, not as a speed optimization.

Where does vLLM's continuous batching start to pay off?

The crossover point is roughly at concurrency 4 for prefill-heavy workloads and concurrency 8 for generation-heavy workloads. The table below sketches the shape.

These are aggregate figures across all in-flight requests; per-request latency for the winning runtime is different. vLLM's win at concurrency 16 is impressive but note the concurrency itself — you need a workload that actually spawns 16 concurrent requests to see it. A single human at a keyboard does not.

How badly does a long context hurt on 12GB?

KV-cache growth is linear in context length and quadratic in the number of layers × heads × head dimension. For an 8B model with 32 layers, 32 heads, and 128 head dim at fp16, the KV-cache footprint is roughly:

At 32K context on fp16 KV, you have used two-thirds of the 12GB card on the KV-cache alone — your model has to fit in the remaining 4GB, which forces q3 or q4 on an 8B and rules out 14B entirely. This is why every runtime on a 12GB card exposes some form of KV-cache quantization (fp8, q8, q4). Turn it on before you start blaming the runtime for slowness at long context.

Prefill time also scales super-linearly with context — a 32K prefill is not 4x a 4K prefill, it is more like 6-8x, because attention itself is quadratic. If your workload is short prompts and long generations, context length costs you VRAM but not much time; if your workload is long prompts and short generations (RAG, code with large included files), prefill dominates and a long context is expensive on any runtime.

What hardware do you need around the GPU?

The host side of an inference rig is easy to underspec. Two components matter:

CPU for prompt-processing offload and tokenization. Anything llama.cpp leaves on the CPU runs at CPU memory-bandwidth speed. Tokenization itself is CPU-bound, sampling is CPU-bound, and any layers left un-offloaded run at CPU-native speed. An eight-core part like the AMD Ryzen 7 5800X is comfortably enough for the single-user case, and its strong per-core throughput helps prompt-processing on offloaded layers. A four-core or dual-channel-RAM APU is where the CPU-side floor starts to bite.

Model-library storage. Quantized weights are large and get read in full at load. A 1TB SATA drive like the Crucial BX500 1TB sustains ~550 MB/s and loads an 8B q4 file in seconds. An NVMe drive cuts that further. What you actually want to avoid is a spinning disk or a nearly-full drive where load times balloon and page-cache thrash makes model switching feel broken.

Cooler headroom. A 12GB card at sustained inference load is not the same thermal problem as a card at sustained gaming load — the load is bursty over the length of a generation but hits full power in short spikes. A quiet tower cooler like the Noctua NH-U12S on the CPU keeps the case airflow honest, and a case with a real front-to-back path keeps the GPU's ambient inlet temp reasonable during hour-long agentic sessions.

Which specific 12GB cards make sense in 2026?

The two realistic entry points for a used or budget-new build are the ZOTAC GeForce RTX 3060 Twin Edge 12GB and the MSI RTX 3060 Ventus 3X 12G. Both are Ampere-generation (SM_86) cards with 12GB of GDDR6, 360 GB/s of memory bandwidth per the TechPowerUp GPU database, and TDP figures in the 170W neighborhood.

At batch size 1, memory bandwidth is the throughput ceiling. The RTX 3060's 360 GB/s is meaningfully lower than a 4060 Ti 16GB (288 GB/s but wider bus effects) or a 4070 Super (504 GB/s), and generation tok/s scales roughly with bandwidth on the same model. Prefill is compute-bound and depends more on the tensor-core throughput, which favors the newer generations. For local LLM specifically, if a 4060 Ti 16GB is within budget it is a better card than a 3060 12GB, mostly for the extra VRAM headroom on 14B models.

Multi-GPU scaling. With two 12GB cards, llama.cpp's row-split mode lets you run a 14B or 27B model split across both cards' VRAM. vLLM's tensor-parallel does something similar but wants NVLink or PCIe 4.0 x16 per card for meaningful scaling — a consumer board with two PCIe 4.0 x8 slots works, but the scaling factor is closer to 1.6x than 2x. Ollama exposes multi-GPU via the same llama.cpp underneath.

Perf-per-dollar and perf-per-watt

At street pricing in 2026, a used RTX 3060 12GB averages about $220 and delivers roughly 60 tokens/sec on an 8B q4 model at 170W. That works out to ~0.27 tok/s/$ and ~0.35 tok/s/W. A 4060 Ti 16GB at ~$450 delivers ~75 tok/s at 165W — ~0.17 tok/s/$ and ~0.45 tok/s/W. The 3060 12GB wins on perf-per-dollar; the 4060 Ti wins on perf-per-watt and on VRAM ceiling. If the electricity bill matters — and on an inference rig that stays warm eight hours a day, it matters — the newer card pays back the price gap in a few years.

Verdict matrix

Get llama.cpp if… you want the maximum control over quantization, KV-cache dtype, and offload strategy; you run models that do not fit fully in the card; you want to embed inference in a C++/Rust process; you plan to move between CUDA, ROCm, Metal, and CPU-only without changing runtimes.

Get Ollama if… you talk to models by hand and want the model-registry ergonomics; you have multiple models you swap between and want automatic lifecycle management; you want an OpenAI-compatible API on port 11434 with zero configuration; you are fine trading a small amount of flag visibility for a great UX.

Get vLLM if… you serve concurrent traffic — 4-plus in-flight requests as a matter of course; you can hold the model entirely in VRAM; you want the highest aggregate throughput for a serving cluster; you have separate cards to dedicate to serving.

Recommended default for the single-user 12GB case: Ollama. The throughput is identical to bare llama.cpp on the same settings, the ergonomics are dramatically better, and the "leave Ollama for vLLM to go faster" argument does not survive contact with a batch-size-1 workload. If you outgrow Ollama, migrate for a feature — structured output schemas, custom sampling, a runtime you can embed — not for tokens per second.

Bottom line. On a 12GB card serving one person, the runtime is not the constraint. The card's memory bandwidth is. Spend the effort on quantization choice, KV-cache dtype, and context-length discipline; those move tokens per second more than any runtime swap does at concurrency one. When the workload changes shape and concurrent requests appear, come back and try vLLM — that is the moment its design pays off.

FAQ

Does vLLM work at all on a 12GB consumer card?

It runs, but the fit is tight. vLLM was designed around unquantized or AWQ/GPTQ weights held entirely in VRAM, with a preallocated KV-cache block pool sized by gpu_memory_utilization. On 12GB that comfortably hosts a 7-8B model at 4-bit and little else; a 14B class model needs aggressive quantization and a trimmed max_model_len. If you routinely spill to system RAM, llama.cpp's partial-offload path is the better-engineered option.

Is Ollama slower than llama.cpp, given it wraps llama.cpp?

Ollama's inference core is llama.cpp, so per-token throughput on identical weights and identical offload settings should be close. Gaps that show up in community measurements usually trace to defaults rather than the engine: context length, flash-attention flags, the number of layers offloaded, and keep-alive model unloading. If Ollama looks slower, compare the effective launch parameters before concluding the wrapper costs you throughput.

When is switching runtimes not worth it?

If you are one person typing into a chat window, batch-size-1 generation is memory-bandwidth-bound and no runtime can conjure bandwidth the card does not have. Continuous batching, the main reason to adopt vLLM, only helps when concurrent requests exist. Stay on whatever you have and spend the effort on quantization choice and context length — those move tokens-per-second far more than a runtime swap does at concurrency one.

How much system RAM should sit behind a 12GB GPU?

Plan 32GB minimum and 64GB if you intend to run models larger than the card can hold. Partial offload keeps the un-offloaded layers in host memory, so those layers execute at CPU memory-bandwidth speed and the CPU becomes the bottleneck. Model files also load through the page cache, so spare RAM cuts reload time between models. Dual-channel population matters here more than raw kit speed.

Does the CPU matter for local inference?

Less than the GPU, but not zero. Tokenization, sampling, and any layers left on the host all run on the CPU, and prompt-processing on offloaded layers is meaningfully core-count-sensitive. An eight-core part with strong per-core throughput covers the single-user case; an APU with no discrete card ties inference to shared system memory bandwidth, which is the real ceiling in that configuration.

Where should model files live?

On the fastest drive with enough free space, because quantized weights are large and get read in full at load. A 1TB SATA SSD holds a serious library at roughly 550 MB/s, so an 8B q4 file loads in seconds; an NVMe drive cuts that further. What you should avoid is a spinning disk or a nearly-full drive, where load times and page-cache thrash make model switching feel broken even though inference itself is fine.

Citations and sources

• vLLM documentation — the design intent for PagedAttention and continuous batching, in the authors' own words (accessed 2026-08-08)
• TechPowerUp — GeForce RTX 3060 12GB specifications — memory bandwidth, TDP, and tensor throughput used in the perf-per-watt math above (accessed 2026-08-08)
• llama.cpp GitHub discussions — community-reported throughput numbers, flag combinations, and the ongoing conversation about speculative decoding limits at batch size 1 (accessed 2026-08-08)

This piece is an editorial synthesis of publicly documented specifications, primary vendor documentation, and community-measured throughput data — no proprietary benchmarks were run for this article.

Related guides

• Running Qwen 3.6 27B on a Single RTX 3060 12GB: Quantization, Context, and Real Tok/s
• Is the RTX 3060 12GB Still Worth It for 1080p Gaming in 2026?
• Best 24GB GPU for Local LLM Inference in 2026
• AMD Ryzen AI Max 395 Box: Can a 128GB Unified-Memory APU Replace a Dual-3090 Local LLM Rig?

— Mike Perry · Last verified 2026-08-08