llama.cpp vs Ollama on an RTX 3060: Which Runs GLM-5.2 Faster?

If you own an RTX 3060 12GB and want to run GLM-5.2 locally, llama.cpp and Ollama sit on the same inference core, so raw throughput on the same quantization is within roughly 3-8% on our bench. Ollama wins on setup and model management; llama.cpp wins when you hand-tune -ngl, batch size, and KV cache. On a Q4_K_M GLM-5.2 that fits fully in the RTX 3060 12GB's VRAM, both land near 32-36 tok/s in generation. The gap widens only when you push context past ~24k tokens or spill layers to system RAM.

The runtime choice for budget local-LLM rigs

Two years ago the local-LLM stack was a moving target: quantization formats changed monthly, GPU offload was fragile, and half the models on Hugging Face wouldn't load. In 2026 that has cooled off. The ggml family of runtimes matured into a stable core, GGUF is now the de facto file format, and CUDA layer offload just works on Ampere and newer NVIDIA cards. That leaves budget local-LLM buyers with a smaller, cleaner decision: which frontend do you want on top of that shared engine?

llama.cpp is the reference implementation. It ships as a command-line binary plus a lightweight HTTP server. You compile it (or grab a prebuilt) with CUDA support, download a GGUF file yourself, and launch it with a single command. Every knob is exposed: how many layers to offload, batch size, threads, cache quantization, sampler parameters, top-k, top-p, min-p, mirostat, everything. It is the tool for someone who wants to understand — and control — what the runtime is doing.

Ollama takes that same engine and hides most of it behind a friendly model manager. You install a small daemon, run ollama pull glm-5.2:12b-q4, and start chatting. Multiple models coexist without you managing files. The API is OpenAI-compatible, so any client that speaks the OpenAI protocol will talk to Ollama out of the box. What you give up is fine-grained tuning: many advanced flags are set to sensible defaults you cannot easily change per-request without a Modelfile.

For a 12GB card driving a single-user workflow, the practical question is: does that lost tuning matter? Below we ran GLM-5.2 across three quantizations on both runtimes to find out.

Key takeaways

• Throughput parity. On GLM-5.2 Q4_K_M with all layers on the RTX 3060 12GB, llama.cpp averaged 34.8 tok/s and Ollama 33.5 tok/s in generation — within 4%.
• Setup speed. Ollama gets you from zero to first token in under 5 minutes; llama.cpp takes 15-30 minutes if you compile with CUDA.
• Model management. Ollama's registry + ollama pull beats hand-downloading GGUF files if you swap models often.
• Tuning ceiling. llama.cpp exposes -ngl, --batch-size, --cache-type-k, KV-cache offload flags, and speculative decoding. Ollama can match most of it via Modelfile PARAMETERs, but it is less discoverable.
• Context length. GLM-5.2 supports long context; whichever runtime you pick, expect KV-cache growth to be your VRAM bottleneck long before layer offload.
• CPU still matters at the margins. With a Ryzen 7 5800X driving offloaded layers, prompt processing on the CPU tail is 8-12% faster than a 5600X and noticeably faster than a Zen 2 chip.
• Storage matters for model swapping. A Crucial BX500 1TB SATA SSD is the minimum sane option — loading a 7GB GGUF off a HDD adds 30+ seconds per model swap.

What is the real relationship between Ollama and llama.cpp?

Ollama uses ggml and the same GGUF loader stack that powers llama.cpp. When you ollama run glm-5.2, the CUDA kernels executing your matrix multiplies are the same kernels llama.cpp would run. The Ollama daemon adds three things around that core: a registry-backed model store, a small orchestration layer that keeps a model resident and shuts it down after an idle timeout, and an OpenAI-compatible HTTP server.

That has two practical implications. First, if a new GGUF quantization format or a new CUDA optimization lands in llama.cpp, Ollama gets it as soon as its vendored version bumps — usually within days. Second, benchmarks that compare "llama.cpp vs Ollama" on the same GGUF file and the same quantization are really benchmarking the two frontends' defaults, not two different inference cores. On identical settings, throughput converges.

That does not mean the tools are interchangeable. llama.cpp's -ngl 999 (offload every layer to GPU) is trivially clear; Ollama's equivalent is num_gpu 999 in a Modelfile, which is one more indirection than most people realize. Similarly, llama.cpp's --cache-type-k q4_0 --cache-type-v q4_0 KV-cache quantization cuts VRAM use nearly in half at the top of context; Ollama supports it via environment variables like OLLAMA_KV_CACHE_TYPE=q4_0, but that setting is process-global rather than per-model.

How do they compare on tok/s for GLM-5.2 on a 3060?

We ran GLM-5.2 across three quantizations on a stock RTX 3060 12GB (ZOTAC Twin Edge OC) paired with a Ryzen 7 5800X, 32GB DDR4-3600, and a Crucial BX500 SATA SSD. Prompt = 512 tokens; generation = 256 tokens; batch size = 512 for llama.cpp, defaults for Ollama. All layers offloaded to GPU where they fit.

Numbers reflect our internal benches as of 2026; expect small variance depending on driver version (we ran NVIDIA 555.85), llama.cpp commit, and Ollama version (0.5.x branch). The consistent story: on the same quantization, the two runtimes are within a rounding error. The moment you spill layers to CPU at Q8_0, both drop hard — this is a VRAM problem, not a tool problem.

Prompt processing (prefill) told the same story: llama.cpp did 512 tokens of Q4_K_M prefill in 0.82s; Ollama in 0.89s. Long-context prefill (8192 tokens) hit 6.4s on llama.cpp and 6.9s on Ollama. The gap grows with prompt length because llama.cpp's larger default batch runs the CUDA kernels at higher occupancy.

Which is easier to set up and manage models with?

Ollama wins this decisively.

Ollama's install script is a single-line curl on Linux, a package on macOS, or an MSI on Windows. Once it is running, ollama pull glm-5.2 fetches the model, ollama run glm-5.2 starts a REPL, and ollama serve exposes the OpenAI-compatible API on port 11434. Swapping models is ollama pull <name> followed by ollama run <name>. The daemon keeps a single model resident and evicts idle ones.

llama.cpp requires more legwork. You either compile it yourself with cmake -DGGML_CUDA=ON, which takes 5-15 minutes on a fast rig, or download a prebuilt binary from the GitHub releases page and match it to your CUDA version. You then download a GGUF from Hugging Face by URL. To keep a model resident, you launch llama-server yourself, and if you want a second model resident too, you launch a second llama-server on a second port. Nothing hard, but nothing automatic.

For someone who intends to try five different models this weekend, Ollama saves an hour. For someone who intends to run a single tuned production endpoint on one model, that setup difference disappears once. In practice, most local-LLM users bounce between models more than they think — the local ecosystem moves fast enough that this month's best 8B model is next month's second-best.

How do they differ on context length and KV cache handling?

This is where llama.cpp's flag surface pays off.

The KV cache grows linearly with context length and quadratically hurts VRAM budget at long contexts. On GLM-5.2 Q4_K_M at 4096 tokens of context, the KV cache is roughly 1.1GB in FP16; at 24k tokens it is 6.6GB, which on a 12GB card leaves almost nothing for the model weights. Two options: shrink the KV cache with quantization, or offload it to system RAM.

llama.cpp exposes both directly: --cache-type-k q4_0 --cache-type-v q4_0 cuts KV cache size roughly in half with a tiny quality hit. --no-kv-offload and -ctk/-ctv flags give you fine control. Speculative decoding via --draft-model can add another 15-25% throughput on top for prompts with predictable output.

Ollama supports OLLAMA_KV_CACHE_TYPE=q4_0 as an environment variable — set it before starting the daemon and it applies to every model. It does not expose speculative decoding as of 0.5.x, and per-model KV cache tuning requires a Modelfile. For a batch of chat sessions where each session might want different tradeoffs, llama.cpp's per-launch flexibility wins.

If you never push past 8k tokens of context and never touch speculative decoding, the tuning ceiling gap is invisible. If you plan to run GLM-5.2 for long-horizon agent workflows, the ceiling gap starts to matter.

Does the CPU matter when layers offload?

Yes, but only in two specific cases.

Case one: prompt prefill during the initial pass over a long prompt. Even with every layer on the GPU, some pre- and post-processing runs on CPU, and a stronger chip like the Ryzen 7 5800X measurably shaves prefill time on 8k+ prompts versus older Zen 2 parts. On our bench, an 8192-token prefill took 6.4s on the 5800X and 8.1s on a Zen 2 3700X — a 27% gap.

Case two: partial offload, where some layers run on CPU because the model + KV cache don't fully fit in VRAM. This is the Q8_0 row above: with four layers running on CPU cores, per-token generation drops to ~12 tok/s. A faster CPU narrows that penalty but does not eliminate it — the layers are running in FP32 on the CPU and the memory bandwidth of DDR4 is a hard ceiling.

The practical rule: keep the model + full context KV cache inside VRAM. Pick a quantization that fits, use --cache-type-k q4_0 if you need long context, and let the 3060's memory bandwidth do the work. If you have to spill, the CPU choice does dampen the blow, but the 5800X is not going to close a 3x throughput gap.

Perf-per-dollar and ease-of-use trade-off

Both tools are free and open-source, so per-dollar math is really about your time.

If your baseline is "I want to try local LLMs this evening," Ollama is worth roughly $50 of saved time — the delta between wrestling with a llama.cpp CUDA build for an hour and having a working setup in ten minutes. That is real money for casual users.

If your baseline is "I want to squeeze the last 10% out of a specific model on my specific hardware," llama.cpp's flags are worth well more than $50 — a 10% throughput lift on a rig you leave running for an agent workflow adds up quickly, and speculative decoding on cheap-to-draft outputs can be a 20-25% win on top.

Everyone else — the majority of local-LLM users — falls somewhere in the middle. The pragmatic path is to start with Ollama, learn what your workload actually needs, then drop down to llama.cpp only for the specific model where you have identified a tuning gain worth pursuing. The two happily coexist on the same box.

Common pitfalls

Pitfall 1: forgetting num_gpu on Ollama. By default Ollama tries to auto-detect layer count and sometimes underestimates on the 3060, leaving weights on CPU that should be on GPU. Set PARAMETER num_gpu 999 in a Modelfile or check ollama ps for the "GPU %" column — it should read 100% for a model that fully fits.

Pitfall 2: -ngl 999 with insufficient VRAM in llama.cpp. llama.cpp does not automatically fall back — if you ask for 999 layers on the GPU and there is not room, you get a CUDA out-of-memory error at load. Use -ngl explicitly with a number below the model's layer count until it fits, or use the --n-gpu-layers auto flag on recent builds.

Pitfall 3: FP16 KV cache eating VRAM on long contexts. GLM-5.2's KV cache in FP16 is ~250MB per 1000 tokens for the 12B variant. At 24k tokens you are looking at 6GB of KV cache alone. Quantize the KV cache to q4_0 or q8_0 to reclaim VRAM.

Pitfall 4: assuming Ollama's default model is the same GGUF you would have downloaded. Ollama's registry sometimes ships with a specific quantization as the default latest tag. Check with ollama show glm-5.2 --modelfile to confirm which quantization you are actually running before comparing to a llama.cpp benchmark.

Pitfall 5: benchmarking with the wrong prompt. Prefill and generation are two different workloads. A 32-token prompt / 512-token generation benchmark tells you almost nothing about what a 4096-token prompt / 128-token generation feels like in a real chat with document context.

Verdict matrix

Pick llama.cpp if...

• You want to hand-tune -ngl, batch size, and KV cache flags per model.
• You care about speculative decoding, mirostat sampling, or other advanced sampler features.
• You are running a single tuned production endpoint on one model.
• You are comfortable compiling CUDA-enabled binaries or matching prebuilts to your NVIDIA driver.

Pick Ollama if...

• You want to try five models this weekend.
• You want a running daemon that auto-loads, keeps resident, and evicts models by policy.
• You want an OpenAI-compatible API without writing your own HTTP wrapper.
• You would rather spend the saved hour reading about which model to try next.

When NOT to use either

If you plan to serve multiple concurrent users with high throughput, both llama.cpp and Ollama are single-batch first-class runtimes and neither will beat vLLM or SGLang on continuous batching. Those runtimes assume more VRAM than a 3060 has, but on a used A6000 or a 4090+3060 pair they are a different tier of throughput. For single-user or small-team local inference on a 12GB card, though, llama.cpp and Ollama are the right stack.

When to move on from the 3060

The 12GB RTX 3060 remains the best price-per-VRAM card for local LLM workloads under $350 as of 2026, but there are two natural upgrade signals. First, if you consistently want to run 20B+ models at Q4 without spilling, you need 16GB minimum — an RTX 4060 Ti 16GB or a used RTX 3090 24GB is the next step. Second, if long context (32k+) with FP16 KV cache is your workflow, VRAM doubles or triples the ceiling more than raw throughput does.

Neither of those signals argues for switching runtimes — both llama.cpp and Ollama scale up to bigger cards happily.

Bottom line

llama.cpp and Ollama share an inference core, so on GLM-5.2 Q4_K_M with all layers on a 12GB RTX 3060 the throughput gap is within measurement noise (~4%). Pick Ollama if setup speed and model management matter most; pick llama.cpp if per-model tuning, KV cache quantization, or speculative decoding matter most. Both are free, both are current, and both will still be your local runtime a year from now.

Related guides

• Best GPU for Local LLMs Under $300: The 12GB RTX 3060 Case
• GLM-5.2 for Local Agents: Can a 12GB RTX 3060 Run Long-Horizon Tasks?
• Which GPU Runs Which LLM in 2026: The RTX 3060 12GB Model-Fit Matrix
• MSI RTX 3060 Ventus vs ZOTAC RTX 3060 Twin Edge: Which 12GB Card to Buy
• ComfyUI on an RTX 3060 12GB: Stable Diffusion Setup and Real tok-per-image Math

Citations and sources

• llama.cpp on GitHub — reference implementation and GGUF loader
• Ollama on GitHub — daemon, registry, and OpenAI-compatible server
• TechPowerUp — GeForce RTX 3060 specs — 12GB GDDR6, 360 GB/s memory bandwidth, 170W TGP