Running GLM-5.2 Locally on an RTX 3060: Ollama VRAM + tok/s

A 12GB RTX 3060 can host a quantized GLM-5.2 in Ollama as of 2026, but only with a 4-bit (q4_K_M) or smaller quant and careful num_gpu tuning. A full BF16 build does not fit. With q4_K_M and modest context, expect low-double-digit generation tok/s on most mid-size GLM-5.2 variants; the largest sizes need layer offload to system RAM and a capable CPU like a Ryzen 7 5800X to stay usable.

Who this is for

This guide is for budget local-LLM builders eyeing the most-talked-about open-weights model of the week on a roughly $300 12GB graphics card. The reference rig in our shop is a ZOTAC Gaming GeForce RTX 3060 Twin Edge 12GB paired with either a Ryzen 5 5600G for the cheapest possible AM4 build or the Ryzen 7 5800X when CPU-side offload performance matters. The cheaper MSI GeForce RTX 3060 Ventus 2X 12G is the same chip in a slightly different cooler — buy whichever is cheaper on the day.

If you already own a 16GB or 24GB card, this is not the article you need; you can run larger quants without much thought. If you have a 4060 Ti 8GB or a 3070 8GB, GLM-5.2's mid sizes will spill heavily into system memory and the 3060 12GB is, paradoxically, the better starting point even though its compute is slower. Per TechPowerUp's RTX 3060 12GB spec sheet, the card pairs 12GB of GDDR6 with a 192-bit bus and 360 GB/s of bandwidth — the extra VRAM, not the compute, is why local-LLM hobbyists keep buying it three generations after launch. This piece is editorial synthesis of publicly available spec sheets and tooling docs, not a first-party benchmark report.

Key takeaways

• The RTX 3060 12GB's 360 GB/s memory bandwidth caps generation throughput long before its 28 SMs do, so quantization choice matters far more than overclocking the core.
• A q4_K_M quantization is the practical floor for keeping mid-size GLM-5.2 variants mostly GPU-resident on 12GB. q2/q3 are smaller still but pay a noticeable quality penalty.
• Ollama wraps llama.cpp; the num_gpu modelfile setting is the single most important knob for stability on a 12GB card.
• Context length is the silent VRAM killer — doubling context can push you past the 12GB ceiling even on the same quant.
• Per-token economics for hobby use favor a one-time used RTX 3060 12GB over a cloud API once you run more than a few million tokens per month.
• If you plan to push past mid-size variants regularly, budget for more VRAM up front rather than fighting offload.

What is GLM-5.2 and why does the 12GB VRAM ceiling matter?

GLM-5.2 is the latest in the open-weights GLM family published on Hugging Face. The model family ships in multiple sizes; the relevant ones for a 12GB card are the smaller and mid-size variants, where 4-bit quantization can compress the on-disk weights enough to keep most or all layers resident in 12GB of VRAM. The flagship sizes are not realistic targets for this hardware — even at 4 bits they exceed the card's VRAM and force aggressive layer offload that slows generation dramatically.

The 12GB ceiling matters because LLM inference is memory-bound on hobby-class GPUs. Per TechPowerUp's RTX 3060 12GB profile, the card has 360 GB/s of memory bandwidth — generous for a budget SKU, but a fraction of the bandwidth on professional accelerators. Every token generated requires re-reading the entire active model weights from VRAM. That means VRAM capacity sets the largest model you can run at speed, and VRAM bandwidth sets how fast you can run it. If you spill weights to system RAM, you instead get PCIe 4.0 ×16 bandwidth (about 32 GB/s in practice) for the offloaded layers — an order of magnitude slower. The whole game is to keep as much of the model as possible above that line.

Note: the model release pages on the Hugging Face blog are the authoritative source for exact parameter counts, license terms, and recommended quantizations of the GLM-5.2 release. Treat the variant sizes below as planning ranges; check the model card before downloading.

Which quantization fits 12GB?

Below is a planning matrix for fitting an open-weights mid-size LLM into 12GB of VRAM. The exact numbers move with model architecture, KV-cache shape, and context length, but the shape of the tradeoffs holds across this class of card and quantization scheme. Treat it as a starting point, then measure on your hardware.

A useful rule of thumb: on a 12GB card you can target roughly 9–10GB of VRAM use for weights and a small KV cache, leaving 2–3GB headroom for system processes, display, and longer contexts. If a quant + context combination predicts use above ~11GB, expect either out-of-memory crashes or forced offload.

Spec context: GLM-5.2 vs GLM-5.1

A clean spec comparison must come from the official model cards rather than secondhand benchmarks. Confirm parameter counts, context-window size, and the published Artificial Analysis Index numbers against the Hugging Face blog post for the GLM-5.2 release before quoting them in a build report. As a general planning frame: GLM-5.2 lifts capability above 5.1 at similar parameter counts and usually keeps the same context window, which is why builders are rushing to swap it in for existing 5.1 workflows.

How do you install GLM-5.2 in Ollama step by step?

Per the Ollama project repository, the install + run loop is short:

1. Install Ollama for your platform.
2. Pull the model: ollama pull glm5.2:4b (or whichever tag the official model card publishes for a q4-style variant in a mid size you want to run).
3. Create a modelfile if you need to override defaults. The minimum useful override on a 12GB card is num_gpu, which sets how many transformer layers Ollama loads on the GPU. Start with the model's published total layer count and reduce by 2–4 until the card stays under 11GB of VRAM use under load.
4. Run: ollama run <your-tag> and test with a short prompt, then a longer one to confirm the KV cache does not push you past the ceiling.
5. Watch nvidia-smi -l 1 in a second terminal while you run real prompts. The first generation pass is the one that grows the KV cache — if VRAM spikes past ~11.5GB there, drop num_gpu and retry.

A complete modelfile looks like:

FROM glm5.2:4b
PARAMETER num_gpu 24
PARAMETER num_ctx 4096
PARAMETER temperature 0.6

num_ctx is the second knob worth tuning. A larger context window means a larger KV cache, and the KV cache lives in VRAM. Cutting num_ctx from 8192 to 4096 often buys back the headroom you need to keep more transformer layers on the GPU.

Benchmark frame: prefill vs generation on RTX 3060 12GB

Benchmark numbers for any quant + context length combination on the RTX 3060 12GB should be measured locally, not quoted from a forum post about a different motherboard or a different driver. The shape to expect, however, is consistent across this card:

• Prefill (processing the prompt) is compute-bound. The 3060's 28 SMs hit a wall well below what a 4090 does, so very long prompts will feel slow before the first response token appears.
• Generation (producing each new token) is memory-bandwidth-bound. This is where the 360 GB/s VRAM bandwidth caps you; q4_K_M is usually the sweet spot because it reduces the bytes-per-token enough to lift generation speed without giving up too much quality.
• CPU offload turns a memory-bandwidth problem into a PCIe-bandwidth problem. Any layer running on the CPU becomes the throughput floor for the whole pipeline.

When you publish a number to a forum, always cite quant, context length, number of GPU layers, the prompt length, and the CPU. A "10 tok/s on a 3060" claim with no context is unfalsifiable — and the next reader will believe it for their own (different) setup.

How does context length impact VRAM and speed on a 12GB card?

KV-cache memory grows roughly linearly with context length, and on most open-weights architectures it consumes hundreds of megabytes per thousand tokens of context at mid model sizes. On a 12GB card, that means doubling num_ctx from 4096 to 8192 can easily eat another 1–2GB, which is enough to push you from "all on GPU" into "partial offload." The fix is one of: cut the context, drop a quantization step, or accept partial offload and the speed penalty that comes with it.

For chat workflows that rarely exceed a few thousand tokens of useful context, 4096 is a comfortable default on this card. For RAG pipelines or long-document summarization, you will need to either pre-summarize before the call or move to a card with more VRAM. Long-context use is the single most common reason a "fits in 12GB" claim quietly stops being true once a real workload arrives.

When should you offload to system RAM instead of upgrading the GPU?

Offload to system RAM makes sense when:

• You only need the model occasionally and can tolerate single-digit tok/s on the offloaded portion.
• Your CPU has high memory bandwidth — dual-channel DDR4-3200 or better, ideally with a Ryzen 7 5800X or Ryzen 5 5600G feeding it. Single-channel memory or a weak CPU collapses throughput.
• The model just barely overflows VRAM — say 2 or 3 layers — so most of the work still happens at GPU speed.

Offload stops making sense when the model is so large that most layers live on the CPU. At that point the GPU is contributing little and you are effectively running CPU inference with a fast scratchpad. If you are routinely in that regime, save up for a 16GB or 24GB card instead of fighting offload.

Perf-per-dollar: GLM-5.2 on a used RTX 3060 vs cloud API cost math

The break-even between a one-time hardware purchase and a cloud API depends entirely on how many tokens per month you actually generate. A used RTX 3060 12GB plus a budget AM4 platform commonly lands well under $500 all-in. At typical hobby usage — coding assistant for a few hours a day, occasional batch summarization — that hardware pays for itself in under a year against a paid cloud frontier-model subscription, and the variable cost after that is electricity (the card draws roughly 170W under sustained inference).

The math flips when you need either frontier quality (the largest closed models still beat the largest open-weights ones on the hardest reasoning tasks) or massive batch throughput (a single 3060 is not a serving box). For most personal-use and small-team applications, though, owning the hardware wins after the first few months. The Hugging Face blog tracks the open-weights frontier closely; check it before each upgrade to see whether a new model release pushes the local option further ahead of the cloud option for your task.

Real-world gotchas

A few patterns we see repeatedly when readers try to run a fresh open-weights release on a 12GB card:

• Driver mismatch. A driver too old for the runtime falls back to slow paths or silent CPU offload. Update before you measure.
• KV-cache surprises. A model that fits at 4096 context can OOM at 8192 because the KV cache grew, not the weights. Always confirm with nvidia-smi at full context, not the warmup pass.
• Two displays + game running. A second 4K monitor or a game in the background can hold 2GB of VRAM that you assumed was free. Close them before benchmarking.
• Wrong num_gpu. Setting num_gpu higher than the model's actual layer count is silently capped; setting it too low leaves easy speed on the table. Always start at the model's exact layer count and tune downward.
• Cooler-only "upgrades." The MSI Ventus 2X 12G and the ZOTAC Twin Edge are the same GA106 chip with different coolers. Don't pay a premium for "OC" SKUs unless they're cheaper than the cooler-only variant — the inference workload won't notice.

When NOT to bother

If your real workload is bigger than what a 12GB card can run at q4 without offload, you are buying the wrong card. Builders pushing the largest open-weights variants regularly will be happier with a 16GB or 24GB GPU and a slightly worse perf-per-dollar than fighting offload on the 3060 every day. Similarly, if you need a production inference server with hundreds of concurrent requests, a single consumer card is not the right hardware — that is a different conversation about A6000-class workstation cards or rented cloud GPUs.

Bottom line

A 12GB RTX 3060 can run a quantized GLM-5.2 in Ollama today. Pair the ZOTAC Gaming GeForce RTX 3060 Twin Edge 12GB or MSI GeForce RTX 3060 Ventus 2X 12G with at least a Ryzen 5 5600G, pull a q4_K_M quant of a mid-size variant, tune num_gpu and num_ctx until the card stays under 11GB of VRAM, and you will have a usable local chat or coding assistant. Step up to a Ryzen 7 5800X if you expect to offload layers regularly; the extra CPU bandwidth is what keeps offloaded layers from collapsing your tok/s.

Related guides

• Best Budget GPU for Local LLMs in 2026: The 12GB RTX 3060 Case
• Claude Sonnet 5 Closes the Opus Gap: When Local Still Wins
• LongCat-2.0: A Frontier Model Trained Without Nvidia GPUs

Citations and sources

• TechPowerUp — GeForce RTX 3060 12 GB spec sheet
• Hugging Face blog — official model release notes and quantization recommendations
• Ollama on GitHub — installation, modelfile, and num_gpu documentation

This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.