vLLM on Windows in 2026: What Actually Works on a 12GB Card

vLLM on Windows in 2026: What Actually Works on a 12GB Card

Direct answer. As of 2026, vLLM does not run natively on Windows in a way most single-user builders would call reliable. Per the official vLLM installation docs, the supported path on Windows is WSL2 with a CUDA-capable NVIDIA driver, an Ubuntu 22.04 or 24.04 image, and PyTorch built against CUDA 12.x. On a 12GB card like the MSI GeForce RTX 3060 Ventus 3X, that setup will serve 7B–8B parameter models at 4-bit or 8-bit quantization comfortably, and 13B models only with aggressive quantization and short context windows.

Why single-user builders on Windows keep hitting the vLLM wall

vLLM was built as a high-throughput serving engine for GPU clusters, not as a hobbyist chat runner. The project's north star is high concurrent request throughput on datacenter cards. Its engineering blog and GitHub issues make that lineage clear: PagedAttention, continuous batching, tensor parallelism, and prefix caching are all optimizations that pay off when many requests share a GPU, not when one person types a prompt every 30 seconds.

That mismatch is why so many Windows users on 12GB cards end up frustrated. The install fails, or it installs but throws CUDA out-of-memory errors on models that llama.cpp loads without complaint, or it runs but shows the same tokens-per-second as Ollama at higher setup cost. This synthesis is for the builder who has an RTX 3060 12GB, Windows 11, and wants to know whether vLLM is worth the friction — or whether a different runtime is the right call. It draws on public benchmarks, the project's documentation, and community measurements from r/LocalLLaMA rather than any first-party testing.

Key takeaways

• Native Windows support is not the official path. Per project docs, Linux and WSL2 are the supported install targets. Native Windows wheels exist but are considered experimental as of 2026.
• 12GB is enough for 7B–8B at 4-bit with room for KV cache. Community measurements on r/LocalLLaMA place Llama-3-8B-Q4 comfortably inside 8–9 GB of VRAM at 4k context.
• 13B is possible but tight. 4-bit 13B weights alone occupy roughly 7–8 GB; add KV cache and you are one long conversation away from an out-of-memory error.
• vLLM's throughput advantage is a concurrency story. For single-user chat, llama.cpp and Ollama match or beat vLLM on wall-clock latency because they skip vLLM's batching overhead.
• CUDA 12.x is table stakes. Older toolchains hit JIT fallback paths that cost 30–50% throughput per community reports.

Does vLLM run natively on Windows 11 or 10?

Per the vLLM installation guide, the supported install paths are Linux with CUDA, Linux with ROCm, and Linux with CPU. Windows is not on that list. The project maintainers have historically closed native-Windows install issues on GitHub with a pointer to WSL2, and the Windows-native code paths remain community-experimental as of 2026.

There is a persistent minority of builders on r/LocalLLaMA who claim to have vLLM running natively under Windows, but the recipe involves pinning specific PyTorch nightly builds, hand-patching setup scripts, and accepting that any point release can break the build. That is not a load-bearing production posture. If you need vLLM's specific advantages — PagedAttention, high-concurrency batching, prefix caching — WSL2 is the honest recommendation. If your workload is one chat session at a time, native Windows support is not the reason to pick vLLM.

How do you set up vLLM under WSL2 on an RTX 3060 12GB?

The install has four dependencies you have to get right: a recent NVIDIA driver on the Windows host, WSL2 with GPU passthrough, a CUDA 12.x toolkit inside the Ubuntu guest, and a matching PyTorch build. Miss any one and the install either fails at compile time or silently falls back to a slow JIT path.

1. Update the NVIDIA host driver. Per NVIDIA's WSL2 GPU documentation, the Windows-side driver ships with the WSL2 CUDA userspace bundled in. Use a Studio or Game Ready driver from 2025 or newer.
2. Install WSL2 and an Ubuntu image. From an elevated PowerShell, wsl --install -d Ubuntu-24.04. Per Microsoft's WSL install guide, no additional GPU driver goes inside the guest — the userspace comes from the host.
3. Confirm GPU visibility. Inside the Ubuntu shell, nvidia-smi should show the RTX 3060 12GB with 12,288 MiB of memory. If it does not, the WSL2 CUDA userspace is not initialized — reboot Windows.
4. Install CUDA 12.4 or newer inside the guest, followed by pip install vllm in a fresh Python 3.11 virtual environment. Per the vLLM docs, this pulls the prebuilt CUDA 12.x wheel.

WSL2 shares memory with the Windows host by default, so a Windows build with 32 GB RAM and 12 GB VRAM will have plenty of headroom for vLLM plus the model. If your model triggers CUDA out-of-memory during startup rather than during generation, the fix is almost always a lower quantization or a smaller context window — not a driver problem. Companion parts on this build tier commonly pair with an AMD Ryzen 7 5700X or Ryzen 7 5800X and a WD_BLACK SN770 NVMe or Crucial BX500 SATA SSD for model storage.

Spec table: vLLM vs llama.cpp vs Ollama on Windows

The three runtimes solve slightly different problems. This table compresses their Windows story into one look. All rows draw on the projects' own docs and repeatable community numbers.

The pattern is consistent across community reports: vLLM's setup is heaviest and its per-token latency for a single request is not measurably better on a 12GB card. Its win is throughput under load — if you are serving five simultaneous chats to a home network, its continuous batching pays off. If you are the only user, llama.cpp and Ollama are strictly easier and no slower.

Quantization matrix on a 12GB card

Weight quantization is the lever that turns "won't fit" into "runs cleanly." The numbers below aggregate community measurements posted to r/LocalLLaMA and the Hugging Face documentation on GGUF quantization.

The practical takeaway is that Q4_K_M is the sweet spot on 12GB. Q5 and Q6 buy back a small quality bump for a big VRAM cost, and Q8 puts 13B models out of reach on this class of card. This mirrors the guidance in the llama.cpp quantization documentation and matches the defaults Ollama ships with for consumer GPUs.

Prefill vs generation throughput on 12GB

vLLM's page-attention scheduler splits work into prefill (processing the prompt) and decode (generating output tokens). On a 12GB card those two phases have very different profiles.

Prefill is compute-bound and benefits from vLLM's batched attention kernels — it is where vLLM's throughput advantage over llama.cpp is most visible in benchmarks like those in Anyscale's original vLLM post. Decode is memory-bandwidth-bound on consumer cards and is where vLLM's advantage over llama.cpp evaporates for single-user workloads.

Context length changes the math. KV cache grows roughly linearly with context and with batch size. A 4K context on 8B-Q4 costs a few hundred megabytes of KV cache; an 8K context doubles that; a 32K context can push a model that fit at 4K into out-of-memory territory. The TechPowerUp GPU database shows the RTX 3060 12GB's memory bandwidth is 360 GB/s, which is the ceiling on decode throughput regardless of runtime.

Perf-per-dollar: is the RTX 3060 12GB still the entry ticket?

The RTX 3060 12GB has spent nearly its entire retail life as the cheapest 12GB NVIDIA card. Its street price is well under most 16GB cards, and as of 2026 it remains the answer to the question "what is the least I can spend to run 8B-class local LLMs?"

The upgrade path is a steep step. Moving to 16GB puts a 13B model at Q4 firmly in range with headroom for context; moving to 24GB opens up 30B-class quantized models. Neither jump is small in dollars. If your workload is 7B–8B chat and quick prototyping — the modal workload for r/LocalLLaMA — the 3060 12GB is the entry ticket. If you have a specific 13B-Q5 or 30B-Q4 target, plan to skip the 3060 12GB entirely.

Common pitfalls

• Installing vLLM without CUDA 12.x. The older CUDA 11.x path triggers a JIT compile fallback that community benchmarks put at 30–50% slower than the CUDA 12 wheel.
• Trying to run FP16 weights on 12GB. A 7B FP16 model needs ~14 GB before KV cache. Use Q4 or Q5 unless you have a specific reason not to.
• Setting --max-model-len too high. The default context length in vLLM's model configs can be 32K+, which reserves KV cache assuming that ceiling. Reduce it to what you actually use.
• Forgetting NVIDIA driver updates. Older Windows drivers ship older WSL2 CUDA userspace. If nvidia-smi inside WSL shows an unexpectedly old CUDA version, update the Windows-side driver.
• Treating vLLM like Ollama. vLLM does not auto-download models from a curated catalog. You are pulling weights directly from Hugging Face and passing paths on the command line.

When NOT to use vLLM on Windows

If you fit any of these, do not run vLLM — pick llama.cpp or Ollama instead:

• You are the only user of the endpoint.
• Setup time matters and you do not want a WSL2 layer.
• Your models are all in GGUF format and you are not planning to migrate to safetensors.
• You need offline model management with a curated catalog.
• Your workload is chat + short bursts of coding help, not batched serving.

vLLM's value is high concurrency throughput. If you do not need concurrency, its cost curve is worse than the alternatives on a 12GB Windows box.

Related guides

• Best GPU for Local LLMs in 2026: Why the RTX 3060 12GB Still Wins on Value
• Ollama vs llama.cpp on a 12GB GPU in 2026: Which Local Runtime to Pick
• Jan.ai vs LM Studio vs Ollama: Easiest Local-LLM App for a 12GB Card in 2026

Bottom line

vLLM is the right runtime when you are serving many concurrent requests and can afford a WSL2 install. It is the wrong runtime for a single builder chatting on a Windows box with an RTX 3060 12GB. Ollama and llama.cpp deliver the same wall-clock latency for one user with a fraction of the setup burden. If you already have WSL2 running for other reasons and want to learn PagedAttention hands-on, vLLM is worth the afternoon. Otherwise, pick a simpler runtime and spend the saved time on prompt engineering or fine-tuning.

Citations and sources

• vLLM installation guide
• Microsoft WSL2 install guide
• NVIDIA CUDA on WSL2 documentation
• TechPowerUp — GeForce RTX 3060 12GB spec sheet
• vLLM engineering blog — PagedAttention
• llama.cpp quantization discussion
• Hugging Face GGUF documentation
• r/LocalLLaMA community

This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.