You set up a self-hosted ChatGPT alternative with Open WebUI and Ollama on an RTX 3060 12GB by installing Ollama on the host, running Open WebUI in a Docker container pointed at the Ollama endpoint, and pulling a 7B-14B quantized model. The 12 GB card handles Llama 3.1 8B, Qwen 2.5 14B, and DeepSeek-Coder-V2 Lite comfortably at 15-25 tokens/second. It won't match GPT-4-class capability, but it delivers a private, always-on chat interface with unlimited tokens.
Why Open WebUI + Ollama is the default self-hosted chat stack in 2026
The self-hosted LLM ecosystem consolidated fast between 2023 and 2026. A hundred half-finished projects narrowed to a small number of production-quality options, and at the top of that shelf sit two projects that consistently show up in every "how I run local models" thread: Ollama as the model server, and Open WebUI as the browser interface. Together they form the closest thing to a plug-and-play self-hosted ChatGPT that currently exists.
The hardware most people run this stack on is a 12 GB consumer GPU. Not a 24 GB workstation card, not a rented cloud A100 — a $300-500 ZOTAC RTX 3060 12GB, MSI Ventus 2X 12G, or Gigabyte Gaming OC 12G tucked into an existing tower. The 12 GB VRAM budget covers 7B-14B class models at 4-bit or 5-bit quantization, which is the exact tier where open models became genuinely useful. This synthesis walks through what a 3060 can serve, which models are worth pulling, and how to stand up the stack. Draws on the Open WebUI repository, the Ollama repository, and the TechPowerUp GeForce RTX 3060 specs page.
Key takeaways
- Ollama is the inference backend; Open WebUI is the browser interface. Both are open source and Docker-friendly.
- A 12 GB RTX 3060 comfortably runs 7B-14B models at 4-5 bit quantization at 15-25 tokens/second.
- Multi-user support ships built-in — a household or small team can share one server.
- The stack is faster to set up than most people expect: a 30-60 minute weekend afternoon.
- Concurrent requests queue on a single GPU. For 3-6 casual users at a time, the 3060 handles the load. Heavy simultaneous use wants more GPU.
What does Open WebUI add on top of Ollama?
Ollama alone is a fine tool for CLI users. ollama run llama3.1:8b and you're chatting from a terminal. But the terminal is the wrong interface for most people who want a ChatGPT alternative. Open WebUI wraps Ollama with the feature set most users actually associate with a modern chat product:
- Chat history and organization. Named conversations, folders, search, tags, favorites. You can go back to a chat from three weeks ago and continue it.
- Multiple user accounts. Household members or small-team users each have their own accounts with private chats. Admin can control who has access.
- Document upload and RAG. Drop a PDF, a text file, or a codebase folder in and ask questions grounded on it. Uses Ollama's embedding models for retrieval.
- Model switcher in the chat window. Try the same prompt against Llama 3.1 8B and Qwen 2.5 14B side-by-side. Compare, keep whichever you like.
- Web search integration. Optional, off by default. Point at SearXNG for privacy-preserving web-grounded answers.
- API compatibility. Open WebUI exposes an OpenAI-compatible endpoint, so tools that speak the OpenAI API (Continue, Cody, LibreChat plugins) can point at your local install and work.
- Model registry and pulls from the UI. Browse and download new Ollama models without touching the CLI.
- Custom prompts and personas. System prompts saved and reusable per conversation or per user.
- Voice input and output. Optional integration with local Whisper for STT and Piper or similar for TTS.
None of these are technically hard to add yourself. All of them are things you'd otherwise never get around to configuring. Open WebUI is the reason the stack feels like a product and not a script.
Which models run well on an RTX 3060 12GB via Ollama?
The 12 GB VRAM budget puts you in the fat middle of the open-weight shelf. The best-fitting models change every few months as new releases land; this table captures the durable picture as of mid-2026.
| Model | Params | Recommended quant | VRAM used | tok/s on RTX 3060 | Notes |
|---|---|---|---|---|---|
| Llama 3.1 8B Instruct | 8B | Q4_K_M | ~5-6 GB | 30-45 | Fast, capable, general-purpose |
| Llama 3.2 3B Instruct | 3B | Q4_K_M | ~2-3 GB | 60-90 | Very fast for lightweight tasks |
| Qwen 2.5 7B Instruct | 7B | Q4_K_M | ~5-6 GB | 30-45 | Strong at code and reasoning |
| Qwen 2.5 14B Instruct | 14B | Q4_K_M | ~9-11 GB | 15-25 | Best balance on 12 GB |
| DeepSeek-Coder-V2 Lite | 16B (MoE) | Q4_K_M | ~10-11 GB | 20-30 | Best code assistant in class |
| Mistral 7B v0.3 Instruct | 7B | Q4_K_M | ~5 GB | 30-45 | Longtime workhorse |
| Gemma 2 9B Instruct | 9B | Q4_K_M | ~6-8 GB | 25-40 | Google's open model, strong at Q&A |
| Phi 3.5 Mini Instruct | 3.8B | Q4_K_M | ~3 GB | 60-90 | Microsoft, punches above weight |
| Nemotron Mini 4B | 4B | Q4_K_M | ~3 GB | 55-80 | Nvidia's tuned mini |
For a general chat replacement, start with Qwen 2.5 14B or Llama 3.1 8B. For a coding assistant, start with DeepSeek-Coder-V2 Lite. For a fast lightweight assistant that runs alongside other workloads, Phi 3.5 Mini or Llama 3.2 3B is right.
Spec-delta table
| Model | Params | Quant | VRAM Used | tok/s on RTX 3060 |
|---|---|---|---|---|
| Llama 3.2 3B | 3B | Q4_K_M | ~2.5 GB | ~70 |
| Llama 3.1 8B | 8B | Q4_K_M | ~5.5 GB | ~35 |
| Qwen 2.5 14B | 14B | Q4_K_M | ~10 GB | ~20 |
| Mistral 7B | 7B | Q5_K_M | ~5.5 GB | ~32 |
| DeepSeek-Coder-V2 Lite | 16B MoE | Q4_K_M | ~10.5 GB | ~25 |
Prefill (context ingestion) speeds are roughly 5-10× the generation speed for these models — the RTX 3060's 360 GB/s memory bandwidth is the ceiling. Loading a 4,000-token codebase into context takes a second or so; generating a 500-token response takes 15-30 seconds. That's the interactive latency you'll actually feel.
Quantization matrix for 7B-14B models
Choosing the right quantization is the highest-leverage decision after picking the model. This is the durable picture in Ollama's GGUF ecosystem.
| Quant | Bits per weight | Quality vs fp16 | VRAM for 7B | VRAM for 14B | Fits 12 GB? |
|---|---|---|---|---|---|
| Q8_0 | 8 | very close to fp16 | ~8 GB | ~15 GB | 7B only |
| Q6_K | ~6 | very close | ~6 GB | ~11 GB | both |
| Q5_K_M | ~5.5 | close, slight drop | ~5.5 GB | ~10 GB | both |
| Q4_K_M | ~4.8 | small perceptible drop | ~5 GB | ~9 GB | both, sweet spot |
| Q3_K_M | ~3.5 | noticeable drop | ~4 GB | ~7.5 GB | both |
| Q2_K | ~2.5 | severe drop | ~3.5 GB | ~6 GB | both but avoid |
For a 12 GB 3060, Q4_K_M is the practical sweet spot on both 7B and 14B classes. Q5_K_M gives you a small quality bump on 7B models with headroom; on 14B models it's tight. Q8_0 fits 7B comfortably but doesn't fit 14B.
Prefill vs generation and context length
Ollama exposes a num_ctx parameter for context window size. Longer context costs more VRAM and slower prefill. For a 12 GB 3060:
| Context length | VRAM overhead (14B Q4) | Generation tok/s | Prefill (~90% utilization) |
|---|---|---|---|
| 4,096 tokens | ~0.5 GB | ~22 | ~130 tok/s |
| 8,192 tokens | ~1.5 GB | ~20 | ~110 tok/s |
| 16,384 tokens | ~3 GB | ~17 | ~90 tok/s |
| 32,768 tokens | ~6 GB | won't fit on 14B | — |
For a code assistant that ingests large files, favor a smaller model at a longer context (Llama 3.1 8B at 16k) over a bigger model at a shorter context (14B at 4k). The right tradeoff depends on your specific workload.
How to install the stack
1. Install Ollama. On Linux, run the official install script from ollama.com or use your distro's package. On Windows, the installer is a straightforward .exe. On macOS, brew install ollama. Ollama runs as a service and listens on localhost:11434 by default.
2. Verify GPU acceleration. Run ollama pull llama3.1:8b (this downloads a few GB). Then ollama run llama3.1:8b "Say hi". It should respond within a second or two. Run nvidia-smi in another terminal during generation — you should see Ollama's process consuming GPU memory.
3. Install Docker if you don't have it. Docker Desktop on Windows/macOS, or apt install docker.io on Ubuntu. Verify with docker run hello-world.
4. Run Open WebUI. The one-liner from the project's README:
That starts Open WebUI on port 3000, mounts persistent storage, and points it at your host's Ollama at host.docker.internal:11434.
5. First browser access. Open http://your-server:3000. Create the admin account. Open WebUI auto-discovers your Ollama models and lists them in the model selector.
6. Pull additional models. From the Open WebUI admin panel, browse the Ollama registry and pull whichever models you want. Or from CLI: ollama pull qwen2.5:14b, ollama pull deepseek-coder-v2:16b, etc.
7. Create user accounts. In the admin panel, invite household members with their own logins. Assign default model, permitted models, and chat limits per user if desired.
8. Set up HTTPS if remote-accessible. If the server is only on your LAN, plain HTTP is defensible but not recommended. If it's remote-accessible, put Open WebUI behind a reverse proxy (Caddy, Traefik) with a Let's Encrypt cert.
9. Enable systemd auto-restart. So Ollama and the Open WebUI container come up on reboot. Docker's --restart always handles the container; systemctl handles Ollama.
10. Update discipline. Pull new Open WebUI images and update Ollama every few weeks. New model releases land constantly.
You now have a private ChatGPT alternative running on your own hardware. Whole setup is under an hour once Docker is installed.
Perf-per-watt for a 24/7 chat box
The RTX 3060 12GB draws 170 W under sustained load and 15-25 W at idle. A conversational Open WebUI use pattern — a few dozen requests a day, most of the time idle — spends most of its life at idle. Realistic 24/7 electrical draw for a Pi-sized or mid-tower host + 3060 running Ollama:
| State | Approx watts | $/year at $0.15/kWh |
|---|---|---|
| Idle (most of the day) | 40-70 W total | $50-90 |
| Modest use (~2 hours/day active) | Peaks at 200 W, avg ~80 W | $105 |
| Heavy use (~8 hours/day active) | Peaks at 250 W, avg ~150 W | $200 |
Even at heavy household use, a self-hosted stack costs less in electricity than a single-user cloud subscription — before you factor in privacy and per-token cost.
Verdict matrix: run this stack, or use a cloud model?
| Situation | Run local Open WebUI + Ollama | Use cloud model |
|---|---|---|
| You want private chat over sensitive material | yes | — |
| You want unlimited tokens and no per-message cost | yes | — |
| You want offline capability | yes | — |
| You need top-tier reasoning on hardest problems | — | yes |
| You use it once a week for a specific question | — | yes |
| You want zero maintenance | — | yes |
| You share access across a household or small team | yes | — |
| You need multi-modal image + text reasoning | — | yes (mostly) |
| You care about model transparency and auditing | yes | — |
| You want the very fastest response times | — | yes |
Common pitfalls setting up the stack
- Running Ollama without GPU acceleration. If
nvidia-smidoesn't show Ollama consuming GPU memory, you're on CPU inference, which is 10-30× slower. Verify CUDA drivers and Ollama build. - Wrong context length. Setting
num_ctxtoo high on a 14B model OOMs the card mid-response. - Not persisting Open WebUI data volume. Restart without the volume flag and users lose all chat history and accounts. Docker volume is mandatory.
- Exposing to the internet without HTTPS. Fine on LAN, catastrophic on WAN.
- Skipping Ollama updates. New releases add model support, quantization formats, and speed improvements. Update monthly.
- Assuming multi-user is truly parallel. The single 3060 serializes GPU requests. Two users hitting it simultaneously still queue. Fine for household use, tight for heavy team use.
- Not budgeting SSD storage for model files. Each model is 3-10+ GB. Have 100+ GB free for a comfortable model library.
When NOT to run this stack
If you need the absolute best reasoning on the hardest problems, cloud frontier models still win. If you use LLM chat only occasionally, a paid ChatGPT subscription is cheaper than a GPU and easier to maintain. If you can't leave a GPU-equipped system on 24/7, cloud is more reliable. But for privacy-sensitive work, heavy daily use, teams sharing one instance, or any use case where per-token cost or vendor policy risk matters, local wins.
Bottom line
Open WebUI + Ollama on a 12 GB RTX 3060 is the current default self-hosted ChatGPT alternative for good reason: it takes an afternoon to set up, it handles 7B-14B models at interactive speed, it supports multiple users, and it never sends a prompt to anyone else's server. Grab a ZOTAC 3060 12GB, MSI Ventus 2X 12G, or Gigabyte Gaming OC 12G, pair it with a fast SSD like the Crucial BX500 1 TB for model storage, run the stack, and pull Qwen 2.5 14B. You'll have a private, always-on chat assistant with unlimited tokens for less than a year of paid cloud service.
Related guides
- Best GPU for 1440p Local Image Generation in 2026
- Reve 2.0 vs Local Image Gen on an RTX 3060
- Hidden Code in Claude Code and Local-LLM Privacy
Citations and sources
This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.
