Best Parts for an Always-On Local LLM Server in 2026

See the SpecPicks review methodology.

Best Parts for an Always-On Local LLM Server in 2026

By Mike Perry · Published 2026-09-09 · 27 min read

An always-on inference box is not a gaming PC that happens to run models. It is judged on completely different numbers. Nobody cares what it does in a benchmark burst; they care what it draws sitting idle at 3 a.m., whether it is audible from the next room, how many models it can hold on disk without pruning, and whether it will still be running unattended in three years. Peak throughput is close to the least important specification on the list.

That reframing changes every part choice. The flagship GPU that wins every tokens-per-second chart is the wrong pick for a machine whose duty cycle is mostly waiting — it costs three times as much, draws more at idle, and buys throughput on a workload that is bounded by how fast you type. The high-end AIO cooler that dominates thermal reviews introduces a pump that will fail before the CPU does. The blazing NVMe drive is solving a problem that occurs once per model load and never again.

What matters instead: enough VRAM that your daily model stays resident, because offloading to system RAM is the single largest performance cliff in local inference and it applies to every token of every request. Low idle power, because idle is where the machine spends its life and idle watts compound over 8,760 hours a year. A noise floor low enough that the box can live where you actually have room for it. Enough storage that you are not deleting a model to try another one. And a platform boring enough to run for years without attention.

The picks below are built around those constraints. The headline choice is a $329-class 12 GB GPU rather than anything faster, and the reasoning for that runs through the whole guide.

Prices are catalog listings as of 2026-09-18 and change daily — check the live listing before you buy.

Top picks

#1 🏆 Best Overall: MSI GeForce RTX 3060 Ventus 2X 12G

12 GB GDDR6 · 192-bit · 3,584 CUDA cores · 170 W TGP · $329 launch MSRP

✅ 12 GB holds a 14B model at Q4_K_M entirely in VRAM — the practical dividing line between a useful assistant and a slow one ✅ 170 W board power per TechPowerUp is roughly half a current flagship's, and it idles low ✅ Two-slot, two-fan design fits small cases and mini-tower builds where triple-slot cards do not ✅ Mature CUDA support: every inference runtime has targeted this architecture for years, so nothing needs a bleeding-edge driver branch

❌ 360 GB/s of memory bandwidth caps generation throughput well below newer cards ❌ At the ~$500 currently listed in the SpecPicks catalog, it is priced far above its $329 MSRP — shop the listing, not the recommendation

This is the pick because of what fits, not what is fast. TyoLab's 12 GB VRAM benchmark records Llama 3.1 8B at q4_K_M running 64.50 tok/s with an 8K context window, and Qwen3 14B at q4_K_M holding 33.40 tok/s at 4K — both entirely resident. Hardware Corner's 3060 12 GB page puts Qwen3 8B at q4_K_XL at 55.20 tok/s using 6.0 GB of VRAM, which leaves substantial headroom for context.

Both of those numbers are faster than anyone reads. For a machine serving one or two users, throughput past roughly 30 tok/s is invisible — the response arrives faster than you consume it. What you notice instead is the difference between a model that fits and one that does not, and 12 GB is where the useful 14B class lands. llmrun.dev shows the ceiling clearly: a 14B model at q4_K_M reports 9.5 GB of VRAM and 24.60 tok/s, still comfortable, while singhajit.com's comparison has Qwen3 14B dropping to 22.70 tok/s once context reaches 16K.

The card also idles quietly and cheaply, which on a machine that is idle most of the time is worth more than a benchmark win.

See price & specs (price may vary — verify at checkout) · See Full Details →

#2 💰 Best Value: AMD Ryzen 5 5600G

6 cores / 12 threads · 65 W TDP · 61 W PPT · Radeon 448SP integrated graphics · AM4 · $259 MSRP

✅ Integrated graphics means no second GPU for display output — one less card idling, one less thing to fail ✅ 65 W TDP with a 61 W package power tracking limit per TechPowerUp's CPU database keeps platform idle low ✅ AM4 boards, DDR4-3200 and coolers are cheap and abundant — the whole platform is a solved problem ✅ Six cores are enough to feed a single GPU and run the rest of a homelab's services alongside

❌ PCIe Gen 3 with 16 CPU lanes limits multi-GPU expansion later ❌ 16 MB of L3 and no chiplet design means it trails the 5800X badly on CPU-side offload

The case for an APU host on an always-on box is about what you leave out. A discrete GPU for display duty is a card that draws power continuously to render a login screen nobody looks at. The 5600G's Radeon 448SP integrated graphics handles console, a remote-desktop session and any monitoring dashboard, leaving the RTX 3060 free to hold nothing but model weights — which, on a 12 GB card, is worth 1–2 GB of reclaimed VRAM that would otherwise go to the desktop compositor.

The 65 W TDP figure is the other half. On a machine running 8,760 hours a year, a 30 W platform difference is roughly 263 kWh annually — about $48 at the June 2026 US residential average of 18.34¢/kWh, every year, forever. That is a meaningful fraction of the CPU's price recovered in the first two years, and the 5600G's 4.4 GHz boost is more than adequate for tokenization and request handling when the GPU does the inference.

See price & specs (price may vary — verify at checkout) · See Full Details →

#3 🎯 Best for Quiet 24/7 Operation: Noctua NH-U12S

120 mm NF-F12 PWM fan · 22.4 dB(A) max · 1,500 RPM max · 158 mm height · 6-year warranty

✅ 22.4 dB(A) at maximum fan speed per Noctua's own specification — and an always-on inference box rarely runs the CPU near maximum ✅ No pump. Nothing circulates, nothing permeates, nothing seizes at 3 a.m. in year four ✅ 158 mm height and a single 120 mm tower clear most mid-towers and many small cases without RAM conflicts ✅ Six-year warranty on a part expected to run continuously — the failure math is the point

❌ Costs more than adequate air coolers with similar thermal performance ❌ On a 105 W chip under sustained all-core load, a 140 mm tower or a 240 mm AIO has more headroom

The acoustic argument for air on a 24/7 machine is not really about decibels at load. It is about failure modes. An AIO's pump is a moving part with a duty cycle of 100% and a lifespan measured in years, and when it degrades it does so audibly first and catastrophically second. A tower cooler's only moving part is a fan you can replace in five minutes for $20.

The load profile helps too. On a build where the GPU handles inference, the CPU spends its time on tokenization, request routing and waiting — a light, steady load, not a sustained all-core burn. Under that profile the NF-F12 sits far below its 1,500 RPM ceiling, and the 22.4 dB(A) rating is a worst case you rarely reach. The options are compared directly in Cooling a 24/7 Local LLM Rig: Air vs 120mm AIO vs 240mm AIO.

See price & specs (price may vary — verify at checkout) · See Full Details →

#4 ⚡ Best Performance: AMD Ryzen 7 5800X

8 cores / 16 threads · 105 W TDP · 4.7 GHz boost · 32 MB L3 · AM4 · $449 MSRP

✅ Eight cores and 32 MB of L3 give real CPU-offload throughput when a model spills past the GPU's 12 GB ✅ 4.7 GHz boost per AMD keeps prompt tokenization off the critical path on long inputs ✅ Drops into the same AM4 platform, so nothing else in the build changes ✅ Headroom to run inference plus the rest of a homelab — containers, a media server, backups — without contention

❌ 105 W TDP against the 5600G's 65 W: a 40 W gap under load, worth about $64/yr only if the box ran flat out around the clock at 18.34¢/kWh — idle sets the real gap, so measure it ❌ No integrated graphics, so you need a display output from the discrete card

This is the pick when the machine's job description includes models larger than 12 GB. Offloading is not free: the layers that live in system RAM are executed by the CPU, and their throughput sets the pace for the whole request. Set expectations with real numbers. In llamafile's public CPU benchmark thread, a 16-core Zen 3 Ryzen 9 5950X running Llama 3.1 8B at Q4_K_M entirely on the CPU records 100.20 tok/s of prompt processing and 10.53 tok/s of generation. The 5800X uses the same Zen 3 core with half the core count, so budget for less on prompt processing. Generation on the CPU side is bounded by memory bandwidth rather than cores. Dual-channel DDR4-3200 moves about 51.2 GB/s, and every generated token reads each CPU-resident weight once. A 70B model at Q4_K_M, around 42 GB, that lives mostly in system RAM therefore tops out near 1 tok/s on any AM4 chip. Offload is for models a little past 12 GB, not a way to run 70B at a usable pace.

The second argument is contention. An always-on box tends to accumulate jobs: a vector database, a document ingester, a cron that re-embeds a folder every night. Six cores get tight when inference requests overlap with those. Eight cores and sixteen threads leave room for the machine to be a server rather than a single-purpose appliance. Pair it with 64 GB of DDR4-3200 if offloading is part of the plan — the offload path is bounded by system memory capacity and bandwidth, as covered in Real Productivity on 32-64GB RAM for Local LLMs.

See price & specs (price may vary — verify at checkout) · See Full Details →

#5 🧪 Budget Pick: Kingston A400 960 GB SATA SSD

960 GB · SATA III · 2.5-inch · ~$152 listed 2026-09-18

✅ Just under a terabyte for the price of a mid-range cooler — capacity is the specification that matters here ✅ SATA III works in any machine, in any bay, with no PCIe lanes consumed by the model library ✅ Frees the boot NVMe for the OS and containers rather than a 40 GB GGUF collection ✅ Silent, low-power, and cool — three properties that matter more on an always-on box than sequential throughput does

❌ SATA's ~550 MB/s ceiling makes cold-loading a large model slower than on NVMe ❌ DRAM-less controller means sustained large writes taper; this is storage for reading models, not a scratch disk

The number people get wrong is how much space a real model library takes. A 14B model at Q4_K_M is roughly 8.5 GB. A 32B at the same rung is about 19 GB. Then you keep two quantization rungs of your daily driver to compare quality, add an embedding model, a vision model, a coding-specialized fine-tune, and a couple of things you downloaded to test and never deleted. Thirty models is an ordinary collection for anyone experimenting seriously, and it lands well past 500 GB.

Cold-load time is the only performance axis that matters, and it is dominated by file size rather than interface generation — a model loads once per session and then lives in VRAM for the rest of the day. On a machine that never restarts, that cost is paid approximately never. Buying capacity instead of speed here is the correct trade, and the full comparison is in NVMe vs SATA SSD for Local LLM Model Libraries in 2026. Kingston's own A400 product page lists the drive's SATA III interface and 2.5-inch form factor.

See price & specs (price may vary — verify at checkout) · See Full Details →

Alternate GPU if the top pick is out of stock

The ZOTAC Gaming GeForce RTX 3060 Twin Edge OC 12 GB is the same GA106 silicon, the same 12 GB of GDDR6 and the same 192-bit bus as the top pick, in a slightly different two-fan shroud. Every benchmark figure quoted above applies to it unchanged, because the differences between board partners on this card are a factory clock bin and a cooler design, not capacity or bandwidth. It is listed at $499.99 in the SpecPicks catalog as of 2026-09-18 — prices move, so compare both listings on the day. If you are considering two cards for 24 GB aggregate, a matched pair is the cleaner path: Best Parts for a Dual RTX 3060 24GB Local-LLM Build in 2026.

See price & specs (price may vary — verify at checkout)

What to look for in an always-on LLM server

VRAM before everything else

Every other specification is a tuning parameter; this one is a cliff. A model that fits runs at the GPU's speed. A model that does not fit runs at the speed of whichever layers landed in system RAM, and that penalty applies to every token of every request for as long as you keep the model loaded. Twelve gigabytes is the practical floor for a machine meant to host a 14B-class assistant, and there is no clever configuration that recovers the gap.

Idle power is the number that compounds

An inference server's duty cycle is mostly waiting. Board power ratings describe a state your machine will occupy for minutes a day. What runs for 8,760 hours a year is idle draw — the platform, the drives, the fans, the GPU sitting at its floor clock. A 30 W difference in idle is roughly 263 kWh and $48 a year at 18.34¢/kWh. The measured numbers are in the power section below. Measure at the wall with a plug meter rather than adding up TDP figures, because the two rarely agree.

Noise floor and where the box lives

A machine that is inaudible can live in an office. A machine that is not has to live in a closet, which means worse airflow, higher temperatures and more fan speed — a loop that makes the problem worse. Choose components with a low noise floor at low load rather than good acoustics at full load, because low load is where this machine lives. Large slow fans, a tower cooler and a case with unrestricted intake do more than any single premium part.

Model-library storage and cold-load time

Plan for more capacity than seems reasonable. Quantization rungs multiply, models accumulate, and nothing gets deleted. Cold-load time scales with file size, and on a machine that stays up it is a once-per-session cost. Capacity per dollar beats sequential throughput here by a wide margin.

Uptime, headless operation and remote access

The best always-on box is one you never sit in front of. Run it headless with SSH and a reverse proxy in front of the inference endpoint. Set the BIOS to power on after AC loss so an outage does not require a trip. Put the service under systemd or a container runtime with a restart policy. Log token throughput and GPU temperature to something you can look at later — the failure you want to catch is the slow one, where thermals creep and clocks sag over months.

Quantization and VRAM: what a 12GB server can hold

Those weight figures are the floor. Add roughly 0.5 GB of KV cache per 4K of context on an 8B model and about double that on a 14B, plus a compute buffer. The measured numbers line up: llmrun.dev reports 9.5 GB of VRAM for a 14B model at q4_K_M on this card, and Hardware Corner reports 6.0 GB for Qwen3 8B at q4_K_XL — leaving 3 GB and 6 GB of headroom respectively for context on a 12 GB board.

The practical recommendation is Q4_K_M on a 14B model, or Q4_K_M on an 8B model if you want a 32K context window. Quantizing the KV cache to 8-bit reclaims roughly half the cache footprint at modest quality cost and is the first setting to change when you run out of room.

Power draw and tokens per watt: what the wall meter says

Spec-sheet TDP tells you what a part is allowed to draw. What you pay for is what the whole box pulls from the wall, and that number only comes from measurement. Jeff Geerling's ai-benchmarks project is one of the few public data sets that records whole-system power next to tokens per second. The watts are peak readings taken at the wall, so treat every per-watt figure below as a conservative lower bound on efficiency.

The RTX 3060 rows come from his RTX 3060 12 GB test log. They were run with llama.cpp's Vulkan back-end on two hosts: an Intel Core Ultra 7 265K desktop, and a Raspberry Pi CM5 driving the card over PCIe. Neither host is the AM4 platform in this guide, so read them as what the card does, not as a quote for the full build.

Two things stand out. First, the 3060 box pulls the same ~224 W whether it is generating a 7B or a 13B model, so tokens per watt falls almost exactly in step with model size. The card runs flat out either way; a bigger model just takes longer per token. Second, per token the GPU is not the energy hog it looks like. At 1.74 J per token on a 3B model, the 3060 system is more efficient than a Raspberry Pi 400 running the same model on its CPU (3.75 J per token). The Pi only wins at the one thing it is built for: sitting idle.

Why idle, not throughput, sets the bill

Put numbers on a realistic day. Say the box serves 200 requests averaging 400 generated tokens each. That is 80,000 tokens. On the 3060 at 61.53 tok/s on a 7B model, generation takes about 22 minutes, a duty cycle of roughly 1.5%. At 3.64 J per token, those 80,000 tokens cost about 0.08 kWh a day, or around 30 kWh and $5.42 a year at the EIA's June 2026 US residential average of 18.34¢/kWh.

Idle runs the other 98.5% of the day, and it is billed by the hour, not by the token:

A desktop idling at 40 W spends about 12 times more on waiting than on the tokens it generates in that example. That is why the picks above favour a 65 W APU host over anything faster. It is also why the most useful thing you can own for this build is a $15 plug-in meter. Measure your own idle before and after every change, because the platform, drives, fans and BIOS power settings move it more than any single spec does.

Power-limit the GPU before you shop for a smaller one

Token generation is bound by memory bandwidth, not core clocks. The 3060's 360 GB/s of GDDR6 is the ceiling long before its 3,584 CUDA cores are. That leaves slack you can take back as watts. NVIDIA's nvidia-smi documentation covers the switch: nvidia-smi -pl <watts> sets the board power limit, and nvidia-smi -q -d POWER reports the minimum and maximum limits your card's firmware allows.

The procedure takes ten minutes:

1. Run your normal model with llama-bench or your serving stack's own timing and note generation tok/s at the stock limit.
2. Lower the limit in 10 W steps, re-running the same test each time.
3. Keep the lowest limit at which your measured tok/s has not dropped, then step back up one notch for margin.

Prompt processing is compute-bound and will slow down first. If your workload is long documents rather than chat, weigh prefill time too. The limit does not survive a driver unload or a reboot, so apply it from a systemd unit at boot. And because the limit only caps load draw, it does nothing for idle. The two optimizations are separate, and you want both.

Size the PSU for where the box actually sits

Every 80 Plus certificate is a promise at specific load points. The 80 Plus tiers on Wikipedia show that for 115 V internal supplies, Gold guarantees 87% at 20% load and 90% at 50% load, and defines nothing below 20%. Titanium is the only consumer tier with a 10% requirement (90%). An always-on inference box idles far below 20% of any sensible PSU. A 750 W unit feeding a 40 W idle load is running at about 5%, below the lowest point any Gold certificate covers.

In practice, don't oversize. The 3060 systems above peaked at about 224 W at the wall. A quality 450–550 W Gold unit puts that peak near the 50% sweet spot, and it keeps idle as close to the 20% line as a small unit allows. Go bigger only if a second GPU is actually on the roadmap. If idle efficiency matters most to you, a Titanium unit is the only tier certified at 10% load.

The Raspberry Pi tier: always-on front end, not the inference engine

Geerling's Raspberry Pi 4 Model B 8GB test measured 1.6 W idle at the wall and 5.0 W under a full stress-ng load, with memory copy bandwidth of about 2.7 GB/s. That idle figure is why a Pi 4 costs about $2.57 a year to leave running. The bandwidth figure is why it is the wrong place to generate tokens. The Pi 400 in the table above uses the same BCM2711 chip at a higher clock and manages 1.60 tok/s on a 3B model, so a Pi 4 will be no faster.

The pattern that gets the best of both is a split stack:

• The Raspberry Pi 4 Model B 8GB runs the chat UI, the request queue and the model catalog around the clock. A walkthrough is in Open WebUI on a Raspberry Pi 4.
• The RTX 3060 machine sleeps and is woken over the network (Wake-on-LAN) when a request needs real throughput.
• Small classification or routing jobs that tolerate a few tokens per second stay on the Pi.

The cost is orchestration work and a cold-start delay of several seconds while the GPU box resumes and loads the model. If you need instant first-token latency around the clock, keep the GPU box awake and spend your effort on its idle draw instead. The Pi 4 8GB listing is $178.81 in the SpecPicks catalog as of 2026-09-18, well above the $75.00 Geerling paid in 2020, so compare it with an authorized Raspberry Pi reseller before you buy. For how a Pi 4 compares with a 5600G host on small models directly, see Qwen3 4B on a Raspberry Pi 4 8GB vs a Ryzen 5 5600G.

Common pitfalls

• Buying the GPU for peak tokens per second. On a single-user box, anything past ~30 tok/s outruns your reading speed. Capacity is the specification that changes what the machine can do.
• Adding up TDP figures instead of measuring at the wall. Spec-sheet numbers describe load states this machine rarely enters. A $15 plug meter tells you what you are actually paying.
• Letting the inference GPU also drive the display. A desktop session can hold 1–2 GB of VRAM. On a 12 GB card that is the margin between a 14B model fitting and not.
• Sizing storage for today's model collection. It doubles. Then it doubles again when a new quantization format lands.
• No restart policy. A power blip at 4 a.m. that leaves the inference service down until you notice defeats the entire point of "always-on." Configure BIOS power-on-after-AC-loss and a systemd or container restart policy.

When NOT to build this

Do not build it if your model needs are occasional. A few hours a month of frontier-quality output is cheaper to rent, and an idle machine costs money whether you use it or not. Do not build it if what you actually want is a resident 32B model at a comfortable quantization — 12 GB will not do that, and the correct answer is a 24 GB card or two 12 GB cards, not this parts list with a bigger PSU. And do not build it if the box has to live in a bedroom and you have not budgeted for acoustics; a machine you turn off at night to sleep is not an always-on server, and every design decision here assumes it never stops.

Sources

1. TechPowerUp — GeForce RTX 3060 12 GB specifications (accessed 2026-09-09)
2. TechPowerUp — AMD Ryzen 5 5600G review (accessed 2026-09-09)
3. Noctua — NH-U12S specifications (accessed 2026-09-09)
4. Hardware Corner — RTX 3060 12 GB LLM benchmarks (accessed 2026-09-09)
5. TyoLab — 64 GB RAM, 12 GB VRAM: the honest local LLM benchmark (accessed 2026-09-09)
6. Jeff Geerling — ai-benchmarks, whole-system power and tokens per second (accessed 2026-09-18)
7. Jeff Geerling — RTX 3060 12 GB llama.cpp test log (accessed 2026-09-18)
8. Jeff Geerling — Raspberry Pi 4 Model B 8GB power and memory tests (accessed 2026-09-18)
9. US EIA — Electric Power Monthly, Table 5.6.A, June 2026 (accessed 2026-09-18)

Related guides

• RTX 3060 12GB Local LLM Guide: Which Models Actually Fit (2026)
• Best GPU for a NAS or Homelab Running Local LLMs (2026)
• Best Parts for a Dual RTX 3060 24GB Local-LLM Build in 2026
• Budget Ollama Build Under $500: What Actually Runs in 2026
• Ryzen 5 5600G vs Ryzen 7 5700X for a 24/7 LLM Server

— Mike Perry

Citations and sources

• TechPowerUp — GeForce RTX 3060 12 GB specifications (accessed 2026-09-09)
• TechPowerUp — Ryzen 5 5600G power consumption and efficiency (accessed 2026-09-09)
• TechPowerUp — AMD Ryzen 5 5600G review (accessed 2026-09-09)
• AMD — Ryzen 7 5800X product page (accessed 2026-09-09)
• Noctua — NH-U12S specifications (accessed 2026-09-09)
• Kingston — A400 SATA SSD product page (accessed 2026-09-09)
• Hardware Corner — RTX 3060 12 GB LLM benchmarks (accessed 2026-09-09)
• TyoLab — 64 GB RAM, 12 GB VRAM: the honest local LLM benchmark (accessed 2026-09-09)
• llmrun.dev — RTX 3060 12 GB model throughput (accessed 2026-09-09)
• singhajit.com — LLM inference speed comparison (accessed 2026-09-09)
• Jeff Geerling — ai-benchmarks, whole-system power and tokens per second (accessed 2026-09-18)
• Jeff Geerling — RTX 3060 12 GB llama.cpp test log (accessed 2026-09-18)
• Jeff Geerling — Raspberry Pi 4 Model B 8GB power and memory tests (accessed 2026-09-18)
• US EIA — Electric Power Monthly, Table 5.6.A, June 2026 (accessed 2026-09-18)
• NVIDIA — nvidia-smi documentation, power limit options (accessed 2026-09-18)
• Wikipedia — 80 Plus certification levels (accessed 2026-09-18)
• llamafile — CPU benchmark discussion #450 (accessed 2026-09-18)

This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.