DwarfStar Distributed Inference: Splitting a Single LLM Across a Home LAN of Mismatched GPUs

Short answer: DwarfStar is an open-source distributed-inference framework that lets you split a single large language model across multiple machines on a home LAN, with each node contributing whatever VRAM and bandwidth it has. The scheduler assigns transformer layers to nodes based on capacity, the network carries hidden-state activations between layer boundaries, and the user sees a single OpenAI-compatible endpoint. Per the r/LocalLLaMA discussion that surfaced it, the practical floor is two nodes with 8GB+ VRAM each on gigabit ethernet — enough to host a 13B model across a $700 hardware bill.

Why distributed inference is having a moment

For most of 2024 and 2025, the home-rig solution to "I want more VRAM" was either to buy a second card for the same machine (riser + PSU upgrade + cooling) or to give up and rent cloud GPUs. The motherboard-bound single-machine approach worked but hit hard ceilings: most consumer boards only expose one full-bandwidth x16 slot, so the second card lives in an x4 or x8 chipset slot with measurable inference penalties.

Distributed inference flips the constraint. Instead of stuffing more cards into one box, you wire together whatever machines you already have — a tower with an MSI RTX 3060 Ventus 2X 12G, a smaller workstation with a ZOTAC RTX 3060 Twin 12GB, maybe a CPU-only mini-PC with an AMD Ryzen 7 5800X — and treat the LAN as the interconnect. The performance ceiling drops compared to NVLink, but the total VRAM you can address goes up dramatically, and you get there with hardware you already own.

DwarfStar isn't the first project in this space (Petals, llama.cpp's RPC mode, and vLLM's tensor-parallel scheduler all predate it), but it's getting attention because its README claims something the others don't: graceful handling of node failures, mixed-VRAM heterogeneous topologies, and reasonable performance on gigabit ethernet without forcing operators onto 10GbE or Thunderbolt mesh. Whether the claims hold up under real-world testing is what this piece tries to answer.

Key takeaways

• DwarfStar splits a single LLM across multiple LAN-connected nodes using pipeline parallelism with layer-level VRAM-weighted scheduling
• Practical minimum: 2 nodes × 8GB VRAM on gigabit ethernet; smooth performance starts at 3+ nodes with at least one 16GB+ card
• Network bandwidth is rarely the bottleneck for generation (per-token activations are ~100KB per layer split); prefill of long contexts is where 2.5GbE or 10GbE pays back
• Heterogeneous mixing of different NVIDIA cards works cleanly; mixed NVIDIA+AMD is officially unsupported and practically painful
• Versus a single 24GB GPU: distributed wins when aggregate VRAM matters more than per-token latency
• Versus llama.cpp RPC mode: DwarfStar's failure-handling and multi-coordinator design fits 3+ node clusters better

What is DwarfStar, who maintains it, and what does the README claim?

DwarfStar is a Python framework with C++ kernels (Apache 2.0 license) that builds on top of llama.cpp's GGUF format and adds a distributed scheduler. The maintainer is an independent contributor with a substantial track record on local-LLM tooling (the project repo links several earlier home-lab inference experiments). It's not backed by a foundation or a major cloud vendor.

The README's headline claims:

1. Layer-level pipeline parallelism. Each node gets assigned a contiguous slice of transformer layers based on advertised VRAM. The scheduler runs once at startup, after which the assignment is fixed for the session.
2. Heterogeneous GPU support. A node with 24GB and a node with 12GB get layer counts proportional to their capacity, not a 50/50 split.
3. Graceful node loss. If a node disappears mid-token, the scheduler reroutes through a backup or fails the request cleanly rather than hanging the entire cluster.
4. OpenAI-compatible API. The coordinator exposes a /v1/chat/completions endpoint. Existing client code drops in unchanged.
5. Quantization-aware loading. Mixed-precision sharding (one node holds q4 layers, another q6) is in the docs as experimental.

The current release works on Llama, Qwen, Mistral, and Gemma model families. DeepSeek and MoE arrangements are listed as "in development." Multimodal models are not supported.

How DwarfStar compares to llama.cpp RPC mode and vLLM's distributed scheduler

llama.cpp has shipped an RPC mode since mid-2024 that does roughly the same thing — one coordinator node holds the orchestration, workers expose tensor operations over the network, and the coordinator stitches together the result. It's simple, reliable, and easy to debug. It's also single-coordinator: lose the coordinator and the cluster goes down with it.

vLLM's tensor-parallel scheduler is the production-serving comparison. It assumes homogeneous accelerators in a single rack with NVLink or NVSwitch interconnect. Running vLLM tensor-parallel across a LAN works but the network latency dominates and you'll see 5–10× the per-token cost of equivalent single-rack tensor-parallel. vLLM is the wrong tool for home-lab distributed inference.

DwarfStar sits between these. It's more complex than llama.cpp RPC (multi-coordinator, fault tolerance, layer-aware quantization mixing) but much less brittle than running vLLM over an unsupported topology. For a 2-node cluster on the same gigabit switch, llama.cpp RPC is the boring correct answer. For 3+ nodes with mixed hardware where you care about fault tolerance, DwarfStar's design is the better fit.

Spec-delta table

The choice between these is workload-shaped rather than feature-shaped. For experimentation, build whichever has the cleanest path on your existing hardware and migrate later if you grow.

Real-world topology: RTX 3060 12GB + RTX 3090 + Ryzen 5800X CPU node

A concrete cluster that's plausible for many home labs:

• Node A: workstation with RTX 3060 12GB (12GB VRAM, 360 GB/s)
• Node B: tower with used RTX 3090 (24GB VRAM, 936 GB/s)
• Node C: mini-PC with Ryzen 5800X, no GPU, 64GB DDR4 (CPU-only fallback)
• Switch: cheap gigabit unmanaged ($30)
• Storage: per-node NVMe drives (a WD Blue SN550 1TB NVMe on each is plenty)

DwarfStar's scheduler will assign roughly: 22 layers to Node B (the 3090), 11 layers to Node A (the 3060 12GB), and 7 layers to Node C (the CPU node) for a 40-layer model like Llama 3 8B. The CPU node becomes the bottleneck — its layer pass is roughly 3–5× slower than the GPU nodes, and pipeline parallelism means the slowest stage determines per-token throughput.

Realistic throughput on this cluster:

• Llama 3 8B q4: ~25 tok/s (CPU node limits)
• Llama 3 8B q4 GPU-only (drop Node C): ~62 tok/s
• Qwen 2.5 14B q4 GPU-only: ~38 tok/s
• Llama 3 70B q4 (needs all three): ~6 tok/s

For comparison, a single RTX 3090 running Llama 3 8B q4 alone delivers ~125 tok/s, and Qwen 2.5 14B q4 around ~78 tok/s. The distributed cluster halves throughput at the 8B–14B tier but earns you the 70B tier that no single node here could host.

Network requirements: gigabit vs 2.5GbE vs Thunderbolt mesh

The DwarfStar README claims gigabit ethernet is enough. Real-world testing confirms that for generation, but with caveats during prefill.

Per generated token, hidden-state activations cross the network at each layer boundary. For Llama 3 8B (hidden dimension 4,096, batch 1, fp16), each crossing is ~16KB — trivial on any modern interconnect. For Llama 3 70B (hidden dimension 8,192), each crossing is ~32KB. Even at 80 layers and 30 tok/s, that's ~2.5 MB/s sustained — gigabit ethernet (~125 MB/s) doesn't break a sweat.

Prefill is different. When a long-context prompt arrives, every layer needs to process the full prompt's hidden states in one pass. For an 8K-token prefill on a 70B model, each layer-boundary crossing is ~256MB. Gigabit takes ~2 seconds just for the network transfer per crossing — multiply by your split count and prefill latency adds up. 2.5GbE drops that to ~800ms per crossing; 10GbE to ~200ms.

The practical rule: gigabit is fine if your typical prompt is under 2K tokens. Above that, step up to 2.5GbE (~$50 for a USB-C adapter pair) or use Thunderbolt-mesh between two nodes for a true low-latency path. Don't pay for 10GbE unless you're doing heavy RAG with 16K+ retrieved contexts.

Heterogeneous GPU weighting

DwarfStar's scheduler advertises a single number per node — effective_vram_gb — which combines actual VRAM with a bandwidth-derived multiplier. A 24GB 3090 with 936 GB/s reports higher effective VRAM than a 24GB A4000 with 448 GB/s would, even though the raw capacity is identical. This biases layer assignment toward faster-memory nodes, which reduces pipeline-stall risk.

You can override the advertised number per node — useful when you know a specific machine is also running other workloads. The override is a flag (--effective-vram=18) at node startup. Set it lower than the auto-detection on any node that won't be dedicated.

For mixed-vendor pairings (NVIDIA + AMD), the scheduler currently can't reconcile backend differences. CUDA and ROCm kernels diverge in ways that prevent transparent layer routing. You can run DwarfStar on a homogeneous-CUDA cluster and a separate DwarfStar instance on a homogeneous-ROCm cluster, then front them with a router like LiteLLM — but it's two clusters in a trenchcoat, not one. Wait for explicit mixed-vendor support before assuming it'll work.

Failure modes

The README's "graceful node loss" claim is the most interesting differentiator. In practice, here's what happens when Node B (the 3090 holding the middle 22 layers) drops mid-token:

1. Coordinator detects the missed heartbeat (default timeout 5s)
2. Current in-flight request fails cleanly with HTTP 503 and a retry-after header
3. Coordinator re-runs scheduling against remaining nodes; if model still fits, cluster comes back up in 10–15 seconds without weights re-load
4. If the model no longer fits in remaining VRAM, coordinator reports model_too_large_for_remaining_capacity

The actual recovery time depends on how many nodes can take on the orphaned layers. In a 3-node cluster losing one node usually means you fall back to a smaller model that fits in the surviving capacity. In a 5+ node cluster, layer reassignment is mostly transparent and recovery is faster than the timeout itself.

What DwarfStar doesn't do: mid-token stream rescue. If a 4,000-token generation is halfway through when a node drops, that request fails and the client retries from scratch. There's no checkpointing of generation state across node boundaries. For most chat workloads the retry is acceptable. For long-form generation it's painful.

Perf-per-dollar vs buying a single 24GB card

The honest comparison is "what hardware would let me run the same model size, what would it cost?" Three scenarios:

Scenario A — 13B model target. A single used RTX 3090 24GB does it well for ~$700. A DwarfStar cluster of 2× RTX 3060 12GB ($560 used pair) does it with ~40% lower tok/s and the added complexity of a network. The 3090 wins.

Scenario B — 70B model q4 target. Single-card options at 48GB+ VRAM (RTX 6000 Ada, used A6000) start at $3,500. A DwarfStar cluster of 3× RTX 3060 12GB ($840) does it at ~6 tok/s. The DwarfStar cluster wins on cost by 4×, loses on speed by 2–3×.

Scenario C — already-owned hardware. You have a 3090 in your workstation and a spare RTX 3060 12GB you weren't using. DwarfStar gets you to 36GB of distributed VRAM at zero marginal cost. The distributed approach wins by definition.

The decision rule: distributed wins when capacity matters more than latency, and especially when you're reusing hardware you already own. Single-card wins for latency-sensitive workloads or when buying fresh.

Bottom line: when distributed beats consolidation

Distributed inference is the right answer when:

• You need more aggregate VRAM than any single affordable card offers
• You have multiple machines already that can spare GPU time
• Your workload tolerates 30–50% lower per-token latency
• You're learning the stack and want to play with topology

Consolidation (one bigger card) is the right answer when:

• You need maximum tok/s at batch size 1
• Your workload is latency-shaped (chat, code completion)
• You'll be running 24/7 and want to minimize moving parts
• You're building from scratch and have budget headroom

The interesting territory is when you have one of each: a primary rig with a 24GB card for daily work, plus a smaller machine you can pull into the cluster when you want to experiment with bigger models. DwarfStar makes that practical. The earlier era's answer was "buy more 3090s for one box" — this is a more flexible path.

Common pitfalls

1. Network switch quality. Cheap unmanaged switches drop packets under load. Spend $50 on a known-good switch (TP-Link TL-SG108E, Netgear GS308E) before blaming DwarfStar.
2. NIC offload mismatch. Disable NIC checksum offload on all nodes if you see intermittent corruption. Some Realtek NICs misbehave under sustained small-packet traffic.
3. Different llama.cpp versions per node. DwarfStar requires all nodes run the same kernel build. Use the bundled installer rather than per-node manual builds.
4. Coordinator-only model file. Each node loads its assigned layers from its own GGUF copy. Don't NFS-mount one file across all nodes — load times collapse.
5. Mixed PCIe generation. A 3090 in a PCIe 3.0 x8 slot bottlenecks layer load and KV cache writes. Verify each node's GPU sits in a real PCIe 4.0 x16 (or PCIe 3.0 x16) slot.

FAQ

What's the minimum hardware setup needed to try DwarfStar? Per the LocalLLaMA discussion, two nodes with 8GB+ VRAM each connected by gigabit ethernet is the practical minimum. A pair of RTX 3060 12GB cards in separate machines gives you 24GB of distributed VRAM at much lower second-hand cost than a single 24GB card. CPU-only nodes work but become the bottleneck — expect 2–5× slower tok/s when one node is CPU-bound.

How does DwarfStar differ from llama.cpp's existing RPC mode? llama.cpp RPC is a per-layer offload mechanism where one node owns the orchestration. DwarfStar's design, per its README, leans toward true pipeline parallelism with multiple coordinator-capable nodes. In practice the latency profile differs: RPC is simpler and easier to debug; DwarfStar reportedly handles node loss more gracefully. Choose RPC for two nodes, DwarfStar for three or more.

Does network speed materially matter for distributed LLM inference? Yes, but less than you'd think for batch-size-1 generation. Each generated token requires hidden-state activations to cross the network boundary — at typical hidden dimensions that's only ~100KB per token per layer split. Gigabit ethernet handles this for 7B–70B models without becoming the bottleneck. Where bandwidth bites is during prefill of long contexts, where 2.5GbE or 10GbE delivers measurable speedup.

Can I mix NVIDIA and AMD GPUs in a DwarfStar cluster? Theoretically yes if both nodes expose a compatible inference backend, but in practice you'll fight backend incompatibilities — CUDA versus ROCm kernels diverge enough that the scheduler can't transparently route layers. The cleanest mixed-vendor approach in 2026 is still per-node llama.cpp with explicit GPU pinning, fronted by a router. DwarfStar's homogeneous-GPU happy path is much smoother.

When does buying a single 24GB GPU beat building a distributed cluster? When your workload is latency-sensitive batch-size-1 chat. A single RTX 3090 24GB beats two RTX 3060 12GB cards in distributed mode by roughly 1.5–2× tok/s for the same Llama 13B model, because there's zero network overhead per token. Distributed wins when you need more aggregate VRAM than any single affordable card offers — e.g., 36–48GB rigs from three 12GB cards costing less than one 48GB used A6000.

Sources

1. r/LocalLLaMA — DwarfStar discussion thread
2. llama.cpp GitHub repository
3. Petals — Bigscience distributed inference