Qwen 3.6 27B at 2x tok/s: Luce DFlash on One RTX 3090
Luce DFlash is a speculative-decoding scheme that pairs a tiny draft model with Qwen 3.6 27B on a single 24GB card, so accepted drafted spans skip the full forward pass. On an RTX 3090 the published result is roughly a 2x tok/s jump versus vanilla decoding, as of 2026, with no change to the output distribution. The gain is largest when the draft and target both fit in VRAM.
Editorial intro
The single-24GB-GPU crowd sits in an awkward spot. You are past the 12GB card that can only really host 7B–13B models at useful context lengths, but you are also priced out of dual-GPU or workstation setups that host 70B checkpoints without heavy quantization. On paper, an RTX 3090 with 24GB of GDDR6X is the cheapest way to keep a modern 27B–34B model resident in VRAM, and second-hand pricing has kept it the community's default cost-per-GB-of-VRAM winner for local inference, per public data from TechPowerUp.
That leaves throughput as the bottleneck, not model fit. Qwen's 27B tier — hosted on Hugging Face and increasingly used as a mid-size instruction target — pushes a single 3090 to somewhere in the 25–35 tok/s range at 4–5 bit quantization on stock runtimes, according to community measurements posted on r/LocalLLaMA and the llama.cpp issue tracker. That is fine for one user but painful for interactive agents, retrieval pipelines, and batched evals.
Speculative decoding is the escape hatch. Instead of buying a second GPU, you spend a few gigabytes of VRAM on a tiny draft model that predicts the next few tokens, then let the 27B verify them all in one batched forward pass. When the draft guesses right — and for related-model draft/target pairs it usually does — you get multiple accepted tokens per big-model step. Luce DFlash is one recent implementation aimed at this exact 27B-on-24GB use case, and its "2x tok/s" claim on a 3090 is what makes it worth wiring into a build in 2026. This synthesis walks through the mechanism, the hardware around it, and where the free lunch stops.
Key takeaways - Luce DFlash roughly doubles Qwen 3.6 27B throughput on a single RTX 3090 by drafting several tokens ahead with a small model, then batching a single verification pass through the 27B. - The full win requires both draft and target in VRAM. A MSI GeForce RTX 3060 Ventus 3X 12G can only host the target with CPU offload, which caps the speedup. - 4-bit and 5-bit quantization (q4_K_M, q5_K_M) keep the 27B under ~18GB on a 3090 while leaving room for a 1B–3B draft and KV cache. - Sustained inference pins the 3090 near its 350W TDP, so an AMD Ryzen 7 5700X plus strong airflow and a fast Samsung 970 EVO Plus 250GB NVMe for model weights is the practical baseline pairing. - At 32k context DFlash still helps; at 128k the KV cache dominates VRAM and the effective speedup shrinks.
What is DFlash speculative decoding and how does it cut latency?
Speculative decoding, in the abstract, splits generation into two stages. A cheap draft model — often one or two orders of magnitude smaller than the target — proposes a short run of tokens, say 4 to 8, autoregressively. The target model then takes the full drafted prefix and runs a single batched forward pass that scores every drafted token in parallel. Any prefix of drafted tokens whose distribution matches the target's within the acceptance rule is accepted in one shot. The first mismatch is resampled from the target's own distribution, and drafting resumes from that token.
DFlash is the "flash-style" variant of that pattern, borrowing tricks from FlashAttention-family kernels to keep the verification pass memory-bound rather than latency-bound. In practice, that means the target's KV cache and the drafted-tokens' logits stay in on-chip SRAM as much as possible, cutting HBM round-trips on the 3090's GDDR6X, whose bandwidth ceiling of about 936 GB/s is documented on the TechPowerUp GA102 page. Because Ampere's tensor cores are already well-fed for the batched verification step, the effective work-per-token drops when acceptance is high.
Two implementation details matter for the RTX 3090 case. First, the draft model runs on the same GPU, so you spend ~1–3 GB of VRAM on the draft weights plus its own KV cache. Second, the acceptance rate depends heavily on how close the draft is to the target — a distilled or same-family draft (for example a 1.5B Qwen 3.x tuned to mimic the 27B) will land 60–80% of drafted tokens on typical prose, per implementer notes tracked in the llama.cpp speculative-decoding threads. Above that ballpark, roughly 2x throughput is the realistic ceiling on a single 24GB card. Below 40% acceptance the batched verification overhead can actually hurt.
Spec delta table: RTX 3090 vs RTX 3060 12GB vs RTX 4090
| Card | VRAM | Bandwidth | TDP | Approx. street price (2026) |
|---|---|---|---|---|
| RTX 3090 (used) | 24GB GDDR6X | 936 GB/s (TechPowerUp) | 350W | Varies by workload; used-market only |
| RTX 3060 12GB (new) | 12GB GDDR6 | 360 GB/s (TechPowerUp) | 170W | ~$630 for the MSI GeForce RTX 3060 Ventus 3X 12G |
| RTX 4090 (new) | 24GB GDDR6X | 1008 GB/s (TechPowerUp) | 450W | Varies by workload |
The takeaway from the delta is that the 3060 12GB and the 3090 are not in the same class for a 27B target. The 3060's 12GB can't hold the model in 5-bit even before you add a KV cache, so any DFlash-style speedup on a 3060 forces CPU offload of layers, which nukes the batched-verification benefit. The 3090 and 4090 both fit the model comfortably; the 4090 has more compute and slightly more bandwidth, but the 3090's used pricing is what makes DFlash on a single card interesting in the first place.
Benchmark table: baseline vs DFlash tok/s across 8B/27B/prompt sizes
Numbers below are community-reported ranges from public runs on stock llama.cpp, cross-checked against Qwen model cards on Hugging Face. Absolute tok/s varies by prompt content, sampler settings, and quantization; treat these as directional.
| Model / prompt | Baseline (RTX 3090) | DFlash (RTX 3090) | Approx. speedup |
|---|---|---|---|
| Qwen 3.6 8B, 512-token prompt | ~90 tok/s (llama.cpp discussions) | ~140 tok/s (llama.cpp discussions) | ~1.5x |
| Qwen 3.6 27B, 512-token prompt | ~30 tok/s (Qwen HF) | ~60 tok/s (Qwen HF) | ~2.0x |
| Qwen 3.6 27B, 4k prompt | ~25 tok/s (llama.cpp) | ~48 tok/s (llama.cpp) | ~1.9x |
| Qwen 3.6 27B, 16k prompt | Varies by workload | Varies by workload | Reported ~1.6–1.8x |
Two patterns fall out. First, the 8B baseline is already high enough that the drafting overhead eats some of the gain; DFlash still helps but the multiplier is smaller. Second, the 27B is the sweet spot: the target is expensive enough per token that even a modest acceptance rate pays for the batched verification cost. The gain scales roughly with the ratio of target-cost to draft-cost, which is why the 27B numbers are where the "2x" claim lives.
Quantization matrix for the 27B on 24GB: q4_K_M/q5/q6/q8 VRAM + tok/s + quality loss
For a 27B model on a 24GB card, quantization choice is what makes speculative decoding physically possible — you need headroom for the draft, the KV cache, and CUDA workspace on top of the target weights.
| Quant | Weight VRAM (approx.) | Baseline tok/s | Quality delta vs fp16 |
|---|---|---|---|
| q4_K_M | ~16 GB (llama.cpp GGUF docs) | ~30 tok/s (llama.cpp) | Small; noticeable on math benchmarks per community measurements |
| q5_K_M | ~18 GB (llama.cpp GGUF docs) | ~27 tok/s (llama.cpp) | Very small in MMLU-style evals |
| q6_K | ~21 GB (llama.cpp GGUF docs) | ~24 tok/s (llama.cpp) | Essentially fp16 parity on typical prompts |
| q8_0 | ~27 GB (llama.cpp GGUF docs) | Does not fit in 24GB | Reference quality; requires >24GB |
Practically, q4_K_M and q5_K_M are the two viable slots on a 3090 running Qwen 3.6 27B with DFlash on top. q4_K_M leaves ~6–7 GB for the draft model, its KV cache, and the target's KV cache, which is enough for a 1.5B–3B draft at moderate context. q5_K_M is tighter but preserves quality on math and code prompts, at the cost of shorter context before the KV cache spills.
How does prefill vs generation split change with speculative decoding?
Standard autoregressive decoding is dominated by the generation phase: each token is one full forward pass through the 27B, and the per-token cost is roughly constant. Prefill (the initial pass over the prompt) is a single batched op that is fast on Ampere because the tensor cores stay saturated.
Speculative decoding does not change prefill much — the prompt still runs through the target once, and the Ryzen 7 5700X or Ryzen 7 5800X handling the tokenizer and PCIe transfers is not the bottleneck there. What changes is the shape of the generation phase. Each "big step" now does one target forward pass that scores a batch of K drafted tokens, and if J of them are accepted, you have effectively produced J tokens for one target pass plus K cheap draft passes. When J is close to K, the amortized cost per token drops close to (draft cost) / J plus (target cost) / J — which is where the 2x number comes from.
The Ryzen 7 5700X's 8 cores and 65W TDP keep the tokenization and prefill orchestration off the critical path without heating the case as much as a 5800X, per AMD's product page. If you already own the AMD Ryzen 7 5800X — the 105W version of the same die — the extra clocks buy you very little in this workload because the GPU is the bottleneck.
Context-length impact: does DFlash hold up at 32k and 128k?
Context length is where the speculative-decoding story gets nuanced. The KV cache for a 27B model at 4-bit weights and fp16 KV grows roughly linearly with sequence length, and at 32k it already consumes several gigabytes. Community measurements on llama.cpp put a 27B q4 model at roughly 4–5 GB of KV cache at 32k with GQA-friendly settings, meaning your 24GB budget is starting to look crowded once you add a draft model and its own cache.
At 32k, DFlash still gives a meaningful speedup — reports in the 1.5–1.8x range are typical, versus the ~2x on shorter prompts — because the verification pass gets more expensive per token but drafting cost is still amortized. At 128k, the picture shifts: KV-cache-friendly attention kernels help, but the batched verification pass now touches so much cache per step that on-chip SRAM residency drops and DFlash's memory-bound advantages fade. Practical guidance is that 32k is safely inside the speedup envelope on a 3090; 64k is workable at q4_K_M; 128k is only realistic with aggressive KV-cache quantization and shorter draft windows.
Fast NVMe helps here indirectly: swapping model shards for A/B testing between drafts is faster off a Samsung 970 EVO Plus 250GB NVMe, and warm cold-starts from the SSD are shorter, per the drive's published sequential-read spec.
Perf-per-dollar and perf-per-watt: 3090 second-hand vs new 3060 12GB
The 3060 12GB is genuinely useful in this space — it's the cheapest way to run 7B–14B models fully in VRAM and to prototype a DFlash pipeline. But it's not the 27B target it's built for. Per TechPowerUp, the 3060 12GB has a 170W TDP and 360 GB/s of memory bandwidth, roughly a third of the 3090's. That bandwidth gap is what determines batched-verification cost for a 27B, so even if you could squeeze the model on with heavy CPU offload, DFlash's payoff shrinks quickly.
Perf-per-watt on the 3090 is worse than the 3060 in absolute terms, but per-tok/s on the 27B target the 3090 pulls comfortably ahead. Perf-per-dollar depends entirely on second-hand 3090 pricing in your region — the used market has settled well below MSRP for several quarters, per community trackers linked from r/LocalLLaMA. A new MSI GeForce RTX 3060 Ventus 3X 12G at ~$630 as of 2026 is a great "start here" card for local LLM tinkering, but for the specific 27B + DFlash workload, a used 3090 delivers throughput a 3060 can't reach at any wattage.
The AIO cooling angle is secondary but real. Sustained 350W on the 3090 heats the case, and the CPU handling prefill and sampling adds a few tens of watts on top. A capable air or 240mm liquid cooler on the Ryzen 7 5700X is enough; you don't need exotic cooling, but you do need airflow across the 3090's GDDR6X memory to keep junction temps in check during long batched runs.
Verdict matrix: Enable DFlash if… / Skip it if…
| Enable DFlash if… | Skip DFlash if… |
|---|---|
| You run Qwen 3.6 27B (or another same-family 27B) on a single 24GB card | Your target is already an 8B model with tok/s in the 100+ range |
| Interactive latency or agent throughput is the pain point | You care more about peak quality on long-context reasoning than throughput |
| You have room for a 1.5B–3B draft alongside q4_K_M or q5_K_M weights | You're on a 12GB card and would need CPU offload for the target |
| Prompts sit at ≤32k tokens most of the time | You routinely push 128k contexts and can't KV-quantize aggressively |
| You can pin a same-family draft to the target for high acceptance | Your draft model diverges stylistically (low acceptance kills the gain) |
Bottom line closing paragraph
Luce DFlash is the closest thing to a free lunch that single-GPU local-LLM builders have seen in a while, but the "free" part is conditional. On an RTX 3090 with 24GB of VRAM and Qwen 3.6 27B at q4_K_M or q5_K_M, community-reported tok/s roughly doubles versus a stock decode, and the output distribution is unchanged. On a 12GB card the physics don't line up — you'd be offloading the target and losing the batched-verification advantage. If you already own a 3090 or can get one on the used market, DFlash is the single highest-leverage change you can make in 2026 without buying a second GPU. If you're building fresh on a 3060 12GB, treat it as a 13B-class rig and revisit DFlash the day you upgrade the card.
Related guides
- Buying guide: best CPUs for local LLM inference
- RTX 3060 12GB benchmarks and product roundup
- Ryzen 7 5700X vs 5800X
- NVMe SSD picks for AI workstations
- GPU buying guide for AI workloads
Frequently asked questions
What is speculative decoding in plain terms?
A small fast draft model proposes several tokens ahead, and the large target model verifies them in one batched pass. When the draft guesses correctly the system accepts multiple tokens per big-model step, which is where the throughput gain comes from without changing the final output distribution meaningfully.
Do I need an RTX 3090 or will a 3060 12GB work?
The 27B model comfortably fits on the 3090's 24GB, so it is the natural target. A 12GB RTX 3060 can still run the model with CPU offload at lower speed, but the speculative-decoding upside is largest when both draft and target models sit fully in VRAM, favoring the 24GB card.
Does speculative decoding hurt output quality?
No, correctly implemented speculative decoding is lossless with respect to the target model because every drafted token is verified against the full model before acceptance. Any rejected draft token is simply discarded and recomputed, so you get the same distribution as standard decoding, only faster on accepted spans.
Why does an RTX 3090 need strong cooling for this?
Sustained inference keeps the GPU near its 350W ceiling for long stretches, and the CPU handling offloaded layers adds heat inside the case. A capable AIO like the Cooler Master ML240L keeps the CPU thermals in check, while good case airflow protects the 3090's GDDR6X memory junction temperatures during long batched runs.
Is a used 3090 a better value than a new mid-range card?
For local LLM work the 3090's 24GB is its headline advantage, letting it host models a 12GB card cannot fit. Second-hand pricing has fallen, so per-gigabyte-of-VRAM it often beats new mid-range cards, though you take on warranty and thermal-pad-age risk that a new card avoids.
Citations and sources
This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.
