On-Device AI Keyboards: What a Sub-2GB LLM Needs to Run Local

Yes — but with important caveats. A "phone keyboard" LLM is a 1-3B-parameter model at 4-bit quantization, which needs roughly 1-3 GB of VRAM. Any modern desktop GPU with 8 GB or more of VRAM can host it, and a 12 GB card like the ZOTAC RTX 3060 Twin Edge has enough headroom for a longer context window and simultaneous models. What you gain in speed over a phone NPU, you give up in efficiency — the 3060 draws roughly 170 W under sustained load versus the phone's sub-2 W envelope.

Why on-device keyboard agents matter for local-inference builders

Acti's push to run its keyboard agent locally on-device is only the most visible example of a broader trend. Model vendors have spent 2025 and 2026 shrinking useful language models to fit the memory and thermal envelope of a phone, which means the same class of model — sub-2 GB weights, sub-3 W runtime — is now available to anyone who wants to self-host a comparable stack on a desktop. That is a very short list of very specific hardware.

The obvious question for anyone building a local-inference rig is whether "phone-class" translates to "trivial on desktop." It does — but not the way most people assume. A phone runs one small model at a time, quantized aggressively, on a specialized NPU tuned for low-power sustained inference. A desktop, even a cheap one, can host that same model on a discrete GPU running one or two orders of magnitude faster. What changes is the cost model: watts, dollars, and the amount of context you can hold in memory before latency spikes.

This piece walks through what a phone-class keyboard LLM actually needs on desktop hardware, why the RTX 3060 12 GB has become the reference bench for this workload, and where paired options like the AMD Ryzen 5 5600G fit for always-on hosting. Public benchmarks referenced throughout are from TechPowerUp, llama.cpp release notes, and Phoronix compute reviews — no first-party lab measurements are reported.

Key takeaways

• A 1.5-3B-parameter model at q4_K_M fits in 1.5-2.5 GB of VRAM for weights, plus ~0.5-1 GB for KV cache at short context.
• A 12 GB RTX 3060 will run circles around any phone NPU on raw tok/s, but the phone wins on watts by roughly two orders of magnitude.
• Quantization is the most important lever: q4_K_M and q5_K_M are the sweet spots for a keyboard-class model; q2/q3 usually degrade coherence too much.
• Keyboard latency is dominated by prefill on short prompts, so tok/s numbers understate what the user actually experiences.
• If you need one card to prototype phone-class LLMs on a budget, the RTX 3060 12 GB remains the cheapest 12 GB path in 2026.

How small are "phone keyboard" LLMs, and what VRAM do they need?

Vendor and researcher disclosures put "phone-class" language models in the 1B-3B-parameter range. The models Acti and its peers have deployed for on-device autocomplete and short-form generation live at the low end of that band, typically 1B-2B parameters with aggressive quantization to keep weights under 2 GB. That number is not accidental — it is chosen so the model can co-reside in RAM with the operating system and application state on a mid-range phone.

Here is what those sizes actually cost on a desktop GPU, weight-only:

Numbers are approximate weight sizes as reported by llama.cpp release notes and community-published gguf catalogs. Actual on-disk sizes vary slightly with tokenizer size and how the metadata is packed.

You have to add the KV cache to any of these to get realistic VRAM footprint. For a 2B model at q4_K_M with a 4,096-token context window and fp16 KV cache, budget another ~500 MB. For 8,192 tokens, budget ~1 GB. At the 32k-token context that a phone keyboard rarely needs but a desktop bench sometimes tests, KV can push the working set well past 2 GB.

The upshot: even a 6 GB desktop GPU can host any of these models comfortably at q4_K_M. A 12 GB card is objectively over-provisioned for a single phone-class model — the reason it's still the reference bench is that 12 GB is the cheapest tier where you can hold two models, or one model plus a heavier vision component, without any offload to system RAM.

Which desktop GPU class matches a phone NPU on these models?

You can approach this two ways: match throughput, or match efficiency. They point in very different directions.

For raw tok/s, essentially any modern discrete GPU with FP16 tensor throughput above roughly 10 TFLOPS clobbers a phone NPU on a 1B-3B model. The bottleneck on desktop is memory bandwidth, not compute. A Zotac RTX 3060 12 GB — 360 GB/s of GDDR6 bandwidth per TechPowerUp — reads a 2B q4_K_M weight file (~1.2 GB) many times per generation step and returns a token every few milliseconds under load. A phone NPU with an internal memory hierarchy in the tens of GB/s cannot compete on this axis.

For watts per token, the picture inverts. A phone NPU generating 5-15 tok/s at under 2 W of package power is roughly 10-30x more energy-efficient per token than a desktop 3060 doing 40-60 tok/s at 170 W board power. The desktop wins on wall-clock; the phone wins on the total energy bill. Both matter, depending on whether you are prototyping or shipping.

Approximate generation throughput on a 12 GB desktop card vs a phone NPU running the same class of model:

Desktop tok/s figures are extrapolated from public llama.cpp CUDA benchmarks and community reports on Phoronix — see the Phoronix RTX 3060 compute review for CUDA throughput baselines. Phone NPU tok/s are from vendor demo videos and third-party teardowns; exact numbers vary by SoC generation.

If you are building the same model that a phone runs and want to iterate on prompt engineering, tool integration, or fine-tuning, the 3060 is roughly the price floor of "meaningfully faster than the phone." Cards with less than 12 GB (a 3060 8 GB, a 6600 XT) work for single-model prototyping but leave no room for a longer context or a second model.

Quantization matrix: what fits and what breaks quality

Quantization is the single lever with the biggest impact on VRAM per model. For a keyboard-class LLM (1.5B-3B parameters), here is what each format costs and what it buys you:

Relative tok/s numbers are from llama.cpp benchmarks on CUDA — the win from lower-bit quants is a bandwidth reduction, not a compute reduction, so throughput scales roughly with weight-file size.

For a phone-class keyboard model, q4_K_M is the standard target and the smallest quant that reliably preserves output quality for autocomplete-length responses. q5_K_M is the safer choice on desktop where the extra VRAM is essentially free. Anything at q3 or below is worth testing before you rely on it — the quality loss is small in isolation but compounds over multi-turn context.

Prefill vs generation: why keyboard latency is a prefill story

Keyboard agents rarely produce more than 5-30 tokens per request. That changes what "latency" means. The user's felt latency is dominated by two things: the round-trip to first token (which prefill cost drives), and the pause between the input and the visible completion.

A 3060 12 GB will prefill a 200-token prompt on a 2B model in the low tens of milliseconds. On a phone NPU, prefill on the same prompt takes hundreds of milliseconds. The desktop wins big on this axis, because prefill is dense matrix work that maps well to GPU tensor cores. Generation, by contrast, is a memory-bandwidth-bound loop — the phone loses relatively less on generation because it never has to keep a large batch of parallel work in flight.

For a keyboard-style workload (short prompt, very short completion), the practical implication is that a good desktop card feels dramatically faster than a phone even on the same model, because the delay before the first character shows up is roughly an order of magnitude smaller. The phone catches up on total energy per interaction, but never on wall-clock.

Context length: where 12 GB starts to pay off

Small LLMs are cheap to host at 2k-4k context. They are expensive to host at 32k or 64k, because the KV cache scales linearly with context length and the constant factor is set by hidden size and head count.

For a 2B model with a hidden size of 2,048 and 16 heads, the fp16 KV cache is roughly 128 KB per token per layer, times layer count — this puts you at ~1 GB of KV for an 8k window and closer to 4 GB for a 32k window. Cache the KV in q8 or int8 and those numbers halve.

An 8 GB card can hold the model and comfortably support 4k-8k context. A 12 GB card supports 32k-64k context on the same model without offload, which is why keyboard-agent developers who want to test long-context autocomplete tend to buy the bigger card. On a phone NPU, context past a few thousand tokens is generally impractical because there is no separate memory to spend on it.

Perf-per-dollar and perf-per-watt: why the RTX 3060 12 GB became the reference

The RTX 3060 12 GB is not the fastest 12 GB card you can buy in 2026. It is, however, still the cheapest 12 GB card with real llama.cpp CUDA support, adequate memory bandwidth, and a mature software stack. That combination makes it the reference bench for anyone benchmarking small LLMs against a fixed 12 GB target.

Per Zotac's B08W8DGK3X listing and the MSI Ventus 2X alternate, the practical difference between AIB partners is cooling and warranty. Either is fine as a benchmark card.

Compared to a phone NPU: the 3060 costs perhaps 30x what a phone SoC costs, draws roughly 100x the power under load, and delivers roughly 10x the raw throughput on the same model. The wattage story is why phones are still the target platform for these models, and the throughput story is why desktops are still the target platform for iterating on them.

What to buy for an on-device-agent test bench

For a builder who wants to test the same class of model a phone keyboard agent runs, the parts list is short and cheap:

1. ZOTAC Gaming RTX 3060 Twin Edge 12 GB — the reference 12 GB card. Twin Edge cooler is quiet enough to leave on a desk, and Zotac's warranty is competitive at this tier.
2. MSI RTX 3060 Ventus 2X 12G — near-identical performance, better price fluctuation windows in 2026. Same 12 GB, same CUDA support.
3. GIGABYTE RTX 3060 Gaming OC 12G — three-fan variant if your case can fit it; cooler and slightly quieter under sustained load.
4. AMD Ryzen 5 5600G — the ideal always-on host CPU for a headless inference rig. Its Radeon iGPU handles display so the discrete card can dedicate itself to inference, and the 5600G is efficient enough to leave running overnight.
5. Crucial BX500 1 TB SATA SSD — the cheap storage tier for holding a dozen quantized model files without eating an NVMe slot.

That bench, complete, lands around $600-800 in 2026 depending on channel — a fraction of any workstation card, and the fastest way to iterate on phone-class LLMs at desktop speed.

Common pitfalls when moving a phone-class model to desktop

Three problems trip up new builders more than any other:

• Assuming tok/s is the felt latency. For a keyboard agent, prefill dominates. Optimize the prompt size before you obsess over generation throughput.
• Under-quantizing "to be safe." Running fp16 or q8 for a phone-class model on a 12 GB card wastes VRAM you could spend on context or a second model. q5_K_M is the safer default if you have not tested the workload.
• Ignoring the CPU host. Small-model inference on a discrete GPU is often bottlenecked by CPU-side tokenization and sampling loops. A cheap Ryzen 5 5600G is fast enough not to be the bottleneck; older Intel chips sometimes are.

When NOT to build this

If your goal is to ship a keyboard agent to end users on their existing devices, the desktop bench does not help you meet the shipping target. Phones are the constraint. The desktop is only useful for iterating on the model itself. Similarly, if you need to run a 7B or larger model — the class of local assistant chat models rather than phone keyboards — a 12 GB card is the floor, not the sweet spot, and a 16 GB or 24 GB card is a better long-term investment.

Bottom line

A 12 GB RTX 3060 will host any phone-class keyboard LLM at throughput a phone cannot match, for the cost of a mid-range GPU and about 100x the power draw. That trade — dollars and watts for speed — is the whole reason desktops remain the reference platform for iterating on small models even when the final shipping target is a phone. If you want to test what an on-device keyboard agent can and cannot do at reasonable latency, this is the cheapest way in as of 2026.

Related guides

• Best Budget Streaming Starter Kit for Twitch & YouTube in 2026
• Ryzen 5 5600G vs Ryzen 7 5700X for a Budget 1080p Gaming Build
• Building a Local AI-Agent Eval Rig After AISI's Benchmark Warning

Citations and sources

• TechPowerUp — GeForce RTX 3060 specifications
• llama.cpp — release notes and benchmark tables
• Phoronix — RTX 3060 compute review

This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.