Yes — an RTX 3060 12GB can fine-tune small models in the 500M–2B parameter range that ship in on-device AI keyboards, using QLoRA at q4 with a 4K context window. It cannot pretrain one from scratch, but for the actual workflow that matters — LoRA-tuning an open small model like Qwen 2.5 0.5B or Phi-3.5-mini on your own writing style — the 12GB card gets the job done in a weekend.
Why on-device AI keyboards are suddenly the story
Every mainstream phone announcement in the first half of 2026 has led with an on-device AI feature. Samsung's Galaxy One UI 8, Google's Pixel Feature Drop, Apple Intelligence, and the AOSP fork Nothing ships all now include some form of AI autosuggest, next-sentence completion, or on-device summarization. The interesting technical point buried in every one of those launches is that the models running the show are small — very small — because they have to fit in a phone's shared LPDDR memory pool alongside the OS and every other running app.
Google is running Gemma 2 2B and 4B distillations. Samsung is running Qwen 2.5 1.5B for text tasks. Apple's on-device model is undisclosed but analysis of the model file sizes puts it around 3B. All three are quantized to 4-bit weights (or Apple's mixed-precision variant) and use grouped-query attention to keep KV cache tiny.
The reason a homelabber should care: these are the smallest models actually shipping in production, they are permissively-licensed for the open ones, and they are the exact class of model an RTX 3060 12GB can fine-tune fully on-box.
Key takeaways
- Production on-device models sit in the 0.5B–4B range. Anything larger will not fit alongside a modern phone OS.
- Weight sizes at q4: 0.5B ≈ 350 MB, 1.5B ≈ 950 MB, 3B ≈ 1.9 GB, 4B ≈ 2.5 GB. All fit trivially in an RTX 3060.
- QLoRA fine-tuning of a 3B model on the RTX 3060 12GB runs at ~2,100 tokens/sec forward+backward, closing a 50k-example run in roughly 6 hours at 4K context.
- Pretraining is out of reach on 12 GB — you want at least an RTX 3090 24GB and preferably 2× A6000 for even a small 0.5B pretrain.
- Ollama and llama.cpp already support all the shipping on-device architectures for inference; fine-tuning is a
torch+bitsandbytes+peftjob.
The models that actually ship in 2026 keyboards
| Model | Params | Weight size (q4) | On-phone use |
|---|---|---|---|
| Gemma 2 2B | 2.6B | ~1.6 GB | Google Pixel autosuggest |
| Gemma 3 4B | 4B | ~2.5 GB | Google Assistant on-device |
| Qwen 2.5 1.5B | 1.5B | ~950 MB | Samsung One UI 8 text tasks |
| Qwen 2.5 0.5B | 494M | ~350 MB | Wearable / earbud tier |
| Phi-3.5-mini | 3.8B | ~2.4 GB | Windows Copilot on-device fallback |
| SmolLM2 1.7B | 1.7B | ~1.1 GB | Nothing OS, custom AOSP builds |
| Llama 3.2 1B | 1.2B | ~750 MB | Meta on-device edge |
| Llama 3.2 3B | 3.2B | ~2.0 GB | Meta on-device edge (bigger tier) |
All of the above ship on Hugging Face with permissive licenses; several are already indexed by the Ollama model library for one-line local pulls. The keyboard-shipping variants are the same open weights the vendors distill from — you can grab exactly the model your phone is running and fine-tune it at home.
What the RTX 3060 12GB can inference
Fast. Small-model inference is trivial on this card. Community measurements on llama.cpp at q4_K_M with the RTX 3060 12GB:
| Model | Tokens/sec | VRAM used |
|---|---|---|
| Qwen 2.5 0.5B q4 | 480 | ~1.1 GB |
| Llama 3.2 1B q4 | 340 | ~1.6 GB |
| Qwen 2.5 1.5B q4 | 220 | ~2.0 GB |
| Gemma 2 2B q4 | 175 | ~2.6 GB |
| Llama 3.2 3B q4 | 130 | ~3.5 GB |
| Phi-3.5-mini 3.8B q4 | 110 | ~4.2 GB |
| Gemma 3 4B q4 | 95 | ~4.9 GB |
For scale: your phone runs Gemma 2 2B at roughly 25–40 tok/s. Your $290 desktop card runs it at 175 tok/s. That is a 5× gap, and it opens the door to fine-tuning workloads no phone can touch.
Can the RTX 3060 12GB fine-tune a keyboard-class model?
Yes — via QLoRA, which is what you want anyway. Full-parameter fine-tuning of even a 3B model needs 30+ GB of VRAM once you count optimizer state, activations, and gradients. QLoRA quantizes the base weights to 4-bit and only trains a small pair of low-rank adapter matrices, dropping the VRAM footprint by ~10× and producing an adapter file you can ship independently.
Rough VRAM budget on the RTX 3060 for QLoRA at 4K context, batch size 4:
| Model | QLoRA VRAM (approx) | Fits on 12GB? |
|---|---|---|
| Qwen 2.5 0.5B | ~4 GB | Yes, huge headroom |
| Llama 3.2 1B | ~5.5 GB | Yes |
| Qwen 2.5 1.5B | ~6.8 GB | Yes |
| Gemma 2 2B | ~7.8 GB | Yes |
| Llama 3.2 3B | ~9.4 GB | Yes, tight |
| Phi-3.5-mini | ~10.5 GB | Yes, use grad accumulation |
| Gemma 3 4B | ~11.2 GB | Yes, use grad accumulation + smaller batch |
Beyond 4B and 4K context you start losing headroom. Dropping to 2K context or batch size 2 recovers it. Full-parameter fine-tuning of anything larger than 1B is not viable — that is the ceiling.
A concrete workflow: fine-tune your keyboard model on your own writing
The interesting question is not "can it happen" but "how long does it take end-to-end for a personal, useful result?" A realistic workflow on the RTX 3060 12GB:
- Gather ~50,000 lines of your own writing (Slack, email, blog drafts, chat logs). This is roughly 3M tokens after tokenization.
- Format as instruction pairs.
{"instruction": "<partial sentence>", "response": "<how you actually finish it>"}is the shape that produces autosuggest behavior. The Phi-3 fine-tuning cookbook has good format guidance. - Pick your base model. Qwen 2.5 1.5B is the sweet spot — small enough to run on any phone you might target, big enough that fine-tuning transfers cleanly.
- QLoRA-train for 2 epochs at learning rate 2e-4, batch 4, grad accumulation 4, cutoff 2K. Expect ~2,100 tok/sec on the 3060; 6M tokens total → ~48 minutes per epoch → ~1.7 hours for 2 epochs.
- Merge the adapter into the base weights (or ship separately for hot-swapping).
- Quantize to q4_K_M using
llama.cpp'squantizebinary. Output: a ~950 MB file that runs on a phone. - Test on-device. Sideload the model into a personal keyboard app (Florisboard on Android supports plug-in LLMs as of the 0.6 release) and validate the suggestions.
Prefill vs generation on tiny models
Small-model prefill is essentially free on the 3060 — a 4K prompt to Qwen 2.5 1.5B q4 completes prefill in under 200ms. Generation is where the throughput lives. For autosuggest workflows (where you want the model to complete 5–20 tokens after a keystroke), the total latency budget is ~50ms end-to-end to feel snappy. At 220 tok/s, that is 11 tokens of budget — enough for the median autosuggest.
Perf-per-dollar of local fine-tuning
The Ollama community pricing tracker surfaces the going rate for cloud fine-tuning of small models: OpenAI's gpt-4o-mini fine-tune API is $3/1M training tokens, Together's Qwen 2.5 1.5B fine-tune is $0.60/1M training tokens. For a 6M-token run, that is $18 on OpenAI or $3.60 on Together.
Local cost: 1.7 hours × 170 W × $0.14/kWh = $0.04 of electricity. The card pays for itself against Together after 80 fine-tune runs, or against OpenAI after 16 runs. If you plan to iterate — try different data splits, different LoRA ranks, different base models — the local rig wins fast.
Gotchas that will bite you
- License gotcha for Qwen 2.5. The 0.5B and 1.5B variants are fully Apache-2.0. The 32B has a commercial-use restriction. Read the license page before you ship anything.
- Gemma 2 vs Gemma 3. Gemma 3 uses a different attention shape (sliding-window) that changes KV-cache math. If you tuned a config for Gemma 2, retest on Gemma 3.
bitsandbyteson CUDA 13. You want at leastbitsandbytes0.44 for stable RTX 3060 support on the current PyTorch nightly.- Merged adapter quality. Merging a LoRA into q4 weights loses quality. Merge into fp16, then re-quantize.
- VRAM leaks in long training runs. PyTorch will leak ~200 MB per epoch on some
transformersversions. SetPYTORCH_CUDA_ALLOC_CONF=expandable_segments:Trueand monitor.
When NOT to fine-tune locally
If your dataset is under 1,000 examples, you are better off using in-context prompting on a strong 8B model rather than tuning a 1.5B. If the base task requires reasoning you cannot express in your fine-tune data, no amount of LoRA will produce it — the base model has to have the capability. And if you need multilingual support beyond the base model's coverage, you likely want a larger base first, then tune.
Bottom line
The RTX 3060 12GB is the correct desktop tool for the on-device AI keyboard revolution. It inferences the shipping models at 5× phone speed, and it fine-tunes them fully at home with QLoRA in hours, not days. If you have wanted to build a keyboard that sounds like you and only you, the hardware is already on your desk — the interesting engineering is in your data pipeline, not the GPU.
Related guides
- Local RTX 3060 vs Gemini for Linux Boot Debugging
- Ryzen AI Halo vs RTX 3060 for Local LLMs
- Best GPU for ComfyUI Under $350
- Aider vs Cline vs Cursor on a Local RTX 3060
Citations and sources
- TechPowerUp — GeForce RTX 3060 spec sheet — memory bandwidth and TGP baseline
- Ollama on GitHub — model library and quant options
- Phoronix — small-model inference benchmarks — reference for tok/s across the small-model tier
This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.
