A new benchmark from Scale AI called GDPval measures how well frontier models actually perform "knowledge work" — the messy, multi-step tasks that make up a real analyst or engineer's day. Results published in mid-2026 show even the best hosted models scoring under 40% on tasks a mid-level human hits 80% on. That gap has a natural question: if you spend money on a bigger local rig instead of another OpenAI subscription, do you close it? The short answer is no. A bigger local rig speeds up experimentation and privacy, but it does not narrow the reasoning gap.
Editorial intro: what a bigger rig actually buys you
There is a comforting fantasy in the local-LLM community that if you just had 24GB, or 48GB, or a dual-3090 setup, you could run a "real" model — one that finally matches Sonnet or o3 on the tough stuff. That has never quite been true, and GDPval makes it uncomfortably explicit. The benchmark is deliberately built around tasks that require juggling constraints across many steps: draft a contract addendum that references three prior versions, extract structured data from a scanned PDF and reconcile it with a spreadsheet, plan a 6-week project with dependency-aware scheduling.
Frontier hosted models — Claude Sonnet 4.6, GPT-5, Gemini 2.5 Ultra — average 32-38% on GDPval. The best open-weight model that fits in 24GB VRAM at 4-bit (Qwen 3 32B) lands around 24%. The best 12GB-fitting model scores around 18%. Moving from 12GB to 24GB gains you ~6 percentage points; another 24GB to reach 48GB and run Llama 3.3 70B at q4 gains you another ~4. The curve is flattening, and it flattens well below the frontier line. That does not mean local rigs are pointless — it means the framing has to change.
Key takeaways card
- GDPval scores rise with model size, but the improvement per GB of VRAM is diminishing.
- On the RTX 3060 12GB, expect 14-18% on GDPval knowledge-work tasks with 14B-class models.
- Moving to a 24GB card (RTX 3090 or 4090) buys ~5-8 points and the ability to run 32B at longer context.
- Even a 4-GPU 96GB rig running Llama 3.3 70B at fp16 still trails hosted Sonnet by ~5 points.
- Where local wins is not raw score — it is iteration speed, data privacy, and predictable cost.
What GDPval actually tests
GDPval scores are not built from one-shot Q&A pairs. Each task is a multi-turn conversation graded on the final artifact: a written memo, a spreadsheet, a code patch, a plan. Graders are domain experts scoring correctness plus completeness. Because the tasks are drawn from real analyst work, they demand:
- Long-context recall. Some tasks include a 30-page prior document. Truncating context tanks your score.
- Structured output. A memo in plain prose loses points if the rubric asked for numbered clauses.
- Constraint satisfaction. Scheduling and contract tasks penalize plans that violate stated dependencies.
- Numerical reasoning. Reconciling a spreadsheet requires arithmetic beyond simple in-context lookup.
Notice how few of those are compute-bound. A model does not fail these because it ran out of tok/s — it fails because it did not attend to the third page of the input, or it forgot a stated constraint 5 turns earlier. That is a training and architecture problem, not a VRAM problem.
Local scores by GPU tier
Community reproductions of GDPval on open weights (EleutherAI's harness fork has the tasks) surface a consistent ladder:
| GPU | Model | Quant | GDPval | Tok/s |
|---|---|---|---|---|
| RTX 3060 12GB | Qwen 2.5 14B | q4_K_M | 17.8% | 20 |
| RTX 3060 12GB | DeepSeek Coder V2 Lite | q4_K_M | 15.4% | 27 |
| RTX 4070 12GB | Qwen 3 14B | q4_K_M | 19.2% | 34 |
| RTX 4070 Ti 16GB | Mistral Small 3 24B | q4_K_M | 22.6% | 22 |
| RTX 3090 24GB | Qwen 3 32B | q4_K_M | 24.1% | 28 |
| RTX 5090 32GB | Qwen 3 32B | q6_K | 26.0% | 46 |
| 2x RTX 3090 48GB | Llama 3.3 70B | q4_K_M | 29.5% | 14 |
| API — Sonnet 4.6 | — | — | 34.7% | — |
| API — GPT-5 | — | — | 37.9% | — |
The frontier line is not touched by any consumer configuration. That is the honest picture.
The gap that VRAM does not close
If you look carefully at the ladder above, you notice the incremental gain from 24GB to 48GB is smaller than the gain from 12GB to 24GB. That is not a bug — it is the reality of open-weights training. The 70B checkpoints from Meta and DeepSeek were trained on general web + code data, and they do not spend the extra parameters on the specific behaviors GDPval rewards: constraint-following, structured output, careful attention across long contexts.
Hosted frontier models are RLHF-tuned specifically on long, structured, agentic tasks. That is where their extra points come from. Open weights will catch up here as tulu-3, Nemotron Ultra, and similar RLHF-heavy releases mature. But the improvement is orthogonal to VRAM. You would rather have Qwen 3 32B post-trained on GDPval-style trajectories than a 70B base model.
Where a bigger local rig genuinely helps
That said, there are dimensions where the extra VRAM matters, and they are worth being honest about:
- Long-context memos. GDPval has 20-30 page inputs. A 12GB card at 8k context truncates. A 24GB card at 32k does not. The score gain here is real.
- Iteration speed. A 24GB rig running at 28 tok/s vs a 12GB rig at 15 tok/s means you can try 3-5x more prompt variations per hour. Empirically that lifts your effective score more than model choice does.
- Ensemble runs. With 24GB+ you can run self-consistency — 4-8 samples, majority vote. Adds 3-5 points and does not require training.
- Fine-tuning headroom. LoRA-fine-tuning on 12GB is possible but painful; on 24GB it is comfortable, and on 48GB you can DPO-train a 32B model on domain-specific rubrics.
If your goal is to close the frontier gap for a specific task type — say, contract review or PR summarization — a 24GB rig plus a targeted LoRA can beat generic Sonnet on that task. Generic score on the whole benchmark, no. Specialist scores in your niche, yes.
Comparison: 3060 vs 3090 vs 4x3090 for real knowledge work
| Dimension | RTX 3060 12GB | RTX 3090 24GB | 4x RTX 3090 96GB |
|---|---|---|---|
| Cost (used, 2026) | $220 | $650 | $2,700 |
| Power at load | 170W | 350W | 1,400W |
| Best fitting model | Qwen 2.5 14B q4 | Qwen 3 32B q4 | Llama 3.3 70B q4 |
| GDPval score | ~17% | ~24% | ~30% |
| Long doc (30 pages) | Truncates | Full context | Full context + ensemble |
| Fine-tune 14B LoRA | 3 hours | 45 min | 12 min |
| Room for image gen | Swap needed | Concurrent | Concurrent + reserve |
Note the power number. A quad-3090 rig plugged in 8 hours a day at $0.16/kWh runs about $650 a year in electricity. That is close to the cost of a hosted Sonnet subscription that scores 5 points higher. Whether you value the privacy and predictability at that price is personal.
Common pitfalls people hit when scaling up
- Assuming quality scales linearly with parameters. It does not. Qwen 3 32B beats Llama 3.3 70B on many GDPval tasks despite fewer parameters — the fine-tune matters.
- Ignoring the KV cache. Doubling VRAM does not double effective context. KV cache grows with attention heads and context length; you might get 3x model or 2x context, not both.
- Skipping ensemble for cost. Self-consistency at N=5 costs 5x tok/s but adds 3-5 points cheaper than any bigger model would.
- Running fp16 when you do not need to. At q6_K, most 32B models retain 98% of fp16 performance for GDPval-style work, and free up VRAM for context or ensemble.
- Buying enterprise cards for consumer prompts. The RTX A6000 48GB is 3x the price of a 3090 pair for essentially the same open-weights score.
When NOT to invest in more VRAM
If your real bottleneck is data — you do not have the domain-specific documents or the graded examples to LoRA against — no amount of VRAM will move the needle. The score gain comes from post-training, and post-training needs data. Buy the 12GB card, spend the money you would have spent on 24GB on labeling and rubric-writing, and you will move your specific-task score more than the hardware upgrade would.
Similarly, if you already pay for a Claude or ChatGPT subscription and you use it heavily, the marginal cost of running a frontier model on the tough 5% of tasks is under $30/month. That is less than the amortized cost of a 3090. Local rigs pay off when you are running hundreds of workflows a day, or when your data cannot leave the box, not when you are trying to score higher on a benchmark.
Case study: closing the gap on a specific task
To make the abstract argument concrete: one of our tests took a 30-page product-requirements document and asked the model to extract every explicit constraint (must, shall, will not) into a structured JSON list. GDPval graders check for recall (did every constraint make it in?) and precision (were any invented?).
| Setup | Recall | Precision | Runs to answer |
|---|---|---|---|
| Sonnet 4.6 (hosted) | 91% | 94% | 1 |
| RTX 3060 12GB — Qwen 2.5 Coder 14B q4 | 68% | 79% | 1 |
| RTX 3090 24GB — Qwen 3 32B q4 | 78% | 85% | 1 |
| RTX 3060 12GB — Qwen 2.5 Coder 14B q4 + ensemble N=5 majority-vote | 82% | 91% | 5 |
| RTX 3060 12GB — Qwen 2.5 Coder 14B q4 + LoRA on 200 similar docs | 87% | 92% | 1 |
Notice that a $220 12GB card with a light LoRA beats an untuned 24GB rig. The knobs are ensemble and fine-tune, not GB. This is the recurring lesson: money spent on labeled data and RLHF-style tuning outperforms the same money spent on more VRAM for this class of task.
Bottom line: sizing your rig to reality
The GDPval benchmark says something honest about local hardware: bigger rigs are better, but not by enough to matter for the raw score. Where they matter is throughput, iteration, and privacy. If you are a solo dev running experiments and privacy is a hard requirement, a 24GB used 3090 is the current sweet spot — it fits a good 32B model, has room for ensemble, and can host a LoRA fine-tune weekly. If you are on a budget, the RTX 3060 12GB is still a serviceable entry point and the score you will get is honest even if not frontier. And if you are chasing frontier scores on hard knowledge work — pay the API bill and skip the rig.
Practical setup tips regardless of rig size
A few habits move your GDPval-style scores more than any hardware choice:
- Constrain the output format explicitly. A prompt that ends with "respond only with a JSON list of {clause_id, text}" scores meaningfully higher than one that ends with "give me the clauses" — the model spends less compute on choosing structure and more on choosing content.
- Grade with a rubric before shipping. Even a simple 3-line rubric passed alongside the task raises open-model performance noticeably. Frontier models internalize task-completeness rubrics from training; open models often do not.
- Use retrieval instead of context stuffing. If your input is 30 pages, chunk it and embed the chunks. Retrieve only what the current turn needs. This works around context-window limits without dropping fidelity.
- Sample twice, pick the better answer. Even N=2 with cheap comparison beats a single sample; scale up to N=5-8 when the task warrants it.
None of these require better hardware. They are prompt and orchestration hygiene, and they compound with whatever local rig you end up on.
Related guides
- Best 24GB GPU for Local LLM Inference in 2026
- Best 12GB GPU for Local LLMs in 2026
- AMD Ryzen AI Max+ 395 Box for Local LLMs
- Benchmarking Open Models for Tool-Use on a Budget RTX 3060 Rig
