Building a Local AI-Agent Eval Rig After AISI's Benchmark Warning

A single 12 GB RTX 3060 paired with a modern 8-core CPU and a 1 TB SATA SSD is enough to run overnight agent-eval batches on an 8B model at 4-bit quantization. That combination is the cheapest way to run the kind of scaffolded, tool-using benchmarks the UK's AI Security Institute has said public leaderboards systematically underrate — and to run them on your own scaffolding rather than someone else's abstract task suite.

Why AISI's warning pushes builders to run their own agent evals

The UK's AI Security Institute — successor to the AI Safety Institute — spent 2025 and early 2026 publishing findings that standardized agent benchmarks understate what capable scaffolding, better prompting, and tool access can extract from the same underlying model. The practical takeaway for anyone building agents is not that the benchmarks are broken. It is that they measure a specific, narrow slice of capability, and shipping decisions made purely against public leaderboards are decisions made against the wrong measurement.

Running your own eval is the alternative. It is also cheap. An 8B model at q4_K_M — the size class that dominates open-weight local-agent development in 2026 — fits in about 5-6 GB of VRAM. Add another 2-3 GB for KV cache during long tool loops and you are still well inside a 12 GB card. The rest of the eval harness — sandboxed tool execution, log capture, prompt orchestration, result grading — runs on the CPU host. That is a workload where a mid-tier consumer PC is genuinely sufficient, not merely a stopgap.

This piece walks through what an agent-eval workload actually asks of hardware, why the RTX 3060 12 GB plus a Ryzen 7 5800X has emerged as the reference build for local agent research, and where a 1 TB Crucial BX500 fits into the storage story. Benchmark numbers below are synthesized from TechPowerUp, llama.cpp release notes, and Phoronix's 5800X compute reviews — no first-party lab measurements are reported here.

Key takeaways

• An 8B model at q4_K_M fits comfortably in a 12 GB RTX 3060, leaving room for a moderate context and tool output.
• Agent eval loops are more CPU-active than plain chat because of tool spawning and result parsing — a Ryzen 7 5800X is a sensible host.
• Public benchmarks tend to underrate agents because they hold scaffolding constant; local eval lets you measure the scaffolding you actually deploy.
• Storage matters less than you think — model loading is the main storage load, and a cheap 1 TB SSD handles it.
• 12 GB VRAM caps you at ~8B-13B models comfortably; larger models need offload and slow the eval loop enough that overnight batches become impractical.

Why public agent benchmarks systematically underrate capability

AISI's published findings can be summarized bluntly: fix the model, vary the scaffolding, and the same model produces very different scores. On some benchmarks the delta between "vanilla model" and "the same model with better tool wrappers, retry-with-critique, and richer prompt structure" is larger than the delta between the model and its next-generation successor.

This matters because most public leaderboards freeze the scaffolding to enable apples-to-apples model comparisons. That is a defensible choice for a leaderboard. It is the wrong measurement if you are trying to answer "will this model succeed at the actual task I will deploy it against, with my prompts and my tools."

The consequence is that public benchmarks understate agents that ship with sophisticated scaffolding and overstate agents that lose that scaffolding. You cannot fix this by picking a different leaderboard. You fix it by running your own eval — same model, your scaffolding — and comparing to the leaderboard result as a floor rather than a ceiling.

What an agent-eval workload demands from a GPU

Agent eval is not chat. A single "task" involves the model producing many turns, each of which invokes a tool, receives output, and continues. The distinguishing features:

• Long, sparse context. Tool outputs are frequently long — a shell command's stdout, a web page, a JSON blob. The eval agent's context balloons across turns.
• Bursty generation. Between tool calls the model produces short bursts of tokens, then pauses while the tool runs, then continues.
• Multi-hour loops. A single eval might run hundreds of tasks. Overnight is not unusual.

That workload profile is different from a chat benchmark and different from single-shot LLM benchmarks. It stresses KV cache management and prefill latency more than raw generation tok/s. On a 12 GB RTX 3060, an 8B q4_K_M model with an 8k-16k context window fits comfortably and processes tool outputs without needing to swap weights to system RAM.

The upper bound on model size is set by whatever leaves ~3-4 GB free for KV cache and short-lived working buffers. Empirically, that means 8B-13B on the 3060 12 GB. 8B is the size class the reference agent scaffolds (Autogen, LangGraph, SWE-agent forks) target most heavily, which is why it has become the reference size for local eval work.

Agent-loop throughput on the RTX 3060 12 GB

Rough tok/s figures for representative open-weight agent models at q4_K_M and q5_K_M on the RTX 3060 12 GB, synthesized from public llama.cpp benchmarks and the Phoronix 3060 compute review:

Generation tok/s numbers depend on context length — longer context slows generation because more KV must be attended to per step. The prefill figures assume a ~1k-token context; longer context reduces prefill throughput linearly on a memory-bandwidth-bound card.

Practical implication: an eval loop that averages ~800 tokens generated per task and ~2,000 tokens ingested per task completes at roughly 30-40 seconds per task on an 8B model. A 500-task overnight eval finishes in 4-6 hours. That is well within an overnight window and a fraction of the time and cost the same job would take against a cloud API.

Quantization matrix for an 8B agent model

Quantization for agents deserves more attention than it usually gets. Agent workflows are more sensitive to small model errors than chat is — a hallucinated tool argument breaks the loop in a way a hallucinated fact in casual conversation does not.

The sweet spot for a 12 GB card running an 8B agent model is q5_K_M or q4_K_M. q5_K_M is the safer choice if you have the headroom; q4_K_M is the choice when you want the largest possible context window on the same card. Below q4, tool-argument fidelity degrades noticeably enough that eval results diverge from the fp16 reference — not because the model is dumber, but because syntactic tool-call outputs become fragile.

Prefill vs generation in multi-turn agent loops

Chat benchmarks report a single generation number. Agent eval loops need three:

• Time to first token on a fresh context — dominated by prefill.
• Sustained generation tok/s between tool calls.
• KV-cache growth cost across a long tool loop, where re-attending to past turns slows each subsequent step.

For a typical agent turn — 200 tokens of new input from the previous tool call, 400 tokens of assistant reply, then a tool invocation — the CPU-side loop (spawning the tool, waiting for I/O, parsing the response, appending to context, invoking the next generation) can easily dominate wall-clock. That is why a fast CPU host matters. On a Ryzen 7 5800X, the CPU side of the loop is not the bottleneck. On an older 4-core chip, it often is.

Context length: how tool-output length changes VRAM

Every tool call risks appending a large blob of text to the context. A curl command dumping a 20 KB HTML page pushes 5,000+ tokens into the KV cache for the rest of the loop. If your eval agent is expected to do web browsing or command-line exploration, budget a 16k or 32k context window.

At 16k context on an 8B q4_K_M model with fp16 KV cache, you're using roughly 5 GB for weights and 2 GB for KV. On a 12 GB RTX 3060, that leaves ~4 GB free — enough for CUDA scratch, tokenization buffers, and driver overhead without offload. At 32k context you are pushing 8 GB total, which starts to feel tight on 12 GB during long runs and argues for either dropping to 16k or moving to a 16 GB card.

The practical trick most eval harnesses use is context trimming: keep the system prompt, the last N assistant messages, and a rolling window of tool outputs. This keeps VRAM under control at the cost of losing long-range coherence, which is fine for most tasks and a problem for a few.

Perf-per-dollar: what you actually pay for a local eval bench

For an under-$1,000 all-in build, the parts list is:

Total for the reference bench lands around $700-800 depending on the AIB partner. That is roughly what a single week of medium-scale cloud eval spend costs on frontier-model APIs. Break-even is measured in weeks, not months.

Comparison to the alternatives is unflattering to cloud for iterative research:

• A cloud A100 rental at 2026 pricing is roughly $1-2/hour. A 6-hour overnight eval is $6-12.
• Running the same eval locally on the 3060 12 GB costs the marginal power (~1 kWh, well under $0.50 in most US markets).
• The cloud A100 is faster in wall-clock. It is much more expensive per iteration once you cross a few hundred iterations.

Local eval hardware pays for itself quickly if you actually iterate. If you run one eval a quarter, cloud is cheaper.

What to buy to assemble the rig

For a first-time local-eval builder targeting the AISI-flagged research pattern, the parts are:

1. ZOTAC Gaming RTX 3060 Twin Edge 12 GB — reference GPU. 12 GB is the practical minimum for 8B q4_K_M with 16k context.
2. AMD Ryzen 7 5800X — 8 cores, 16 threads, plenty for the CPU-side agent loop. Runs hot, so budget a competent cooler.
3. Crucial BX500 1 TB SATA SSD — cheap storage for many quantized model files plus accumulated eval traces. 1 TB fills up faster than you expect once you keep old runs.
4. MSI RTX 3060 Ventus 2X 12G — same-tier alternate GPU when Zotac stock is thin.
5. GIGABYTE RTX 3060 Gaming OC 12G — three-fan alternate for cases with room, quieter under sustained load.

Common pitfalls in agent-eval workloads

• Under-provisioning storage. A 500-task eval trace with tool outputs can easily exceed 5 GB. 1 TB is the floor.
• Assuming chat tok/s translates. Agent loops are context-heavy — the sustained tok/s under 8k-16k context is what matters, not the marketing tok/s at 1k.
• Skipping determinism. Local eval is only useful if it is reproducible. Fix the seed, log the sampling parameters, and check point-releases of both the model and the scaffold code.
• Ignoring the CPU host. A weak CPU turns tool spawning into the bottleneck. The 5800X is the floor for keeping the GPU busy.

When NOT to build local eval hardware

If you are running one eval per quarter, cloud is cheaper and less operational overhead. If you need to eval a 70B+ model, a single 12 GB card is not the right hardware — you need 24 GB, 48 GB, or multi-GPU. If your eval requires large-batch parallel task execution rather than sequential turn-taking, a cloud GPU cluster wins.

Local eval is the right choice when you iterate weekly or faster, when 8B-13B is your target size, and when you need control over scaffolding, sampling, and seed for reproducibility.

Bottom line

A 12 GB RTX 3060, a Ryzen 7 5800X, and a 1 TB SATA SSD is enough to run your own scaffolded agent evals on any 8B-class open-weight model — the exact class of measurement AISI has said public leaderboards systematically underrate. The rig pays for itself in a few weeks of iteration and gives you the reproducibility public leaderboards cannot.

Related guides

• On-Device AI Keyboards: What a Sub-2GB LLM Needs to Run Local
• Ryzen 5 5600G vs Ryzen 7 5700X for a Budget 1080p Gaming Build
• Best Cooler for the Ryzen 7 5800X: Noctua vs DeepCool vs CoolerMaster

Citations and sources

• TechPowerUp — GeForce RTX 3060 specifications
• llama.cpp — release notes and benchmark tables
• Phoronix — Ryzen 7 5800X compute review
• Phoronix — RTX 3060 compute review
• UK AI Security Institute

This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.