Ask a large language model to write a new function from a plain-English prompt, and current frontier models are, by now, reasonably good at it. Ask the same model to look at a compiled binary — or a stack of scattered documentation — and rebuild the entire program well enough that it compiles, runs, and behaves like the original, and the results get a lot less consistent. That gap is what a growing cluster of evaluations grouped under the name ProgramBench are trying to measure.
This piece lays out what program reconstruction actually tests, what the broader public research on the adjacent task of decompilation shows about how far LLMs have gotten, and — because this is SpecPicks — what hardware actually matters if you want to run coding-capable models locally instead of renting API tokens.
What Is ProgramBench, and What Is It Actually Testing?
ProgramBench-style evaluations sit downstream of a well-established idea: code benchmarks like HumanEval measure whether a model can write a short, self-contained function that passes unit tests, while SWE-bench measures whether a model can resolve a real GitHub issue inside an existing, unfamiliar codebase. Program reconstruction pushes further in a different direction: instead of writing new code against a spec, the model has to recover an entire working program — architecture, control flow, and all — from a compiled binary, partial documentation, or an API trace, with no access to the original source.
That framing overlaps heavily with academic decompilation research. The LLM4Decompile project is the most visible public effort in this space: it trains and evaluates models specifically on turning compiled C binaries back into compilable, re-executable source, using a benchmark built for that purpose (HumanEval-Decompile) rather than a repurposed code-generation set. Evaluations like it typically score on two axes — does the reconstructed program even compile, and does it reproduce the original's runtime behavior on the same test inputs — which is a stricter bar than the pass@1 metric HumanEval uses for greenfield generation.
Where ProgramBench specifically differs from HumanEval-Decompile is scope: it isn't limited to single binaries. Reconstruction tasks discussed under that name also cover rebuilding a program from documentation and partial API surfaces alone, closer to how a contractor might rebuild a legacy internal tool with no source access and only a requirements doc. C, C++, and Rust projects at varying complexity tiers show up most often in this kind of test, since compiled, statically-typed codebases make "does it actually compile and run" an unambiguous pass/fail signal in a way that dynamically-typed scripting languages don't.
How Do LLMs Actually Perform on Reconstruction-Style Tasks?
Here's the honest state of the evidence: there is no single, versioned, publicly audited ProgramBench leaderboard the way there is for SWE-bench or HumanEval, so any specific percentage attributed to a named model on "ProgramBench" circulating without a citation should be treated skeptically rather than repeated as fact. What the adjacent, peer-reviewed decompilation research does show consistently is directional: re-executability rates on decompiled or reconstructed C programs trail well behind pass rates on greenfield code-generation benchmarks, and that gap widens sharply as function count, pointer arithmetic, and control-flow complexity increase. Reconstruction from a stripped binary with no debug symbols is measurably harder than reconstruction from a binary that still carries type and variable-name information — models lean heavily on those hints when they're available.
Model choice still matters, of course, and results shift with every major release cycle. If you're evaluating models for this kind of task yourself rather than trusting a scraped leaderboard number, the practical move is to build a small held-out set of programs you already have working binaries and source for, and score compile-and-match rates directly rather than trusting an unlabeled benchmark screenshot. For a sense of how differently LLMs can be scored depending on what a benchmark is actually optimizing for, our breakdown of 42 LLMs tested for apocalypse-compliance benchmarks is a useful reminder that benchmark names promise more standardization than most of them deliver.
| Benchmark | What it measures | Public leaderboard status |
|---|---|---|
| HumanEval | Function-level pass@1 on greenfield prompts | Long-established, widely reported |
| SWE-bench | Resolving real GitHub issues in existing repos | Long-established, actively maintained |
| LLM4Decompile / HumanEval-Decompile | Binary-to-source re-executability | Active research area, growing but narrow |
| "ProgramBench"-style evals | Full-program reconstruction from binaries or docs | Emerging; no single audited public leaderboard yet |
Which Hardware Actually Matters If You Want to Run This Locally?
If you're experimenting with local coding models — for reconstruction-style tasks or ordinary day-to-day code generation — VRAM headroom and context length matter more than raw compute. A quantized 7B-13B coder model runs comfortably on a 12GB card, but reconstruction tasks tend to involve longer inputs (whole binaries, disassembly listings, multi-file documentation), which pushes VRAM and context requirements up fast compared to a single-function HumanEval-style prompt.
| Card | VRAM | Reasonable fit for local coding work | Read more |
|---|---|---|---|
| RTX 3060 12GB | 12GB | Quantized 7B-13B coder models, short-context tasks | RTX 3060 12GB vs Arc B580 for local LLMs |
| Intel Arc B580 | 12GB | Budget-tier alternative with comparable VRAM | Intel Arc B580 for local LLMs |
| Intel Arc Pro B60 | Workstation-class VRAM headroom | Larger context windows, professional workloads | Arc Pro B60 vs RTX 3060 12GB |
| RTX 4090 / 3090-class, 24GB | 24GB | Larger local coder models, longer context reconstruction tasks | TechPowerUp GPU specs database |
System RAM matters too, especially once you're loading large disassembly listings or multi-file source alongside the model itself; our real-world RAM usage guide for local LLM setups covers where 32GB starts to feel tight versus where 64GB actually pays off. And if your budget or footprint tops out at a Raspberry Pi rather than a discrete GPU, our Raspberry Pi local LLM home-lab guide is a realistic starting point — a small board like the Pi-based SunFounder PiDog kit (ASIN B0CGJF4SSC) is a fun way to get hands-on with running an LLM locally at the edge, though it's a hobby-scale platform, not something you'd point at a full binary-reconstruction workload.
Where Program Reconstruction Still Breaks
The consistent theme across public decompilation and reconstruction research is that models are still weakest exactly where human reverse engineers are strongest: recovering intent, not just syntax. A few recurring failure patterns show up across the research:
- Obfuscated or stripped binaries. Removing debug symbols, variable names, and type information removes exactly the signal models lean on most, and reconstruction quality drops accordingly.
- Legacy and undocumented API surfaces. When a program calls into an internal library with no public documentation, models tend to guess plausible-looking but incorrect signatures rather than flagging the gap.
- Memory-management edge cases. Manual memory management, custom allocators, and pointer arithmetic in C/C++ remain a disproportionate source of reconstructed programs that compile but behave incorrectly under specific inputs.
- Long-range architectural coherence. A model can often reconstruct an individual function correctly while losing the broader architectural pattern the original program used, producing code that passes narrow tests but wouldn't hold up to a maintainer's review.
These failure modes are also why security researchers are cautious about treating LLM-reconstructed code as trustworthy without independent verification — a theme that shows up in adjacent work like the security-focused local LLM tooling covered in our Cisco open security LLM piece, where local vulnerability scanning is explicitly framed as an assist layer rather than a replacement for a human review pass.
What's Next for Program-Reconstruction Benchmarks
The research direction that shows up most often in the decompilation and reconstruction literature isn't "bigger model, better score" — it's hybrid pipelines that pair an LLM with symbolic execution or a traditional decompiler (Ghidra, IDA) so the model handles pattern recognition and naming while a deterministic tool handles control-flow recovery. That combination is reported to outperform either approach alone on harder binaries, which tracks with a broader pattern in AI tooling: symbolic and neural methods tend to cover each other's blind spots rather than one fully subsuming the other.
It's worth noting this isn't a new instinct — the retro-gaming community has been doing manual "program reconstruction" for decades, reverse-engineering old console binaries into buildable source for 1:1 remakes of classic games long before LLMs existed. The difference now is speed and cost, not the fundamental difficulty of the task, and the failure modes read remarkably similar: obfuscation, missing documentation, and architectural drift are exactly what human reverse engineers have always fought too.
For now, treat any single benchmark number attached to "ProgramBench" the way you'd treat any unaudited leaderboard claim — useful as a directional signal, not as a substitute for testing on your own held-out programs if the result actually matters to you.
Citations and sources
- SWE-bench — public benchmark and leaderboard for LLM resolution of real GitHub issues
- HumanEval — OpenAI's function-level code-generation benchmark
- LLM4Decompile (arXiv:2403.05286) — research on training and evaluating LLMs for binary-to-source decompilation
- TechPowerUp GPU Specs Database — GPU VRAM and specification reference
This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.
