The panic was real for about eighteen months. Through 2023 and early 2024, researchers and executives openly worried that large language models were approaching a hard ceiling: the internet only contains so much high-quality human-written text, and frontier models were consuming it at a pace that outran the web's growth rate. Headlines asked whether GPT-5-class models were even possible without a new source of training material.
That crisis didn't resolve with a single breakthrough. It dissolved into several parallel workarounds — some technical, some commercial — that together took the pressure off without fully retiring the underlying constraint. Here's what actually happened.
The data wall was a real, measured constraint
The concern wasn't hype. Researchers at Epoch AI modeled the trajectory of publicly available human-generated text against the growth of training datasets used by frontier models and found the two curves were on a collision course, with the highest-quality text (books, curated web content, code) plausibly running short before the low-quality indexed web did (Epoch AI).
By late 2024, the concern had reached the labs building the largest models. Reuters reported that OpenAI, Google, and other frontier developers were seeing diminishing returns from simply scaling pretraining runs on existing data, and were shifting research effort toward new techniques rather than betting everything on ever-larger scraped corpora (Reuters, Nov. 2024). Ilya Sutskever, speaking around the same period, put it more bluntly: pretraining on the open web the way the industry had been doing it was reaching a natural limit, even as compute kept getting cheaper and more available. That framing — a plateau on the data axis, not the compute axis — is what shifted lab strategy over the following year.
Synthetic data filled part of the gap — carefully
The most direct response was to generate more training material rather than only scrape it. Larger "teacher" models produce worked examples — code with test cases, math proofs, step-by-step reasoning chains — that get filtered and fed back into training runs for smaller or newer models. This works especially well in domains where correctness is automatically checkable: a generated math solution can be verified by a solver, generated code can be run against unit tests. It works far less reliably for open-ended factual writing, opinion, or creative text, where there's no automatic check for correctness.
That distinction matters because of a real failure mode documented in a 2024 Nature paper: when models are trained recursively on their own (or prior models') outputs without enough fresh, human-grounded data mixed in, quality and diversity degrade over successive generations — a phenomenon the paper's authors termed "model collapse" (Shumailov et al., Nature 2024). The practical response across the industry has been to treat synthetic data as a supplement with strict filtering and provenance tracking, not a wholesale replacement for human-generated text. Readers curious how these tradeoffs show up in practice at the model level can compare outputs across current open-weight releases in our breakdown of 42 LLMs tested against safety and reasoning benchmarks.
The bigger shift: spending compute at answer time instead of only at training time
The more consequential change wasn't about finding more data at all — it was about extracting more capability from the data already on hand. Reasoning models like OpenAI's o1 and o3 and DeepSeek's R1 introduced "test-time compute": rather than a fixed forward pass, the model generates and evaluates intermediate reasoning steps before answering, spending more inference compute on harder problems (OpenAI; DeepSeek-R1 paper, arXiv). DeepSeek's paper in particular showed that reinforcement learning on top of an existing pretrained base — rather than a larger pretraining corpus — could produce large reasoning gains, using far less additional raw text than another full pretraining scale-up would have required.
This reframed the scaling conversation. Instead of "we need 10x more tokens," labs started asking "how much more can we get out of the tokens and compute we already have." For hobbyists running models locally rather than in a hyperscaler cluster, that same logic shows up at a smaller scale — reasoning-heavy open-weight models trade raw parameter count for inference-time reasoning steps, which changes what hardware actually matters. Our guide to running Kimi K3's open weights locally and our dual-GPU llama.cpp speedup benchmarks both cover the practical hardware side of that shift.
Publishers started getting paid
The other lever was commercial rather than technical: paying for data that used to be scraped for free. Through 2024 and into 2025, OpenAI and other labs signed content-licensing agreements with news organizations, forums, and other platforms sitting on large volumes of fresh, high-quality, and — critically — legally clean text. This didn't solve the volume problem (licensed corpora are tiny next to the open web), but it addressed two things scraping couldn't: provenance and freshness. Licensed feeds give labs a defensible, continuously updating stream of human-written text that postdates their last training cutoff, which is exactly the category Epoch AI's models flagged as scarcest.
What labs are still doing differently
| Approach | What changed | Where it shows up |
|---|---|---|
| Synthetic + verified data | Teacher-model-generated examples, filtered by automatic checkers (code tests, math solvers) | Math, code, and agentic-tool-use training sets |
| Test-time/reasoning compute | More inference-time compute per query instead of only more pretraining tokens | o1/o3, DeepSeek-R1, and their open-weight successors |
| Licensed, dated corpora | Paid deals for continuously updating human text instead of one-time scrapes | News, forum, and publisher partnerships |
| Multimodal expansion | Video, audio, and image-text pairs treated as a separate data pool from web text | Frontier multimodal model training runs |
| Curation over volume | Smaller, higher-quality, deduplicated datasets outperforming raw scale in ablations | Base-model pretraining recipes since 2024 |
None of these fully retired the constraint Epoch AI described — the supply of fresh, high-quality human text is still growing more slowly than frontier training runs would like. What changed is that the industry stopped treating "more scraped tokens" as the only lever and started pulling several smaller ones at once: better use of existing data, synthetic augmentation in checkable domains, paid access to fresh text, and — the biggest structural change — getting more usable intelligence out of the same pretraining budget via reasoning at inference time.
For anyone building or running models outside a hyperscaler's cluster, the practical upshot is that the hardware calculus shifted too: inference-heavy reasoning workloads favor different GPU/VRAM tradeoffs than a straight pretraining scale-up would have. At the consumer end that puts the GeForce RTX 5090 — 32 GB of GDDR7, enough to host a mid-size open-weight reasoning model without offloading — into the picture for local inference. Serious homelab or on-prem builds move up to workstation cards where VRAM stops being the bottleneck: the RTX PRO 6000 Blackwell Workstation at 96 GB, the PNY RTX A6000 at 48 GB, and the datacenter-tier Tesla L40S 48GB all keep a full context-length reasoning trace resident instead of paging weights on every token. If you're speccing a homelab around current-generation open-weight reasoning models, it's worth checking what that hardware is actually worth before buying, in our piece on downsizing a homelab and what the parts are worth on resale, and keeping any internet-facing local inference stack locked down per our homelab security notes.
Frequently asked questions
Did AI companies actually run out of training data? Not in the literal sense. No frontier lab hit a hard stop. But growth researchers at Epoch AI showed the stock of high-quality human text was on track to stop scaling with model size within the decade, which is the constraint that drove every response described above.
Does synthetic data actually work for training LLMs? It works well in domains where outputs can be automatically verified — math, code, formal reasoning — and is riskier in open-ended factual or creative text, where errors from a generating model can compound rather than cancel out.
What is model collapse? The degradation that occurs when models are trained recursively on their own or prior models' outputs without enough fresh, human-grounded data mixed in, documented in a 2024 Nature paper. It's the main reason synthetic data is used as a filtered supplement rather than a full replacement.
How does test-time compute relate to the data shortage? Reasoning models like o1, o3, and DeepSeek-R1 spend more compute at answer time rather than relying purely on larger pretraining corpora, extracting more capability from existing data instead of requiring proportionally more of it.
Are companies paying for training data now? Yes — through 2024 and 2025, several labs signed licensing deals with publishers and platforms for continuously updating, legally clean text, addressing freshness and provenance rather than raw volume.
Will data scarcity slow future frontier models? It's reshaping how models scale — toward curated/synthetic data mixes, multimodal sources, and inference-time reasoning — rather than halting progress outright.
Citations and sources
- Reuters — OpenAI, rivals seek new path to smarter AI as current methods hit limitations (Nov. 2024)
- Shumailov et al., "AI models collapse when trained on recursively generated data," Nature (2024)
- Epoch AI — Will we run out of data? Limits of LLM scaling based on human-generated data
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, arXiv
- OpenAI — Learning to Reason with LLMs
This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.
