Large language models generate reasoning as strings of words — chain-of-thought text — rather than operating purely in the continuous vector space their weights live in. The short answer is that word-based reasoning is interpretable, auditable, and compatible with the tooling the entire LLM ecosystem is built on, while vector-space reasoning is not, even though it can be more compute-efficient in principle. Vector-space (or "latent") reasoning is an active research direction, not vaporware, but it hasn't displaced text-based chain-of-thought in any model people actually use.
What 'Reasoning in Vector Space' Would Even Mean
Inside a transformer, every token — word, sub-word, or otherwise — is represented as a high-dimensional vector (a hidden state) at each layer. Standard chain-of-thought prompting, described in Wei et al.'s 2022 paper Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (arxiv.org/abs/2201.11903), works by having the model decode that hidden state into a discrete token, append it to the context, and re-encode the whole sequence before predicting the next token. That decode-then-re-encode step is the expensive, seemingly wasteful part: the model collapses a rich vector into one word from a fixed vocabulary, then has to re-derive most of that vector's information on the next forward pass.
'Reasoning in vector space' means skipping that collapse — feeding the last hidden state directly back into the model as the next reasoning step, instead of forcing it through a token first. In principle, this lets the model carry forward more information per step (a continuous vector can encode much more than one vocabulary entry can) and reason over multiple candidate paths in superposition rather than committing to one word at a time.
Why Production Models Don't Do This
Interpretability is the dealbreaker
Every frontier lab that ships a reasoning model — OpenAI's o1-class models, Anthropic's Claude, DeepSeek's R1 — externalizes the reasoning trace as text specifically so humans (and automated safety systems) can read it. OpenAI's own framing of this design choice, in its overview of teaching models to reason (openai.com/index/learning-to-reason-with-llms/), treats the visible chain-of-thought as a monitoring surface, not just a scratchpad. Vector-space reasoning has no equivalent: a sequence of hidden states isn't something a human reviewer, a content filter, or a downstream tool call can inspect.
Anthropic's interpretability team has gone further, mapping what happens inside a model's hidden states during reasoning. Their attribution-graph research, published as "On the Biology of a Large Language Model" (transformer-circuits.pub/2025/attribution-graphs/biology.html) and summarized in "Tracing the thoughts of a language model" (anthropic.com/research/tracing-thoughts-language-model), found that a model's stated chain-of-thought can diverge from what its internal circuits are actually computing — the text explanation is sometimes a post-hoc rationalization rather than a faithful trace. That's a real limitation of text-based reasoning. But it's a limitation you can at least detect and study, because the text exists to compare against the internals. Pure vector-space reasoning removes the text entirely, so there's nothing left to compare against.
Discrete tokens are the interface the rest of the stack expects
Function calling, tool use, guardrails, content moderation, citation extraction, and every agent framework built on top of LLMs all operate on token sequences. A model reasoning in raw vector space has no natural point to emit a tool call, a citation, or a refusal — those all require collapsing back to discrete symbols eventually anyway. Token-based chain-of-thought reasoning fits directly into an ecosystem that was already built around text.
Training a model to reason well in continuous space is genuinely harder
Language-model pretraining is next-token prediction over a discrete vocabulary — an objective with a clean, well-understood loss function and decades of tooling behind it. Feeding continuous hidden states back into the model as "thoughts," as Meta's Coconut (Chain of Continuous Thought) research explores (arxiv.org/abs/2412.06769), changes the training dynamics: there's no discrete target to compute a standard cross-entropy loss against for the latent steps, and the paper describes needing a multi-stage curriculum to get the approach to train stably at all. That's a solvable research problem, but it's still an open one, not a drop-in replacement for how every shipping model is trained today.
Language Reasoning vs. Vector-Space Reasoning, Side by Side
| Property | Token-based chain-of-thought | Continuous vector-space reasoning |
|---|---|---|
| Human-readable | Yes — every step is a word | No — hidden states aren't directly interpretable |
| Compatible with tool calls / guardrails | Yes, natively | No — requires collapsing to tokens anyway |
| Training objective | Standard next-token cross-entropy | Requires custom curricula (see Coconut) |
| Compute per reasoning step | Full decode + re-encode round trip | Avoids the decode/re-encode round trip |
| Production deployment today | GPT, Claude, DeepSeek-R1, Qwen reasoning models | Research prototypes only |
| Auditability for safety review | Traceable, though not always faithful | Effectively opaque |
The Efficiency Argument Isn't Wrong — It's Just Not the Whole Story
The intuition behind wanting vector-space reasoning is sound: decoding to a token and re-encoding it is redundant work, and a continuous representation can in principle pack in more information per reasoning step than one word can. That's the entire premise of the Coconut research. But 'more efficient per step' has to be weighed against everything reasoning models are also expected to do — be inspectable during safety review, integrate with tool-calling frameworks, produce citable outputs, and train reliably at scale with existing infrastructure. Text-based chain-of-thought wins on all of those secondary requirements even where it loses on raw compute efficiency, which is why it's the production default.
What This Means If You're Running Reasoning Models Locally
The reasoning models people actually run on consumer hardware are all token-based, and that has a direct hardware consequence: a 'thinking' model doesn't just answer your prompt, it also generates — and you pay VRAM and time for — the entire visible reasoning trace before the final answer. GLM-5.2's reasoning outputs have been observed running to roughly 37,000 tokens on some prompts, a token-explosion effect covered in SpecPicks' breakdown of what that means for local rigs. Smaller open reasoning models trade some of that verbosity for a footprint that fits modest GPUs:
| Model | Class | Consumer GPU it fits | SpecPicks coverage |
|---|---|---|---|
| VibeThinker-3B | 3B reasoning model | RTX 3060 12GB | Reasoning in 3 billion params |
| VibeThinker-3B | 3B reasoning model | RTX 3060 12GB (any 12GB card) | Fits any 12GB GPU |
| VibeThinker-3B | 3B reasoning model | RTX 3060 + Raspberry Pi 4 | Split across RTX 3060 and Pi 4 |
| VibeThinker-3B | 3B reasoning model | RTX 3060 12GB, local-first setup | Local reasoning walkthrough |
| Bonsai 27B | Open reasoning model | RTX 3060 12GB | Can a 12GB RTX 3060 run it? |
| GLM-5.2 | Large reasoning model, long CoT | Higher-VRAM local rigs | 37k-token reasoning outputs explained |
Every one of those models still reasons in words, not vectors — which is exactly why the token count (and therefore the VRAM and time budget) climbs the way it does on harder prompts. If vector-space reasoning ever reaches production, one of its selling points would be cutting that token overhead; until then, budgeting VRAM for the full visible reasoning trace, not just the final answer, is the practical reality of running these models locally.
Will Vector-Space Reasoning Ever Take Over?
Hybrid approaches — models that reason in continuous space internally but still surface a text summary for auditability — are a plausible middle path, and they're an active area of research following work like Coconut. But replacing text-based chain-of-thought outright would mean giving up the interpretability properties that labs like Anthropic are actively building tooling around, and giving up native compatibility with the tool-calling and guardrail infrastructure the rest of the ecosystem depends on. Until a vector-space approach can match those properties, not just match raw accuracy, natural-language reasoning stays the default — inefficiency and all.
Citations and sources
- Wei et al., "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" — https://arxiv.org/abs/2201.11903
- "Training Large Language Models to Reason in a Continuous Latent Space" (Coconut) — https://arxiv.org/abs/2412.06769
- Anthropic, "On the Biology of a Large Language Model" — https://transformer-circuits.pub/2025/attribution-graphs/biology.html
- Anthropic, "Tracing the thoughts of a language model" — https://www.anthropic.com/research/tracing-thoughts-language-model
- OpenAI, "Learning to Reason with LLMs" — https://openai.com/index/learning-to-reason-with-llms/
This piece is editorial synthesis based on publicly available information. No independent first-party benchmarking is reported.