Reasoning models: how they work, what they improve, and measured limits

Reasoning models are language models trained to handle certain multistep tasks more effectively, often by spending more computation on each answer. They improve results in areas such as mathematics and programming, but their explanations guarantee neither a correct result nor an accurate account of how they reached it. Their value has to be measured on a specific task, with an explicit budget and evaluation procedure.

What makes a model a reasoning model

The term describes a family of behaviors and training methods, rather than a single architecture. A model may break down a problem, explore a hypothesis, spot a contradiction, and start again before delivering its answer. Here, reasoning describes an evaluated capability; the term alone does not establish equivalence with human thought.

The answer shown to the user should be distinguished from the intermediate steps generated to prepare it. Those steps often form a chain of thought. Depending on the system, they may be visible, hidden, or replaced with a summary. The interface therefore does not necessarily reveal all the computation involved.

Research by Jason Wei and his coauthors, initially published in January 2022, shows that including worked examples in the context can improve performance on arithmetic, symbolic, and commonsense tasks. This method, known as chain-of-thought prompting, changes how an already trained model is prompted.

Specialized training goes further by trying to make these useful behaviors occur more reliably. Simply telling a model to think through a problem does not automatically reproduce the capabilities of a model trained for that purpose.

This page focuses on these mechanisms and their observable results. The related article Do LLMs really reason? looks more directly at the terminology and the interpretations it allows.

How training encourages useful intermediate steps

Pretraining gives a model patterns in language, knowledge, and examples of problem solving. Post-training then aims to guide its behavior: following a request, presenting a usable answer, or solving more problems successfully. Reasoning models build on this foundation, using training recipes that vary across research teams.

Supervised learning trains a model on examples of answers or solution trajectories. Reinforcement learning has it generate attempts, then uses a reward signal to change the probabilities of its future behaviors. A verifiable mathematical answer or a program checked against tests provides a more directly assessable signal than a general judgment about writing style.

In their initial January 2025 publication, DeepSeek researchers describe DeepSeek-R1-Zero, trained through reinforcement learning without a preliminary supervised fine-tuning stage. They observe checking and backtracking behaviors, alongside readability problems and language mixing. DeepSeek-R1 subsequently combines several training stages with initial problem-solving data. These models serve here as documented historical examples, rather than recommendations of current products.

A reward based on the final answer does not, however, guarantee that every intermediate step is valid. A result may be correct despite a flawed justification. An incomplete verifier may also accept an unsuitable solution. You therefore need to examine what the training process actually rewards.

Distillation offers another route: one model learns from answers generated by another. Improvements achieved this way indicate which capabilities have been transferred, without demonstrating that the systems use exactly the same internal mechanisms.

Spending more computation on an answer

Training compute is spent before you submit your request. Inference compute is spent when a system processes that request. Extending the preparation of an answer therefore shifts some of the computational effort to usage, where the cost recurs with each request.

Several strategies are available:

• Develop an attempt further: generate intermediate steps, check a result, or revise an approach.
• Generate more attempts: produce several solutions and compare them.
• Select with a verifier: evaluate candidates using tests, rules, or a specialized model.
• Use a tool: run a calculation or program, then use its output.

These strategies are not interchangeable. Multiple attempts can repeat the same mistake, and a majority vote does not establish a proof. A verifier is most useful when it can reliably distinguish correct solutions from those that merely look plausible.

In their August 2024 study, Charlie Snell and his coauthors show that the effectiveness of additional computation depends on the problem’s difficulty and how that computation is allocated. Their work on inference-time compute examines approaches including verifier-guided search and answer revision. It does not support a universal rule that waiting longer always produces a better result.

Additional computation must also be distinguished from additional learning. Generating a lengthy solution does not necessarily change the model’s weights. Our page on test-time adaptation explains this distinction between using context, searching for solutions, and adapting parameters.

Direct answers, extended reasoning, and tools

When choosing a configuration, compare complete systems rather than labels. A direct answer may be enough for an extraction task. A longer search may help with an ambiguous problem. An exact calculation may require a specialized tool.

Criterion	Direct answer	Extended reasoning	Reasoning with tools
Work performed	Produces an answer quickly	Explores and revises intermediate steps	Adds external observations or execution results
Uses to evaluate	Rewriting, extraction, simple requests	Problems with constraints, multistep solutions	Exact calculations, code tests, data lookup
Available checks	Comparison with a reference	Checking the result and useful intermediate steps	Inspecting tool outputs and the final result
Resources to count	Input and generation	Additional preparation and generation	Model calls, tools, and infrastructure
Typical limitation	Answer produced too quickly	Lengthy but incorrect exploration	Wrong tool, wrong input, or incorrect interpretation

This table is a framework for choosing an approach, rather than a performance ranking. A single system may also move between these approaches internally, adjusting its effort to the request.

Consider a hypothetical example: you need to explain a discrepancy between two accounting totals. The model can suggest possible causes, but those hypotheses must be checked against the relevant entries. A calculation run on the actual data provides a stronger check than a convincing explanation based on assumed amounts.

The appropriate sequence is to identify the necessary data, perform the reconciliation, and present differences that can be verified. Reasoning organizes the work; records and calculations establish the result.

What published improvements actually demonstrate

Tasks with verifiable success criteria provide particularly clear evidence. A numerical answer, a solution that satisfies constraints, or a program that passes tests makes it possible to measure more than the quality of the writing.

In the initial DeepSeek-R1 report, January 2025, the authors report that R1-Zero’s performance on AIME 2024 rose from 15.6% to 71.0% during their reinforcement learning experiment. The same report gives R1 a score of 79.8% on that evaluation. These are results reported by the authors under their historical protocol, rather than measurements of models available in October 2026.

The metric pass@1 estimates the success rate of a single attempt under the evaluation’s sampling conditions. It should not be confused with a procedure that generates several candidates and then selects an answer. The number of attempts and the selection process are part of the result.

These findings support a bounded conclusion: certain training methods substantially improve success on verifiable problems. They do not establish the same level of reliability for a document audit, a business decision, or an open-ended software task.

For your own use, identify what the score measures: a final answer, working code, a complete proof, or a selection from several proposals. Check the experimental conditions as well. The guide to reading LLM rankings explains why similar percentages can reflect different evaluation procedures.

Limits become clearer as complexity changes

An average score does not show how errors change with difficulty. A system may solve a family of simple problems, improve on intermediate cases, and then fail as the number of constraints or the length of the solution increases.

Apple’s study The Illusion of Thinking, June 2025 uses controlled puzzles to vary this complexity. The authors report distinct performance regimes: standard models have an advantage on some simple tasks, extended reasoning helps at intermediate difficulty, and performance then collapses at high difficulty in their experiments. They also observe a decline in generated reasoning effort beyond certain thresholds.

The scope of these findings depends on the experimental setup. Asking for an exhaustive list of moves does not assess exactly the same capability as asking for an algorithm that can generate that list. An output cut short, a rule violation, and an impossible problem need to be distinguished.

In Rethinking the Illusion of Thinking, July 2025, Iñaki Dellibarda Varela and his coauthors reexamine Tower of Hanoi and river-crossing puzzles. They report both persistent difficulties with Hanoi after changes to the evaluation procedure and problems involving unsolvable configurations in the river-crossing tests.

This disagreement rules out sweeping verdicts. Measured failures require an explanation, and their causes may combine model capabilities, output constraints, and test design. For a useful evaluation, vary difficulty, answer format, and tool access separately. Then classify the errors before combining them into an overall score.

A detailed explanation can obscure what caused an answer

An answer’s correctness and the faithfulness of its explanation are separate questions. An answer can be right while its explanation is incomplete. A coherent explanation can also rationalize a choice influenced by a hint without acknowledging that influence.

In a study published in April 2025, Anthropic researchers insert hints into requests, then examine whether models acknowledge their influence. Under the conditions tested, Claude 3.7 Sonnet mentions the hints it used in an average of 25% of cases, compared with 39% for R1. These figures describe that experiment, rather than a universal rate of transparency.

Reading a chain of thought can therefore provide useful information without amounting to a complete audit. A seemingly reasonable step does not, by itself, establish why the model chose its answer.

For professional use, request evidence you can check: explicit assumptions, sources consulted, reproducible calculations, tests performed, and unresolved uncertainties. These support verification. They do not provide exhaustive access to the model’s internal workings.

A hallucination remains a problem even when surrounded by logical argument. When a task depends on missing or recent information, the priority is to obtain an appropriate source. Extending the reasoning from a false premise may simply make the error more elaborate.

Reasoning and generalization remain separate questions

Solving a difficult exercise demonstrates a capability under that exercise’s conditions. Evaluating generalization also requires examining how unfamiliar the task is, what prior knowledge is available, and which resources are needed to reach a solution.

In On the Measure of Intelligence, published in November 2019, François Chollet proposes a definition of intelligence centered on the efficiency of acquiring new skills. He distinguishes that efficiency from the level of skill achieved after extensive experience. His proposal calls for taking initial knowledge, experience, and generalization difficulty into account.

Applied to reasoning models, this distinction suggests examining search costs and available examples alongside the final score. This is an analytical approach drawn from that framework, rather than a judgment made by the earlier publication about systems developed later.

The work by Snell and DeepSeek puts more emphasis on a different experimental question: which additional capabilities can computation and training produce? These approaches yield useful results without answering the full question of general intelligence on their own.

An improvement can therefore matter to a user while its scientific interpretation remains open. The page on generalization in François Chollet’s framework explores this perspective further. The AGI and research hub brings together other approaches to measurement.

Measuring cost per acceptable result

Longer reasoning uses additional resources, but the amount of visible text is not enough to estimate them. You need to examine the usage counters actually provided and the billing rules of the service you use. A short final answer may have required substantial preparation.

To compare two configurations, measure the total cost of the tasks, then divide it by the number of acceptable results. A configuration with a higher cost per call may be worthwhile if it avoids repeated attempts. Conversely, a faster configuration may be preferable when checking is straightforward and mistakes are easy to correct.

Evaluate latency separately. A delayed solution may be suitable for preparatory analysis but disruptive in an interaction where the user expects an immediate answer. The maximum acceptable delay therefore depends on the work, as well as the model’s performance.

Set stopping rules: the budget has been exhausted, the time limit has been reached, no progress is being made, or essential information is missing. Another attempt has value only if it contributes a different approach or a useful observation.

The LLM pricing calculator helps estimate spending from measured usage volumes. Add tools, infrastructure, and review time afterward: token pricing accounts for only part of the cost of completed work.

Evaluating a model on your own tasks

Start by defining an acceptable result before comparing configurations. For extraction, that might mean accurate fields and explicit handling of missing values. For code, it includes the expected behavior, relevant tests, and the absence of unwanted changes.

Build a representative set of cases: routine requests, conflicting constraints, missing information, and examples that require external verification. Hold some cases back while adjusting your setup, so you do not optimize solely for requests you already know.

Then compare a simple configuration, a configuration with more reasoning, and, if the task warrants it, a configuration with tools. Keep the data and success criteria consistent. If you change the model, context, and tools at the same time, you will be evaluating the whole system without being able to isolate the cause of an improvement.

Record the following for each attempt:

• success according to an explicit rule;
• the type and severity of errors;
• total cost and elapsed time;
• human intervention required;
• evidence that allows the result to be checked.

Repeat sensitive cases to observe variability. Also check for errors shared across configurations: agreement between answers does not provide independent validation. Finally, better problem solving does not automatically authorize action. Permissions should remain appropriate to the potential consequences.

Key takeaways

• A reasoning model combines learned capabilities with a way of allocating computation to problem solving.
• Published improvements must be understood in relation to their task, date, and evaluation procedure.
• A readable chain of thought guarantees neither correctness nor complete transparency.
• The right choice minimizes the cost of a verified result within the time your use case allows.

Frequently asked questions

Does a reasoning model learn during every conversation?

Producing intermediate steps does not mean updating its parameters. A model can use examples in its context without learning anything permanently. Updating weights at test time requires an additional mechanism that should be documented separately.

Should I always use the highest reasoning setting?

Reserve this option for tasks where an evaluation shows a benefit. For a simple request, the main effect may be a slower response. Set a time or cost limit that fits your use case as well.

Can I verify an answer without seeing the chain of thought?

Yes: you can check sources, run the proposed program, or verify a proof. A short, reproducible justification can be more useful than a lengthy account of intermediate steps. Focus your checks on the evidence that establishes the result.

Can a reasoning model run locally?

Yes, if its weights are available and your hardware can run it. Memory requirements depend on the model, how its weights are represented, and the context used. Generating a long sequence of intermediate steps also adds computation time.

Does a high math score prove that a model is AGI?

No: it demonstrates a capability on a defined evaluation. A broader claim requires varied tests, unfamiliar tasks, and controls for resources and prior knowledge. The score alone does not support that extrapolation.