Test-time adaptation: methods and results

Test-time adaptation adjusts a system to a task’s data while the system is solving it, often by temporarily changing some of the model’s parameters. Inference-time compute is a broader category: it also includes repeated attempts, search, and verification without changing the weights. Published results show gains under certain protocols, but their significance depends on the available data, the budget, and how the final answer is selected.

Distinguishing adaptation, learning, and additional compute

Inference means using a model to produce an output. It does not require the entire system to remain fixed: an application can make several calls, maintain state, or run an optimization procedure before returning its answer.

The terminology nevertheless covers different mechanisms. Test-time scaling increases the resources devoted to solving a problem. Test-time adaptation adjusts the system to the observations it encounters. Test-time training refers more specifically to a learning procedure performed during this phase, usually through parameter optimization.

Mechanism	What changes	Signal used	Possible persistence
In-context learning	Context and activations	Instructions and demonstrations	Within the retained context
Generating multiple candidates	The solutions explored	Sampling, voting, or a verifier	Within the search state
Parameter adaptation	Selected weights or adapters	A local loss function	Temporary or continuous
Program search	Candidate programs	Execution on examples	Within the retained solutions

These categories can work together. A system might adapt a model, generate several programs, and then execute them to select an answer. Its result therefore reflects the complete pipeline. To interpret a score, identify the components actually used instead of inferring the mechanism from the word “reasoning” alone.

Building a learning signal without the final answer

Local learning requires an objective the system can compute. If the desired solution is unknown, the system must use something else: available demonstrations, a property of the data, or an auxiliary task whose answer can be generated automatically.

Yu Sun and his coauthors describe this approach in Test-Time Training with Self-Supervision for Generalization under Distribution Shifts, published at ICML in 2020. Their method turns an unlabeled test observation into a self-supervised problem, updates the parameters, and then makes the prediction. They report improvements on image classification evaluations involving distribution shifts.

This principle does not, however, provide a correct answer for free. An auxiliary objective can become easier to satisfy without improving performance on the main task. The reduction in auxiliary loss and the improvement in the desired answers need to be measured separately.

Consider a hypothetical system receiving documents from a new scanner. It might adapt to visual patterns in those documents, but that does not guarantee more accurate extraction of monetary amounts. The evaluation must measure which fields were extracted correctly. The local objective provides a way to learn; the business criterion remains the measure of success.

Adjusting confidence does not guarantee accuracy

Another family of methods works directly with the model’s predictions. Here, entropy measures how spread out the prediction distribution is: a prediction concentrated on a particular class has lower entropy than an uncertain prediction.

In Tent, presented at ICLR in 2021, Dequan Wang, Evan Shelhamer, and their coauthors propose minimizing this entropy during testing. The method estimates normalization statistics and optimizes channel-wise affine transformations. Among their findings, the authors report lower error rates on classification datasets containing corrupted images.

This approach applies to a specific setting. It does not establish a general rule that increasing confidence makes an answer more accurate. If a system has already settled on an incorrect interpretation, strengthening its preference could instead make that error persist.

An evaluation therefore needs to track errors and confidence separately. Also check what happens when the data changes abruptly, when classes are imbalanced, or when observations arrive individually. A method evaluated on batches cannot be assumed to behave the same way in a service that handles each request independently.

The choice of parameters that can change also matters. Adapting a small subset limits the scope of the updates, but it does not remove the need to compare against the fixed model or provide a procedure for restoring the original state.

Learning from the available demonstrations on ARC

ARC grid tasks provide input-output examples alongside a new problem. The system must infer a transformation that applies to an additional input. The reference page on ARC-AGI explains this setup and the differences between evaluation sets.

Adaptation uses the demonstrations as local training data. For example, a system can hold back a known example to test a hypothesis developed from the others. Changes to colors or orientation can also produce variants, provided they preserve the relationship the system needs to learn.

The paper by Ekin Akyürek and his coauthors was initially titled The Surprising Effectiveness of Test-Time Training for Abstract Reasoning. Its March 25, 2025 revision is titled The Surprising Effectiveness of Test-Time Training for Few-Shot Learning. It reports 53.0% accuracy on ARC’s public validation set with an 8-billion-parameter model, rising to 61.9% when combined with program synthesis methods.

These scores therefore do not describe either a general model used without preparation or adaptation operating in isolation. The paper examines factors including prior preparation, local data, and inference variants. The hybrid result should not be attributed solely to weight updates.

What public results support

Public experiments help researchers understand mechanisms and reproduce comparisons. They also allow researchers to refine their methods. This dual role makes it necessary to distinguish a development set from an evaluation kept separate from technical decisions.

The official ARC Prize report published on December 6, 2024 highlights progress from deep learning-guided program synthesis, test-time training, and combinations of the two. It reports, among other results, a private score of 53.5% for the ARChitects team. That result comes from a different dataset and protocol than the public scores cited above.

Useful comparisons involve systems evaluated under the same conditions. They should specify the examples available, the attempts allowed, the time available, and the success criterion. Finding a correct answer somewhere among many candidates does not automatically mean the system can select it correctly.

Avoid turning performance on grids into a claim about every kind of new task. Discovering a visual transformation, drafting a contract, and planning actions involve different constraints. A gain on a benchmark establishes, first of all, a result on that benchmark.

How to read model leaderboards offers further guidance: a percentage becomes meaningful when the accompanying protocol and resources are known.

Using more compute with fixed weights

The alternative to parameter adaptation is to make better use of an existing model. A system can generate independent answers, revise an attempt, explore several branches, or ask a verifier to score intermediate steps.

Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar study these trade-offs in Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters, submitted on August 6, 2024. On mathematical problems, they report an improvement in test-time compute efficiency of more than a factor of four over a best-of-N baseline. Their conclusion depends in particular on problem difficulty and budget allocation.

This result does not mean that extending every generation automatically improves its quality. A search procedure must be able to reach good solutions and then recognize them. If its proposals remain incorrect, or its verifier favors convincing errors, the additional computation may be poorly spent.

More computation does not necessarily mean more text, either. Scaling up Test-Time Compute with Latent Reasoning, published in February 2025, explores an architecture that repeats a recurrent block in latent space. This approach separates the depth of internal processing from the length of the visible answer.

For users, a long response is therefore an insufficient measure of useful effort. What matters is the system’s performance, time requirements, and resource use.

The verifier and stopping rule determine usefulness

A generator proposes candidates; a selection mechanism decides what to return. Selection can rely on execution, voting, a consistency measure, or an evaluator model. Each option brings its own limitations.

In a hypothetical table transformation task, several programs might reproduce every available example. Yet they could behave differently on a new input. Matching the demonstrations rules out some errors without necessarily identifying the intended rule.

Majority voting presents a similar difficulty. Answers that agree can share the same false assumption. To assess the value of repeated attempts, measure how much useful diversity they produce, then whether the selection mechanism can retain the correct proposals.

The stopping rule must be defined before evaluation. It might depend on a maximum budget, external validation, or a lack of progress. A model’s stated confidence is not enough to establish that a solution has been verified.

For an application, define explicitly what should happen when uncertainty remains: return an answer with specific qualifications, request missing information, or hand the task to a human. The system must be able to stop without manufacturing certainty. Do LLMs really reason? examines the difficulties involved in interpreting observed behavior.

Measuring the full cost rather than tokens alone

Additional compute shifts part of the expense to each problem being processed. To compare strategies, measure their total cost per task and then their cost per correctly solved task.

Cost area	Generation and search	Parameter adaptation
Main computation	Generating and evaluating candidates	Optimizing, then producing an answer
Memory	Contexts, activations, and branches	Adapted parameters and optimizer state
Latency	Sequential or parallel attempts	Preparation and local updates
Expenses to include	Generator, verifier, and tools	Adaptation, inference, and restoration
Operational challenge	Coordinating selection	Isolating and managing modified states

The same token count does not necessarily represent the same amount of computation. The model, optimization passes, tools, and hardware all affect the expense. If energy use has not been measured, reporting runtime and configuration is preferable to making an arbitrary conversion into an energy estimate.

Usage volume also changes the trade-off. Shared preparation can be amortized across many requests, while adaptation performed separately for each problem remains a recurring cost. Conversely, reserving expensive processing for difficult requests can avoid slowing down simple ones.

The LLM pricing calculator helps estimate calls billed by token usage. An experimental cost assessment must also include expenses absent from that billing: local computation, verification, temporary storage, and discarded attempts.

Designing an evaluation that isolates the gain

To decide whether a method justifies its additional cost, start with a documented, fixed baseline. Set the available data, selection rule, and budget before inspecting the answers in the final evaluation set.

Then compare changes separately: adding in-context examples, making multiple attempts, applying local adaptation, and finally combining methods where appropriate. Without this separation, an improvement could come from a better input format or more effective selection rather than test-time learning.

A useful report should state:

Final success under the official scoring rule.
Resources used, including discarded attempts.
Typical latency and the slowest cases.
New errors introduced by the method.
Stability across runs and the effect of task order.

State management must be explicit. With episodic adaptation, every task starts from an initial state. With continuous adaptation, changes accumulate. These results answer different questions because the latter system benefits from a history of previous observations.

Also separate the data used to tune the learning rate, choose transformations, or define stopping rules from the data used for the final assessment. A task’s permitted demonstrations can support adaptation; its hidden answer must not be used to choose the settings.

The guide to evaluating an AI agent provides a complementary framework for connecting these measurements to a practical use case.

Different interpretations of generalization

François Chollet frames intelligence as the efficiency of skill acquisition relative to prior knowledge, experience, and generalization difficulty in On the Measure of Intelligence, published in November 2019. This perspective calls for distinguishing the skill level achieved from the information used to reach it.

Test-time adaptation raises precisely this question: what can a system learn from observations specific to a new problem? The number of examples matters, but so do previously acquired knowledge and the choices made by the system’s designers. The page on generalization explores these distinctions.

The studies discussed here nevertheless emphasize different approaches. Akyürek and his coauthors show the value of local parameter optimization for certain new tasks. Snell and his coauthors investigate how to allocate computation between search and revision. These results do not establish that any particular architecture is necessary or sufficient.

The distinction also concerns the outcome being sought. For an operational service, solving more tasks within an acceptable budget may be enough to justify a method. Research into general intelligence must additionally examine novelty, prior knowledge, and transfer across domains.

A useful evaluation makes these objectives explicit. It can demonstrate a practical improvement without claiming to provide a complete measure of intelligence.

Choosing based on the task and available feedback

Start with the kind of information available while solving the problem. Relevant local demonstrations may justify adaptation. A verifiable property may make search useful. Additional documents may simply call for better context, without fine-tuning.

For an extraction task guided by client-specific examples, first compare context alone against local adaptation. For code generation supported by reliable tests, investigate the value of generating and executing multiple candidates. For a question whose answer cannot be checked against any source, more attempts do not resolve the lack of evidence.

Plan how the data and resulting state will be managed as well. If adapted parameters are retained, determine who can reuse them and how to restore the initial state. If processing is temporary, check that restoration covers every relevant state, rather than only the conversation text.

When comparing methods, keep these points in mind:

Identify what changes: context, search, statistics, or parameters.
Require results tied to an explicit protocol and budget.
Measure final answer selection, beyond the existence of a good candidate.
Check regressions, persistence of changes, and the full cost.

The AGI and research hub places these mechanisms alongside other approaches to learning and solving new tasks. Their value depends on their measured ability to turn available information into correct answers within the constraints of the intended use.

Frequently asked questions

What is the difference between test-time adaptation and test-time scaling?

Test-time adaptation adjusts a system to data it encounters during use, often by changing parameters or internal statistics. Test-time scaling increases the computation devoted to producing an answer. Searching through several possible solutions falls into the second category without necessarily involving the first.

Does a model learn when you ask it to think harder?

That instruction provides no evidence that the model's weights change. Depending on the system, it may alter the generated text or the generation strategy. To establish that parameter learning occurs, you need to identify an optimization procedure and the parameters it updates.

Can a model adapt without knowing the correct answer?

Yes, if the available data supports an auxiliary objective or the task includes demonstrations. The final answer can remain hidden. The challenge is checking that the objective helps solve the problem rather than reinforcing an error.

Do ARC results prove that adaptation leads to AGI?

They show that certain methods help solve visual abstraction problems under a particular evaluation protocol. A claim about AGI would require broader evaluations involving new tasks, controlled resources, and an analysis of prior knowledge.

Should changes be retained after each task?

That depends on the protocol. Episodic adaptation restores the initial state, while continuous adaptation retains changes across observations. This choice must be explicit because it affects reproducibility and the risk of information carrying over between users.

Test-time adaptation: how it works and what results show