Choosing an LLM: criteria, pricing, and benchmarks

Choose the language model that meets your quality requirements on real tasks while staying within your cost, response time, and data handling constraints. Use benchmarks to shortlist candidates, then compare them through a representative trial. Check context limits and licensing for the exact model and deployment method you intend to use.

Define the work before comparing models

Start by describing the result you need without naming a provider. Extracting references from invoices, drafting a sales response, and fixing a program require different capabilities. Fluent writing does not establish that a model can accurately use documents or reliably carry out an action.

Specify the inputs: French text, scanned documents, tables, images, conversation history, or tool results. Describe the output as well: a free-form paragraph, structured data, a file change, or a proposal for someone to approve. This distinction keeps your comparison focused on the intended work rather than an unrelated demonstration.

Next, write down the acceptance criteria. A summary must preserve certain qualifications; an extraction must leave a field empty when the value is missing; a code fix must pass tests without removing functionality. Also define when the system should ask for clarification or hand the case over to a person.

The language model profiles can help you build an initial shortlist. Keep advertised capabilities separate from performance observed in your application, however. A feature listed in the documentation still needs to be tested with your files and operating constraints.

Turn your use case into observable criteria

Divide your criteria into requirements and preferences. Incompatibility with your documents or data policy rules out a candidate, even if it produces good answers elsewhere. Writing style can then help distinguish models that meet the requirements.

Use case	Quality to measure	Operational constraint	Failure to look for
Writing and translation	Meaning, tone, terminology	Review time	Adding a claim absent from the source
Document extraction	Field accuracy, handling of missing information	Usable output format	A plausible but invented value
Searching documents	Answers supported by relevant passages	Access to the right sources	A citation that proves nothing
Software development	Functional correctness, preservation of behavior	Integration with tests	A change that conceals a defect
Agent with tools	Correct action selection and arguments	Permissions, duration, budget	An unnecessary or unauthorized action

For each relevant row, define what would demonstrate success. An assessment such as “satisfactory answer” leaves too much room for the reader’s impression. Prefer verifiable criteria: references retained, fields extracted correctly, no unsupported additions, passing tests, or a decision appropriately put on hold.

Evaluate refusals and clarification requests as well. A system that always answers may appear useful while introducing errors that go unnoticed. Conversely, excessive refusals can make a tool impractical for tasks it is authorized to perform.

Read benchmarks with their scope in mind

A benchmark measures performance under a specific evaluation protocol. Before looking at the score, identify the task, language, data, and execution conditions. A strong problem-solving result does not demonstrate the quality of a specialized translation or table extraction.

Stanford’s HELM approach advocates evaluation across multiple scenarios and dimensions, including robustness and efficiency. It also recognizes that an evaluation remains incomplete. Look for measurements that address your needs rather than relying on a single overall ranking.

For coding, the official SWE-bench leaderboard distinguishes between evaluation sets and testing conditions. Check whether you are comparing a model alone or a system that includes tools, instructions, and an execution environment. A different setup can change what the result tells you.

Our article on understanding model rankings explains how to interpret these results; the benchmark comparison tool helps you examine available measurements.

Before relying on a result, check:

the exact identifier of the model evaluated;
the tools and documents it could access;
the reasoning settings and execution budget;
whether answers could be retried or selected;
how closely the benchmark tasks match your use case.

A difference in ranking is not enough to justify a migration. It needs to translate into an observable, reproducible improvement that matters to your application.

Build a trial that exposes errors

Prepare a representative set of cases before refining your instructions. Include common requests, malformed inputs, missing information, and situations where answering would be premature. Use data you are authorized to process and remove unnecessary sensitive information.

Keep the examples used to improve instructions separate from those used for the final comparison. Otherwise, you risk choosing the model best suited to examples you have already optimized for. Also retain difficult cases drawn from errors discovered during the pilot.

First, run candidates under comparable conditions. Then allow documented adjustments to the instructions for each model. A shared prompt supports the initial comparison; a tailored prompt shows what each option can deliver in practice. Keep a record of both results.

Where possible, hide the model’s name from the people reviewing its answers. Assess correctness, completeness, and presentation separately: polished writing can conceal an omission. For structured data and code, add automated checks against the expected result.

The guide to evaluating an AI agent develops this method for workflows involving successive calls and actions. In these cases, also record where the first failure occurred. The final answer alone rarely explains what went wrong.

Distinguish advertised context from usable context

The context window describes how much context a request can accommodate under the service’s counting rules. Check input limits, output limits, and any shared budgets separately. A large advertised capacity does not guarantee that a long response is possible.

A token is a unit into which the model breaks down content; it is not a word. The same document can produce different token counts with different models. Use the token counter for your chosen service when estimating usage, rather than a fixed conversion from pages or characters.

Google’s long-context documentation distinguishes, among other things, retrieving an isolated fact from handling requests that require several pieces of information. It notes that performance varies with the content. A successful lookup therefore does not establish that a model can produce a complete synthesis of a document.

Test documents at their actual length. Put necessary information in different places, include conflicting versions, and request an answer that draws on multiple passages. Check qualifications and exceptions, which often determine whether the result is useful.

Finally, compare sending the full document with sending a selection of relevant passages. The former can simplify processing; the latter can reduce the volume transmitted. Measure selection errors too: a model cannot use a passage your application has withheld.

Calculate the cost of an accepted task

The price per million tokens is a starting point, not the final cost. Add up inputs, outputs, any billed reasoning tokens, paid tools, and retries. Then include the time needed to validate or correct the result.

As a dated reference, Claude Sonnet 5.5 is listed at $2 for input and $10 for output per million tokens in the site’s October 2026 pricing record. The official Anthropic pricing page, accessed October 24, 2026, shows the same base rates. These rates do not cover every billing option.

That pricing page also lists different rates for Claude Haiku 5.5 depending on prompt length. This example illustrates why a headline price must be read alongside its conditions: context tiers, caching, batch processing, and additional fees can change the calculation.

Use this relationship to compare candidates:

Cost per accepted task = total cost of the batch, including corrections, divided by the number of accepted tasks.

Always report this figure alongside the success rate. Otherwise, a model that abandons difficult cases could appear economical. Include resources consumed by failed attempts and the time people spend taking over the work.

The LLM pricing calculator helps you explore budgets. Supply the volumes observed during your pilot, then model different levels of activity. A growing conversation history or more detailed answers can make an initial projection too optimistic.

Measure the speed of the complete service

Distinguish the delay before the first content appears from the time needed to obtain a usable result. An answer that starts quickly but requires extensive correction may be less useful than a slower answer that is immediately acceptable.

For a form, measure the time until validated data is available. For an agent, measure the full task: planning, tool calls, waiting, checks, and retries. Text generation speed covers only part of the process.

Also examine slow requests, temporary errors, and capacity limits. An average hides delays that disrupt users. Run trials in the intended region and at the expected level of concurrency, rather than testing only from a development workstation.

Some settings let you trade quality, cost, and response time within the same model. Anthropic’s model selection documentation recommends evaluating reasoning effort as well. Compare complete, documented configurations rather than product names alone.

Finally, define what should happen when the service is unavailable. Waiting, retrying later, and switching to another model have different consequences. Test the fallback before treating it as an assurance of service continuity.

Check formats, tools, and supported inputs

Examine a claim of “structured output” in terms of the features actually supported. Check field types, accepted constraints, missing values, and refusal handling. Your application must be able to detect an unusable response.

Google’s structured output documentation describes generation guided by a schema. This mechanism constrains the format; it does not establish that the values match the document. Validate structure and data accuracy separately.

For tools, prepare cases where the correct behavior is to make no call. Check function selection, arguments, and the response to an error. In particular, test ambiguous identities and missing authorization before any action that affects an external system.

For images and scanned documents, use your actual files: skewed pages, complex tables, small annotations, and poor-quality scans. Identify any transformations performed before the model call. A document conversion problem may explain a failure too quickly attributed to the model.

Technical compatibility remains a prerequisite. The application must enforce permissions, approval requirements, and action limits. Good test performance does not replace these controls.

Review licensing and where the model runs

Open weights provide access to a model’s distributed parameters under their applicable terms. They do not, by themselves, establish that the whole system is open source. The OSI’s Open Source AI Definition also includes requirements concerning code and information about training data.

For each candidate you can deploy yourself, read the license for the exact version. Check commercial use, redistribution, modifications, and attribution obligations as they apply to your project. Keep the license with the deployed files and refer ambiguities to someone qualified to assess them.

Comparison point	Hosted API	Deployment under your control
Operations	Provider manages the infrastructure	You organize capacity and maintenance
Data	Review data flows and service terms	Check local flows and supporting services
Changes over time	Subject to the provider’s availability lifecycle	Version can be retained under applicable terms
Budget	Usage and contractual options	Hardware, hosting, and operations
Responsibilities	Clarify the contract and allocation of duties	Take responsibility for security and monitoring

Running locally does not, on its own, guarantee privacy. Examine telemetry, logs, backups, and connected tools. For an API, check retention terms, possible uses of your data, and the contractual commitments that apply to your plan.

The guide to running an LLM locally explains hardware constraints. Evaluate a quantized version in its actual configuration: results from another variant are not sufficient.

Decide, document, and reassess

First, eliminate candidates that fail a mandatory requirement. Then compare the remaining options on quality, cost per accepted task, completion time, and operational workload. Avoid an overall score that would allow a low price to offset a critical error.

A single model may be enough. If you are considering routing between models, define observable conditions: an invalid format, a failed check, or an identified task category. A model’s self-reported confidence is not, on its own, a reliable rule for switching.

Keep a decision record covering the model identifier, provider, settings, instructions, evaluation set, and observed limitations. Record why you chose it and what would justify revisiting the decision. This documentation makes migrations and budget discussions easier.

Reassess when the model, contract, documents, or tools change. A successful pilot supports a monitored rollout; it does not demonstrate permanent reliability. Retain a way to return to a known configuration when conditions allow.

Key takeaways

Define your tasks, critical errors, and acceptance criteria before selecting models.
Use benchmarks to guide trials, then compare results on your own data.
Measure quality, cost per accepted task, and total completion time together.
Check usable context, licensing, and data handling terms for the exact configuration.
Document your choice and repeat the evaluation when the system or its use changes.

Frequently asked questions

What is the best language model for French?

A general ranking cannot answer that on its own. Compare candidates using your French documents, including abbreviations, industry terminology, and tone requirements. Assess factual accuracy and writing quality separately.

Does an assistant subscription include API calls?

Do not assume it does: an application subscription and API usage are separate offerings. Check which rights and credits your contract explicitly includes. When comparing budgets, separate user licenses from the API calls your software makes.

Should I choose the model with the most parameters?

Parameter count does not directly measure how useful a model will be for your task. Data, training, settings, and available tools also affect the result. For local deployments, model size also affects hardware requirements.

Can I switch models without changing my application?

A common interface can make switching easier, but it does not guarantee identical behavior. Supported schemas, output limits, and tool calls may differ. Plan to validate functionality before every migration.

Can a model know about information published today?

Text generation alone does not guarantee access to recent publications. For an up-to-date answer, provide search access or a current document source. Then check that the cited references actually support the claims.

Choosing a language model: criteria, benchmarks, pricing, context, and licensing