Open-source or closed models for your AI agents?
Choose an open model if you need control over its hosting, want to retain its files or adapt how it works, and have the resources to operate it. A closed model accessed through an API is a practical option when you prefer infrastructure managed by a provider, provided its data terms meet your requirements. For an agent, base the decision primarily on task success, total cost, and the permissions it needs. For the same trade-off on the agent software side, see open source or proprietary AI agent; for a vendor that mixes both, see Mistral AI.
Open-source LLMs: distinguish openness from access to weights
The term open-source LLM covers different arrangements. A model may offer downloadable files without publishing its training code or allowing every use. Open weights means access to the parameters learned during training; it does not guarantee an open-source license or the ability to reproduce that training.
The Open Source Initiative’s Open Source AI Definition requires the freedom to use, study, modify, and share a system. It also requires the parameters, necessary code, and sufficiently detailed information about the training data. Publishing weights alone therefore does not automatically meet this definition.
Before selecting a model, examine the license for the exact release you plan to use. Check commercial use, modification, redistribution, attribution requirements, and any restrictions. Review the terms for the model, the software running it, and the hosting service separately. An open-source interface can call a closed model, while an open model can run on proprietary infrastructure.
Finally, open and local do not mean the same thing. You can pay for a remote service that runs open weights. Conversely, downloading a model does not mean your computer can run it with acceptable performance. The language model profiles can help you identify candidates, but you still need to read the original licensing and operating terms.
Compare hosting arrangements as well
The decision has two dimensions: what you are allowed to do with the model, and who operates the machines. Comparing a closed API with a self-hosted open model combines those decisions. A hosted service running open weights offers an option between them.
| Criterion | Closed model through an API | Hosted open weights | Self-hosted open weights |
|---|---|---|---|
| Machine operations | Provider | Hosting provider | Your organization |
| Access to parameters | Generally unavailable | Available depending on the release | Available depending on the release |
| Where data goes | Provider’s infrastructure | Hosting provider’s infrastructure | Infrastructure you configure |
| Modifying weights | Depends on the offering | Depends on the offering and license | Depends on the license and your resources |
| Expenses to track | Usage and associated services | Usage or reserved capacity | Hardware, energy, operations |
| Continuity | Depends on the service | Depends on the service and available files | Depends on your ability to operate the system |
Managed hosting does not remove your responsibilities. You still configure access, tools, and the agent’s rules. Self-hosting gives you more control over deployment, but also makes you responsible for availability, updates, and monitoring.
The Hugging Face Inference Endpoints configuration documentation illustrates this distinction. It includes choices for hardware, authentication, networking, and capacity scaling. These settings describe the service that runs the model, independently of how open its weights are.
If you are preparing to deploy on your own hardware, the guide to running LLMs locally explains the constraints to assess. Start with your actual workload: long documents, concurrent users, and acceptable response times.
For an agent, evaluate actions as well as conversation
A model that writes well may still fail when it needs to choose a tool, supply its arguments, or recover from an error. The quality of its tool calling therefore matters as much as the quality of its answers. Being open or closed provides no guarantee here.
Build a set of tasks that represents your intended use. For a document agent, check whether it retrieves the right material, answers faithfully, and declines to draw conclusions when information is missing. For an agent that edits files, inspect both the intended result and any unwanted changes.
Also examine the steps the agent takes: unnecessary tool calls, repetitive loops, ignored errors, or bypassed human approval. A task completed after many retries may still be too slow or too expensive for your work.
Anthropic’s guide to evaluating agents distinguishes checks performed by code, models, and people. It emphasizes evaluating both the outcome and the execution traces. The same approach applies when comparing open and closed models.
Results from the Benchmark Lab can help you build a shortlist. The article on understanding model rankings explains how to interpret their limitations. To make your final decision, rerun your own tasks using the model and configuration you actually intend to deploy.
Calculate the cost per successful task
A free download does not mean free operation. With self-hosting, account for available capacity, actual utilization, storage, monitoring, and time spent handling incidents. An underused machine still costs money when your agent is idle.
An API converts some of those expenses into usage charges. However, an agent task can involve many calls, an expanding conversation history, and repeated steps. The advertised token price alone is therefore insufficient to predict the bill.
Compare the options using the same scope:
- Tasks completed correctly and accepted.
- Calls, retries, and failures that consume resources.
- Any external tools that carry separate charges.
- Human time spent reviewing and correcting results.
- Capacity needed during busy periods.
The cost per successful task relates those expenses to useful outcomes. A model that costs less per call can end up costing more if it needs more attempts or corrections. A slower model may also prevent you from meeting the required turnaround time.
The LLM pricing calculator helps estimate the API component. For self-hosting, build a comparable budget that includes operations. Do not assume that a particular usage volume always makes local deployment cheaper: the break-even point depends on the model, hardware, concurrent requests, and your organization.
Privacy: trace the entire path your data takes
Open weights do not tell you where your documents go. An open model running at a service provider receives whatever data you send it. A local model may stay within your infrastructure while its agent subsequently sends excerpts to a web search service or remote tool.
Map the entire path: user input, document preparation, model, tools, logs, and backups. Identify who receives each piece of content and who may have administrative access. Review the data retention and usage commitments of every service involved.
The Hugging Face Inference Endpoints security documentation, for example, distinguishes payloads from logs and describes several access arrangements. This shows why a general privacy statement cannot replace reading the terms of the specific service you choose.
The article on AI agent data and the GDPR explores this analysis in more detail. For your technical decision, first define concrete requirements: permitted data categories, acceptable recipients, restricted access, and a deletion process you can verify.
Finally, every hosting arrangement requires limits on permissions. A local agent can delete a file or transmit a secret if its tools allow it. Enforce approval for sensitive actions in the program that executes them, rather than relying solely on instructions given to the model.
Choose an architecture you can maintain
For an initial deployment without an infrastructure team, a managed API can reduce operational work. That choice still depends on your privacy, availability, and budget requirements. Plan what to do if the service becomes unavailable or its terms change.
Self-hosting is worth considering when control over deployment is an explicit requirement. Assign someone responsibility for updates, incidents, and evaluations after changes. Possessing the files lets you retain a model; it does not guarantee that you can keep its service running over time.
A hybrid architecture can assign some tasks to a local model and others to an API. However, any handoff to the remote service must comply with your data rules. A local outage must not automatically trigger the transfer of confidential documents to an unauthorized destination.
Anthropic describes routing between specialized workflows as a possible architecture pattern. Its value needs to be measured: adding a routing decision also introduces potential errors and maintenance work. Keep that complexity only if it improves your results.
The guide to choosing a language model expands on this process. For comparisons between specific candidates, use the model comparisons. This guide focuses on the decision criteria that apply across those choices.
Key takeaways
- Distinguish licensing, access to weights, and execution location before comparing offerings.
- Evaluate the agent on complete tasks, including its tools, errors, and approval steps.
- Compare the cost of accepted results, including retries and operations.
- Choose an architecture whose data flows you can verify and whose maintenance you can sustain.
Frequently asked questions
Can an open-source LLM run without an internet connection?
It can run offline if the necessary files are available on the machine and its runtime environment supports offline use. However, an agent that uses web search or remote software will still need a network connection for those actions. Check the model's connections and the tools' connections separately.
Can you use an open-weight model commercially?
Access to the files does not establish your commercial rights. Read the license for the exact release, including any restrictions and obligations that apply to redistribution. The license for the software running the model does not replace the license for its weights.
Do you need to train your own model to run a local agent?
No. You can run a pretrained model and give it instructions and tools. Additional training addresses a separate need and should be considered after checking the existing model's limitations. It is not a prerequisite for self-hosting.
Can you switch models without rebuilding your agent?
A shared access layer can reduce the changes needed in your code. It does not guarantee identical behavior: tool-call formats, context handling, and responses to errors all need to be checked again. Keep a set of reference tests to determine whether a replacement is acceptable.