Computer use: how it works, tools, and security

A computer use agent completes tasks through a graphical interface: it observes a screen, clicks, types, and checks what changed. It can operate a browser or an entire desktop, but its reliability depends on the workflow, safeguards, and permissions involved. Give it clearly bounded tasks with verifiable results, and require approval before consequential actions.

What computer use covers

Computer use is an action capability, rather than a single product. A model can propose actions while another program executes them. A ready-made application brings these components together; an API integration leaves more of the responsibility with you.

Distinguish what the agent can access from how it observes its environment. An agent restricted to a browser does not necessarily have access to files on the computer. A desktop agent can switch windows and use multiple applications, subject to its permissions. Either type may observe screenshots or use a structured description of visible elements.

Approach	What it observes	How it acts	Main limitation
Visual desktop control	Screenshots of the authorized screen	Mouse, keyboard, scrolling	Sensitive to windows and their arrangement
Structured browsing	Page elements and structure	Actions on identified controls	Depends on the structure exposed
API integration	Data made available by the service	Explicitly supported operations	Coverage limited by the API
Fixed-rule automation	Predefined states and conditions	Programmed sequence	Requires maintenance when the workflow changes

These approaches can work together. A graphical interface might be used to retrieve a document, followed by a structured operation to process its contents.

For workflows that run entirely on the web, the article on browser agents provides additional detail. The central decision remains the same: give the agent the most controllable path to the result without unnecessarily expanding its access.

How the action loop works

The process follows a loop: receive the objective, observe the environment, choose an action, execute it, and observe again. The model does not move the mouse directly. It issues a request that a program translates into an action within the authorized environment.

An instruction such as preparing an export involves several decisions: identifying the correct account, selecting the period, starting the download, and locating the file. An early mistake can produce a result that still looks plausible. The agent therefore needs to check the details that determine subsequent steps, especially the account identity and selected filters.

The next screenshot may reveal a dialog box, a loading state, or an action that had no effect. Continuing from an outdated observation risks clicking the wrong place. Keyboard shortcuts can simplify some actions, but the agent still needs to check which field has focus.

Anthropic’s computer use documentation, accessed on October 24, 2026, describes screenshot and mouse-and-keyboard capabilities whose calls are executed by the client application. It also warns that the model may assume an action succeeded without checking.

Completing a sequence does not prove that the objective was achieved. Clicking export proves neither that the file exists nor that it contains the correct period and data. Define the success criteria before execution, ideally with a check that is independent of the agent’s visual interpretation.

Available options: application, model, or infrastructure

As of October 24, 2026, comparing offerings requires identifying what you are buying: a ready-to-use interface, a model capability, a programmable agent, or an execution environment. These components take care of different parts of the work.

Offering or component	Type	Documented scope	Responsibility to examine
Computer Use in ChatGPT	Desktop application feature	Authorized local interfaces	Computer permissions and regional availability
Claude computer use	Capability available through an API	Environment supplied by the integrator	Action execution and safeguards
Browser Use	Programmable agent and hosted services	Web browsing	Model selection, sessions, and data
Cua	Drivers and infrastructure	Computing environments and isolation	Assembly and operation
Playwright MCP	Tool server	Browser and accessible page structure	Calling agent and session permissions

The ChatGPT Computer Use documentation lists availability on macOS and Windows, in supported regions, with ChatGPT Work and Codex. It distinguishes this local access from tasks performed in a remote environment.

Older product names need careful handling. The ChatGPT agent help page now states that this mode is no longer available and directs readers to ChatGPT Work. It retains historical sections, so their former limits do not reliably describe the current offering. Operator is another former name; its capabilities had been incorporated into agent mode.

This comparison describes documented capabilities and scope. It does not rank the offerings by task success.

Choosing a programmable component

The official Browser Use repository, accessed on October 24, 2026, distinguishes its library, command-line interface, and cloud services. A hosted browser supplies the browsing environment; a hosted agent offering also handles running the agent. That distinction determines what your team needs to operate.

The official Cua repository presents drivers, isolated environments, and tools intended for purposes including evaluation. Treat it as infrastructure to integrate, rather than assuming it provides a finished application suited to your business needs.

The official Playwright MCP repository describes interaction based on structured accessibility representations. This lets an agent target web elements without relying solely on their position in a screenshot. It does not, by itself, determine whether an action is appropriate.

When comparing these components, examine:

Who chooses the actions and which model receives the observations.
Who hosts the session and retains its data.
How to pause, resume, and close a run.
Which operations can be blocked through technical controls.
What evidence remains available after an incident.

The listings in the AI agent directory help you find the projects. Popularity and project activity can indicate ongoing maintenance, but they do not measure reliability on your task.

When to choose an API or conventional automation

Computer use makes sense when an interface is the only practical route to an operation. It can also help navigate a changing workflow or check a result as a user sees it. That flexibility comes at a cost: more observations, waiting, and uncertainty.

When a suitable API exists, examine it first. An API can make fields, identifiers, errors, and permissions explicit. This makes checks easier, although it does not eliminate business errors. A structured integration can still modify the wrong record if the objective is ambiguous.

Fixed-rule automation suits stable workflows with known conditions. Introducing a model to choose every click does not necessarily add value. Conversely, accumulating fragile rules for every possible variation can become difficult to maintain.

Consider a hypothetical example: retrieving a document from a portal without a usable API. The agent can navigate to the download, after which deterministic processing checks the filename, format, and expected fields. Final acceptance then depends on explicit criteria.

Start with the actual need: does the task require interpreting a screen, choosing a variable path, or simply moving known data? Allow the agent to make decisions where judgment is needed. Predictable steps benefit from simple, repeatable checks.

Evaluating reliability across a complete task

An agent can identify buttons correctly and still fail to deliver the required business result. It may also ask for help at the right moment: that interruption reduces autonomy but can prevent an error. Measure success, human intervention, and incidents separately.

Prepare representative situations: slow loading, an existing file, an expired session, an unexpected window, several records with the same name, or a missing result. Also include a workflow in which the agent must stop because the available information is insufficient.

Dimension	Evidence to retain	Decision it informs
Result	Independently verified final state	Was the task completed?
Intervention	Reason for and duration of human takeovers	How much supervision is needed?
Time	Total duration, including waiting	Does the workflow save time?
Cost	Calls, environment, and recovery work	What budget is actually needed?
Incident	Unintended action, duplicate, or exposure	Is the access scope acceptable?

Public benchmarks help establish capabilities within a particular evaluation protocol. OSWorld 2.0, accessed on October 24, 2026, covers long computer tasks in real environments. A score must remain tied to its protocol, permitted tools, and execution conditions.

Do not convert a public result into a probability of success on your own software. A selected demonstration tells you little about routine failures, either. The guide to evaluating an AI agent explains how to build a suitable test set.

Common errors and risky recovery attempts

The model is not the only source of difficulty. An application may move a button, display a notification, or apply a saved filter. A download may still be running even though the screen appears to have returned to its initial state.

Ambiguous labels create another trap. Depending on the context, a confirmation button may save a draft, publish a document, or commit to a purchase. Reading the label needs to be accompanied by checking the item involved, the recipient, and the consequences.

Recovery after an incident needs its own rule. If the connection drops after a submission, immediately trying again may create a duplicate. The agent must first look for evidence of the previous action: a created document, the record’s status, or confirmation from the service.

Define recovery checkpoints that describe the state reached, rather than simply listing clicks performed. A useful checkpoint identifies the open record, confirmed changes, and operations whose outcome remains uncertain. This lets a person take over without guessing.

Finally, limit retries and total run time. When a blockage persists, the expected output should be an actionable report: the attempted action, the observation received, and the remaining uncertainty. Retrying blindly consumes budget and can make the error worse.

Security: displayed content remains data

An agent reads pages, messages, and documents that may contain hostile instructions. A prompt injection attempts to make it treat that content as an authorized instruction. The attack can appear in text or in an image the agent observes.

Browser access adds another concern: a signed-in session already carries authorization. Even without knowing the password, the agent may be able to perform the actions available to the account. Protecting credentials alone is therefore insufficient.

Separate viewing, preparation, and commitment as much as possible. An account that can read records does not necessarily need permission to delete them. A preparation environment can produce a draft without being able to send it. Enforce these restrictions through permissions, rather than relying only on instructions in the prompt.

Isolation limits some consequences for the computer, but it does not remove the permissions of a remote account. An isolated session that can send messages retains that ability. Similarly, an allowed-domain list reduces exposure without guaranteeing that the content on those domains is trustworthy.

Anthropic recommends measures including a dedicated environment, minimal privileges, and human confirmation for decisions with significant consequences. The guide to securing an AI agent helps turn these principles into deployment rules.

Also provide an accessible stop control that actually pauses execution and allows the relevant access to be closed.

Privacy: track screens, files, and sessions

A screenshot may expose more than the intended document: notifications, customer names, tabs, file paths, or data from another record. Preparing a minimal environment reduces this accidental collection. Close unnecessary applications and use a separate profile where possible.

Map the places data passes through separately: the local computer, remote environment, model provider, trace storage, and monitoring tool. A supposedly local run may still send observations to a remote API. A cloud browser may retain a signed-in session.

Ask which data is retained, for how long, in which region, and with what administrative access. Also check how to delete a session and revoke authorizations. These questions need to cover screenshots, downloaded files, and logs.

Keep logging proportionate. Logs help explain an error, but they can also create another copy of sensitive information. Retain useful audit evidence with restricted access and a defined retention period, rather than recording everything indefinitely by default.

For a sensitive task, inspect the visible content before starting. Screen capture permission does not automatically distinguish information relevant to the objective from information belonging to unrelated work. Preparing the computer is therefore part of privacy control.

Cost: calculate it per accepted result

Total cost includes model calls, the execution environment, supporting services, integration, and supervision. A library available at no charge is not necessarily free to operate. Nor does a general subscription prove that the feature you want is included.

Screenshots, repeated observations, and history can contribute to model usage. Loading and recovery extend the time spent occupying the environment. Depending on the offering, these items may be billed in different ways: check exactly what is covered before comparing.

The useful metric is cost per accepted result. Divide spending across a batch by the number of tasks actually validated, including the cost of abandoned attempts. Add the time spent reviewing, correcting, and recovering from incidents.

A workflow that is cheap to start can become poor value if every result requires a complete inspection. Conversely, brief, targeted supervision may suit a repetitive task, provided the result is easy to verify.

Set limits on spending, duration, and attempts. Then compare the workflow with human execution or conventional automation over the same scope. The budget needs to include maintenance when the interface changes, as well as the first successful demonstration.

Running a pilot you can control

Start with a reversible task using fictitious data or data suitable for testing. The prerequisites are a separate environment, an account with limited permissions, an expected result, and a person able to take over. You do not need to authorize sending or publishing anything to test whether an agent can prepare a deliverable.

Describe the result. Specify the account, period, format, and acceptance criteria.
Limit access. Allow only the applications, folders, and destinations needed.
Define stopping conditions. Cover ambiguous situations, persistent errors, and consequential actions.
Test variations. Include a normal workflow and situations that should trigger an interruption.
Check independently. Verify the deliverable or service state, then record discrepancies.
Expand gradually. Add a workflow once you understand the previous results.

Common problems

If the agent loses track of the active window, require a fresh observation before any typing. If it encounters authentication or a CAPTCHA, arrange for an authorized person to take over. If a submission’s outcome remains uncertain, check the remote system’s state before trying again.

Checklist before regular use

The result can be verified without trusting the agent’s report.
Permissions match the task.
Sensitive actions require explicit approval.
Incidents have a recovery procedure.
Traces and sessions have a defined retention period.
Limits and the stop control have been checked.

Key takeaways

Computer use provides access to an interface; the quality of its results still needs to be demonstrated on your workflow.
Distinguish the model, agent, browser, and environment before comparing offerings.
Prefer structured operations when they meet the need and make checks easier.
Limit permissions, verify actual effects, and measure supervision costs alongside execution costs.

Frequently asked questions

Does a computer use agent always control my personal computer?

No. It can operate in a hosted browser, a virtual machine, or an authorized local session. Before connecting anything, check which environment it controls and which accounts are already signed in.

Can I use a computer use agent without knowing how to code?

A ready-made application can spare you from building the action loop yourself. Integrating the capability through an API, however, requires you to manage execution, errors, and permissions. Either way, you need to define a result you can verify.

Does computer use work with every application?

Access to a graphical interface does not guarantee that an agent can complete a task. Compatibility depends on factors including the operating system, permissions, screen readability, and authentication mechanisms. You still need to test your own workflow.

Can an agent work while I use the same computer?

In a shared session, your mouse movements and window changes can disrupt the agent's work. Use a separate environment when the tool supports it. Also check whether human intervention actually pauses its actions.

Does an open source project keep all my data local?

Having access to the code does not determine where data goes. A locally installed program may send screenshots to a remote model or use a hosted browser. Examine each component and its settings separately.

Computer use agents: how they work, available tools, reliability, and security