Computer use: an AI that operates a computer the way you do

Definition

Put simply: you give the AI a virtual screen, mouse and keyboard. Instead of calling functions built for it, an AI agent with computer use works through the interface designed for humans. It relies on a multimodal model that can read a screenshot, spot a field or a button and estimate its coordinates.

Anthropic introduced the term when it announced the capability on October 22, 2024 as a public beta for Claude 3.5 Sonnet, presented at the time as experimental, still slow and error-prone.

The agent loop

It works as a loop repeated until the task is done:

1. the application sends the model the instruction and a screenshot;
2. the model chooses an action: click, type, key press, scroll, zoom, wait;
3. the application performs the action in the environment;
4. it returns the result and a fresh screenshot, and the model decides what comes next.

A key point: the model never touches the machine directly. The developer’s code executes each action, which means actions can be filtered, logged or blocked. The official tool documentation lists these actions and the recommended precautions.

A concrete example

An accounts team receives invoices every week that must be keyed into a legacy business application with no API. An agent running in a dedicated virtual machine opens each PDF, copies supplier, date and amount into the application’s fields, then stops before the final submission so an accountant can review and confirm. The agent has no access to email or banking credentials.

Why it matters, and what to watch

Computer use extends automation to software that no integration covered. It also widens the attack surface: a web page can carry a prompt injection that pushes the agent to act against its user. Official guidance points the same way: a sandbox with minimal privileges, no sensitive data, an allowlist of domains and a human in the loop for actions with real-world consequences. Our agent risk assessment helps you walk through these points.

Common misconceptions

• “The agent sees the screen like we do.” It reasons over still images captured between actions and can miss what moves or sits outside the frame.
• “It’s the best way to automate.” It is often the slowest and most expensive; an API remains preferable when one exists.
• “Running it on my own laptop is fine.” It would inherit your open sessions and files: isolate it.

Related terms: AI browser, AI agent, prompt injection, sandboxing.