Anthropic releases Claude Computer Use Developer Best Practices Guide

Anthropic launched Claude The Computer Use Developer Best Practices Guide covers a complete solution from screenshot preprocessing and model selection to security defense. (Targeting...) Claude 4.6 Family and Opus 4.7 Model: Core recommendations include pre-scaling screenshots to 1280×720 to improve click accuracy, prioritizing text over screenshots, and replacing traditional tutorial mode with a more advanced one. Prompt The project integrates three layers of injection defense and long-dialogue context compression strategies to help build production-grade... Agent automaticChemical system.

Click accuracy is the foundation of Computer Use integration. If a click is off-target, all subsequent workflows will fail. The article points out that the most impactful optimization is also the most...SimpleThe screenshot is downsampled before being sent to the API.

API internal processing limitations

Claude The 4.6 family of APIs is limited to...:

• Maximum length of the long side: 1568 pixels
• Maximum total pixels: 1.15MP
• If any limit is exceeded, it will be internally silent downsampled.

Opus 4.7 supports higher resolutions.:

• Maximum length of the long side: 2576 pixels
• Maximum total pixels: 3.75MP
• Exceeding the limit will also result in silent downsampling.

core issues

When the screenshot exceeds the API limit, the model sees a compressed image, and the returned coordinates are based on the compressed size. The client-side harness still performs clicks at the original resolution. This mismatch between the coordinate space and the image perceived by the model is the primary reason for inaccurate clicks at high resolutions.

recommendresolution

• General Default1280×720. Uses approximately 80% of the pixel budget, common in training data, and has good compatibility.
• Opus 4.7 recommend1080p achieves a better balance between token consumption and performance.
• Maximize API compatibilityCalculate the optimal resolution based on the native aspect ratio to avoid proportional distortion caused by forced stretching.

Resolutions to be avoided

• Native resolution (unscaled)Unless it happens to be below the limit, this is the most common reason for inaccurate clicking.
• Too low resolution (below 960×540)Too many details are lost, and the model cannot recognize small UI elements.
• macOS NoteScreenshots often have a 2x device pixel ratio, so a 1440p screen actually outputs 2880p, far exceeding the API limit.
• 4.6 Family members should avoid having a birth rate of 1920×1080 or higher.: It will be silently compressed if it exceeds the pixel limit; Opus 4.7 has a higher limit, 1080p and 1440p are within budget, but native 4K should still be avoided without scaling.

Coordinate scaling

After automatically resizing the screenshot before sending, the coordinates returned by the model are based on the display_width_px / display_height_px at the time of sending. These coordinates must be proportionally restored to the actual screen coordinates before execution.

Sorting message array contents

When constructing the message content array, place the text command before the screenshot.

content = [ {"type": "text", "text": "Click on the Submit button"}, {"type": "image", ...}, ]

content = [
 {"type": "image", ...},
 {"type": "text", "text": "Click on the Submit button"},
]

Model selection

Based on internal testing at Anthropic, different models each have their own strengths:

• Sonnet 4.6Mechanical clicking offers the highest accuracy, precise spatial positioning, fewer errors at close range, and better tolerance for heavy image compression. It is suitable for most mechanically executed tasks, achieving the optimal balance between accuracy, inference, and cost.
• Opus 4.7Opus 4.7 offers stronger inference capabilities and its click accuracy has caught up with Sonnet 4.6. Furthermore, with a larger resolution budget (3.75MP vs 1.15MP), it requires less compression. If the task requires both inference and precise clicks, Opus 4.7 is currently the optimal solution.
• Haiku 4.5The latency-first option is suitable for scenarios with extremely high speed requirements.

Advanced Mode: Commander Mode (Orchestrator + Sub-agent)

In complex workflows, a model with strong reasoning capabilities (such as Opus) can be used as the "commander" to be responsible for planning and decision-making, while Sonnet or Haiku can execute specific click operations, thus dividing the work and cooperating.

Large buttons, input boxes, standard menu items,Claude Accuracy is generally reliable with small elements such as checkboxes, system tray icons, drop-down arrows, and small switches.

Reason: When a 4K screen (3840×2160) is compressed to 720p, the checkboxes that were originally 16 pixels are reduced to about 5 pixels, making them difficult to hit accurately.

Solution

• Turn on Zoom:Claude Versions 4.6 and 4.7 support zoom functionality, allowing you to zoom in on a specific area of the model before clicking. Add "enable_zoom": True to the tool configuration.
• Enlarge the targetIf the UI is controllable, increasing the size of the click target (lowering the system DPI, increasing the browser zoom, adjusting the UI scale) can disproportionately improve accuracy.
• Keyboard replacementFor very small elements, tab navigation or keyboard shortcuts are more reliable than mouse clicks.
• Consider source resolutionCompression of a 4K+ monitor to 720p will result in a significant loss of detail. If using a 4.6 model, try lowering the DPI or focusing the screenshot on the relevant screen area; if using Opus 4.7, a higher resolution budget can reduce the need for compression.

Optimization of Thinking Effort

Claude Supports adaptive thinking, with the intensity set via the thinking parameter: low, medium, high, xhigh (Opus 4.7 only), and max.

Opus 4.7

In OSWorld Verified benchmark tests:

• high-endThe accuracy is close to the maximum, but the output tokens are only about half of the required values.
• low-endThe score is similar to the high/max of Opus 4.6, and the token usage is about 1/10.
• max fileThe highest score corresponds to a significant increase in token cost.

• default:high (best cost-effectiveness for complex, multi-step interactions)
• High throughput/cost sensitive:low
• SimplefastTaskTry Sonnet 4.6
• Complex one-time task:max

Claude 4.6 Family

Tests show that the success rate of medium-level tasks is close to its maximum, and further increases in thinking effort yield diminishing returns.

• recommenddefaultmedium
• Low is better than shutting down thinking.Because of the reduction in errors and retries, the actual total token consumption is actually less.
• NorecommendUsing max in Computer UseThe test showed no improvement in accuracy compared to high; it only increased token costs.

let AI Agent When directly operating a computer, security is paramount. Anthropic's defense system consists of three layers:

First layer:Training Immunity

During training, the model is exposed to a large number of web pages and application interfaces containing injected content, and learns to identify and reject malicious commands through reinforcement learning.

Second layer:Real-time classifier

Parallel scanning enters on each request Claude The contextual content is used to detect hidden instructions in text, instructions embedded in images, and attempts to deceive. Agent Fake UI elements.

When using the official computer_20251124 tool type, a prompt appears indicating the need to inject a classifier.automaticRuns in parallel with model inference, with zero additional latency and zero additional cost. This layer is not available if you implement your own tool instead of using the official types.automaticProtect.

Third layer:Human-in-the-loop

Before performing irreversible operations (submitting a form, making a payment, sending a message, or modifying data), allow... Agent Pause, please confirm. The original text emphasizes: the most effective defense is actually human-in-the-loop.

Computer Use tasks are often very long, with each screenshot consuming 1000–1800 tokens. A 200K context window fills up with less than 100 screenshots, and a 1M window is also in short supply.

First layer: Cache Breakpoints

The API supports up to 4 cached breakpoints.recommendProcedure: Place one in the system prompts and tool definitions (fixed content), and place the other three in...up to dateIn the `tool_result`, old tags are cleared and new tags are placed in each round. This way, the API will not reprocess the entire dialog prefix every time.

Second layer: Rolling Buffer

Only retain the complete data of the most recent N screenshots; earlier screenshots are replaced with text placeholders [Image omitted].

Key details: Batch replacement is used instead of individual image cleanup. The most recent 3 images are kept by default (keep_n=3), and cleanup is performed every 25 images (interval=25). If images are replaced one by one, the change of the dialog prefix in each round will cause the cache to be invalidated; batch replacement can keep the prefix byte consistent across multiple rounds, maintaining the cache hit rate.

Third layer:LLM compression

When the scroll buffer can no longer hold the content, the model itself summarizes the dialogue history and then discards the original content.

The compressed prompt template is required to retain 8 types of information: complete user instructions (retaining all constraints such as "must", "do not", "always", etc. word for word), task template, constraint rules, executed operations, error and repair records, progress tracking, current status, and next step plan.

The most critical principle: All user commands must be preserved verbatim. User commands are the most critical element; their loss will lead to… Agent Deviating from the mission.

serverautomaticcompression

Add the `context_management` parameter and the `compact-2026-01-12 beta` flag to your API request. The server will then respond when the number of input tokens reaches a threshold.automaticTrigger compression. Upon receiving the compression response, the client simply truncates its local message array to the same position to maintain cache alignment.

The traditional approach is to describe the task in words, but this is tedious and prone to errors in model understanding. The original paper proposes a new approach: Don't tell... Claude How to do it? Just "demonstrate" it directly.

Recording stage

The user manually performs the task once, and the system records each step of the operation (click coordinates, input content, page navigation), and provides a screenshot for each step, with the click location marked by a blue circle on the screenshot.

Replay stage

Claude Received complete operation demonstration: "Step 1, click the expense type drop-down menu" with a screenshot; "Step 2, select travel type" with the next screenshot...Claude The same sequence is executed in the current real-world environment, but it will not rigidly replay according to coordinates. If the UI layout changes, buttons are moved, or menus are rearranged,Claude Based on the demonstration, it will understand "what to do" and find the corresponding element on the current screen.

Three playback modes

• Strict modeFollow the steps exactly, and stop and report if there are significant UI changes. Suitable for compliance-sensitive scenarios.
• Adaptive modeUse the example as a reference but make flexible adjustments to accommodate minor layout changes, button renaming, and menu rearrangement.recommendDefault mode.
• Goal-oriented modelThis method focuses solely on the final result; the recording steps are for reference only. It is suitable for scenarios where the UI frequently changes but the goal remains the same.

Computer use mostly involves mechanical operations (clicking, typing, scrolling), making Sonnet a low-cost and fast solution. However, occasional deeper considerations are needed: Should I click this button? What if the information is incorrect? How do I undo a mistake in the process?

practiceSonnet performs routine operations autonomously, while Opus 4.7 is invoked as an advisor when strategic decisions are needed. After Opus makes a decision, it returns the process to Sonnet to continue executing the repetitive steps, achieving low cost and high efficiency.intelligentThe balance.

Ineffective optimization (ineffective after internal testing)

The original text explicitly lists the following methods that did not produce consistent improvements in internal evaluations:

• The screenshot is divided into smaller tiles (quadrants/regions) and sent separately.
• Overlaying coordinate grids onto the screenshot helps with model localization.
• Changing the scaling algorithm (PIL LANCZOS, sips, etc. yielded no difference in results).

The original text provides a systematic diagnostic framework:

Special scenariosSome dropdown menus may call system-level UI elements that the browser viewport cannot capture—the model appears to fail, but in reality, the menu is simply not visible. In this case, the model should be changed to use JavaScript execution, keyboard navigation, or direct DOM manipulation, instead of clicking.

The guide provides ready-to-use Python code templates, covering:

• compute_max_api_fit(): Calculate the optimal resolution based on the native aspect ratio using the API
• prepare_screenshot(): Zoom in on the screenshot and convert it to base64
• scale_coordinates()Restore the coordinates returned by the API to the actual screen coordinates.

And a complete API call example: the entire process from screenshot capture, pre-scaling, message construction (text before image), tool configuration to coordinate restoration execution.

The core engineering logic of the guide can be summarized as follows: pre-scaled screenshots eliminate coordinate misalignment → text precedes screenshots for better understanding → scenario-based selection (Sonnet execution/Opus inference) → zoom or keyboard input for small goals → optimal medium/high level of thinking → three-layer defense for security → three-layer funnel for context management → recording instead of writing to reduce costs → advisor mode to save tokens. "Pre-scaled screenshots" represent the single-point optimization with the highest return on investment, while "teaching mode" represents... Prompt A new paradigm for engineering to evolve towards demonstration learning.

• Official website addresshttps://claude.com/blog/best-practices-for-computer-and-browser-use-with-claude