Computer use

Computer use is the Computer tool. The model reads the application's accessibility tree or takes a screenshot, acts in the application in front, and checks the result. With the default asks access type, the first act in each application asks by the application's name. Existing grants, permission rules and the selected access type determine whether later actions need a question. The tool is on by default on a machine that has the desktop driver.

Targeting controls and taking fewer screenshots

Read tree when a control's name answers the question; use a screenshot when pixels matter. Mouse actions accept target: { role, name, text } instead of x and y: role and name match exactly, ignoring case and repeated whitespace; text matches a substring of the title or value. Supply only the fields needed to identify one element. An ambiguous match lists up to eight numbered candidates and nth: 2 on the target picks the second of them; a missing match names the three nearest by text; a truncated match or an element without bounds refuses with a remedy. Targets, find and wait with until are resolved afresh before the act at depth 32 and 2000 elements unless you pass smaller depth or limit values; larger values clamp to those caps with a note in the result. find accepts the same target and returns matching numbered rows without an image. ref: 7 on a mouse action uses row [7] from the latest tree or find. References are checked again before input; a missing element or a new tree read makes the old reference unusable. root: 7 or root: { role: "AXApplicationDialog", name: "Settings" } limits a tree or find read to that subtree.

For example, click with target: { role: "AXButton", name: "Save" } clicks that control's centre. move hovers over a target; scroll scrolls at it; drag accepts a start target and a toTarget destination. Pixel coordinates remain available. click also accepts count: 3 for a triple-click.

On macOS, tree, find and wait with until use the focused window by default. scope: "app" includes the other windows, with the focused window first, but omits hidden menu entries. The driver requests web-content accessibility on the application; a browser may take about two seconds to expose its page after the first read. A result says when page content is still starting or unavailable. Use a longer wait with until while it starts; use a screenshot when no page tree is exposed. An incomplete tree never establishes that a target is unique or absent.

settleMs caps post-action settling time (default 300 ms, cap 5000 ms); a captured act finishes early as soon as two consecutive captures match, and a mouse act checks only the region around its target so unrelated animation elsewhere does not delay it. An act's result ends with a second line read once after the act settles, for example changed · now: window "Automations" · dialog "Webhook" · focus AXTextField "Name" · url example.test/automations: changed when the window title or the focused control differs from before the act, unchanged when nothing moved within settleMs. That line tracks the title and the focus; a page can still be loading behind an unchanged title, so use wait with until to wait for a specific control. Intermediate frames are neither saved nor sent to the model, capture: false skips the capture, and any act accepts an optional label. With capture: true, scale, full and display describe that capture.

Passing verify: true on an act compares the tree and front window before and after the action, reports whether a change was observed (or if the result is unknown), names the focused control when available, and suggests the next read. Verification does not prove that a save or form submission succeeded and never retries an act automatically.

wait accepts until with { role, name, text } for an element, { title } for the window title, { url } for the address a browser window shows, or gone: true beside one of those to wait for it to leave, bounded by durationMs (default 1000 ms, cap 10000 ms). The result names the condition that passed, or that the time ran out with the size of the last tree read; if the front window changes the call reports that instead of succeeding. capture: false makes this an image-free check.

read with target or ref returns that element's labelled value or text (up to 20,000 characters), and read with clipboard: true returns the text copied last, under its own permission card. focusApp: { name } brings a running application to the front by name or bundle id. Both are macOS reads today; on other platforms they say so.

A screenshot, or the screenshot after an act, can take region: { x, y, width, height } in pixels of the last screenshot. The saved and transmitted image contains only that rectangle; its top-left becomes (0, 0) for subsequent pixel actions. Mercury captures the full display locally and crops the image before saving and sending it, which reduces image size without changing display capture time.

sequence takes steps, an ordered array of acts, tree, find, read or wait reads, and screenshot steps that capture at that point. A fast browser interaction, after reading the control names, is:

{
  "action": "sequence",
  "steps": [
    { "action": "click", "target": { "role": "AXTextField", "name": "Search" } },
    { "action": "key", "key": "cmd+a" },
    { "action": "type", "text": "release notes" },
    { "action": "key", "key": "Enter", "label": "search" },
    { "action": "wait", "until": { "text": "Results" }, "durationMs": 5000 },
    { "action": "find", "target": { "text": "Results" } }
  ]
}

Use the roles and names your tree actually returned; the example's cmd+a is the macOS select-all chord (ctrl+a on other platforms). Each step goes through the same permission and safety checks as an individual call, including each read. A refusal, timeout or interruption stops the sequence and reports how many steps completed; completed acts are not undone. One screenshot follows the final step unless that step is a screenshot, and the after-state line of the last step closes the result. Set capture: false on the sequence to keep the interaction image-free. Set capture, region and settleMs on the sequence itself; steps do not take them. Nested sequences are refused.

find, read and wait with until read the tree and therefore ask like tree. A timed wait without until remains a never-ask read. Targeted mouse actions keep their existing act permission class. The access types and sovereign mode work identically for individual acts and sequences.

Typing pause and captcha hand-off

Before acting, Mercury waits for a 1400 ms quiet window without recent keyboard activity, for at most 3000 ms. Continued typing refuses the act (nothing done; try again in a moment), and a final check refuses if typing resumes while the target is being checked. This typing pause applies in every access type, including Sovereign mode, checking composer activity in the current process and, on macOS, HID keyboard activity reported by the desktop driver.

Mercury never solves a captcha. Named reCAPTCHA, hCaptcha, Turnstile and human-check challenges in the accessibility tree stop input and hand the step to you. When a challenge is visible only in screenshot pixels, the model is instructed to report captcha: true and the tool refuses that input as well (Mercury does not run OCR on screenshots, so a purely visual challenge is caught only when the model reports it). Complete the challenge yourself, then let Mercury read the screen again.

Access types

The Boot Menu's Computer use row has an Access type row beneath it with three values.

asks: The default with Sovereign mode off. The first act in each application asks, and a Yes covers that application for the session.
permissive: Takes the application in front at the turn's first Computer call as the turn's home application. Acts in it never ask; an act in any other application asks once. A home application leaves no grant behind.
full: Never asks.

Unset, the access type follows Sovereign mode: full with the Sovereign mode row on and asks with it off. Under Sovereign mode nothing asks, whatever the row saves, and the saved value applies when Sovereign mode is off. MERCURY_COMPUTER_ACCESS=asks, =permissive or =full sets it for one session; any other value produces a diagnostic note and the default applies.

Reads never ask, except tree, find, read and wait with until, which read the accessibility tree and therefore ask like an act. A timed wait without until remains a never-ask read. Under asks, the first act or tree read in each application asks by the application's name and identity, names the action, and asks how long to allow. The five answers, in this order, are:

Yes: Allows this act and every later act in this application for the rest of the session.
No, and tell Mercury what to do differently: Refuses the act and hands the composer back to you.
Yes, for 1 hour, every application: Allows every act in every application for one hour.
Yes, for 24 hours, every application: Allows every act in every application for twenty-four hours.
Enable sovereign mode to avoid further permissions by default: Turns Sovereign mode on for this session and saves the Boot Menu's Sovereign mode row for later sessions.

Under Sovereign mode the card never appears. A timed grant lives with the session, is never written into settings, and leaves no per-application grant behind. An application that moved in front between the ask and the act is not driven; the model takes a new screenshot.

One read asks like an act: tree, the accessibility tree of the application in front. It answers an outline, [n] role "title" value, indented by kept depth. Empty non-interactive wrappers disappear and adjacent static text fragments join into one row. A static-text child repeating its parent's value is omitted; the text stays in the parent's row. Zero-size, outside-window and duplicate text rows return with all: true. points: true adds the centre as (x,y)px when it is inside the last screenshot. The default budget is 12 kept levels and 400 kept rows, up to depth 32 and 2000 rows. root starts depth at the chosen subtree; find keeps the outline indentation. A cut says to narrow with root or find, or raise the depth or count. Result text over 20,000 characters is saved with a preview even when an image accompanies it. It is read on macOS through the Accessibility API, under the same Accessibility grant as the keyboard; it sends no input and reads no clipboard. Because it reads what an application shows (its controls and their text), the first tree in an application asks by the application's name like a first act, and the grant it leaves is the same one: later trees and acts in that application ride it. On Windows and Linux the action says plainly that this release reads the tree on macOS only.

Stop computer use

Give keyboard focus to the terminal running Mercury and press Esc or Ctrl+C. Mercury ends the act in flight, releases every key and button the model was holding, and ends the turn.

The stop key reaches Mercury only while its terminal has keyboard focus. While a turn drives, the footer shows a hands-off line naming the application Mercury is driving and the key that stops it. Your own mouse and keyboard keep working while a turn drives.

Disable computer use for new sessions

Turn the Boot Menu's Computer use row off and start a new session. The new session's catalogue omits the Computer tool and touches no desktop driver.

MERCURY_COMPUTER_USE=0 in the environment before the session starts does the same. A running session keeps the tool list it started with until the next compaction or /clear. On a machine without the desktop driver the tool is absent as well.

Application rules

Computer(app:<identity>): A rule in guardrails.allow grants an application beyond the session. You write the rule; the card writes none.
guardrails.deny: Refuses the matching application before any ask, in every mode.
guardrails.ask: Asks for the application in every mode that asks and does not apply under Sovereign mode.
Application identity: A bundle identifier on macOS such as com.apple.Safari, an executable name on Windows such as chrome.exe, or a window class on Linux such as firefox. A rule written on one platform matches only that platform's identities.

The model never types into the terminal running Mercury: with that terminal in front, typed text and held keys are refused, a key chord is allowed only to switch applications, and a click must land outside the terminal's window. Another window of the same application can be driven when both window identities are known. On macOS, Apple Terminal and iTerm2 map the terminal UI's tty to its window through their scripting bridge; titles and window positions are not ownership evidence. A worker uses the tty supplied by the attached terminal UI (the daemon that spawned it has none). Each guard check resolves that tty afresh, so moving a tab between windows does not leave stale ownership behind. The lookup runs only when the terminal application is in front, in a separate helper with a 1000 ms deadline. MERCURY_DESKTOP_OWNER_TIMEOUT_MS accepts 1–5000 ms; an invalid value keeps the default. A failed or timed-out lookup keeps the application-wide guard and does not block the event loop.

Keyboard rules

Text is typed as given: a newline presses Enter and a tab presses Tab, so a newline at the end of typed text submits whatever field it lands in. Key chords are spelled cmd+shift+s: cmd, ctrl, alt and shift joined by + with a named key such as Enter, Escape, Tab, the arrows or F1 to F12, or one character. Keyboard shortcuts on a non-US layout land on the US key positions on macOS in this release; typed text is always layout-correct.

Screenshots and retained state

Every screenshot is written to <config-home>/desktop-shots/, the newest 200 kept, and sent to the model at the image budget of its route (downscaled only when its route requires it; the file on disk keeps its full size). The result reports the pixel-to-point scale and warns if the coordinate frame changed, in which case earlier pixel coordinates no longer apply. The conversation file carries only a line naming the file, and after a compaction or a resume the model takes a new screenshot before acting.

Platform requirements

macOS: The terminal application needs the Screen Recording and Accessibility grants under Privacy & Security in System Settings. The first capture or act registers the terminal in both lists; restart the terminal application after allowing.
Windows: An interactive desktop session (not a locked screen or background service). Windows drops input into a window that runs elevated when Mercury does not.
Linux: An X11 session with DISPLAY set. A Wayland session is refused by name.

The tool needs a model that receives images. A text-only route refuses and names itself, and a model that refuses images on its own route is refused for the rest of the session on that model, with the provider's words, until you switch models. Crewmates and MCP serve clients do not have the tool, and a bare mercury run cannot drive the desktop and receives a policy denial directing you to open an interactive session. mercury health and /health have a Computer use row naming the driver pack and where it was found, including through MERCURY_DESKTOP_PACK_DIR, the grants, the access type and the session driving the desktop. A missing grant is reported as information and the check passes.

Permission modes · Workspace trust and permissions · Health check · Sessions