* feat(computer): send tree diffs after actions and add get-app-state --query
Action results now carry only the accessibility-tree lines that changed since
the caller's last view of the same window (~ changed, + added, - removed as
index ranges, or an explicit no-change line), plus the focused element. The
runtime sidecar keeps the last tree per session/worktree and app, so all three
providers get it. The CLI opts in with treeDiff; older runtimes strip the
param and keep returning the full tree, and --full-tree opts out.
The macOS helper now keeps an element's index across snapshots of a window:
AX identity first, then the next unclaimed index with the same signature for
views that rebuild their AX objects, and never reuses a removed index.
get-app-state --query prints matching elements with their ancestors and
--max-elements caps the output, both in the CLI over the rendered tree.
Custom action names with raw newlines no longer split an element over lines.
* fix(computer): preserve parent changes in snapshot diffs
* fix(computer): refuse stale index reuse for lookalike controls
* fix(computer): print full trees in action replies
* fix(computer): keep controls located in snapshot replies
* fix(computer): fail set-value when its read-back shows a different value
A value_mismatch came back as ok with exit 0, so callers that check only
the exit status took an ignored or misplaced write as success. The runtime
now turns it into a verification_failed error at the sidecar boundary,
which covers every provider and every caller of the RPC.
Part of #33511
* fix(computer): pair macOS AX windows with their CGWindowID
AppKit and WebKit windows don't answer AXWindowNumber, so matching a
capture to its accessibility window fell back to frames. Two windows with
the same frame then paired the screenshot of one with the tree and title
of whichever AX listed first, under the first one's id. Read the id with
_AXUIElementGetWindow, as yabai and Hammerspoon do, and keep AXWindowNumber
as the fallback.
Fixes#27252
* fix(computer): fail fast when a macOS app stops answering accessibility
Every request runs behind one helper lock, and a hung app cost a timeout
per AX read, so one get-app-state on it held every agent for up to the 60 s
RPC timeout. Pin the AX messaging timeout at 1.5 s, refuse an app that
doesn't answer a root probe, and stop a tree walk at the first read that
waits out the timeout, each with an action_timeout error.
A press or write that the app received but didn't answer now fails the
same way, saying it may have taken effect. Before, the click fell through
to AXConfirm and then a synthetic click, so a button whose handler blocked
ran twice, and type-text typed the text again as keys.
* fix(computer): report macOS accessibility actions that changed nothing
AXPress and other accessibility actions came back as plain unasserted
success even when the app ignored them, as Chromium does for a role=button
whose handler sits on a child. After such an action the helper now compares
the target's tree, frame and on-screen windows with the snapshot it took
just before. If they still match after re-reading for about a second, the
result says unverified/suspected_noop; if they change late, the result
carries the settled state instead.
The new reason is additive; older CLIs print it like any other reason.
Refs #31758
* fix(computer): propagate late macOS observation failures
* fix(computer): select existing text before replacement
* fix(computer): share native macOS window identity
* feat(computer): drive macOS apps in the background by default
Clicks no longer activate and raise the target first. Pointer and key
input go to the target window's pid through a new BackgroundInputTransport:
mouse and scroll events carry the window id in field 51 and a window-local
location, keys run inside a focus-without-raise key-window scope, and
chords carry no Unicode string so menu key equivalents fire. Scroll and
drag now land (#27306); typed text no longer doubles in Electron (#17942).
The HID tap and focus fence stay for --restore-window only. Results
report action.delivery, the handshake reports supports.input.background,
and a missing private symbol or a popup click fails with
background_unavailable instead of falling back to the foreground.
* test(computer): read typed key strings back instead of diffing live-state events
* fix(computer): keep Unicode characters in one key event
* fix(computer): preserve active macOS key windows
Refuse key-window changes in the active app and surface failed
focus records, including cleanup after partially delivered input.
* fix(computer): guard macOS click routes before delivery
* fix(computer): let paste events read the clipboard
* fix(computer): classify clicks in the requested macOS window
* fix(computer): fence macOS scroll and drag recipients
pnpm dev launches the app as com.stablyai.orca.dev, but the macOS helper's
isTrustedOrcaApplication only trusted com.stablyai.orca, com.github.Electron
and ids starting with com.stablyai.orca.dev. (with the trailing dot). Every
computer-use request from a dev build failed with "computer-use agent peer is
not authorized".
Move the bundle id rule into TrustedOrcaApplication in
OrcaComputerUseMacOSCore so it is tested, and trust com.stablyai.orca.dev
exactly as well. Look-alikes such as com.stablyai.orca.devtools stay
untrusted.
Fixes#26498
Co-authored-by: petersindex <287557875+petersindex@users.noreply.github.com>
Co-authored-by: Neil <neil@stably.ai>
The macOS helper serves each sidecar connection on a dedicated thread
that lives as long as the connection, and the sidecar keeps one
connection open while the helper runs. Encoding a screenshot with
NSBitmapImageRep leaves autoreleased objects that keep the captured
image alive, and that thread never drained its pool, so every
screenshot kept a window-sized buffer in WindowServer until the helper
restarted.
Drain an autorelease pool around each request.
* feat(computer-use): support macOS middle click and gate the AX click path
`--mouse-button middle` already validated end-to-end through the CLI, the
zod schema, and the provider validator, and both the Windows and Linux
providers honored it. Only the macOS provider rejected it outright with
"middle-click is not yet supported", so the flag was a dead end on the one
platform that has no fallback.
Two changes:
- Add `.middle` to the macOS button mapping. macOS has no dedicated middle
event family, so it rides `otherMouseDown`/`otherMouseUp` with the button
number carried by `mouseButton: .center`; that constructor argument is
honored for exactly the `otherMouse*` types, so no extra field write is
needed.
- Validate the requested button before the accessibility fast path, and skip
that path for buttons it cannot express. Previously the raw string was read
unvalidated, and `performClickAction` only special-cased `right`, so
`click --mouse-button middle --element-index N` (no modifiers, count 1) fell
through to `AXPress` — a left click — and reported success with
`path: "accessibility"`. Any unrecognized button string did the same. This
matches guards the Windows and Linux providers already had.
The button enum moves into `OrcaComputerUseMacOSCore` so it is unit-testable;
`main.swift` keeps only the CoreGraphics mapping.
Also documents `--mouse-button` in the computer-use skill guide, which never
mentioned the flag, so agents on Windows and Linux had no way to discover it.
* test(computer-use): cover macOS middle click in the real-desktop e2e suite
* test(computer-use): prove macOS middle-click delivery
Mouse events posted with CGEventPostToPid reach the target app with no
window association, so AppKit never routes the press to a view: hover
states fire but the control is never activated, and the mouseUp is
dropped outright when posted back-to-back. Post click events to the HID
event tap instead (as keyboard synthesis already does), pace them, and
stamp mouseEventClickState so multi-clicks register.
Synthetic clicks now also report verification unverified/synthetic_input
from the helper itself, matching the other synthetic actions.
* perf(computer): add mac helper owner-loss benchmark
Measure the release helper's resident memory before and after its owner-session deadline. Record exact revisions, per-trial RSS, retained state, and clean-exit latency so lifecycle reclamation is reproducible.
* fix(computer): reap mac helper after client loss
Bind the detached macOS helper lifetime to authenticated socket ownership. Reap the helper after its final authenticated client disconnects, and add a startup deadline for sessions that never authenticate.
* test(computer): harden owner benchmark cleanup
* test(computer): make owner benchmark cleanup failure-safe
* test(computer): close remaining owner cleanup races
## Summary
- stop the macOS helper from unlinking caller-supplied socket and token paths
- keep parent-owned token cleanup tied to the helper startup that created it
- fail closed for non-socket socket-path collisions and cover helper cleanup races with regression tests
## Verification
- swift test --package-path native/computer-use-macos
- pnpm exec vitest run --config config/vitest.config.ts src/main/computer/macos-native-provider-client.test.ts
- pnpm run typecheck:node
- pnpm lint
- pnpm run build:computer-macos
- Electron/helper startup smoke validation
* fix(computer-use): stop malformed numeric args from crashing the agent
requiredNumber only checked isFinite, and several request handlers cast the
resulting Double straight to Int/UInt32 (elementIndex, clickCount, pages,
from/toElementIndex, windowId, windowIndex). Int(Double) traps when the value
is outside the integer's representable range, so a single malformed request
such as `{"elementIndex": 1e300}` crashed the entire agent process, killing
all in-flight automation.
Add a bounded conversion helper in the Core library and route every untrusted
Double->integer cast through it. boundedInteger truncates toward zero like
Int(Double) but returns nil (via init(exactly:)) instead of trapping when the
value is non-finite or out of range; the request handlers then surface a clean
invalid_argument error (or resolve to nil for the optional window lookups).
The helper lives in the Core library because the test target cannot import the
executable target where the handlers live. A negative-control run confirms the
out-of-range test traps with the previous Int(Double) cast and passes with the
fix.
* review: close remaining numeric crash paths
- parse stale-element validation indexes through bounded conversion
- reject malformed window selectors instead of dropping the target window
- bound synthetic scroll wheel deltas and cover Int32 conversion
---------
Co-authored-by: Jinjing <6427696+AmethystLiang@users.noreply.github.com>
Kelvin Amoaba
F8 Team
Jinwoo Hong
Brennan Benson
Neil
