SwiftUI controls advertise only AXAttributedDescription but answer
AXDescription, so treat AXDescription as readable in that case and give
Calculator keys and System Settings sidebar buttons their labels.
Render custom actions ("Name:X\nTarget:..\nSelector:..") as just their
name, and match perform-secondary-action on the exact pretty name first so
a custom "Delete" is not shadowed by AXDelete.
Refs #42828
Co-authored-by: Neil <neil@stably.ai>
* fix(computer): click macOS windows that refuse focus with --restore-window
A raised window that never becomes the focused window (Archicad's CEF start window) failed every --restore-window click with window_not_focused, because the click fence required the target to be focused as well as topmost at the point. A real click reaches the topmost window whether it is focused or not, so when the target stays unfocused the click now goes ahead while the target is topmost at the point, and is refused before anything is sent when another window covers it. Afterwards the app the user had in front comes back if the raise or the click left the target app in front without the target window.
* fix(computer): restore the front app only after a click on a window that refused focus
* fix(computer): keep the target app in front when an unfocused-window click opened something
* fix(computer): count small popups and re-read the front app last before restoring it
* feat(computer): add opt-in flags for smaller computer-use screenshots
Every observe and action screenshot came back as a full-resolution PNG, so a Retina window cost about four times the pixels it needed. --screenshot-scale caps pixels per window point, --screenshot-max-edge caps the longer side, and --screenshot-format jpeg re-encodes it, on macOS, Windows and Linux. Defaults are unchanged.
* fix(computer): treat requested screenshot sizes as ceilings and skip repeat encodes
* fix(computer-use): put the macOS pointer back after --restore-window clicks, scrolls and drags
HID gestures left the user's pointer at the target. Save it first, and after a
200 ms settle warp it back unless it no longer sits where the last posted event
left it, which means the user moved it. The HID idle counter can't tell: posting
to the HID tap resets it too.
* fix(computer-use): keep the pointer where the user moved it during a macOS HID gesture
* fix(computer-use): save the macOS pointer right before the first HID event
* fix(computer): report a locked screen instead of missing Accessibility
Behind the lock screen (display sleep, screen saver or a manual lock) macOS reports Accessibility as untrusted. Orca read that as a missing permission, and its recovery hint led agents to open System Settings. Check the session lock first and return screen_locked; permission setup refuses while locked.
* fix(computer): recheck the lock under the provider lock and at setup launch
* fix(computer): read the lock fresh for setup and keep locked status reads out of the cache
* refactor(computer): drop the extra lock read at setup; the status and setup helper already check it
* fix(computer-use): observe minimized, hidden and other-Space windows on macOS without a restore
macOS only enumerated on-screen windows and dropped minimized AX windows,
so an app whose window was minimized, hidden or on another Space (any
full-screen app) could not be observed, and agents fell back to
--restore-window, which activated every window of the app and switched
the user's Space.
Enumerate every window, keep the off-screen ones accessibility can still
read (other-Space windows by remote token, closed windows dropped by
their Space membership), and report their real minimized and off-screen
state. Background pointer and key input on such a window refuses before
sending anything. --restore-window now raises only the target window,
once per request, and delivery is measured from whether the target
became frontmost.
* fix(computer-use): pin the restored window and prefer visible windows by default
* fix(computer-use): keep window-index cache keys and stale-element checks across a restore
* fix(computer-use): restore only this app's resolved window and refuse a closed target before raising
* docs(computer-use): say macOS reads minimized, hidden and other-Space windows
* fix(computer-use): refuse macOS background actions that open a menu or dialog over the user
* fix(computer-use): let web comboboxes click in the background and stop probing a hung app
* fix(computer-use): refuse background drags that start on a menu owner
* docs(computer-use): note background drags on menu owners are refused
* fix(computer-use): classify the element under a drag's start and check AXConfirm/AXOpen secondary actions
* fix(computer): verify macOS type-text and paste-text sent as keys by reading the field back
When text can't be written through AXValue (Chromium/Electron fields, fields
that refuse the write), type-text and paste-text send keys or Cmd+V and always
reported unverified. Now the helper predicts the focused field's value from its
value and selection before sending, then reads it back until it settles:
verified on a match, unverified (text_mismatch) otherwise. Secure fields and
fields without a readable value and selection stay unverified as before, and
the result never repeats the typed text or the field's contents.
* fix(computer): keep the caret put after a refused macOS text write and skip unprovable readbacks
A refused AX value write no longer moves the caret before falling back to keys, so the keys and the prediction both use the user's selection. Text with control characters, a value the field already holds, and a field still filling at the 2 s cap stay unverified rather than verified or text_mismatch. The quiet window starts after the first read answers.
* fix(computer): bound macOS text readback size and say a mismatch can mean keys went elsewhere
* fix(computer): treat a macOS text readback past the cap as undetermined
* fix(computer): check macOS click targets are enabled, in view and on top before sending input
* fix(computer): scope macOS click cover checks to the target window
* fix(computer): scroll before trusting container hits and stop hit walks at the shared container
* refactor(computer): share the one-scroll rule in the macOS click check
* fix(computer): select list and table rows when a macOS click leaves them unselected
A plain click on an AXRow/AXCell answered success (AXPress) or unverified synthetic input while nothing got selected: AppKit rows have no AXPress, and background pointer input does not select them. After the click, read the selection back; if the row is still unselected, write it through the owning table/outline/list (AXSelectedCells/AXSelectedRows/AXSelectedChildren), falling back to the row's own AXSelected, and report verified only when the selection reads back on.
* fix(computer): find a cell's row through wrapper groups
* feat(computer): send tree diffs after actions and add get-app-state --query
Action results now carry only the accessibility-tree lines that changed since
the caller's last view of the same window (~ changed, + added, - removed as
index ranges, or an explicit no-change line), plus the focused element. The
runtime sidecar keeps the last tree per session/worktree and app, so all three
providers get it. The CLI opts in with treeDiff; older runtimes strip the
param and keep returning the full tree, and --full-tree opts out.
The macOS helper now keeps an element's index across snapshots of a window:
AX identity first, then the next unclaimed index with the same signature for
views that rebuild their AX objects, and never reuses a removed index.
get-app-state --query prints matching elements with their ancestors and
--max-elements caps the output, both in the CLI over the rendered tree.
Custom action names with raw newlines no longer split an element over lines.
* fix(computer): preserve parent changes in snapshot diffs
* fix(computer): refuse stale index reuse for lookalike controls
* fix(computer): print full trees in action replies
* fix(computer): keep controls located in snapshot replies
* feat(computer): drive macOS apps in the background by default
Clicks no longer activate and raise the target first. Pointer and key
input go to the target window's pid through a new BackgroundInputTransport:
mouse and scroll events carry the window id in field 51 and a window-local
location, keys run inside a focus-without-raise key-window scope, and
chords carry no Unicode string so menu key equivalents fire. Scroll and
drag now land (#27306); typed text no longer doubles in Electron (#17942).
The HID tap and focus fence stay for --restore-window only. Results
report action.delivery, the handshake reports supports.input.background,
and a missing private symbol or a popup click fails with
background_unavailable instead of falling back to the foreground.
* test(computer): read typed key strings back instead of diffing live-state events
* fix(computer): keep Unicode characters in one key event
* fix(computer): preserve active macOS key windows
Refuse key-window changes in the active app and surface failed
focus records, including cleanup after partially delivered input.
* fix(computer): guard macOS click routes before delivery
* fix(computer): let paste events read the clipboard
* fix(computer): classify clicks in the requested macOS window
* fix(computer): fence macOS scroll and drag recipients
pnpm dev launches the app as com.stablyai.orca.dev, but the macOS helper's
isTrustedOrcaApplication only trusted com.stablyai.orca, com.github.Electron
and ids starting with com.stablyai.orca.dev. (with the trailing dot). Every
computer-use request from a dev build failed with "computer-use agent peer is
not authorized".
Move the bundle id rule into TrustedOrcaApplication in
OrcaComputerUseMacOSCore so it is tested, and trust com.stablyai.orca.dev
exactly as well. Look-alikes such as com.stablyai.orca.devtools stay
untrusted.
Fixes#26498
Co-authored-by: petersindex <287557875+petersindex@users.noreply.github.com>
Co-authored-by: Neil <neil@stably.ai>
* test: remove redundant cases and repair behavioral assertions
* docs: reconcile audit records with current main
* fix(deps): apply security updates from #32101
Apply the six-file DOMPurify 3.4.17 and Next.js 16.4.1 patch from PR #32101 to resolve the dependency security gate for this test audit. Original patch by Brennan Benson.
* test: satisfy changed-line quality gates
Add required braces and replace filter indexing with find. Rerun both CI quality gates and the 27 affected relay tests; refresh audit hashes and record the corrections.
---------
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
Co-authored-by: Brennan Benson <79079362+brennanb2025@users.noreply.github.com>
The macOS helper serves each sidecar connection on a dedicated thread
that lives as long as the connection, and the sidecar keeps one
connection open while the helper runs. Encoding a screenshot with
NSBitmapImageRep leaves autoreleased objects that keep the captured
image alive, and that thread never drained its pool, so every
screenshot kept a window-sized buffer in WindowServer until the helper
restarted.
Drain an autorelease pool around each request.
* feat(computer-use): support macOS middle click and gate the AX click path
`--mouse-button middle` already validated end-to-end through the CLI, the
zod schema, and the provider validator, and both the Windows and Linux
providers honored it. Only the macOS provider rejected it outright with
"middle-click is not yet supported", so the flag was a dead end on the one
platform that has no fallback.
Two changes:
- Add `.middle` to the macOS button mapping. macOS has no dedicated middle
event family, so it rides `otherMouseDown`/`otherMouseUp` with the button
number carried by `mouseButton: .center`; that constructor argument is
honored for exactly the `otherMouse*` types, so no extra field write is
needed.
- Validate the requested button before the accessibility fast path, and skip
that path for buttons it cannot express. Previously the raw string was read
unvalidated, and `performClickAction` only special-cased `right`, so
`click --mouse-button middle --element-index N` (no modifiers, count 1) fell
through to `AXPress` — a left click — and reported success with
`path: "accessibility"`. Any unrecognized button string did the same. This
matches guards the Windows and Linux providers already had.
The button enum moves into `OrcaComputerUseMacOSCore` so it is unit-testable;
`main.swift` keeps only the CoreGraphics mapping.
Also documents `--mouse-button` in the computer-use skill guide, which never
mentioned the flag, so agents on Windows and Linux had no way to discover it.
* test(computer-use): cover macOS middle click in the real-desktop e2e suite
* test(computer-use): prove macOS middle-click delivery
Mouse events posted with CGEventPostToPid reach the target app with no
window association, so AppKit never routes the press to a view: hover
states fire but the control is never activated, and the mouseUp is
dropped outright when posted back-to-back. Post click events to the HID
event tap instead (as keyboard synthesis already does), pace them, and
stamp mouseEventClickState so multi-clicks register.
Synthetic clicks now also report verification unverified/synthetic_input
from the helper itself, matching the other synthetic actions.
* perf(computer): add mac helper owner-loss benchmark
Measure the release helper's resident memory before and after its owner-session deadline. Record exact revisions, per-trial RSS, retained state, and clean-exit latency so lifecycle reclamation is reproducible.
* fix(computer): reap mac helper after client loss
Bind the detached macOS helper lifetime to authenticated socket ownership. Reap the helper after its final authenticated client disconnects, and add a startup deadline for sessions that never authenticate.
* test(computer): harden owner benchmark cleanup
* test(computer): make owner benchmark cleanup failure-safe
* test(computer): close remaining owner cleanup races