Commit Graph
7 Commits
Author SHA1 Message Date
Jonatas SoaresandNeil 8221969006 fix(computer-use): read SwiftUI labels and name custom actions on macOS (#42829)
SwiftUI controls advertise only AXAttributedDescription but answer
AXDescription, so treat AXDescription as readable in that case and give
Calculator keys and System Settings sidebar buttons their labels.

Render custom actions ("Name:X\nTarget:..\nSelector:..") as just their
name, and match perform-secondary-action on the exact pretty name first so
a custom "Delete" is not shadowed by AXDelete.

Refs #42828

Co-authored-by: Neil <neil@stably.ai>
2026-11-15 15:33:38 -08:00
Kelvin Amoaba 030c88deec feat(computer): return tree diffs after actions and add get-app-state --query (#36023)
* feat(computer): send tree diffs after actions and add get-app-state --query

Action results now carry only the accessibility-tree lines that changed since
the caller's last view of the same window (~ changed, + added, - removed as
index ranges, or an explicit no-change line), plus the focused element. The
runtime sidecar keeps the last tree per session/worktree and app, so all three
providers get it. The CLI opts in with treeDiff; older runtimes strip the
param and keep returning the full tree, and --full-tree opts out.

The macOS helper now keeps an element's index across snapshots of a window:
AX identity first, then the next unclaimed index with the same signature for
views that rebuild their AX objects, and never reuses a removed index.

get-app-state --query prints matching elements with their ancestors and
--max-elements caps the output, both in the CLI over the rendered tree.

Custom action names with raw newlines no longer split an element over lines.

* fix(computer): preserve parent changes in snapshot diffs

* fix(computer): refuse stale index reuse for lookalike controls

* fix(computer): print full trees in action replies

* fix(computer): keep controls located in snapshot replies
2026-10-29 23:05:41 +00:00
Neil 832caea15d Avoid irrelevant scroll-action searches in Mac snapshots (#24731)
Check the current pure action name before searching for vertical scroll actions in the same immutable array.
2026-10-03 00:14:03 -07:00
Neil 8713f6ee72 perf(native): avoid sanitizing snapshot values twice (#20271) 2026-09-12 18:06:33 -07:00
Jinwoo HongandOrca f3e379d6dc fix: cache macos computer-use snapshot ax reads (#6130)
Co-authored-by: Orca <help@stably.ai>
2026-06-22 20:57:13 -07:00
Jinwoo HongandOrca 76cb846d68 Harden computer use runtime and CLI (#4705)
Co-authored-by: Orca <help@stably.ai>
2026-06-07 17:49:01 -07:00
Jinwoo HongandOrca 0f54103dda Add native computer-use automation (#1683)
Co-authored-by: Orca <help@stably.ai>
2026-05-11 14:20:08 -07:00