mirror of
https://github.com/stablyai/orca.git
synced 2026-10-27 08:02:51 +00:00
* Revert "Enable accessibility tree (`ax`) command on iOS emulator sessions (#10007)"
This reverts commit 43ae014a64.
* fix(emulator): expose iOS accessibility tree
* fix(emulator): support device-only iOS AX
* fix(emulator): normalize iOS ax to 0..1 and heal missing axUrl
serve-sim's helper /ax reports element frames in absolute pixels, but
tap/gesture take normalized 0..1 coords. Normalize the raw AX node tree
into a compact nested shape whose frames are 0..1 over the device screen
(first root's frame), mirroring serve-sim's own normalizeAxTree, so agents
can feed ax output straight back into input commands.
Also heal sessions that were registered without an axUrl: #9924 only
derived /ax at parse time, so already-active sessions had no endpoint.
The bridge now derives it from the session's mjpeg stream URL, guarded to
the /stream.mjpeg suffix so a non-mjpeg URL never fabricates a bogus /ax.
* docs(emulator): mark ax working on iOS with correct raw-AX-tree shape
Both skill guides and the CLI summary described iOS ax as unsupported (or,
via the reverted #10007, as a normalized "screen + elements" shape that
never matched the endpoint). ax works on both backends: Android via
uiautomator, iOS via the serve-sim helper. Document the real iOS output —
a raw AX node tree (labels, roles, nested children) with frames normalized
to 0..1 — and regenerate the bundled skill guides.
* chore(skills): regenerate skill bundle manifests
CI verify failed because generated skill artifacts were stale after version/skill revision bumps.
* fix(emulator): read ax from explicit device without active session
Fall back to udid-keyed session lookup when a worktree has no active emulator,
allowing `--device` targeting to work the same way for ax as it does for tap/type.
Also clarify in docs that AX frames are normalized 0..1 with top-left origin,
and show how to tap an element at its frame center (x+width/2, y+height/2).
* fix(emulator): cap iOS AX tree at 500 nodes
Unbounded accessibility trees can flood agent output. Enforce a 500-node limit (matching serve-sim's snapshot cap) and mark truncated parents so consumers know the tree was cut.
---------
Co-authored-by: 5Hyeons <ohs2251@naver.com>
This commit is contained in:
File diff suppressed because one or more lines are too long
@@ -108,7 +108,7 @@ export const EMULATOR_COMMAND_SPECS: CommandSpec[] = [
|
||||
},
|
||||
{
|
||||
path: ['emulator', 'ax'],
|
||||
summary: 'Dump the device accessibility tree (uiautomator on Android, serve-sim AX on iOS)',
|
||||
summary: 'Dump the accessibility tree (Android uiautomator; iOS serve-sim AX, frames 0..1)',
|
||||
usage: 'orca emulator ax [--device <id>] [--worktree <selector>] [--json]',
|
||||
allowedFlags: [...GLOBAL_FLAGS, 'device', 'emulator', 'worktree']
|
||||
},
|
||||
|
||||
Reference in New Issue
Block a user