Jinjingand5Hyeons
4a9affd6e5
* Revert "Enable accessibility tree (`ax`) command on iOS emulator sessions (#10007)"
This reverts commit 43ae014a64.
* fix(emulator): expose iOS accessibility tree
* fix(emulator): support device-only iOS AX
* fix(emulator): normalize iOS ax to 0..1 and heal missing axUrl
serve-sim's helper /ax reports element frames in absolute pixels, but
tap/gesture take normalized 0..1 coords. Normalize the raw AX node tree
into a compact nested shape whose frames are 0..1 over the device screen
(first root's frame), mirroring serve-sim's own normalizeAxTree, so agents
can feed ax output straight back into input commands.
Also heal sessions that were registered without an axUrl: #9924 only
derived /ax at parse time, so already-active sessions had no endpoint.
The bridge now derives it from the session's mjpeg stream URL, guarded to
the /stream.mjpeg suffix so a non-mjpeg URL never fabricates a bogus /ax.
* docs(emulator): mark ax working on iOS with correct raw-AX-tree shape
Both skill guides and the CLI summary described iOS ax as unsupported (or,
via the reverted #10007, as a normalized "screen + elements" shape that
never matched the endpoint). ax works on both backends: Android via
uiautomator, iOS via the serve-sim helper. Document the real iOS output —
a raw AX node tree (labels, roles, nested children) with frames normalized
to 0..1 — and regenerate the bundled skill guides.
* chore(skills): regenerate skill bundle manifests
CI verify failed because generated skill artifacts were stale after version/skill revision bumps.
* fix(emulator): read ax from explicit device without active session
Fall back to udid-keyed session lookup when a worktree has no active emulator,
allowing `--device` targeting to work the same way for ax as it does for tap/type.
Also clarify in docs that AX frames are normalized 0..1 with top-left origin,
and show how to tap an element at its frame center (x+width/2, y+height/2).
* fix(emulator): cap iOS AX tree at 500 nodes
Unbounded accessibility trees can flood agent output. Enforce a 500-node limit (matching serve-sim's snapshot cap) and mark truncated parents so consumers know the tree was cut.
---------
Co-authored-by: 5Hyeons <ohs2251@naver.com>
8.8 KiB
name, description, license
| name | description | license |
|---|---|---|
| orca-emulator-android | Control an Android emulator / device from inside Orca using the `orca` CLI. Use for listing/booting AVDs, taps, swipes, typing, hardware buttons (incl. Back and Recents), rotation, app install/launch, runtime permissions, the accessibility tree, and logcat — driving a real adb-connected device or emulator. Cross-platform (Windows, Linux, macOS). Complements the orca-emulator (iOS) and orca-cli skills. | Apache-2.0 |
Orca Emulator — Android (adb / emulator powered)
Drive an Android emulator or adb-connected device from within Orca using
ORCA emulator ... commands. The Android backend shells out to the Android SDK
(adb, emulator, avdmanager) that Android Studio installs, so it works on
Windows, Linux, and macOS — unlike the iOS backend (orca-emulator), which is
macOS-only. Device control uses adb shell input, so it works without any extra
streaming server.
Status: device discovery + lifecycle + full input/capability control are live. The embedded 60fps visual pane (scrcpy/H.264) is in development — for now, watch the device in Android Studio's emulator window while you drive it from the CLI.
CLI executable
Choose the Orca executable once: use the ORCA_CLI_COMMAND environment value when set;
otherwise use orca-dev in a dev session exposing ORCA_DEV_REPO_ROOT, orca-ide on
Linux outside an Orca-managed terminal, and orca everywhere else. Never try bare
orca first on unmanaged Linux because it normally resolves to the GNOME screen reader.
In every command example — fenced blocks, tables, and prose — ORCA is a documentation
placeholder. Replace it with the chosen executable before running the command; do not
create a shell variable or run ORCA literally. The command examples are intentionally
shell-neutral for POSIX shells, PowerShell, and cmd.exe.
When to use
- List, boot, and target Android emulators/AVDs and physical devices.
- Tap, swipe, type, press hardware buttons (home/back/recents/power/volume), rotate a running Android device.
- Install an APK, launch an app, grant/revoke runtime permissions.
- Read the accessibility tree (
uiautomator) or capture logcat. - Run an arbitrary
adb shellcommand viaexec.
When NOT to use
- iOS simulators → use the
orca-emulatorskill (macOS only). - Building the app → use Gradle /
./gradlew assembleDebug, theninstall. - Camera/sensor injection → not supported yet (Android virtual-scene is out of scope for now).
- Remote/SSH device control → out of scope; the SDK + device are local to the host.
Prerequisites (surfaced by Orca)
- Android Studio / Android SDK installed, with
ANDROID_HOME(orANDROID_SDK_ROOT) set. Orca also checks the per-OS default location (%LOCALAPPDATA%\Android\Sdk,~/Library/Android/sdk,~/Android/Sdk). adb+emulatoron the SDK path; at least one AVD (create in Android Studio ▸ Device Manager) or a connected device with USB debugging.- A device that is booted and
adb-visible for input/capability commands (an AVD that is still shutdown can be listed but must be booted first).
Orca returns a clear message when the SDK is missing
(Android SDK not found. Install Android Studio and set ANDROID_HOME.).
Mental model
┌────────────────────────┐
│ orca CLI (agents) │ e.g. ORCA emulator tap 0.5 0.7 --device emulator-5554
└───────────┬────────────┘
│ RPC
▼
┌────────────────────────┐ resolves backend by device
│ EmulatorBridge (router)│ ─────────────────────────────► AndroidEmulatorBackend
└────────────────────────┘ │ adb / emulator / avdmanager
▼
Android emulator / device
Orca owns backend routing and the per-worktree active-device registry. The
Android backend converts Orca's normalized 0–1 coordinates to device pixels and
issues adb shell input events; AVD names resolve to running adb serials.
Common operations
Use --json for agent-friendly output. Coordinates are normalized 0..1
(top-left origin) — never pixels; Orca converts using the live screen size.
| Goal | Command | Notes |
|---|---|---|
| List devices + AVDs | ORCA emulator devices --json |
Cross-platform; shows iOS + Android with a platform column, booted vs shutdown. |
| Single tap | ORCA emulator tap <x> <y> --device <serial> |
Normalized 0..1. Preferred for single taps. |
| Swipe / gesture | ORCA emulator gesture '<json>' --device <serial> |
adb approximates the path by its endpoints (start→end). |
| Type text | ORCA emulator type "user@example.com" --device <serial> |
US ASCII; spaces handled. No newlines. |
| Hardware button | ORCA emulator button back --device <serial> |
home, back, recents, power, volume_up, volume_down. |
| Rotate | ORCA emulator rotate landscape_left --device <serial> |
Sets user_rotation (disables auto-rotate). |
| Install an APK | ORCA emulator install ./app-debug.apk --reinstall --device <serial> |
--reinstall passes -r. |
| Launch an app | ORCA emulator launch com.acme.app --activity .MainActivity --device <serial> |
Omit --activity to launch the default LAUNCHER activity. |
| Grant a permission | ORCA emulator permissions grant com.acme.app android.permission.CAMERA --device <serial> |
grant / revoke / reset. |
| Accessibility tree | ORCA emulator ax --device <serial> --json |
uiautomator dump parsed to a node tree. |
| Logcat (one-shot) | ORCA emulator logcat --lines 200 --device <serial> |
Dumps recent lines; parsed to entries. |
| Raw adb shell | ORCA emulator exec --command "getprop ro.build.version.sdk" --device <serial> |
Runs adb -s <serial> shell <command>. |
Critical gotchas (teach agents)
- All coordinates are normalized 0..1 (top-left origin), never pixels — Orca scales to the device's live resolution.
- Target a running device by its adb serial (e.g.
emulator-5554) shown inORCA emulator devices. An AVD name resolves only once that AVD is booted. - The device must be booted and adb-visible before input/capability commands;
a shutdown AVD is listed with
state: shutdownand must be started first (Android Studio, oremulator @<avd>). typeusesadb shell input text— US ASCII, spaces are handled, newlines are not. For unicode-heavy input, use the app UI directly.gestureis a straight swipe between the first and last point (adb limitation); fine for scroll/swipe, not for true multi-touch paths.- Capability verbs
install/launch/permissions/logcatare Android-only and fail against an iOS device withemulator_unsupported.axworks on both, with backend-specific output (Android:uiautomatornode tree; iOS: serve-sim raw AX node tree with frames normalized to 0..1). - No camera/sensor injection yet.
Targeting devices & worktrees
- Explicit device:
--device <serial>(recommended for Android today) or an AVD name once booted. ORCA emulator devicesis global (lists every backend's devices); other verbs target the resolved device's backend automatically.--worktree <selector>scopes to a worktree's active device once the attach/active flow lands for Android.
Examples (agent-friendly)
ORCA emulator devices --json
ORCA emulator tap 0.5 0.85 --device emulator-5554 --json
ORCA emulator type "hello world" --device emulator-5554 --json
ORCA emulator button recents --device emulator-5554 --json
ORCA emulator install ./app-debug.apk --reinstall --device emulator-5554 --json
ORCA emulator launch com.acme.app --device emulator-5554 --json
ORCA emulator permissions grant com.acme.app android.permission.CAMERA --device emulator-5554 --json
ORCA emulator ax --device emulator-5554 --json
ORCA emulator logcat --lines 100 --device emulator-5554 --json
Next action
Run ORCA emulator devices --json to find a booted device, then drive it with
--device <serial> while watching the emulator window.
See also: orca-emulator (iOS, macOS-only), orca-cli (terminals, worktrees,
built-in browser), computer-use (desktop UI outside the emulator).