orca/docs/android-emulation.md

Android Emulation

Problem

Orca ships a built-in mobile emulator surface (live pane + orca emulator CLI + agent skill), but it is iOS Simulator only and macOS only:

• src/main/emulator/emulator-availability.ts:32 hard-returns "unavailable" for any platform() !== 'darwin', so Windows and Linux users get nothing.
• The backend (src/main/emulator/emulator-bridge.ts) is wired directly to serve-sim (serve-sim-*.ts) and xcrun simctl (simctl-simulator-devices.ts), both Apple-only tooling.

Android emulators run on Windows, Linux, and macOS via the Android SDK that Android Studio installs. We want Android emulation as a first-class peer of the iOS feature: full AVD lifecycle management, a live ~60fps pane, the full tap/gesture/type/button/rotate control surface, accessibility tree, app install/launch, runtime permissions, logcat, plus a dedicated orca-emulator-android agent skill.

Current architecture (what we reuse vs. replace)

The existing stack already separates a backend from everything above it. The renderer pane, session registry, RPC/CLI shape, and tab system are effectively backend-agnostic and are reused unchanged:

• Frame transport is main-owned. src/main/ipc/emulator-frame-stream.ts:40 runs the MJPEG socket in the main process and forwards raw JPEG bytes to the renderer over emulator:frameStreamFrame. The renderer (src/renderer/src/components/emulator-pane/use-emulator-frame-stream.ts:74) just wraps each frame in a Blob/<img>. The renderer is a frame consumer, decoupled from the source.
• Per-worktree "active emulator" lives in src/main/emulator/emulator-session-registry.ts (like the active browser tab). Backend-agnostic.
• RPC is declared in src/main/runtime/rpc/methods/emulator.ts and implemented in src/main/runtime/orca-runtime-emulator.ts.
• CLI is src/cli/specs/emulator.ts + src/cli/handlers/emulator.ts.
• Pane is src/renderer/src/components/emulator-pane/** (~50 files).

What is iOS-bound and needs an Android sibling:

• Device management: xcrun simctl → adb / emulator / avdmanager.
• Streaming helper: serve-sim (MJPEG/H.264 over HTTP+WS) → scrcpy-server.jar (H.264 + control over adb-forwarded sockets).
• Input: serve-sim normalized-coord WS → adb-backed Android input commands.
• Availability gate: darwin-only → SDK-present on any OS.

Goals

• Android emulation on Windows, Linux, and macOS (macOS users choose iOS or Android in the same pane).
• Full AVD lifecycle: discover installed AVDs via the SDK, boot/shutdown them from Orca, and attach to already-running emulators + physical adb devices.
• Live ~60fps pane via scrcpy H.264 decoded in the renderer with WebCodecs.
• Control parity: tap, swipe/gesture, type, hardware buttons (Back, Home, Recents, Power, Volume), rotate.
• Extra capabilities: accessibility tree (uiautomator dump), app install/launch (adb install / am start), runtime permissions (pm grant/revoke/reset), logcat capture.
• A dedicated skills/orca-emulator-android/SKILL.md.

Non-goals (v1)

• Camera/sensor injection (the Android emulator's virtual-scene path is a much larger problem than serve-sim's iOS camera injection; defer).
• Remote/SSH device control (matches the current iOS limitation; emulator hardware is local).
• Wear OS / Android TV / Automotive form factors.
• Migrating the iOS backend to H.264 (the interface allows it later; not done now).

Design

Extract an EmulatorBackend interface; make the bridge a router

Today EmulatorBridge is the iOS implementation. Refactor it into a thin router over a backend interface so iOS and Android share the session registry, RPC/CLI shape, frame IPC, and tab system. This is the only change to existing iOS behavior, and it is a pure extraction (no semantic change).

New module src/main/emulator/backends/emulator-backend.ts:

export type EmulatorBackendKind = 'ios' | 'android'
export type EmulatorStreamCodec = 'mjpeg' | 'h264'

export type EmulatorDevice = {
  backend: EmulatorBackendKind
  id: string            // opaque: simulator UDID, adb serial, or AVD name
  name: string
  state: 'shutdown' | 'booting' | 'booted'
  kind?: string         // form factor / api level, display only
  isAvailable: boolean
}

export type EmulatorBackendCapabilities = {
  install: boolean
  launch: boolean
  permissions: boolean
  accessibilityTree: boolean
  logcat: boolean
}

export interface EmulatorBackend {
  readonly kind: EmulatorBackendKind
  readonly capabilities: EmulatorBackendCapabilities
  isSupportedOnHost(): boolean
  checkAvailability(): Promise<BackendAvailability>
  listDevices(): Promise<EmulatorDevice[]>
  bootDevice(id: string): Promise<EmulatorDevice>
  startSession(id: string): Promise<EmulatorSessionInfo> // includes streamCodec
  tap(id, x, y): Promise<void>
  gesture(id, points): Promise<void>
  type(id, text): Promise<void>
  button(id, name): Promise<void>
  rotate(id, orientation): Promise<void>
  exec(id, command): Promise<unknown>
  // capability-gated:
  installApp?(id, path): Promise<void>
  launchApp?(id, pkg, activity?): Promise<void>
  setPermission?(id, op): Promise<void>
  accessibilityTree?(id): Promise<unknown>
  logcat?(id, opts): Promise<...>
  stopSession(id): Promise<void>
  kill(id): Promise<void>
  shutdown(id): Promise<void>
}

• src/main/emulator/backends/ios-emulator-backend.ts — the existing serve-sim
  • simctl logic extracted from EmulatorBridge, implementing the interface with kind: 'ios', streamCodec: 'mjpeg', and capabilities mapped to what serve-sim already supports.
• src/main/emulator/backends/android-emulator-backend.ts — new, orchestrates the Android modules below with kind: 'android', streamCodec: 'h264'.
• EmulatorBridge keeps its public method names (so RPC/runtime callers don't churn) but becomes a router: it holds the available backends, resolves which backend owns a given device/session (via the session registry's recorded backend tag, or by probing listDevices() for an unknown id), and delegates. The session registry record gains a backend: EmulatorBackendKind field; EmulatorSessionInfo gains streamCodec. The existing deviceUdid field is retained as the opaque id to preserve wire-compat across the renderer and CLI.

New Android modules — src/main/emulator/android/

Each module is small and single-purpose with a co-located .test.ts, matching the existing serve-sim-* / simctl-* granularity (no file approaches the max-lines limit):

• android-sdk-discovery.ts — resolve the SDK root and the adb / emulator / avdmanager binaries from ANDROID_HOME, then ANDROID_SDK_ROOT, then per-OS defaults: %LOCALAPPDATA%\Android\Sdk (Windows), ~/Library/Android/sdk (macOS), ~/Android/Sdk (Linux). All paths via path.join.
• adb-devices.ts — adb devices -l, resolve serial, wait-for-device + getprop sys.boot_completed poll, and wm size for the device resolution.
• avd-manager.ts — emulator -list-avds, boot an AVD via a detached emulator @<name> spawn, shut down via adb -s <serial> emu kill.
• scrcpy-server-deploy.ts — push the version-pinned scrcpy-server.jar, start it via app_process, and set up the adb forward tunnel(s).
• scrcpy-stream-session.ts — owns the server process, adb tunnel, and video socket lifecycle.
• scrcpy-video-frame-parser.ts — read the video socket, parse scrcpy frame headers (PTS + length), and emit H.264 access units plus the codec config (SPS/PPS).
• scrcpy-control-protocol.ts — encode scrcpy control messages (touch down/move/up with pointer id + pressure, inject keycode, inject UTF-8 text, scroll, set screen power, rotate, clipboard) for a future low-latency input path.
• android-input-mapping.ts — convert normalized 0–1 ↔ device pixels using the live frame size; map button names → Android keycodes (BACK=4, HOME=3, APP_SWITCH=187, POWER=26, VOLUME_UP=24, VOLUME_DOWN=25).
• android-input-commands.ts — current adb-backed tap/type/button/rotate/gesture command construction.
• uiautomator-tree.ts — adb shell uiautomator dump → parsed XML tree.
• android-app-control.ts — adb install <apk>, am start package/activity.
• android-permissions.ts — pm grant / revoke / reset.
• android-logcat.ts — tail/filter adb logcat with bounded buffering.
• android-availability.ts — SDK present? AVDs + connected devices list, with clear, surfaced messages (mirroring the iOS availability message style).

Streaming & control data flow

Control currently uses adb shell input commands. Coordinates stay normalized 0–1 at every public boundary (CLI, RPC, renderer); the Android backend maps them to device pixels before issuing adb-backed tap/gesture/button commands. The scrcpy control protocol encoders are present for a future low-latency input path, but video streaming does not require that path.

Video path (H.264 → renderer WebCodecs):

1. android-emulator-backend.startSession() deploys + starts scrcpy-server, opens the video + control sockets, and returns EmulatorSessionInfo with streamCodec: 'h264'.
2. scrcpy-video-stream.ts reads access units and the SPS/PPS config and pushes them over a new IPC channel emulator:videoStream{Start,Config,Frame,Stop} (a sibling of the existing emulator:frameStream*), keyed by stream id.
3. New renderer hook src/renderer/src/components/emulator-pane/use-emulator-video-stream.ts feeds the access units to a WebCodecs VideoDecoder, drawing decoded VideoFrames to a <canvas>.
4. src/renderer/src/components/emulator-pane/emulator-screen-stream-content.tsx branches on session.streamCodec: mjpeg keeps today's <img> path untouched; h264 uses the canvas path. No iOS behavior changes.

Coordinate & input mapping

• Public API (CLI/RPC/pane gestures) stays normalized 0–1, top-left origin, as the iOS path already mandates.
• android-input-mapping.ts multiplies by the current device display size before adb input commands are built. Rotation changes the effective frame size; the mapper reads the current size each gesture rather than caching.
• Hardware buttons: adb keyevents inject keycodes. Android adds Back and Recents (no iOS equivalent); the button name → keycode map and the renderer's hardware-button row gain Android variants.

RPC + CLI surface

Extend src/main/runtime/rpc/methods/emulator.ts, src/main/runtime/orca-runtime-emulator.ts, src/cli/specs/emulator.ts, and src/cli/handlers/emulator.ts:

• orca emulator list gains a platform column and shows iOS + Android devices/AVDs together; device selection resolves the backend automatically (by recorded session tag, else by which backend's listDevices() owns the id).
• Existing verbs (attach, tap, gesture, type, button, rotate, exec, kill, shutdown) route unchanged to the resolved backend.
• New capability-gated verbs: install, launch, permissions, ax, logcat. On a backend lacking the capability they fail with a clear emulator_unsupported error rather than silently no-op.
• Existing --worktree / --device targeting is unchanged.

Renderer pane

• src/renderer/src/components/emulator-pane/emulator-phone-hardware-buttons.tsx → Android variant (Back, Home, Recents, Power, Volume) selected by backend kind.
• Android device bezel/frame + Android entries (and a "boot AVD" affordance) in the attach/list UI.
• MobileEmulatorAgentSetupGuide* → Android prerequisites step (install Android Studio / SDK, set ANDROID_HOME).
• Codec-aware stream content (the canvas path above).
• All UI follows docs/STYLEGUIDE.md: existing tokens from src/renderer/src/assets/main.css and shadcn primitives in src/renderer/src/components/ui/; no new color/size/shadow values.
• Shortcut labels and any new accelerators use the platform checks required by AGENTS.md (CmdOrCtrl, ⌘/Ctrl+).

Packaging & dependencies

• Bundle the single, version-pinned scrcpy-server.jar (~80 KB) as an app resource; wire it into config/electron-builder.config.cjs and config/packaged-runtime-node-modules.cjs. Pin the scrcpy version — the server protocol is coupled to the jar.
• Do not bundle adb / emulator / avdmanager (large; Android Studio installs them). Discover them at runtime and surface a clear setup message if the SDK is absent.
• No new runtime npm dependency is required for decode (WebCodecs is built into Electron's Chromium). The wasm-decoder fallback (see Risks) would add a dep only if the WebCodecs spike fails.

Skill

New skills/orca-emulator-android/SKILL.md, mirroring skills/orca-emulator/SKILL.md:

• Prerequisites: Android Studio / SDK installed, ANDROID_HOME (or ANDROID_SDK_ROOT) set, at least one AVD or a connected device.
• The orca emulator ... command table (shared CLI; Android examples).
• Gotchas: Orca handles pixel ↔ normalized conversion (agents always pass 0–1); adb device/serial targeting; no camera injection in v1; scrcpy version coupling.
• Cross-reference from the iOS skill's "When NOT to use" (which already anticipates an Android backend under the same namespace).
• Register it the same way orca-emulator is registered.

Availability & platform gating

src/main/emulator/emulator-availability.ts becomes an aggregator that asks each backend isSupportedOnHost() + checkAvailability():

• iOS backend: supported only on darwin (unchanged behavior/messages).
• Android backend: supported on any OS where the SDK is discoverable.
• The combined result drives the pane's availability UI; Windows/Linux report an available mobile backend for the first time.

Edge cases

• adb device in offline / unauthorized state → clear surfaced error, not a hang.
• AVD boot timeout (cold boot can take minutes) → bounded wait with a cancel/error path; pane shows "booting".
• SDK present but no AVDs and no devices → availability message points to "create an AVD in Android Studio".
• Device rotates while a gesture is mid-flight → mapper re-reads frame size per event; no cached dimensions.
• Multiple Android devices in one worktree → same "one active per worktree" model as iOS; explicit --device <serial> for the rest.
• WebCodecs decoder error / key-frame loss → request a new keyframe from scrcpy and surface a transient "reconnecting" state (parity with the MJPEG reconnect in mjpeg-frame-stream.ts).
• Windows path handling for the SDK and the pushed jar uses path.join only; never assume / or \.
• App quit / pane close cleans up scrcpy-server, the adb tunnel, and (for managed AVDs) the emulator, mirroring EmulatorBridge.onAppQuit() / destroyAllSessions().

Test plan

Unit tests (co-located, node Vitest, matching the module's existing test density):

• android-sdk-discovery — env precedence + per-OS default paths (mock env/fs; assert Windows/macOS/Linux branches).
• adb-devices — parse adb devices -l (booted, offline, unauthorized, physical), boot-complete polling.
• avd-manager — parse emulator -list-avds, boot command construction.
• scrcpy-control-channel — exact byte encoding of touch/key/text/scroll messages.
• android-input-mapping — normalized↔pixel round-trips, rotation, keycode map.
• uiautomator-tree — XML → tree parsing, including malformed input.
• android-availability + emulator-availability aggregation — iOS-only, Android-only, both, neither.
• backend router resolution in emulator-bridge — id → backend, unknown id, cross-backend isolation.

Integration tests mock adb / emulator and the scrcpy sockets the same way the iOS tests mock serve-sim (serve-sim-*.test.ts, emulator-bridge.test.ts).

Electron validation (manual, on a machine with the Android SDK):

• Boot an AVD from Orca; confirm the live pane streams and is responsive.
• tap / swipe / type / Back / Home / Recents / rotate.
• ax, install + launch, permissions grant, logcat.
• Cross-platform smoke on Windows (primary driver) and macOS (iOS + Android coexistence).

Risks / verify-first

• Electron H.264 WebCodecs decode — verify in a step-0 spike that an Electron renderer VideoDecoder decodes scrcpy's H.264. Electron ships proprietary codec decode, so this is expected to pass. Fallback if not: a wasm H.264 decoder (Broadway / tinyh264) or dropping to a main-process H.264→JPEG transcode — neither changes the backend interface, since the session advertises its codec.
• scrcpy-server protocol is version-coupled to the bundled jar (same class of risk as serve-sim's private SimulatorKit APIs). Pin the version; record it next to the bundled jar.
• adb/emulator environment variance — offline/unauthorized devices, cold-boot timeouts, missing SDK. All handled via explicit, surfaced errors.

Rollout

1. Step-0 spike: confirm WebCodecs H.264 decode in the Electron renderer.
2. Extract the EmulatorBackend interface + IosEmulatorBackend (pure refactor); keep all iOS tests green.
3. Android device management (android-sdk-discovery, adb-devices, avd-manager, android-availability) + availability aggregation; surface Android devices in orca emulator list.
4. scrcpy streaming (scrcpy-server-deploy, scrcpy-video-stream) + the video IPC channel + the renderer WebCodecs canvas path; live pane renders.
5. scrcpy control (scrcpy-control-channel, android-input-mapping) + tap/gesture/type/button/rotate end-to-end.
6. Extra capabilities: ax, install/launch, permissions, logcat.
7. Renderer polish: Android hardware buttons, bezel, setup guide.
8. Packaging (scrcpy-server.jar resource) + the orca-emulator-android skill.
9. Tests at each step; typecheck + lint; Electron validation on Windows + macOS.

Open decisions

• Whether orca emulator install/launch/logcat should also be exposed for iOS later (iOS install is xcrun simctl install); v1 leaves them Android-only via capability flags.
• Whether to expose an explicit orca emulator boot <avd> verb vs. folding boot into attach; initial version folds boot into attach (parity with iOS, which boots on attach) and adds a --no-boot opt-out.