mirror of
https://github.com/stablyai/orca.git
synced 2026-09-22 00:02:31 +00:00
Jinwoo Hong
96c1dd8b70
* feat(mobile): paint browser screencast frames through web siblings (OTA phase C, C6.2) The pane's frame path is written against React Native's native-prop writer, which does not exist on React Native Web: a ref there is the DOM node, so both writes throw and the pane never shows a frame. Three `.web.ts` siblings, each for a measured gap. - The image and layer writes move out of `mobile-browser-frame-state.ts` into `browser-frame-layer-paint.ts`, whose sibling paints the frame as a `background-image` on the element RN Web sizes and flips the double buffer with one opacity write per layer. The pane still never re-renders while it streams. - A `background-image` write fires no load event, so the offscreen layer would never become visible. The sibling arms the flip from an image decode instead, and the flip itself is shared with the native `onLoad` path rather than written twice. - The data URI keeps the base64 the bridge already carried instead of encoding the bytes back into the same string. Measured in this tree against the `buffer` shim the page bundle resolves: 0.256 ms per frame at 45,815 bytes and 2.61 ms at 463,942, against under a microsecond for the carried string. Per C6 ruling 5 the pane asks for binary frames only when the shell granted the lane, and renders its existing stream-error state otherwise, so a page never waits on frames a shell without the encoder cannot send. The grant name is a placeholder until C6.1 reports it. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * feat(mobile): budget the mobile view's frame area against the bridge cap (OTA phase C, C6.3) A screencast frame crosses the bridge as one message under BRIDGE_MAX_MESSAGE_BYTES. Measured here: the phone's mobile view at the native device scale factor asks for a 390x712 viewport at 2x, which is 1,110,720 device pixels, and at the worst case JPEG has at quality 72 that is a 807,559-byte message against a 655,360-byte cap — 123% of it. The `.web.ts` sibling holds the mobile view inside that. The budget is computed rather than written down: the cap, less an envelope this measures from the frame's own shape at its widest (435 bytes), is what the base64 may occupy; three quarters of that is the JPEG; divided by one named worst-case constant of 0.545 bytes per pixel it is an area of 901,271 pixels. The phone lands on a device scale factor of 1.80 and a 654,205-byte message, 99.8% of the cap. A cap that moved and a budget that did not would be a pane going dark on a page it could have streamed. Web view mode is untouched, and byte-identical to the native request: there the frame is a desktop viewport letterboxed into maxWidth/maxHeight, which the page cannot predict, so C6 ruling 1's drop-the-over-cap-frame rule is its only protection. Native is unchanged. The constants and the assembly move to a third module because a `.web.ts` cannot import a value from the file it shadows — the bundler resolves the specifier back to the sibling itself — and two copies of them would drift. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * feat(mobile): put the browser pane's text inputs on the web font-size seam (OTA phase C, C6.4) The pane has two text inputs, the address bar at 12px and the key row's "Type on page…" at 14px, and neither went through TEXT_INPUT_FONT_SIZE. In a browser an input under 16px makes iOS zoom the page on focus and never zoom back, and keyboard-occlusion.web.ts reads a visual viewport scale other than 1 as "no keyboard" — so one focus would leave the pane's keyboard lift at 0 for the rest of the typing session. C4.2's failure exactly, on a screen its census does not walk: that census walks the source-control hub and the review route, and the pane is in neither until C7 lists a route that mounts it. The key row's input goes straight onto the seam, whose native value is the theme's body size, so it renders at the 14px it already did. The address bar is a `.web.ts` split instead, so native keeps the 12px meta size it has always shown; the input and the label painted over it move together, or the address would resize on every focus. The address bar also gets inputMode="url" on the web only. keyboardType is a native enum a browser does not read, so the page's address bar was falling back to a plain keyboard; inputMode takes precedence over keyboardType, so it stays undefined on both native platforms. One consequence recorded rather than fixed, and pinned in config/scripts/mobile-web-app-browser-pane-text-inputs.test.mjs: the C4.2 census resolves an import through .ts/.tsx only, never .web.ts, so it reads the native address style that no browser loads and reports it as an offender. Whoever lists the pane's route either teaches resolveLocal the extensions the builder already prefers, or moves the address bar onto the seam natively at 14px. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(mobile): name C6.1's binary screencast grant (OTA phase C, C6.2) C6.1 has decided the name: `screencastBinary`, one camelCase token. Replaces the placeholder this PR landed with while C6.1 was still choosing. The placeholder was also unusable, which the test added here would have caught: `GRANT_NAME_PATTERN` in the manifest contract admits a bare name or a `native.`-prefixed verb and nothing else, so a route declaring `browser.screencast.binary` would have been refused by the bundle before any shell saw it, and the pane would have taken its stream-error branch for a reason no screen could report. The name is now checked against `MobileWebBundleRouteSchema` itself rather than against a restated regex, with the dotted spelling as the failing case beside it. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(mobile): keep the address field test inside the typecheck ratchet (OTA phase C, C6.4) `findByType('TextInput')` does not typecheck: a host-component string is not an `ElementType`, so the file fell out of `tsc -p tsconfig.test.json` and the tests-typecheck ratchet reported it. Found by reading the ratchet's exit code rather than its piped tail, which is how it was missed the first time. The element is looked up by its placeholder instead, and the ratchet is green with 732 test files in the program. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(mobile): answer a frame decode for the frame, not the layer (OTA phase C, C6.2) Round 1 folds on #21754. The undecodable arm freed the pending slot without checking whose frame had failed, while the displayable arm checked. Reproduced: frame 2 goes pending on layer 1, frame 3 repoints the same layer, frame 2's decode rejects and clears the slot layer 1 is holding for frame 3, then frame 3 decodes and the flip is refused because the slot no longer names its layer. The newest frame sits decoded at opacity 0 behind an older one, and a page that has gone still sends no further frame to recover with. Web only; native never calls this. Both arms now answer for the frame they were armed with. The displayable arm's own guard had no test: deleting it left `src/browser/` and the full suite green, because the case that exercised it settled both decodes and asserted an end state both orders produce. The harness now settles one decode at a time, keyed on the source it was given, and the ordered case reds without the guard. Also: the paint sibling's opacity test claimed "no re-render" while asserting two style strings, so it is named for what it checks and the claim is counted where React is — across ten streamed frames the three state setters are called once each, on the mount frame. And the overrides allowlist is rebuilt from main's bytes plus the new entries, so two pre-existing reasons keep their literal em dash instead of a re-serialized escape. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * test(mobile): find the address input by its imported type (OTA phase C, C6.4) `findByType('TextInput')` does not typecheck — a host-component string is not an `ElementType` — so the file dropped out of `tsc -p tsconfig.test.json` and the tests-typecheck ratchet reported it. The imported component is what the element is looked up by now; the react-native mock stands it up as that same string at runtime, so the lookup is unchanged and the file is back inside the ratchet's program. Why the earlier run reported 0: the command was `node scripts/check-tests-typecheck-ratchet.mjs 2>&1 | tail -2; echo $?`, and `$?` after a pipeline is the exit code of `tail`, which is always 0. The banner line that printed was the last line of the failure banner, not the success one. Every gate in this branch's report is now read from the command itself, unpiped. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(scripts): read a text input's size from the file the page loads (OTA phase C, C6.4) The census followed an import through `.ts`/`.tsx` only, while the closure it walks comes from esbuild, which prefers `.web.tsx`/`.web.ts`. So a style module with a platform sibling was judged on the half no browser loads. That fails in the direction that matters: a split whose web half sits under the focus-zoom floor reads as clean because its native half is on the seam, which is the exact shape the seam exists to catch. `resolveLocal` now tries the extensions in the builder's own order. The seam comparison collapses a resolved path onto its module identity before matching, because the seam is itself a split — `text-input-font-size.web.ts` is where the raise lives — and without that every binding in the tree would stop naming the seam: deleting it reds both C4 route closures. The browser pane's own census flips from pinning its address field as an offender to expecting none. The two C4 route closures still answer 0 offenders and 0 unresolved, run with the closure tests enabled. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(mobile): bound the frame envelope above every double it can carry (OTA phase C, C6.4) Round 1 folds on #21760. The envelope estimate serialized each metadata field as a 16-character double, so the bound was 435 where a real frame event at its widest measures 516. Since the budget spends whatever the cap leaves, that 81-byte shortfall was the whole margin: a frame on the budget would have gone over the cap and been dropped. The bound is now the skeleton plus the widest a double can print, for the nine metadata keys imported from the protocol module rather than copied, so a tenth field cannot be added without being paid for. Two corrections to the fold as written, both measured here. The widest is 25 characters, not 24. Exponential form tops out at 24 (`-1.7976931348623157e+308`), but ToString only leaves fixed notation below 1e-6, and just above it a double prints as sign, `0.`, five zeros and seventeen digits: `-0.0000012345678901234567`. A sweep over four million random bit patterns found 25 and nothing longer; a deterministic sweep over both forms is in the test. And the base64 expansion has to count padding. Three quarters of the room claims up to two characters base64 does not have for an image of 3k+1 bytes, which at a margin the budget now spends exactly is a dropped frame. The two agree at today's envelope size because the room happens to divide by four, so this is a latent fix rather than a reproduced one, and the test pins the discrepancy directly instead of implying it. Budget moves from 901,271 to 901,161 pixels; the phone's mobile view stays at a device scale factor of 1.80. Pinning the bound against C6.1's real encoder is C6.5's, once both are on main; the docstring says so, and says what the bound does not cover: the metadata object is loose, so unknown keys and web view mode's letterboxed frame are ruling 1's to drop rather than this budget's to predict. Also: the pane census listed its own closure by hand, so "no unresolved styles" said the walk read those files, not that they are the pane's set. It now scans `src/browser` for every non-test module that renders a `TextInput` and asserts the list matches; a third module planted there reds it. And the seam's native-consumer pin names the key row and the address bar directly rather than transitively, matched at the `fontSize` property instead of anywhere in the file — a file-wide search survives the change, because the import line does. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * test(mobile): check the frame write against a real react-native-web Image (OTA phase C, C6.2) Round 2 folds on #21754. Every test for the web paint sibling handed it a `div > div` of its own making, so the assumption it rests on — that the host's first element child is the one carrying the frame — was only ever checked against a shape written to match it. React Native Web also renders an accessibility `<img>` in there, and a release that reorders those children would keep all of them green while the pane painted nothing. One test now renders the real component, asks it which child it painted, and checks the write lands on that one. Pointing the sibling at `lastElementChild` reds it and leaves the hand-built cases passing, which is the gap. A second case records what the `<img>` does: the streaming path writes styles and never props, so it keeps the source it mounted with for the life of the pane, and that is what a screen reader and the image context menu see. react-native-web ships no type declarations, so the component comes through `createRequire`, whose return is `any` at its own signature; the one prop it renders with is declared rather than asserted, and the file stays inside the tests-typecheck ratchet. The module docstring also claimed more than the code does. The frame path adds no render, but a render from any of the pane's other state — address focus, a dialog, the view mode, zoom — repaints both layers from `renderedFrameSource`, which reads `frameUriRef.current`, so both land on the newest frame whether or not it has decoded. Native clobbers the same way through `setNativeProps`. Said plainly, along with what restores the buffering. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * docs(mobile): say what a render does to the accessibility image (OTA phase C, C6.2) pullfrog is right, and the test carried the same wrong claim. The note said the hidden `<img>` keeps the frame it mounted with for the life of the pane, two paragraphs after saying a render from the pane's other state passes `renderedFrameSource` as `source` — and React Native Web derives that image's `src` from the same prop it paints the background from, so the first such render moves it. Measured here rather than reasoned about: rendering the real component, writing a frame imperatively, then re-rendering with a new source moves the `src` and leaves the background where the imperative write put it. The two halves are now two cases, named for what each one shows, and the note says the streaming writes never touch it while a render does — so it holds the frame the pane last rendered with, not the one on screen. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb
Orca Mobile
React Native companion app for Orca. Monitor worktrees, view terminal output, and send commands from your phone.
Local development uses two processes:
- Orca desktop/Electron from the repo root. This hosts the mobile WebSocket RPC server on port
6768. - Expo Metro from
mobile/. This serves the React Native app on port8081.
Unless a command says otherwise, run mobile app commands from the mobile/ directory.
Prerequisites
- Node.js 24+
- pnpm
- Xcode and/or Android Studio tooling for simulator or device builds
- Expo Go on your phone, or a development client build when native modules are needed
- Phone and desktop on the same LAN when testing a physical phone
Start Desktop Orca
From the repository root:
pnpm install
pnpm dev
Confirm the mobile RPC server is listening:
lsof -nP -iTCP:6768 -sTCP:LISTEN
Restart pnpm dev after changing Electron main-process code. Metro hot reload only applies to the mobile JavaScript bundle.
Start The Mobile App
cd mobile
pnpm install
pnpm start
Scan the Expo QR code with your phone's camera on iOS, or Expo Go on Android.
For a native dev-client build:
pnpm exec expo run:android
pnpm exec expo run:ios
pnpm start --dev-client
Pair With Desktop Orca
- Open Orca desktop.
- Go to Settings > Mobile.
- Scan the pairing QR code from the mobile app.
- Confirm the mobile host endpoint is
ws://<desktop-ip>:6768.
For the Android emulator, use ws://10.0.2.2:6768. For a physical phone, use the desktop LAN IP, for example ws://192.168.0.179:6768.
If the phone has a stale host entry, remove it from the app and pair again.
Development Paths
Android Phone
- Install Expo Go from Google Play
- Run
pnpm start, scan QR with Expo Go - For native modules:
pnpm exec expo run:android - Run with
pnpm start --dev-client
iOS Simulator
- Install Xcode from the App Store
- Run
pnpm start --iosto open in iOS Simulator
Physical Phone Debugging
The phone can be inspected through the connected device tooling:
orca snapshot --json
orca click --element @e3 --json
orca fill --element @e1 --value "ls" --json
orca screenshot --json
Use snapshot first to find the current element refs, then click/fill those refs. After mobile file edits, Metro usually hot reloads automatically, but navigating out of and back into the session screen can be useful because it re-runs terminal.subscribe.
Terminal Streaming Repro Without A Phone
Use this when terminal output does not render on device and you need to split server streaming bugs from WebView/UI bugs:
cd mobile
ORCA_MOBILE_WS_URL=ws://127.0.0.1:6768 pnpm exec tsx scripts/test-subscribe.ts <deviceToken> <serverPublicKeyB64>
You can pass a worktree selector as the third argument:
pnpm exec tsx scripts/test-subscribe.ts <deviceToken> <serverPublicKeyB64> "id:<worktreeId>"
pnpm exec tsx scripts/test-subscribe.ts <deviceToken> <serverPublicKeyB64> "path:/absolute/worktree/path"
pnpm exec tsx scripts/test-subscribe.ts <deviceToken> <serverPublicKeyB64> "name:my-worktree"
The expected result includes:
streamSawMarker: true
readSawMarker: true
If this repro fails, debug the desktop runtime/PTY path before the mobile WebView. If it passes but the phone is blank, debug the session screen or TerminalWebView readiness/queueing path.
Terminal Color Repro Without A Phone
Use this when terminal colors disappear after switching tabs. Open a Claude Code terminal and at least one other terminal in the target worktree, then run:
cd mobile
ORCA_MOBILE_WS_URL=ws://127.0.0.1:6768 pnpm exec tsx scripts/repro-terminal-colors.ts \
<deviceToken> <serverPublicKeyB64> "id:<worktreeId>"
The script captures terminal.subscribe snapshots in an A → B → A sequence and writes raw snapshots to mobile/terminal-color-repro/. If the two A snapshots have different sgrColor counts, the desktop snapshot changed during the switch. If they match, the ANSI color data is still present and the bug is in mobile replay/rendering.
Validation
Run these checks before committing mobile terminal changes:
cd mobile
pnpm exec tsc --noEmit
pnpm run check:tests-typecheck
pnpm lint
cd ..
pnpm typecheck:node
tsc --noEmit reads tsconfig.json, which excludes test files so Metro never bundles them.
tsconfig.test.json puts them back, and pnpm run typecheck:tests shows their errors in full.
check:tests-typecheck is the gate over it: a ratchet against tests-typecheck-baseline.txt, the
127 test files that do not typecheck yet. It fails when a file that checks today stops checking,
and when a baseline entry starts checking (prune it with
node scripts/check-tests-typecheck-ratchet.mjs --prune). The list may only shrink.
The same gate censuses the program first: every *.test.ts(x) on disk must be in it, or named in
the script's TESTS_OUTSIDE_PROGRAM with a reason. Without that, a test excluded from
tsconfig.test.json — or a Foo.test.tsx shadowed by a Foo.test.ts beside it, which a wildcard
include drops for the higher-priority extension — would leave the ratchet silently.
Protocol Version Compatibility
Mobile and desktop talk over a versioned protocol. Because mobile updates lag desktop by 24-48h via the App Store, both sides exchange version numbers on status.get so a genuinely incompatible combo can hard-block instead of silently misbehaving.
Constants live in two files (Metro can't resolve outside mobile/):
src/shared/protocol-version.ts—DESKTOP_PROTOCOL_VERSION,MIN_COMPATIBLE_MOBILE_VERSIONmobile/src/transport/protocol-version.ts—MOBILE_PROTOCOL_VERSION,MIN_COMPATIBLE_DESKTOP_VERSION
Today all four are set so evaluateCompat always returns { kind: 'ok' } — nothing blocks. The wire format is in place to flip a switch when needed.
When to bump
Bump DESKTOP_PROTOCOL_VERSION (and the mobile mirror MOBILE_PROTOCOL_VERSION when relevant) for breaking changes:
- Removed RPC method or required parameter that mobile uses
- Changed meaning (units, nullability) of an existing field mobile reads
- Changed encryption, framing, or auth handshake
Do not bump for additive changes:
- New RPC methods
- New optional fields on existing methods
- New event types in
terminal.subscribe
Set MIN_COMPATIBLE_MOBILE_VERSION (kill-switch) when desktop ships a change that requires a minimum mobile version to function safely. Same for MIN_COMPATIBLE_DESKTOP_VERSION from the mobile side.
When a verdict is blocked, mobile/src/components/ProtocolBlockScreen.tsx renders a screen pointing the user at either the App Store (mobile too old) or GitHub Releases (desktop too old).
To exercise the block screen locally: set MIN_COMPATIBLE_DESKTOP_VERSION = 999 in mobile/src/transport/protocol-version.ts, rebuild, pair to any desktop. Revert before merging.
Mock Server
Develop the mobile app without a running Orca desktop instance:
pnpm mock-server # starts mock WebSocket server on port 6768
Connect from the app using endpoint ws://localhost:6768 and token mock-device-token.
Environment variables
MOCK_NATIVE_CHAT=1— serve the native-chat scenario (one live agent tab, empty transcript, image upload) instead of the default terminal fixtures.MOCK_CHAT_AGENT=omp— withMOCK_NATIVE_CHAT=1, present an OMP tab and four decoded transcript messages, including a tool call and result, instead of the default Claude scenario. It deliberately omitstranscriptPathto exercise legacy-hook readability discovery; current OMP hooks may report a path.MOCK_SERVER_KEY_FILE— persist the server keypair across restarts so a paired device keeps its public-key pin. A missing or invalid file is re-keyed with a warning, which forces a re-pair.
Scenario control files
Read on every request, so behaviour can be flipped mid-session without a restart (a restart would re-key E2EE and force a re-pair). Write the mode into the file, or delete it for the default.
MOCK_SEND_MODE_FILE(defaultorca-mock-send-modein the system temporary directory) —accept(default) accepts the send,errorfails it withmobile_input_floor_unavailable, anything else reports the send as rejected.MOCK_TERMINAL_LIST_MODE_FILE(defaultorca-mock-terminal-list-modein the system temporary directory) —omitreturns an empty terminal list,otherreturns a list that omits the chat handle, anything else lists it.MOCK_TERMINAL_STREAM_MODE_FILE(defaultorca-mock-terminal-stream-modein the system temporary directory) —deadanswers a subscribe withsubscribedthenend(a gone PTY), which is what exercises the rearm bound and terminal prune; anything else streams normally.
Connecting to Real Orca
- Start Orca desktop with WebSocket transport enabled
- In Orca, go to Settings > Mobile and scan the QR code with this app
- The QR encodes the connection endpoint, device token, and TLS fingerprint
Project Structure
mobile/
├── app/ # Expo Router screens (file-based routing)
│ ├── _layout.tsx # Root layout with navigation stack
│ ├── index.tsx # Home screen — paired hosts list
│ └── pair-scan.tsx # QR code scanning screen
├── src/
│ ├── terminal/ # Terminal WebView and xterm bridge
│ └── transport/ # WebSocket RPC client
├── scripts/
│ ├── test-subscribe.ts # Desktop streaming repro without a phone
│ └── mock-server.ts # Standalone mock WebSocket server
└── assets/ # App icons and splash screen