mirror of
https://github.com/stablyai/orca.git
synced 2026-09-21 16:02:20 +00:00
Jinwoo Hong
f5d2d6e757
* feat(mobile): carry screencast frames over the bridge as base64 (OTA phase C, C6.1) `bridge-screencast-binary.ts` landed in C0 as the page's half of the binary lane and named C6 as the owner of the encoder that satisfies it. This is that encoder, plus the host honouring `wantsBinary`: a subscribe that asked for binary gets an `onBinaryFrame` on the native stream, and each frame crosses as the envelope's `event.binary` on the same `seq` ledger as the stream's JSON events, because the page acks by that count. The base64 encoder is grouped rather than per byte or per `fromCharCode` window. Its docstring carries the measurement, including the part that contradicts the design note this came from: on V8 the per-byte form is the fastest of the three, not the quadratic one, and the chunked form it was meant to beat is the slowest. The grouped one is here because its cost does not depend on how an engine ropes `+=`, and Hermes is what the shell runs. No new opcode, no `v` bump, no negotiation added: `wantsBinary` is already in the contract and is the negotiation. Over-cap behaviour is unchanged in this commit — a binary event over the frame cap still ends the stream, which is what C6.2 changes. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * feat(mobile): drop an over-cap screencast frame instead of ending the stream (OTA phase C, C6.1) Measured at the pane's own request parameters, a screencast frame exceeds the 640 KiB envelope on a phone layout whenever the page will not compress: JPEG's worst case is 0.545 bytes per pixel at quality 72, so mobile view mode at 780x1424 is 811,289 bytes, 124% of the cap. Ending the stream there blacks out a browser tab for the life of the pane over one frame. So the two kinds of event part at the cap. A JSON event that will not fit still ends the stream with `overflow`, because its reader cannot see the hole it would leave; a screencast frame is dropped and the stream lives, because the next frame is one throttle interval away and the pane is still showing the last one. Both are asserted side by side so neither turns into the other. A drop leaves no other trace: the diagnostic beside it prints once per host, so a stream shedding a frame a second and one that shed a single frame read the same. The host therefore counts them per stream for the diagnostic and keeps a session total, and the shell's dev facts carry that total — the surface that already shows build state, with the line moved into its own module so what it says is pinned rather than inferred from a template. The 12-character build prefix it has always shown is unchanged. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * feat(mobile): name the binary screencast lane as a grant (OTA phase C, C6.1) Ruling 5's negotiation, and the check it asked for first: no reader of a grant is a closed enum, so there is no blocker and nothing an older page has to tolerate. `BridgeGrantsSchema.native` and the shell's manifest reader are both open string arrays, and the shell reader's own docstring already states the degradation — a grant name a build does not know leaves that one route native rather than refusing the bundle. What does constrain the name is the host contract's `GRANT_NAME_PATTERN`: a grant is one camelCase token or a `native.<domain>.<action>` verb with at least two dot segments. So `browser.screencast` and `native.screencast` are both refused, and the lane is `screencastBinary`. `screencast` alone would be wrong: the page can already subscribe to `browser.screencast` and receive its JSON events, and only the binary frames need the encoder. Added to the shell's implemented set, which is the same list `init.grants.native ` offers, so a route declaring it is served by a shell that has the encoder and left native by one that does not. No route declares it here; C7's session route does. The contract-side case is a characterisation pin, not a red-first one: the pattern already admitted this name, and the test records that the two tempting spellings are the ones it refuses. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * test(mobile): check the dropped-frame total through the bridge hook (OTA phase C, C6.1) The hook gained a required `onBinaryFramesDropped` two commits ago and this test kept calling it without one, so the tests-typecheck ratchet went red on that commit — caught here rather than in CI because an exit code was read off a pipeline's last stage instead of the script. Fixed by wiring the callback into the probe rather than by a cast, and with the case that makes the wiring evidence instead of types: a dropped frame raises the total the screen receives, and the stream stays subscribed while it does. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * refactor(mobile): keep the dropped-frame counter with the ledger it belongs to (OTA phase C, C6.1) Declared between a getter and a method, which is not where this class keeps state: the subscription map is at the top and the counter is the same kind of thing. Move only. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(mobile): serve the binary screencast lane only to a route granted it (OTA phase C, C6.1) Reported as a gap after C6.1's third commit and ruled on: the host honoured `wantsBinary` from any page, so a route that never declared `screencastBinary` could still make the shell encode base64 on its behalf. That is the hole per-route grants exist to close — the same class as a route granted only `navigate` and `storage` reaching the clipboard. The rule now reads the session's resolved list, which is what its route declared narrowed to what this shell implements, and is the same set `init.grants.native` is built from. So the host offers the lane in `init` exactly when it will serve it. Ungranted is not a refusal. The subscription proceeds and its JSON events cross as before, which is the silence every other grant gives at the call site; a page that reads its own grants never reaches that state. Both branches are pinned beside each other, and `grantsForRoute` is pinned dropping a grant this shell does not implement — granted-but-unimplemented and never-granted arrive at the host as the same absence, so its rule reads one case. The grant name moves into the module that holds the rule reading it, so the two cannot drift. `bridge-host.ts` is at 298 of its 300-line cap after this. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * refactor(mobile): move the page's stream-frame rules out of the host (OTA phase C, C6.1) `bridge-host.ts` reached 298 of its 300-line cap, so the next main merge that touched it would have crossed under CI pressure on someone else's PR. Split deliberately instead, at the boundary the growth came from. `bridge-host.ts` is the host's lifecycle and its dispatch. Opening a stream is the only frame kind whose handling is more than one line of delegation — four refusals and, since C6.1, the binary-lane decision — so it moves whole, and `cancel` and `ack` move with it so all three stream frames are decided in one place. The host's `cancel` arm still chooses between a stream and a request where it always did: a page's `cancel` names one or the other, and splitting that choice would leave half an arm in each module. Counted without blank lines or comments, as the rule counts them: bridge-host.ts 298 -> 270, and the new module is 59. A pure move. No test changed and none was added, which is what makes the existing suites the proof: 45 files and 745 tests green on the same assertions as before. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * feat(mobile): report a page that asked for screencast frames it was not granted (OTA phase C, C6.1) An ungranted `wantsBinary` is not a refusal on the wire, so nothing crosses back: the subscription proceeds and its JSON events cross as they always have. That left a page which did ask getting JSON for the life of the document with no side able to say why. `notify-refused` has covered the equivalent notify case since C0; this is the same shape for the one frame kind that lacked it. The rule now answers a verdict rather than a boolean, because `not-asked` and `ungranted` are the same answer for different reasons and only one is worth reporting. So the decision and the report read one rule, and a page that never asked stays silent — pinned, along with a granted route staying silent, so the line cannot start firing on either. The wire is unchanged and pinned unchanged: the case beside this one still asserts one JSON event delivered and zero error frames. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(mobile): reset the dropped-frame total with the host that counts it (OTA phase C, C6.1) Round 1 on #21758, three findings. The real one: the count is per host and the screen's copy was not. A rebuilt host starts its own total at zero, so the screen kept the retired host's number until the new one dropped a frame and then read *lower* — a falling count looks like frames coming back, which is worse than starting over. The hook now announces a fresh count as it builds a host. That also reports zero on the first build, where the screen is already at zero and React bails out of the render; the two hook cases pin that leading zero rather than leave it to be rediscovered. Two docstrings that described nothing: `BUILD_ID_PREFIX_LENGTH`'s stayed behind when the constant moved to the dev-facts module and had drifted above `failureMessage`, and `page-route-policy.test.ts` kept the docstring of the test it replaced above the one that replaced it. Both deleted; the first's text lives on the new module. Red-first for the reset, checked against its final expectations rather than its first: with the one line reverted both hook cases fail on the missing zero, and both pass with it. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(mobile): keep the dropped-frame total out of a production build's render path (OTA phase C, C6.1) CodeRabbit's Major on #21758. The total went into React state on every dropped frame in every build, and outside a development build the line that reads it renders null — so an over-cap page re-rendered the whole shell screen up to ten times a second for a fact nobody can see. Measured, not argued: five drops, five extra renders. Fixed at the seam rather than with a ternary at the call site. The dev-facts module owns the line, so it now owns the number behind it and the rule that the number is only state where something renders it. The screen holds no flag and no counter; it asks for both and passes the reporter on. The reporter is stable, so the bridge host is never rebuilt for it. `isDevelopmentBuild` becomes a call rather than a module constant. A build flag never changes at runtime so this costs nothing, and as a constant the branch was unreachable to anything that did not set the global before the module loaded — which is why the production case could not be written at the screen at all. Also fixed, found while writing that case: the screen test's `usePageHostSnapshot` double returned a fresh object on every render, so the host effect's identity changed each time and the bridge host was torn down and rebuilt on every render of the screen, settling every pending request with it. The real hook holds the snapshot in `useState` and is stable. One object for the file now. This was masking the fold under test — the count reset to zero on every render — and every other case in that file was measuring a rebuild storm. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * perf(mobile): price a screencast frame before encoding it (OTA phase C, C6.1) Round 2 on #21758, two lows. The encode is a base64 pass over the whole image and the window decides whether the frame can be posted at all, so deciding after encoding made a page that had stopped acking pay for every frame the shell then threw away — the reviewer's case is ten 300 KB frames against a closed window, 3 MB encoded and nothing sent. The size is knowable without encoding: base64 is ASCII, so JSON escapes none of it and the frame is its header serialized plus exactly the image's encoded length. `encodeBridgeScreencastFrame` is now built from that header rather than beside it, so the shape measured and the shape sent cannot drift, and the window arithmetic is one rule read before the encode and again on the frame that was. Exact, not conservative, so the drop diagnostic still reports the whole frame and the committed byte pin is untouched. Red-first with the real encoder wrapped in a counter: window full, ten frames, ten encodes before and zero after, with the drop count still ten. An over-cap frame likewise goes from one encode to none. A third case holds the other direction — two carryable frames still encode twice — so the fix cannot pass by encoding nothing. Second low: the dev-facts block sat outside the only `beforeEach` and left `routeGrants` and `client` mutated, inert only because it runs last. The shared setup moves to file level where the mutable dependencies actually live, resets both, and a case at the end of the file pins it — deleting the reset fails there and nowhere else, since nothing else runs after a case that mutates them. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb
Orca Mobile
React Native companion app for Orca. Monitor worktrees, view terminal output, and send commands from your phone.
Local development uses two processes:
- Orca desktop/Electron from the repo root. This hosts the mobile WebSocket RPC server on port
6768. - Expo Metro from
mobile/. This serves the React Native app on port8081.
Unless a command says otherwise, run mobile app commands from the mobile/ directory.
Prerequisites
- Node.js 24+
- pnpm
- Xcode and/or Android Studio tooling for simulator or device builds
- Expo Go on your phone, or a development client build when native modules are needed
- Phone and desktop on the same LAN when testing a physical phone
Start Desktop Orca
From the repository root:
pnpm install
pnpm dev
Confirm the mobile RPC server is listening:
lsof -nP -iTCP:6768 -sTCP:LISTEN
Restart pnpm dev after changing Electron main-process code. Metro hot reload only applies to the mobile JavaScript bundle.
Start The Mobile App
cd mobile
pnpm install
pnpm start
Scan the Expo QR code with your phone's camera on iOS, or Expo Go on Android.
For a native dev-client build:
pnpm exec expo run:android
pnpm exec expo run:ios
pnpm start --dev-client
Pair With Desktop Orca
- Open Orca desktop.
- Go to Settings > Mobile.
- Scan the pairing QR code from the mobile app.
- Confirm the mobile host endpoint is
ws://<desktop-ip>:6768.
For the Android emulator, use ws://10.0.2.2:6768. For a physical phone, use the desktop LAN IP, for example ws://192.168.0.179:6768.
If the phone has a stale host entry, remove it from the app and pair again.
Development Paths
Android Phone
- Install Expo Go from Google Play
- Run
pnpm start, scan QR with Expo Go - For native modules:
pnpm exec expo run:android - Run with
pnpm start --dev-client
iOS Simulator
- Install Xcode from the App Store
- Run
pnpm start --iosto open in iOS Simulator
Physical Phone Debugging
The phone can be inspected through the connected device tooling:
orca snapshot --json
orca click --element @e3 --json
orca fill --element @e1 --value "ls" --json
orca screenshot --json
Use snapshot first to find the current element refs, then click/fill those refs. After mobile file edits, Metro usually hot reloads automatically, but navigating out of and back into the session screen can be useful because it re-runs terminal.subscribe.
Terminal Streaming Repro Without A Phone
Use this when terminal output does not render on device and you need to split server streaming bugs from WebView/UI bugs:
cd mobile
ORCA_MOBILE_WS_URL=ws://127.0.0.1:6768 pnpm exec tsx scripts/test-subscribe.ts <deviceToken> <serverPublicKeyB64>
You can pass a worktree selector as the third argument:
pnpm exec tsx scripts/test-subscribe.ts <deviceToken> <serverPublicKeyB64> "id:<worktreeId>"
pnpm exec tsx scripts/test-subscribe.ts <deviceToken> <serverPublicKeyB64> "path:/absolute/worktree/path"
pnpm exec tsx scripts/test-subscribe.ts <deviceToken> <serverPublicKeyB64> "name:my-worktree"
The expected result includes:
streamSawMarker: true
readSawMarker: true
If this repro fails, debug the desktop runtime/PTY path before the mobile WebView. If it passes but the phone is blank, debug the session screen or TerminalWebView readiness/queueing path.
Terminal Color Repro Without A Phone
Use this when terminal colors disappear after switching tabs. Open a Claude Code terminal and at least one other terminal in the target worktree, then run:
cd mobile
ORCA_MOBILE_WS_URL=ws://127.0.0.1:6768 pnpm exec tsx scripts/repro-terminal-colors.ts \
<deviceToken> <serverPublicKeyB64> "id:<worktreeId>"
The script captures terminal.subscribe snapshots in an A → B → A sequence and writes raw snapshots to mobile/terminal-color-repro/. If the two A snapshots have different sgrColor counts, the desktop snapshot changed during the switch. If they match, the ANSI color data is still present and the bug is in mobile replay/rendering.
Validation
Run these checks before committing mobile terminal changes:
cd mobile
pnpm exec tsc --noEmit
pnpm run check:tests-typecheck
pnpm lint
cd ..
pnpm typecheck:node
tsc --noEmit reads tsconfig.json, which excludes test files so Metro never bundles them.
tsconfig.test.json puts them back, and pnpm run typecheck:tests shows their errors in full.
check:tests-typecheck is the gate over it: a ratchet against tests-typecheck-baseline.txt, the
127 test files that do not typecheck yet. It fails when a file that checks today stops checking,
and when a baseline entry starts checking (prune it with
node scripts/check-tests-typecheck-ratchet.mjs --prune). The list may only shrink.
The same gate censuses the program first: every *.test.ts(x) on disk must be in it, or named in
the script's TESTS_OUTSIDE_PROGRAM with a reason. Without that, a test excluded from
tsconfig.test.json — or a Foo.test.tsx shadowed by a Foo.test.ts beside it, which a wildcard
include drops for the higher-priority extension — would leave the ratchet silently.
Protocol Version Compatibility
Mobile and desktop talk over a versioned protocol. Because mobile updates lag desktop by 24-48h via the App Store, both sides exchange version numbers on status.get so a genuinely incompatible combo can hard-block instead of silently misbehaving.
Constants live in two files (Metro can't resolve outside mobile/):
src/shared/protocol-version.ts—DESKTOP_PROTOCOL_VERSION,MIN_COMPATIBLE_MOBILE_VERSIONmobile/src/transport/protocol-version.ts—MOBILE_PROTOCOL_VERSION,MIN_COMPATIBLE_DESKTOP_VERSION
Today all four are set so evaluateCompat always returns { kind: 'ok' } — nothing blocks. The wire format is in place to flip a switch when needed.
When to bump
Bump DESKTOP_PROTOCOL_VERSION (and the mobile mirror MOBILE_PROTOCOL_VERSION when relevant) for breaking changes:
- Removed RPC method or required parameter that mobile uses
- Changed meaning (units, nullability) of an existing field mobile reads
- Changed encryption, framing, or auth handshake
Do not bump for additive changes:
- New RPC methods
- New optional fields on existing methods
- New event types in
terminal.subscribe
Set MIN_COMPATIBLE_MOBILE_VERSION (kill-switch) when desktop ships a change that requires a minimum mobile version to function safely. Same for MIN_COMPATIBLE_DESKTOP_VERSION from the mobile side.
When a verdict is blocked, mobile/src/components/ProtocolBlockScreen.tsx renders a screen pointing the user at either the App Store (mobile too old) or GitHub Releases (desktop too old).
To exercise the block screen locally: set MIN_COMPATIBLE_DESKTOP_VERSION = 999 in mobile/src/transport/protocol-version.ts, rebuild, pair to any desktop. Revert before merging.
Mock Server
Develop the mobile app without a running Orca desktop instance:
pnpm mock-server # starts mock WebSocket server on port 6768
Connect from the app using endpoint ws://localhost:6768 and token mock-device-token.
Environment variables
MOCK_NATIVE_CHAT=1— serve the native-chat scenario (one live agent tab, empty transcript, image upload) instead of the default terminal fixtures.MOCK_CHAT_AGENT=omp— withMOCK_NATIVE_CHAT=1, present an OMP tab and four decoded transcript messages, including a tool call and result, instead of the default Claude scenario. It deliberately omitstranscriptPathto exercise legacy-hook readability discovery; current OMP hooks may report a path.MOCK_SERVER_KEY_FILE— persist the server keypair across restarts so a paired device keeps its public-key pin. A missing or invalid file is re-keyed with a warning, which forces a re-pair.
Scenario control files
Read on every request, so behaviour can be flipped mid-session without a restart (a restart would re-key E2EE and force a re-pair). Write the mode into the file, or delete it for the default.
MOCK_SEND_MODE_FILE(defaultorca-mock-send-modein the system temporary directory) —accept(default) accepts the send,errorfails it withmobile_input_floor_unavailable, anything else reports the send as rejected.MOCK_TERMINAL_LIST_MODE_FILE(defaultorca-mock-terminal-list-modein the system temporary directory) —omitreturns an empty terminal list,otherreturns a list that omits the chat handle, anything else lists it.MOCK_TERMINAL_STREAM_MODE_FILE(defaultorca-mock-terminal-stream-modein the system temporary directory) —deadanswers a subscribe withsubscribedthenend(a gone PTY), which is what exercises the rearm bound and terminal prune; anything else streams normally.
Connecting to Real Orca
- Start Orca desktop with WebSocket transport enabled
- In Orca, go to Settings > Mobile and scan the QR code with this app
- The QR encodes the connection endpoint, device token, and TLS fingerprint
Project Structure
mobile/
├── app/ # Expo Router screens (file-based routing)
│ ├── _layout.tsx # Root layout with navigation stack
│ ├── index.tsx # Home screen — paired hosts list
│ └── pair-scan.tsx # QR code scanning screen
├── src/
│ ├── terminal/ # Terminal WebView and xterm bridge
│ └── transport/ # WebSocket RPC client
├── scripts/
│ ├── test-subscribe.ts # Desktop streaming repro without a phone
│ └── mock-server.ts # Standalone mock WebSocket server
└── assets/ # App icons and splash screen