starred/orca
1
0
Fork 0
mirror of https://github.com/stablyai/orca.git synced 2026-09-30 16:02:56 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
debug-github-rate-limit
orca/tests/tools/relay-bench/README.md
T
Jinwoo Hong 6729f3a0b0 tools: add a phone-vantage relay connect benchmark (#19251) 
* tools: add a phone-vantage relay connect benchmark

Connect-speed work on the phone had no way to attribute latency to a hop.
Timing the mobile app end to end only says "connect is slow", and a synthetic
WebSocket probe does not exercise the credential check, the E2EE handshake, or
the RPCs the phone blocks on before it publishes connected.

This replays the shipped mobile wire sequence from Node against a real desktop
over the production relay, so each phase gets its own number. The handshake is a
plain-JS port of the mobile client session, which is only trustworthy if it stays
byte-identical to what ships; a parity test runs it against the real desktop
responder in the normal unit suite so drift in the transcript encoding, key
schedule, or frame layout fails there rather than producing a bench that measures
a handshake nobody uses.

Adds a foreground mode for the resume-after-background question the phone lanes
need: connect, go silent past the relay's client silence watchdog, then report
whether the retained socket still answers and what the fallback redial costs.

The bench writes a resume-credential bundle at runtime. That file carries a live
device token for a real paired desktop, so the directory ignores it outright.

* tools: make the relay bench name its target and opt in to dialing

The supporting scripts carried production defaults: the director origin was
hardcoded in both, and the hop-latency probe defaulted to a named production
cell. Running either with no arguments sent live traffic at production, and the
region probe did it on import, before any argument was read. A default like that
is the wrong shape for a bench, because the operator never states what they are
measuring against and a stray invocation is indistinguishable from an intended
one.

Every script now refuses to open a socket unless ORCA_RELAY_BENCH_LIVE=1 is set,
and the director comes from --director or ORCA_RELAY_BENCH_DIRECTOR with no
fallback. The cell origin is a required argument. Refusals print one line of
usage and exit 2, so an accidental run is inert rather than live.

The remaining host-id default is an id no desktop owns, which is the point of
that probe: it measures the cell hop without reaching a desktop at all.

* tools(relay-bench): type refuse() as never so origins are strings

* tools(relay-bench): fail closed on hostile input and bounded arguments

Review found the harness trusted whatever it was handed: the DevTools port
and the director-supplied probe origins went straight into a URL, http
origins were accepted, repeat counts came from a bare Number() cast, and the
state file kept its existing mode.

- Validate the DevTools port as a 1-65535 integer, so '80@attacker.example'
  cannot move the fetch off loopback via URL userinfo.
- Require https for every origin, and refuse loopback, link-local, private,
  and multicast destinations. Region probe origins and the cell URL the
  director returns go through the same check, so a compromised director
  cannot aim the harness at the operator's own network.
- Bound --runs, --rounds, runs, --gap, and --hold as whole numbers, so
  'Infinity' exits 2 instead of looping forever against the relay.
- Report a region as UNREACHABLE when every probe fails, rather than letting
  Math.min([]) spread into NaN and read as ok.
- Bound the director /v1/resolve and /v1/regions fetches and report timeouts.
- Return null openMs when the socket never opened, and clear dial, cell, and
  RPC timers on the first terminal event so Node exits promptly.
- Default handle.rpc() to RPC_TIMEOUT_MS, not DIAL_TIMEOUT_MS.
- Write the state file through a helper that creates the parent directory,
  refuses a symlink, and forces 0600 on an existing file; refuse to read one
  that is readable beyond the operator.
- Read the pairing link from stdin or a 0600 file, never argv.
- Reject missing and invalid positionals with usage and exit 2.

Adds unit tests for the pure guards: argument parsing, bounded integers,
port and origin classification, DNS vetting, state-file modes and symlinks,
region verdicts, and pairing-link decoding. None opens a socket, and every
network path stays gated on ORCA_RELAY_BENCH_LIVE=1.

* tools(relay-bench): settle in-flight rpcs and guard an empty region catalog

Follow-up to the review fixes. Clearing a pending rpc timer without a
resolution swapped a 15 s timeout for an await that never returns, so the
teardown paths now settle each waiter with a closed result. A director that
answers /v1/regions with no regions now reports that and exits 1 instead of
printing an empty round.

* tools(relay-bench): attach the origin-vetting doc to the function it describes

* fix(tools): resolve a director-named cell through DNS and fail cdp-eval clearly
2026-09-07 13:28:47 -04:00

210 lines
11 KiB
Markdown
Raw Permalink Blame History

# relay-bench
Measures how long a phone takes to reach a usable connection with a desktop over the production
relay, without building or instrumenting the mobile app.
`relay-phone-connect-bench.mjs` replays the shipped mobile wire sequence: the relay auth frame,
the E2EE v2 handshake with the same transcript encoding and HKDF key schedule the app uses, then
the RPCs the phone issues before it publishes `connected`. Because it is the real sequence against
a real desktop, the per-phase numbers attribute latency to a specific hop rather than to "connect".
The handshake itself lives in `phone-e2ee-v2-session.mjs`, a plain-JS port of the mobile client
session so it runs outside the React Native bundle.
`phone-e2ee-desktop-parity.test.mjs` pins that port to the desktop responder
in `src/main/runtime/rpc/mobile-e2ee-v2-desktop-session.ts`. It runs in the normal unit suite, so
a change to the transcript encoding, key schedule, or frame layout fails there instead of leaving
a bench that quietly measures a handshake nobody ships. The four other `*.test.mjs` files in this
directory cover the invocation guards, the state file, the region verdicts, and pairing-link
decoding, and none of them opens a socket.
## Security rules
- The pairing link contains a live invite token and a device token. Treat it as a credential. `pair`
reads it from stdin, or from a file named by `--pairing-url-file`, so it never reaches your shell
history or the process argument list. Passing it as an argument is refused.
- `state.json` holds the resume token and device token for a real paired desktop. Never commit it,
paste it, or attach it to an issue. The `.gitignore` in this directory blocks `*.json` and
`state*`, but do not rely on that alone.
- Revoke the bench device when you are done. See "Cleaning up" below.
- Do not point the bench at a desktop you do not own.
No script here has a production default. Every one of them refuses to open a socket unless
`ORCA_RELAY_BENCH_LIVE=1` is set, and the two that talk to the director require its origin from
`--director=<origin>` or `ORCA_RELAY_BENCH_DIRECTOR`. Without those, they print usage and exit 2.
That keeps an accidental or automated invocation inert instead of live traffic.
The guards are in `relay-bench-invocation.mjs` and `relay-bench-state-file.mjs`, and
`relay-bench-invocation.test.mjs` / `relay-bench-state-file.test.mjs` pin them:
| Guard | What it stops |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| https-only origins | An `http:` director or cell, where an on-path observer reads bench credentials |
| Public-destination check | A director aiming the harness at your loopback, link-local, or private network, by literal address or by a name that resolves there |
| Bounded integer arguments | `--runs=Infinity` and friends, which loop forever and generate relay traffic |
| `0600` state file | An existing state file staying group- or world-readable, or being a symlink |
A director you name also _supplies_ URLs: the region catalog's probe origins and the cell URL from
`/v1/resolve`. Those go through the same public-https check as an origin you typed, so a compromised
or spoofed director cannot turn the harness into a probe of your own network. Region entries whose
probe origins are all refused report `REFUSED (no allowed probe origin)` rather than being sampled.
Hostnames are also resolved and checked, which narrows but does not close the DNS rebinding window,
because `fetch()` resolves again.
State-file handling creates the parent directory before writing, refuses a symlink, and forces
`0600` on an existing file. The first of those matters most: `pair` writes only after the desktop
has already provisioned the resume credential, so a failed write loses it.
## Requirements
`ws` and `tweetnacl` resolve from the repo root `node_modules`. Measured against `ws` 8.21.3 and
`tweetnacl` 1.0.3. Run every command from the repo root.
Syntax check after editing:
```bash
for f in tests/tools/relay-bench/*.mjs; do node --check "$f"; done
npx vitest run --config config/vitest.config.ts tests/tools/relay-bench
```
## Getting a pairing link
Start a relay-enabled dev app hidden, with remote debugging on:
```bash
ORCA_BACKGROUND_LAUNCH=1 \
REMOTE_DEBUGGING_PORT=9222 \
ORCA_CLOUD_API_URL=https://login.onorca.dev \
ORCA_CLOUD_CLIENT_ID=orca-desktop \
ORCA_DEV_USER_DATA_PATH=/tmp/orca-relay-bench-profile \
ORCA_RELAY_REGION_OVERRIDE=us-central1 \
pnpm run dev
```
`ORCA_DEV_USER_DATA_PATH` keeps the bench pairing out of your real profile.
`ORCA_RELAY_REGION_OVERRIDE` pins the cell region, which is what you want when comparing a change
rather than comparing regions. Both are optional.
Sign in, then read the pairing offer out of the hidden renderer:
```bash
node tests/tools/relay-bench/cdp-eval.mjs 9222 'window.api.mobile.getPairingQR({})'
```
The `orca://pair?code=...` value in that output is the pairing link.
## Commands
```bash
export ORCA_RELAY_BENCH_LIVE=1
BENCH=tests/tools/relay-bench/relay-phone-connect-bench.mjs
# One-time: dial the invite, provision a resume credential, save the bundle. The pairing link
# comes in on stdin so it stays out of your shell history and out of `ps`.
pbpaste | node $BENCH pair /tmp/relay-bench/state.json
# Or from a file you protect yourself, which `pair` requires to be mode 0600:
umask 077 && printf '%s' '<orca://pair?code=...>' > /tmp/relay-bench/pair.txt
node $BENCH pair /tmp/relay-bench/state.json --pairing-url-file=/tmp/relay-bench/pair.txt
rm /tmp/relay-bench/pair.txt
# Steady-state foreground reconnect, 10 times, 2 s apart, re-resolving the cell each time.
node $BENCH run /tmp/relay-bench/state.json 10 --resolve --gap=2000
# Resume after background: connect, idle 45 s, then probe the retained socket.
node $BENCH foreground /tmp/relay-bench/state.json --hold=45000
# Same, but crossing the relay's ~105 s client silence watchdog.
node $BENCH foreground /tmp/relay-bench/state.json --hold=120000
```
On Linux or Windows, replace `pbpaste` with whatever prints the link to stdout, or use
`--pairing-url-file`. Every count and duration is a whole number: `runs` and `--rounds` are 1-1000,
`--gap` and `--hold` are 0-3600000 ms, and anything else exits 2 rather than running unbounded.
The bench reads the director and cell for a resume dial out of `state.json`, which the pairing
offer supplied, so it takes no `--director`.
`run` prints one JSON row per iteration plus a `SUMMARY` line with medians.
`foreground` prints a single JSON row. Flags:
| Flag | Default | Meaning |
| ---------------- | ------- | -------------------------------------------------------------- |
| `--hold=ms` | `45000` | Idle time with no application traffic after reaching connected |
| `--force-redial` | off | Redial even when the retained socket answered |
| `--resolve` | off | Re-resolve the cell through the director before each dial |
It adds two fields to the per-phase shape. `retainedAnswerMs` is how long the held-open socket took
to answer `status.get`, or `null` if it could not. `redialMs` is the wall clock for a full resume
redial through the same connected sequence, measured on failure or with `--force-redial`.
`closedDuringHold` carries the close code if the relay dropped the socket while it was idle.
Note that the WebSocket library answers protocol-level pings automatically, exactly as the phone's
socket does. The silence watchdog counts application traffic, not pongs.
Two supporting scripts:
- `relay-hop-latency.mjs --cell=<origin> --director=<origin> [--host=<relayHostId>] [--runs=N]`
measures the infrastructure floor with a throwaway credential: director `/v1/resolve` plus cell
WebSocket open to `relay-hello`. It needs no pairing, because a cell answers a bogus credential
without reaching a desktop. `--host` defaults to an id no desktop owns. `openMs` is `null` when
the socket never opened, and a director that stalls is reported as a resolve timeout rather than
hanging the run loop.
- `region-probe-replay.mjs --director=<origin> [--rounds=N]` replays the desktop's region
selection with the same probe, sample count, and spread rule, and prints why each region passed
or failed. A region whose every probe fails reports `UNREACHABLE`, not `ok`.
Both take the director from `--director` or `ORCA_RELAY_BENCH_DIRECTOR`, and both need
`ORCA_RELAY_BENCH_LIVE=1`:
```bash
ORCA_RELAY_BENCH_LIVE=1 ORCA_RELAY_BENCH_DIRECTOR=<director origin> \
node tests/tools/relay-bench/region-probe-replay.mjs --rounds=3
```
## What each phase means
| Phase | Measures |
| ------------------- | ------------------------------------------------------------------------------------ |
| `wsOpen` | DNS, TCP, and TLS to the cell, up to the WebSocket upgrade |
| `relayHello` | Cell-side credential validation and the desktop-side attach, ending at `relay-hello` |
| `e2eeReady` | Desktop's `e2ee_ready`, so one relay round trip plus the desktop's key generation |
| `e2eeAuthenticated` | Device-token check on the desktop, ending the handshake |
| `confirm` | `pairing.getEndpoints` with the resume confirm id, which settles the credential |
| `capabilities` | The client capability advisory the phone sends before publishing connected |
| `status.get` | The first RPC the UI gate blocks on |
| `worktree.ps` | The worktree catalog, and the largest payload in the sequence |
| `session.tabs.list` | Per-worktree tab list for the first worktree |
| `terminal.list` | Per-worktree terminal list for the first worktree |
`totalToConnectedMs` is `e2eeAuthenticated` plus `confirm` plus `capabilities`.
`totalToFirstTerminalListMs` is the whole sequence.
## Reference numbers
Measured 2026-09-07 from a US-East vantage, same desktop and identical sequence, differing only in
which cell region served the connection. The vantage matters: these are not what a phone next to
the desktop would see.
| Cell region | To connected | `relayHello` | `confirm` |
| ----------- | ------------ | ------------ | --------- |
| Asia | 10.5 s | 5.8 s | 3.4 s |
| US | 0.63 s | 0.29 s | 0.14 s |
## Cleaning up
Revoke the bench device from the desktop that granted it:
```bash
node tests/tools/relay-bench/cdp-eval.mjs 9222 'window.api.mobile.revokeDevice({ deviceId: "<id>" })'
```
If you do not know the id, list the paired devices first:
```bash
node tests/tools/relay-bench/cdp-eval.mjs 9222 'window.api.mobile.listDevices()'
```
Then delete `state.json`. If you used
`ORCA_DEV_USER_DATA_PATH`, removing that directory drops the pairing with it.
Reference in New Issue View Git Blame Copy Permalink