Commit Graph
44 Commits
Author SHA1 Message Date
Neil 5df115db15 fix(relay): runtime edge control on the director, safer skipped exports, faster evacuation (#39773)
- Directors with an edge read director/edge.json from the relay-control bucket through the shared
  flag channel: mode on/off/down and rolloutPermille (0..1000). Boot defaults are mode on and the
  env percent; every cohort decision reads the applied flags. Off empties the cohort and hosts
  leave normally; down also skips the export at once (reason mode_down). The permille bucket keeps
  every host of P percent in at P x 10 permille. A control change re-arms the evacuation sweep
  within 5 s.
- A skipped export (120 s outage clock or mode down) removes the host's relay_devices rows before
  placing it, so revokes made on the edge are never undone; logs a count only.
- Evacuation: 100 drains per tick with 10 in flight, each pass starting at a random point of the
  keys and wrapping around; pages stay 500 rows and an idle pass reads each edge row once.
- Fence retries: 56 attempts (last about 25.5 h after the failure), 50 per sweep with 10 in
  flight; fence_abandoned now carries outcome, cell and epoch like fence_failed.
2026-11-07 23:38:33 -08:00
Neil 340c917429 relay: edge export skip on outage, cohort-exit drain sweep, region-scoped account allow (#39707)
* relay: edge export skip on sustained outage, evacuation drain sweep, account allow scoped by region

- Leaving hosts skip the edge export once it has been unavailable for 120 s
  (ORCA_RELAY_EDGE_EXPORT_SKIP_AFTER_MS), keeping Postgres devices and
  fencing best effort, instead of 503ing until the edge is back.
- Directors drain edge hosts that left the cohort (POST /__admin/drain,
  20 per jittered 60 s tick) so connected desktops re-ask and move off.
- Account-allowlisted hosts must also be in the edge regions.
- Tests for the invite-socket edge branch.

* relay: export skip needs continuous failures; evacuation sweep runs soon after start, then every 10 min

Failures more than 30 s apart restart the unavailable clock, so one old failure cannot skip an
export. The sweep reads the edge rows (cell_id has no index): first pass 45-75 s after start (a
cohort change is a new revision), every 45-75 s while hosts are queued, else every 7.5-12.5 min.
2026-11-07 20:09:41 -08:00
Neil 568567f7ad relay: director edge cohort for the relay edge (dark by default) (#39362)
* feat(relay): director edge override for moving hosts to the relay edge (dark by default)

Directors with ORCA_RELAY_EDGE_CELL_URL seed an existing-only 'edge' cell (registered
with the admission selector so deploy workflows carry it) and answer a host cohort
(hash bucket, allow/deny lists, region scope) on the edge before the reserve and
database lanes. The epoch rises only on a cell change; a host still holding units on
its old cell is told to retry. Phone resolve and the invite move answer edge rows
without moving anything; old cells answer WRONG_CELL to phones of moved hosts.
Unset URL keeps today's behaviour.

* feat(relay): director pushes host data to the relay edge on a move and takes it back before any move away

With ORCA_RELAY_EDGE_ADMIN_SIGNING_KEY set (its own key, never the assignment lease key), a move
onto the edge pushes the host's relay_devices and a location hint under the row lock and commits
only after the edge accepts; an edge holding a newer epoch is re-minted above once. Before any
lane may re-place an edge row, the director exports (refused while the desktop is connected; the
edge freezes the host), replaces the host's Postgres devices, places it as today, then fences the
edge with the new cell and epoch. Idle regional rehome skips edge rows. Unset key: unchanged.

* fix(relay): a deferred edge move keeps the host on today's path instead of a 503 loop

A host still holding old-cell units (a pairing invite holds one for its 10 min TTL) or whose push
the edge did not take now stays where it is and moves on a later ask. 503 stays for a held row
lock and for a desktop still connected to the edge on the way out.

* relay: director refuses an edge URL without the edge admin key

The deployed edge keeps host data in its objects; without the key the director could
not push it and hosts would land on an edge holding none of their devices.

* relay: edge cohort by account (ORCA_RELAY_EDGE_ALLOW_USERS)

Every host of a listed account (relay token sub) joins the edge cohort, in any region;
a denied host still stays off. For the team's own accounts before any share rollout.

* relay: edge modules follow main's relay module moves

* relay: satisfy the changed-lines lint gate in the edge cohort code

Braces on every if, T[] arrays, type aliases, one ternary, and a justified max-lines header on
edge-relay.ts. No behaviour change: relay unit 845 passed.

* relay: move edge cohort config and row helpers to edge-relay-cohort.ts

Keeps edge-relay.ts under the 300-line limit without a max-lines disable. Code moved verbatim;
edge-relay.ts re-exports what other modules import. Relay unit 845 passed.
2026-11-07 18:40:06 -08:00
Neil 46fa97a6b4 refactor(relay): group route, forwarding and connection owners (#38374)
* refactor(relay): group route, forwarding and connection owners

* refactor(relay): use explicit defining-owner type imports
2026-11-04 21:59:50 -08:00
Neil e74b40bd4e fix(relay): dead-man boot grace for a freshly booted reserve cell (#37225)
* fix(relay): give a freshly booted reserve cell a dead-man boot grace until its first director contact

A reserve cell's dead-man starts its 60-120 s window when the switch file applies, but no
placing director can reach a new process until its load balancer routes to it. On the
2026-11-01 production same-cap rolls that took up to about 110 s after the switch file
applied; c10 (104 s window) and c16 (71 s) tripped 1-4 s before the first director poll
landed, latched into db, and failed their roll after a clean drain and restart.

Until the process's first contact (a reserver=1 feed poll or a real booking), the window is
the hashed window plus 180 s, matching autohealing's initial delay. After the first contact
nothing changes. A process that never hears from a placing director still falls back to db,
up to three minutes later than before.

* test(relay): assert the server wires the dead-man boot grace

* docs(relay): state the boot gap behind the dead-man grace as inferred, not measured
2026-11-01 20:15:37 -08:00
Neil 6dcbcd6be5 refactor(cloud): colocate relay assignment store and tests (#37098) 2026-11-01 10:24:10 -08:00
OrcaWinandOrcaWin 7d6603aa17 [comment audit] Complete repository-wide comment cleanup (#33249)
* [comment audit] Align legacy failure-read counts with cached display

* [comment audit] Condense shared provider lifecycle and browser recovery comments [skip ci]

* [comment audit] Condense shared generation, path and preview comments [skip ci]

* [comment audit] Condense host, file and Git contract rationale [skip ci]

* [comment audit] Tighten Git lifecycle and provider contract comments [skip ci]

* [comment audit] Clarify shared host authority and launch isolation [skip ci]

* [comment audit] simplify local access mobile and native chat contracts [skip ci]

* [comment audit] simplify native chat presentation and state contracts [skip ci]

* [comment audit] simplify native runtime and provider contracts [skip ci]

* [comment audit] simplify migration and orchestration contracts [skip ci]

* [comment audit] simplify plugin trust and process capture rationale [skip ci]

* [comment audit] simplify terminal delivery and remote compatibility rationale [skip ci]

* [comment audit] clarify remote recovery and launch wire rationale [skip ci]

* [comment audit] shorten shared RPC compatibility and runtime evidence comments [skip ci]

* [comment audit] simplify secure storage shell startup and sidebar rationale [skip ci]

* [comment audit] clarify sidebar, skill and SSH invariants [skip ci]

* [comment audit] condense structured chat delivery and timing rationale [skip ci]

* [comment audit] condense terminal replay and telemetry invariants [skip ci]

* [comment audit] clarify launch permission and terminal delivery rationale [skip ci]

* [comment audit] clarify workspace ownership and layout contracts [skip ci]

* [comment audit] wait for fitted and painted SSH terminal links

* [comment audit] clarify layout persistence and session ownership [skip ci]

* [comment audit] clarify host routing and workspace lifecycle contracts [skip ci]

* [comment audit] clarify startup persistence and app lifetime comments [skip ci]

* [comment audit] clarify recovery and activity boundaries [skip ci]

* [comment audit] condense activity and artifact invariants [skip ci]

* [comment audit] condense automation authority contracts [skip ci]

* [comment audit] simplify automation scheduling and history contracts [skip ci]

* [comment audit] clarify browser chrome and annotation lifecycles [skip ci]

* [comment audit] clarify browser routing and retained guest ownership [skip ci]

* [comment audit] clarify browser recovery and document ownership comments [skip ci]

* [comment audit] restore hosted runners required by native IME fixtures

* [comment audit] shorten preview, palette and dashboard invariants [skip ci]

* [comment audit] clarify terminal preview, dashboard and retention comments [skip ci]

* [comment audit] clarify diff-note recovery and editor lifecycle comments [skip ci]

* [comment audit] shorten rich-text and combined-diff ownership comments [skip ci]

* [comment audit] shorten autosave, draft recovery and model ownership comments [skip ci]

* [comment audit] shorten save-queue, notebook and Markdown constraints [skip ci]

* [comment audit] shorten Monaco, PDF and rich Markdown safeguards [skip ci]

* [comment audit] clarify rich Markdown source and selection safeguards [skip ci]

* [comment audit] clarify editor loading and emulator stream lifetimes [skip ci]

* [comment audit] trim emulator and feature tour narration [skip ci]

* [comment audit] clarify floating workspace and GitHub cache ownership [skip ci]

* [comment audit] trim provider controls and Linear filter explanations [skip ci]

* [comment audit] clarify pairing and chat resume lifetimes [skip ci]

* [comment audit] clarify chat delivery and transcript lifetimes [skip ci]

* [comment audit] clarify session options and host-owned status [skip ci]

* [comment audit] clarify draft durability and live-status precedence [skip ci]

* [comment audit] Trim chat send and model ownership explanations [skip ci]

* [comment audit] Shorten transcript ownership and send recovery comments [skip ci]

* [comment audit] Simplify native chat composer and transcript hook comments [skip ci]

* [comment audit] Shorten native chat scroll and structured session explanations [skip ci]

* [comment audit] Trim workspace picker and onboarding commentary [skip ci]

* [comment audit] Simplify pet rendering, review cache and quick open comments [skip ci]

* [comment audit] Trim sidebar and session history explanations [skip ci]

* [comment audit] Condense session resume and review checks comments [skip ci]

* [comment audit] Shorten checks state and file explorer refresh comments [skip ci]

* [comment audit] Trim plugin bridge and source-control action rationale [skip ci]

* [comment audit] Trim source-control listing and review rationale [skip ci]

* [comment audit] Shorten sidebar lifecycle and explorer safeguards [skip ci]

* [comment audit] Condense Settings account and appearance rationale [skip ci]

* [comment audit] Trim pairing and repository Settings explanations [skip ci]

* [comment audit] Condense Settings search SSH and terminal safeguards [skip ci]

* [comment audit] Shorten Settings voice and provider ownership comments [skip ci]

* [comment audit] Condense Settings draft persistence and host-state rationale [skip ci]

* [comment audit] Trim Settings hooks and sidebar setup rationale [skip ci]

* [comment audit] Condense sidebar deletion and board-filter safeguards [skip ci]

* [comment audit] Condense worktree cards, host ownership and Markdown contracts [skip ci]

* [comment audit] Condense sidebar drag, folder and filtering contracts [skip ci]

* [comment audit] Condense sidebar parity, sleep and host ownership contracts [skip ci]

* [comment audit] Prune focus-read baseline after comment cleanup

* [comment audit] Clarify sidebar visibility, board drops and agent evidence [skip ci]

* [comment audit] Tighten sidebar listing, drag and reveal contracts [skip ci]

* [comment audit] Clarify viewport geometry and skill update safeguards [skip ci]

* [comment audit] Tighten skill inventory, usage and host status comments [skip ci]

* [comment audit] Clarify usage ownership and resource kill safeguards [skip ci]

* [comment audit] Tighten tab selection, rename and search contracts [skip ci]

* [comment audit] Tighten tab-strip geometry and drag ownership comments [skip ci]

* [comment audit] Tighten Tasks recovery, pagination and mutation comments [skip ci]

* [comment audit] Tighten provider loading and terminal recovery comments [skip ci]

* [comment audit] Tighten completion, cadence and PTY ownership comments [skip ci]

* [comment audit] Clarify terminal restore and parked-pane boundaries [skip ci]

* [comment audit] Tighten PTY startup and replay explanations [skip ci]

* [comment audit] Clarify PTY identity and delivery ownership [skip ci]

* [comment audit] Shorten remote reconnect and input replay rationale [skip ci]

* [comment audit] Condense terminal parking clipboard and file link comments [skip ci]

* [comment audit] Clarify IME ownership and trim trace history [skip ci]

* [comment audit] Shorten pane recovery and parked watcher explanations [skip ci]

* [comment audit] Condense snapshot replay and visibility recovery rationale [skip ci]

* [comment audit] Shorten terminal parking and keyboard ownership rationale [skip ci]

* [comment audit] Trim renderer lifecycle and menu rationale [skip ci]

* [comment audit] Condense workspace cleanup ownership and settlement notes [skip ci]

* [comment audit] Clarify renderer hook evidence and source ownership [skip ci]

* [comment audit] Tighten IPC ownership and reconnect explanations [skip ci]

* [comment audit] Trim renderer focus, paste and attention comments [skip ci]

* [comment audit] Shorten scroll, locale and agent lifecycle comments [skip ci]

* [comment audit] Trim agent launch, resume and ownership commentary [skip ci]

* [comment audit] Shorten account, browser and editor comments [skip ci]

* [comment audit] Clarify host-mirror waits, previews and input ownership [skip ci]

* [comment audit] Condense launch recovery and editor tokenization comments [skip ci]

* [comment audit] Condense workspace selection and terminal pane rationale [skip ci]

* [comment audit] Condense terminal fitting and rendering recovery comments [skip ci]

* [comment audit] Condense terminal input and host routing explanations [skip ci]

* [comment audit] Condense ownership and sleeping-session recovery rationale [skip ci]

* [comment audit] Condense structured launch and terminal link explanations [skip ci]

* [comment audit] Condense terminal ownership and workspace file routing rationale [skip ci]

* [comment audit] Condense session persistence and workspace activation rationale [skip ci]

* [comment audit] Condense workspace routing and browser ownership rationale [skip ci]

* [comment audit] Condense mirror recovery and layout authority rationale [skip ci]

* [comment audit] Condense runtime transport and file safety rationale [skip ci]

* [comment audit] Condense runtime graph and structured session rationale [skip ci]

* [comment audit] Condense browser creation and remote tab recovery rationale [skip ci]

* [comment audit] Condense snapshot authority and publisher recovery rationale [skip ci]

* [comment audit] Condense frame, startup and provider cache rationale [skip ci]

* [comment audit] Condense host twin and agent lifecycle rationale [skip ci]

* [comment audit] Condense browser authority and editor cleanup rationale [skip ci]

* [comment audit] Condense editor buffer identity and review cache rationale [skip ci]

* [comment audit] Condense provider lifetime and repository ownership rationale [skip ci]

* [comment audit] Condense reconnect, hydration and terminal retirement rationale [skip ci]

* [comment audit] Condense recovery, reconciliation and persistence rationale [skip ci]

* [comment audit] Condense cleanup ownership and listing admission rationale [skip ci]

* [comment audit] Condense teardown ordering and terminal ownership rationale [skip ci]

* [comment audit] Condense recovery ledger and web host-boundary rationale [skip ci]

* [comment audit] Condense web reconnect and credential lifecycle rationale [skip ci]

* [comment audit] Condense cloud rollout and assignment safety rationale [skip ci]

* [comment audit] Condense relay session and database recovery rationale [skip ci]

* [comment audit] Condense relay placement and protocol compatibility rationale [skip ci]

* [comment audit] Condense build and packaging constraints [skip ci]

* [comment audit] Condense CI gates and validation isolation rationale [skip ci]

* [comment audit] Condense catalog identity and browser validation rationale [skip ci]

* [comment audit] Condense mobile browser and frame-budget rationale [skip ci]

* [comment audit] Shorten mobile route and browser test explanations [skip ci]

* [comment audit] Condense editor and session seam test rationale [skip ci]

* [comment audit] Trim session and terminal closure history [skip ci]

* [comment audit] Shorten terminal browser and census explanations [skip ci]

* [comment audit] Preserve root-only E2E diagnostic baseline

* [comment audit] Condense mobile seam and HTML preview test rationale [skip ci]

* [comment audit] Condense preview instrumentation and render harness rationale [skip ci]

* [comment audit] Condense bundle contracts and native-build constraints [skip ci]

* [comment audit] Shorten native packaging and CI routing rationale [skip ci]

* [comment audit] Condense release tooling and benchmark constraints [skip ci]

* [comment audit] Condense packaging and platform gate rationale [skip ci]

* [comment audit] Shorten mobile route and history rationale [skip ci]

* [comment audit] Condense mobile browser and component rationale [skip ci]

* [comment audit] Shorten drawer and workspace-form rationale [skip ci]

* [comment audit] Trim mobile review and editor-document explanations [skip ci]

* [comment audit] Condense editor lifecycle and Markdown explanations [skip ci]

* [comment audit] Trim mobile file, diagnostics and dictation commentary [skip ci]

* [comment audit] Simplify mobile host-list and workspace-view explanations [skip ci]

* [comment audit] Clarify mobile shell lifetime and bridge delivery constraints [skip ci]

* [comment audit] Condense bridge stream, native request and audio rationale [skip ci]

* [comment audit] Reduce bridge schema and page-client commentary [skip ci]

* [comment audit] Tighten mobile page bridge contracts [skip ci]

* [comment audit] Clarify mobile cache and session constraints [skip ci]

* [comment audit] Condense mobile route and lifecycle rationale [skip ci]

* [comment audit] Tighten mobile shell hook timing rationale [skip ci]

* [comment audit] Clarify route handoff and notification compatibility [skip ci]

* [comment audit] Clarify push registration and dictation cleanup constraints [skip ci]

* [comment audit] Condense keyboard geometry and native media ownership rationale [skip ci]

* [comment audit] Tighten mobile chat scroll and composer concurrency rationale [skip ci]

* [comment audit] Condense mobile reply compatibility and upload ownership rationale [skip ci]

* [comment audit] Condense optimistic echo and scroll ownership rationale [skip ci]

* [comment audit] Tighten stream replay and queued session ownership rationale [skip ci]

* [comment audit] Condense mobile composer and reply-schema constraints [skip ci]

* [comment audit] Clarify mobile transcript recovery and session ownership [skip ci]

* [comment audit] Clarify mobile queue and source-control compatibility [skip ci]

* [comment audit] Condense storage and task launch constraints [skip ci]

* [comment audit] Clarify task schemas and terminal document ownership [skip ci]

* [comment audit] Tighten terminal input and viewport rationale [skip ci]

* [comment audit] Shorten terminal and RPC recorder contracts [skip ci]

* [comment audit] Condense recorder engine and connection contracts [skip ci]

* [comment audit] Shorten host credential and relay recovery comments [skip ci]

* [comment audit] Tighten pairing recovery and bundle transport rationale [skip ci]

* [comment audit] Clarify RPC delivery and workspace catalog boundaries [skip ci]

* [comment audit] Condense catalog and relay entry contracts [skip ci]

* [comment audit] Shorten shutdown and SSH relay delivery contracts [skip ci]

* [comment audit] Condense WSL relay and E2E regression rationale [skip ci]

* [comment audit] Shorten cross-version harness and renderer fixture explanations [skip ci]

* [comment audit] Condense paired-host and E2E lifecycle rationale [skip ci]

* [comment audit] Shorten layout oracle and regression explanations [skip ci]

* [comment audit] Condense paired recovery and ownership regression contracts [skip ci]

* [comment audit] Shorten restart, sidebar and SSH lifecycle contracts [skip ci]

* [comment audit] Condense terminal rendering and IME regression rationale [skip ci]

* [comment audit] Shorten terminal recovery and benchmark contracts [skip ci]

* [comment audit] Condense tooling and Windows lifecycle proof rationale [skip ci]

* [comment audit] Condense workflow routing and release rationale [skip ci]

* [comment audit] Condense mobile security and installer contracts [skip ci]

* [comment audit] Shorten native rationale and stylesheet workarounds [skip ci]

* [comment audit] Shorten Windows visibility probe rationale [skip ci]

* [comment audit] Condense embedded release and signing rationale [skip ci]

* [comment audit] Retain current main test behavior in final reconciliation [skip ci]

* [comment audit] Complete final comment review across current source inventory

* [comment audit] Handle consolidated audit reports before final CI

* [comment audit] Preserve credential storage census detail [skip ci]

* [comment audit] Scale broad changed E2E selections with verified timing shards

* [comment audit] Align E2E assertions with browser close and settled session metadata

* [comment audit] Match existing E2Es to current resource and windowing contracts

* [comment audit] Observe the shared server outage notice in reconnect coverage

* [comment audit] Make existing renderer probes measure settled tab and PTY work

* [comment audit] Align palette E2E selector with task search

* [comment audit] Refresh remaining palette search selectors

* [comment audit] Keep selector correction within selected audit specs

* [comment audit] Await emoji fixture settings inside renderer evaluation

* [comment audit] Cancel fixture fit retries between terminal tests

---------

Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
2026-10-25 04:05:46 -07:00
Jinwoo Hong 0ae42fe594 feat(relay): cap how many reserve cells a director outage trips to the database (#33415)
<!-- orca-pr-loc -->
<!-- Programmatic LoC summary. Do not edit by hand; rewritten on every commit. -->

| | Files | Added | Deleted | Net |
| :--- | ---: | ---: | ---: | ---: |
| Test | 4 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​406 | $\color{#cf222e}{\Huge{\mathbf{−}}}$​4 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​402 |
| Prod | 6 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​165 | $\color{#cf222e}{\Huge{\mathbf{−}}}$​10 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​155 |

<!-- /orca-pr-loc -->

## ELI5

Relay servers ("cells") normally get their seat bookings from the front door (the director). If the director goes quiet for a minute or two, each cell switches itself to an older mode that talks to the shared Postgres database for every connection (the "dead-man" trip). Today every cell switches at once, so the database budget must keep room for every cell's full connection pool. This PR adds a per-cell switch that caps how many cells may trip at the same time. A cell that cannot trip yet stays in its normal mode and keeps serving the desktops it already has.

## What Changed

- New cell flag `deadManTripSlots` (integer 0-64). Absent, which is every cell today, keeps the current behaviour: trip as soon as the dead-man window (60-120 s) passes.
- With the flag set, a cell whose window has passed first claims a slot in Postgres. The claim is one short transaction: it takes an advisory lock without waiting, counts the other reserve cells whose own `relay_cell_admit_effective` row says `db`, and writes this cell's row to `db` only if that count is below the flag. Only then does the cell trip (void bookings, re-register its hosts, run on its full pool).
- A cell that gets no slot is "held". It stays in reserve mode, logs `orca_relay_cell_reserve_dead_man_held` once, keeps writing `reserve` to its row, and asks again every 30 s.
- A slot is held while the holder's own row says `db`, its switch still says `reserve`, and the row was refreshed in the last 10 minutes. A holder that restarts or is re-armed writes `reserve` and frees its slot. One that died tripped frees it after 10 minutes. Heartbeats are not used: they reach Postgres through the director, so in a director outage they all go stale and every slot would read as free.
- A tripped cell skips its 15 s row refresh while its pool has waiters. So that a long brownout cannot age its row out of the count, a tripped cell's row older than 5 minutes is refreshed even when the pool is busy.
- A trip stays latched after the director returns, as today. The tripped cells keep their slots until an operator re-arms them with a new reserve write. Until then, a second outage trips nobody new: the other cells stay held. That is the intended bound, but re-arming after an outage now also frees the slots.
- The claim goes through `CellAdmitEffectiveWriter`, which already serialises every write of that row. A `reserve` write still in flight therefore cannot land after the claim and hide the slot from the next claimant.
- Cells keep their existing per-cell window spread (60-120 s by cell-id hash), so the earliest windows claim first.

What a held cell does during a director outage (unchanged code paths, now for longer than the window):
- Desktops already connected keep their sockets. Nothing is closed or re-registered.
- A desktop that drops and redials at the same epoch is admitted from memory (`seatedHereAt`). Any other hello falls through to the normal database check, as it does in reserve mode today.
- The database placement path places no new desktops on it, because its row says `reserve`. It also never re-places the held cell's desktops, because `liveReserveCell` keys on the switch (`relay_cell_admit_modes`), not on this row.
- New placements during the outage go to cells already in database mode (c32, c33 and c34 today) plus the tripped slot holders.

## Why

The database budget (Cloud SQL `orca-cloud-auth-db`, shared with auth, max_connections 500, 490 usable) currently reserves every relay cell's full pool. The only reason for that is the dead-man trip of the whole fleet at once. Spare is 35 on main and about 3 once C36/C37 land, which caps the number of Asia cells.

Budget arithmetic (`relay-cloud-sql-connection-budget.mjs` on orca-cloud main: directors 15, auth 20, API 50, rollout overlap 65, maintenance 5, explicit reserve 10):

| | Cells line | Operating | Spare |
|---|---:|---:|---:|
| Main today: 6 db/legacy cells at 10, 16 US at 8, 7 Asia at 16 | 300 | 455 | 35 |
| Plus C36/C37 at 16 | 332 | 487 | 3 |
| Same fleet, `deadManTripSlots=2`, Asia reserve cells budgeted at 12 (measured peak) | 76 + 128 + 8x12 + 2x4 = 308 | 463 | 27 |
| Plus two more Asia cells at 12 | 332 | 487 | 3 |

So with two slots, two more Asia cells fit in the room C36/C37 leave today. Two caveats keep this honest:
- The bound holds only once every reserve cell runs this image with the same flag value. An older image ignores the flag and trips freely, though its `db` row still takes a slot from the capped cells. With mixed values the cap is the largest value set.
- Counting reserve cells at their measured peak is a softer guarantee than counting their configured pool: the pool is still 16, and a reserve cell could burst past 12. Making it a hard bound needs a pool ceiling that depends on the mode (reserve-mode ceiling 12, the full 16 only for a slot holder or a deliberate db cell). That is a follow-up; this PR does not change pool sizes.

Alternatives considered:
- Longer or ranked trip delays only spread the trips out. A trip is latched until the switch is rewritten, so after the longest delay every cell is still in database mode. That bounds the burst, not the budget.
- A static per-cell "may trip" list in the switch file needs no database coordination, but a listed cell that is down or draining wastes its slot, and an operator has to keep the list current.
- Removing the fallback (cells never trip) is the target design, but it waits on the director stall fix and two weeks of clean running. `deadManTripSlots=0` gives exactly that per cell, so it is the same switch.

Deliberate db flips (for example, flipping reserve cells to db for an image roll) are outside the cap. Budget each Asia cell flipped that way at its full 16. With a two-cell Asia roll batch running during an outage, the slot row above is 316, which leaves room for one more Asia cell rather than two.

Rollout: the flag is dark. Setting it is an `Operate Relay Production Cell Flags` write, per cell, after every reserve cell runs this image.

## Linked Issue

N/A (internal relay capacity work).

## Visual Proof

N/A: relay server behaviour only, no UI.

## Testing

- `cell-reserve-dead-man.test.ts` (new): trips as before without the flag; held until a permit, then latched; slot 0 never trips; a permit for a silence that contact ended, or for an earlier switch generation, is dropped.
- `cell-admit-effective-writer.test.ts`: the claim waits for an in-flight reserve write; a refused claim keeps the reserve row fresh and retries after 30 s; no claim while the pool has waiters or the switch is off reserve; a tripped row is refreshed past a saturated pool after 5 minutes, a reserve row is not.
- `cell-reserve-dead-man-slots.blackbox.test.ts` (new, real cell on SQLite): a director outage with the only slot taken by another cell that writes nothing but its own row: the cell stays reserve across several claims, re-registers nothing, keeps its desktops connected, and lets a redialing desktop back in from memory; once the peer writes `reserve`, the cell trips and re-registers. Also: slot 0 rides out 30 minutes of outage without tripping; no flag trips as before.
- `cell-dead-man-trip-slot-postgres.test.ts` (new, PostgreSQL): eight cells claiming together get exactly two slots; a live holder keeps its slot; a stale one frees exactly one. Removing the advisory lock fails this test.
- Mutation checks: making the slot count always 0, or removing the lock, each fails a test.
- Full relay suite, including the PostgreSQL suites against a local PostgreSQL 17: 173 files, 1564 passed, 3 skipped. The pr.yml static-analysis steps pass locally.
- Two rounds of adversarial review. Round 1 found that the first version counted holders by director-written heartbeats (fixed). Round 2 found the brownout aging (fixed) and the latched slots after an outage (documented above).

- [x] I manually tested these changes locally
- [x] Automated tests added/updated, or explained why not below

## Review

## Agent skill upstream boundary

- [x] Not applicable, or this change copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.

## Notes

No new table or column. The claim reads `relay_cell_admit_effective` and `relay_cell_admit_modes` (one row per cell). Under a fleet-wide trip, held cells add one short transaction each per 30 s; the advisory lock is held for about one round trip.

## Checklist

- [x] This PR is small and focused
- [x] I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
- [x] Before/after screenshots or videos attached for UI changes, or `N/A` with reason
- [x] Self-reviewed for correctness, security, and performance
- [x] Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
- [x] `pnpm lint`, `pnpm typecheck`, `pnpm test`, and `pnpm build` pass (or CI will cover; local preferred)
2026-10-25 01:46:23 -04:00
Jinwoo Hong e4d1c981a4 feat(relay): rehome reserve hosts into their own region on reconnect (off by default) (#32191)
Rehome on reconnect for reserve-mode cells: a desktop reconnecting on a reserve cell outside its preferred region is placed fresh in that region, only while the region has at least 10% of seats free, at most once a day per desktop, rate-limited, and switched off by default (ORCA_RELAY_RESERVE_REHOME_PERCENT=0); the regional rehome switch stops it.
2026-10-23 18:48:35 -04:00
Jinwoo Hong 07a07a0326 fix(relay): stop reviving a session outside the cell's map, keep the seat feed on live sessions, and log demote outcomes (#32149)
* fix(relay): repair cell seat-log drift, and keep a new session's generation above a listed seat

Directors mirror each cell's seat feed, so the feed must list exactly the cell's live sessions.
The cell now sweeps its seat log against its sessions on a director's poll: a seat no session
backs is left (4409) and the live session listed, and a live session the feed lost is listed
again, each once two sweeps in a row find it. A demotion naming an unbacked seat repairs it at
once. A host's new session now starts above any generation the feed still lists, which closes
the drained-session reconnect race that removed a live seat.

Claude-Session: asia-drain-pulse

* fix(relay): refuse a rebind whose session left the map during activation, and keep a per-host generation floor

A rebind whose database activation outlived the orphan grace (or a drain) revived the removed
session outside the map: connected, still in the seat feed, and unreachable by phones or
demotion for as long as the host stayed up. Activation now rechecks the map after its awaits and
refuses with PEER_DROPPED so the host redials. Generations come from a per-host floor that
outlives the session, and an activation at a generation already used here is refused.

Claude-Session: asia-drain-pulse

* fix(relay): log each demote outcome on the director by host digest

The director now reads the cell's existing {v:1, outcome} demote reply and logs it; the cell's
feed stays the only source of its map. The supersede line now reaches the log as well.

Claude-Session: asia-drain-pulse
2026-10-22 22:16:58 -04:00
Neil d667e54b02 refactor: colocate relay assignment admission owners (#32142) 2026-10-22 18:39:05 -07:00
Jinwoo Hong e49236f1ec feat(relay): count and log phone-path failures, and every attach deadline (#30656)
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode

Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.

* feat(relay): resume paired phones from one device read on memory cells

With phonePath=memory in reserve mode, a phone resume is admitted from one device
read and the cell's own seat, and takes no activity leases. A refused device closes
4401, a failed read 4503, and only a missing device row falls through to the invite
path. A phone whose host has no seat gets one assignment read: 4409 when the row
names another cell or the read fails, otherwise 4404.

* feat(relay): keep a memory resume's basis in the cell and confirm it from one read

A phone resumed on the memory path keeps its connection basis in the cell, not in
relay_connection_bases. Its confirm rereads the device row once, renews it only when
less than 29 days remain, and always answers a current-version confirm renewed:true
with the expiry the database holds. Confirms route by where their basis lives, so a
flip between accept and confirm completes either way. Retries answer from a 10-min
in-memory result map.

* test(relay): roll a memory phone cell back with confirms in flight

Flipping back by admitMode=db, phonePath=db or a dead-man trip leaves in-flight
confirms to finish on their memory basis, writes no phone lease and leaks none, and
evacuation still refuses the cell while Postgres records it as reserve.

* feat(relay): batch a memory cell's resume-confirmed audit rows

Memory confirms buffer their audit rows and the cell writes them as one multi-row
insert every 5 s or 500 rows, idempotent by a per-event id. The buffer holds at most
5,000 rows and drops a row older than 10 minutes rather than retrying it, counting
and logging both kinds of drop.

* feat(relay): count and log phone-path failures, and every attach deadline

The runtime metrics line gains phone refusals by close code, attach deadlines, accept
database errors, resume confirm failures by code, completed accepts by phone path and
the resume audit drop total. Every attach-deadline 4404 now logs what the cell knew of
the desktop's control (pong age, ping outstanding, control age and round trip), so a
half-open control can be told from a slow data dial.

* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes

An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.

* fix(relay): log a failed no-seat read, recheck the switch after the device read

A failed assignment read for a phone with no seat now logs a rate-limited line before
its 4409. A switch that turned off during the device read sends the phone down today's
path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy.

* fix(relay): log a failed memory renewal; test flips, replays and basis cleanup

A renewal write that fails still answers the stored expiry, but now logs a rate-limited
line so a persistently failing update is visible before phones age toward re-pairing.

* test(relay): assert a rolled-back memory splice issues no statement when it closes

* fix(relay): flush resume audit early only on crossing the batch size, drop the shutdown flush

A failing database no longer takes a 500-row insert per confirm; retries wait for the
5 s timer. The shutdown flush raced the pool closing, so it is gone and the at most 5 s
loss is documented. A stopped buffer no longer re-arms its timer.
2026-10-19 13:16:00 -04:00
Jinwoo Hong 1cf7472d1e feat(relay): batch a memory cell's resume-confirmed audit rows (#30655)
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode

Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.

* feat(relay): resume paired phones from one device read on memory cells

With phonePath=memory in reserve mode, a phone resume is admitted from one device
read and the cell's own seat, and takes no activity leases. A refused device closes
4401, a failed read 4503, and only a missing device row falls through to the invite
path. A phone whose host has no seat gets one assignment read: 4409 when the row
names another cell or the read fails, otherwise 4404.

* feat(relay): keep a memory resume's basis in the cell and confirm it from one read

A phone resumed on the memory path keeps its connection basis in the cell, not in
relay_connection_bases. Its confirm rereads the device row once, renews it only when
less than 29 days remain, and always answers a current-version confirm renewed:true
with the expiry the database holds. Confirms route by where their basis lives, so a
flip between accept and confirm completes either way. Retries answer from a 10-min
in-memory result map.

* test(relay): roll a memory phone cell back with confirms in flight

Flipping back by admitMode=db, phonePath=db or a dead-man trip leaves in-flight
confirms to finish on their memory basis, writes no phone lease and leaks none, and
evacuation still refuses the cell while Postgres records it as reserve.

* feat(relay): batch a memory cell's resume-confirmed audit rows

Memory confirms buffer their audit rows and the cell writes them as one multi-row
insert every 5 s or 500 rows, idempotent by a per-event id. The buffer holds at most
5,000 rows and drops a row older than 10 minutes rather than retrying it, counting
and logging both kinds of drop.

* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes

An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.

* fix(relay): log a failed no-seat read, recheck the switch after the device read

A failed assignment read for a phone with no seat now logs a rate-limited line before
its 4409. A switch that turned off during the device read sends the phone down today's
path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy.

* fix(relay): log a failed memory renewal; test flips, replays and basis cleanup

A renewal write that fails still answers the stored expiry, but now logs a rate-limited
line so a persistently failing update is visible before phones age toward re-pairing.

* test(relay): assert a rolled-back memory splice issues no statement when it closes

* fix(relay): flush resume audit early only on crossing the batch size, drop the shutdown flush

A failing database no longer takes a 500-row insert per confirm; retries wait for the
5 s timer. The shutdown flush raced the pool closing, so it is gone and the at most 5 s
loss is documented. A stopped buffer no longer re-arms its timer.
2026-10-19 06:37:35 -04:00
Jinwoo Hong a12c592f75 feat(relay): add the per-cell phonePath switch, effective only in reserve mode (#30629)
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode

Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.

* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes

An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
2026-10-19 05:38:03 -04:00
Jinwoo Hong 88accb2e4a feat(relay): log where the sticky lane's slot time goes, per pooled connection (#28908)
* feat(relay): log what the sticky slot spends its hold on, per pooled connection

* fix(relay): time every sticky round trip at the PostgreSQL client, pool wait apart

* docs(relay): say which round trips the sticky slot tally covers
2026-10-14 20:55:18 -04:00
Jinwoo Hong 2de7381204 fix(relay): supersede a reserve seat from the feed once a later, newer join shows (#28716)
* test(relay): check invariant 4 (one demotion winner) in the step-5 simulation

Every demotion now names its winner (the newer booking, or the row). The model
flags a director that names a different winner for a divergence another already
named, and a host whose two seats are each demoted for the other. A fault where
directors break level ties by their own cell order shows the check fires.
Violations are capped per invariant so one flood cannot hide another class.

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f

* test(relay): bound unseen duplicates by the silence close and assert all invariants on contested ties

A reserve seat the booking director's map had not seen (joined, then half-open within a poll)
is never superseded and has no lease to re-check, so it lasts until the cell's control silence
close (75 s + one 15 s ping). Invariant 3 now uses that bound for such seats, and the
level-divergence config asserts every invariant.

Stale maps now judge a divergence another director already demoted, so the config's
"contested" counter (divergences judged by two or more directors) is the case invariant 4
guards. The silence-close constant now matches production (90 s, not 100 s).

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f

* test(relay): raise the floor on map-judged demotes and count only level contests

A reconcile whose map has not seen another director's demotion still demotes in
production, which raises its epoch floor; the model now does too. The contested
counter counts only same-epoch pairs, the only ones invariant 4 can split. The
keep-this-seat rule is one helper.

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f

* refactor(relay): name the simulation's fault-kept seat check for the fault

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f

* fix(relay): supersede a reserve seat from the feed once a later, newer join shows

A half-open seat the booking director's map had not yet seen was never demoted and held its
cell until the 75 s control silence close. Directors now demote it once both joins are in the
map; the simulation holds invariant 3 at the supersede bound for it.

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f

* fix(relay): outrank only past cell clock skew, reuse demoteSeat, keep invariant 3's loose clock running

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f

* refactor(relay): share the outrank rule with the simulation, pin the skew boundary

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f

* fix(relay): count only confirmed level demotions and every second judgment, and bound duplicates by placement records alone

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f

* fix(relay): a drained seat keeps its drain grace under a demote, and the feed reports each seat once

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f

* refactor(relay): name the outranked-report bound and the reconcile backstop

Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
2026-10-14 17:06:33 -04:00
Jinwoo Hong 7ce15106f3 fix(relay): keep verifying host tokens from the last served keys while auth is down (#27490)
* fix(relay): keep verifying host tokens from the last served keys while auth is down

jose refetches the relay JWKS once its cache is 10 min old and rejects the token when that
fetch fails, so an auth outage turned every still-valid host token into a 4401. The cell now
falls back to the last set auth served for up to an hour, skips refetching for 30 s after a
failure, and still rejects any key that set never had. A fresh set replaces it as soon as auth
answers, so a rotation takes effect at the first successful fetch.

* fix(relay): bound stale relay keys at 30 min and document the emergency purge

Thirty minutes still covers a full auth outage (no token the stale keys could accept outlives
~16 min of one) and halves how long a rotated-out key stays trusted on a cell kept from auth.
A restarted process holds no set and refuses host tokens until a fetch succeeds; the ops doc
names that as the purge after a signing-key leak, and a test pins the fail-closed start.

* test(relay): pin the 30 min stale relay key bound (29 min accepts, 31 min rejects)
2026-10-11 14:25:29 -04:00
Jinwoo Hong 735e6d6c5d fix(relay): let a host leave a dead reserve cell without an operator fence (#27458)
* fix(relay): let a host leave a dead reserve cell without an operator fence

A host whose reserve home cell died got 503 relay_home_cell_unavailable until the
cell came back: the database path demands a committed fence for every capped cell.
On its own /v1/assign, a host now moves off a reserve cell (Postgres records it as
reserve) when the cell's heartbeat is stale and this director's seat-feed polls have
gone unanswered for 45 s, unless more than a third of the polled cells look the same.
Sweeps, migration pins and capped db cells keep the fence.

* fix(relay): guard the dead-cell verdict by region and sample size, and floor the move on the map

- No verdict when half or more of the dead cell's own region fails with it (a cross-region
  partition), more than a third of the fleet does, or fewer than 6 cells are polled. A withheld
  region or fleet verdict logs orca_relay_dead_cell_verdict_withheld, which alerts.
- The move needs the step-5 plan's epoch floor, so it mints above every seat the map knows;
  without the plan the fence is kept. A test revives the old cell's seat above the row and checks
  that the ledger never writes the row back.
- The move's log line carries the fleet and region counts.

* fix(relay): count only feed cells and young streaks in the dead-cell guards; gate the alert

- Denominators hold only cells that served this director a feed page and have capacity: legacy
  rows answering 404 no longer fill the 6-cell sample or dilute the fleet and region ratios.
- A peer counts as failing for the guards on any current streak (a failure in the last 5 s), so a
  staggered partition cannot let its first cell cross 45 s while its neighbours read healthy.
- Every withheld verdict is logged, sample included (a cell missing from the feed cells is one).
- Tests: ready=0 and enabled=0 with failing polls keep the fence (drain then death is fenced).
- The withheld metric and alert are gated by relay_dead_cell_verdict_alert (default false, on in
  staging); the alert docs now describe poll failures, not heartbeats.

* fix(relay): always sample the dead-cell candidate, so a restarted director can still judge it

A director that restarted after a reserve cell died never received a feed page from it, so the
cell was never in the sample and its verdict stayed withheld as 'sample'. The candidate now always
counts when it has capacity (the store asks only about a cell Postgres records as reserve, and only
feed images run reserve); peers still count only once they have served a feed. Tests: a restart
during a lone outage decides, a partition begun before the restart withholds by region, and the
capacity filter. The Terraform comments now say to apply the metric first and the policy minutes later.
2026-10-11 14:24:52 -04:00
Jinwoo Hong a18883130f fix(relay): database sweeps and ops reads skip reserve-mode cells (step-5 census, inert) (#27046)
Rebased onto main as one commit after its parent squash-merged.
2026-10-10 19:22:11 -04:00
Jinwoo Hong fda01facc2 feat(relay): the director that booked a join writes its row about a second later (off) (#27048)
Rebased onto main as one commit after its parent squash-merged.
2026-10-10 19:11:59 -04:00
Jinwoo Hong 5493c5e815 feat(relay): directors place on reserve-mode cells from memory (off) (#27047)
Rebased onto main as one commit after its parent squash-merged.
2026-10-10 02:26:12 -04:00
Jinwoo Hong abff7ab86a feat(relay): flip a reserve-mode cell back without disconnecting anyone (E-mix) (#27043)
Rebased onto main as one commit after its parent squash-merged.
2026-10-10 02:18:37 -04:00
Jinwoo Hong d488ce5bc0 feat(relay): cell reserve mode: book, admit and demote without the database (off) (#27041)
Rebased onto main as one commit after its parent squash-merged.
2026-10-10 02:11:01 -04:00
Jinwoo Hong 452e44464e feat(relay): read-only /v1/admin/host-whereabouts on directors (step 3) (#26816)
On-call can ask a director where a host is: its relay_assignments row and
open migration (primary-key reads only), the shadow map's seats and recent
leaves with close codes, the compare verdict (pure classification, judged as
/v1/resolve sees the host, recording nothing), and each named cell's poll
status and applied flags. shadow is {state:'off'} when the poller is off.
Deploy, monitor and fence identities may read it; capacity and asia-proof
may not. With no user id, the map names the user; an unknown host answers
empty.
2026-10-09 18:58:10 -04:00
Jinwoo Hong 404e54d7f6 feat(relay): dark shadow seat compare on sticky verify and /v1/resolve (step 3) (#26813)
After the database answers a reconnect verify or a phone resolve, the
director compares that answer with the shadow seat map and counts the
result by class, logging one count line per route and class a minute and a
few rate-limited samples of unexplained disagreements (cell and epoch only,
never raw ids). It runs only when the seat poller is on, inside try/catch,
and never changes a response.

A map-behind mismatch is explained only while this director first saw the
newer epoch within 10 s (recorded only by the compare; classification is
pure). A seat on a cell not polled within 10 s is cell-unpolled. A null
answer for a seat on a cell placement's liveness rule calls dead is
map-only-cell-unlive, or cell-unlive-pending when its heartbeat ran out
after the last cell-list read.
2026-10-09 18:47:32 -04:00
Jinwoo Hong 8848ca95d6 feat(relay): key id on assignment leases; cells shadow-check the lease (off) (#26815)
* feat(relay): key id on assignment leases; cells shadow-check the lease behind ticketCheck=shadow (off)

Directors name the lease signing key with a kid. With the per-cell
ticketCheck switch at shadow, a cell verifies the lease a host echoes in
x-orca-relay-assignment-lease, compares it with the database answer that
still decides, and logs per-class counts and the hello's database read
time once a minute.

* feat(relay): flat per-class totals on the lease shadow line, so log metrics can extract them

* refactor(relay): the lease shadow line carries the flat classes and one database-refused count

The per-class-and-verdict map duplicated the flat classes; disagree
already names the valid-lease refusals, and dbRefused gives the total.
2026-10-09 18:40:34 -04:00
Jinwoo Hong f514cb2c66 feat(relay): dark shadow seat directory poller on directors (step 3) (#26810)
Directors can poll each cell's read-only seat feed (#26803) once a second
and keep an in-memory map of which host is seated on which cell, plus a
bounded 24 h recently-left memory. Off unless ORCA_RELAY_SHADOW_SEAT_FEED_CELLS
names cells. Applies a change at or above the seat's generation, follows
cut pages, resyncs on a gap or new incarnation, and checks the map against
counts.seats. A 404 reads as no feed; timeouts and errors keep the last
seats as unverifiable. The cell list is every cell (an existing-only cell
still seats hosts), from RelayAssignmentStore.seatFeedCells with
placement's liveness rule; completeness waits only on live cells, and a
failed list read backs off. Nothing reads the map yet.
2026-10-09 18:33:32 -04:00
Jinwoo Hong de6d49d52f feat(relay): local /ready behind the per-cell readinessLocal switch (off) (#26812)
* feat(relay): local /ready behind the per-cell readinessLocal switch (off)

With the switch on, a cell's /ready answers 200 while it listens and its
host token verifier holds keys, and reports failing probes as degraded.
The heartbeat keeps the SQL-gated verdict, so directors still stop
placing on a cell that cannot reach the database.

* refactor(relay): the app verifies relay tokens with the server's shared key set

One cached JWKS per process now serves the host-control upgrade, HTTP
token checks and local readiness, instead of a second set in the app.
2026-10-09 18:32:59 -04:00
Jinwoo Hong f0dd050a23 feat(relay): per-cell runtime switches read from Cloud Storage every 5 s (#26805)
* feat(relay): per-cell runtime switches read from Cloud Storage every 5 s

Cells read cells/<cellId>.json from the <project-id>-relay-control bucket,
start on all-off defaults, keep their applied values when the object
cannot be read, and report what they applied in the seat feed. No
consumer reads the switches yet.

* test(relay): exempt the flag channel's poll from the floating-promise census; it catches every failure

* fix(relay): release every switch-file reply the reader does not consume, and bound the read while streaming

404, 5xx, malformed-generation and lower-generation replies, and failed
metadata replies, now cancel their bodies. The object read stops and
cancels at the first byte past 64 KiB instead of buffering it whole.

* feat(relay): runtime status reports the applied cell flags for the flag workflow's read-back

* docs(relay): the switch channel reads per-cell objects only
2026-10-09 18:08:24 -04:00
Jinwoo Hong 37d2154765 feat(relay): read-only seat feed on cells for the directors' shadow directory (#26803)
* feat(relay): read-only seat feed on cells for the directors' shadow directory

Cells keep an in-memory log of host joins, leaves and drain notices and
serve it at GET /v1/admin/cell-seats, authenticated with the directors'
existing rehome identity. Nothing reads it yet.

* test(relay): read the seat feed through its published shape so the tests typecheck

* fix(relay): serve the seat feed's full snapshot from the change log's own seats

counts.controls drops when a socket starts closing, but the leave lands
only at its close event, so a full snapshot built from open sockets could
disagree with a reader that applied every change. The log now keeps its
own seats under the readers' generation rule and reports counts.seats.

* fix(relay): report auth-expiry drain-only and its refresh as seat-feed changes

The heartbeat's auth-expiry drain-only and an auth refresh back to active
changed session state without a feed entry, so a reader's seat state
drifted from the cell's. Both now go through the log; the refresh is a
new 'active' kind.

* refactor(relay): cut unread seat-feed fields; verify a feed poll once

fixLevel, counts.splices, counts.enforcedUnits and counts.hardCap had no
reader, so they leave before they become a cross-version contract. The
admin middleware no longer verifies /v1/admin/cell-seats polls, which the
route already gates on the rehome identity.
2026-10-09 16:58:53 -04:00
Jinwoo Hong 80d7e2dded fix(relay): spread drain placement over a load band instead of one empty cell (#26802)
* fix(relay): spread placement over a load band so a drain does not land on one empty cell

Placement took the single least-loaded cell, so a just-rolled cell took a
whole drain and every follow-up serialized on its relay_cells row. Directors
now pick at random among cells within 5% of capacity of the minimum, and skip
a cell that already took over 40% of its pool's placements in the last 10 s
(floor 4). It only reorders: the least-loaded cell is the fallback, the hard
cap and region preference are unchanged.

* fix(relay): size the load band's intake window for the 1,200 s drain pace

At ~0.5 picks/s per director, a 10 s window with a floor of 4 let the floor
decide and the empty cell took ~65% of a c27-sized drain. A 60 s window and
a floor of 2 keep it at 40% at both the 300 s and 1,200 s paces across five
directors. Adds a once-a-minute orca_relay_placement_load_band line (picks,
budget skips, fallbacks, band size, picks per cell).

* chore(relay): log the load-band summary at info, not warning
2026-10-09 16:58:07 -04:00
Jinwoo Hong 3062b9d8d0 fix(relay): refuse host hellos fast when the database pool is timing them out; renewals jump the queue (#26362)
* fix(relay): refuse host hellos fast when the database pool is saturated; renewals jump the queue

A reconnect herd (2026-10-06 18:34Z, 1,046 asia-east2 hosts dropped by the
load balancer at once) queued hundreds of host hellos behind each cell's
16-connection pool. They timed out after 2 s, the desktops redialled into
the same queue, and control-lease renewals starved behind them.

- Host control upgrades get an immediate 503 (Retry-After: 2) once 32
  callers wait for a pooled connection. Shipped desktops already treat
  that as a connect error and back off with jittered exponential retry.
- General work now queues in PostgresPoolPressure instead of pg-pool, so
  renewal statements (priority lane) take the next released connection.
- A renewal batch that never got a connection no longer fans out into
  one statement per host.
- hostHellosShedDelta counts refusals in the runtime metrics.

* fix(relay): shed host hellos on the pool's oldest wait, not its queue length

asia-east2 queues of 50-196 waiters are routine and keep moving, so a
32-waiter limit would refuse hellos that were going to succeed. Shed only
while the oldest pool waiter has waited 1.5 s of the 2 s acquire timeout,
when a new hello would time out anyway; a refused desktop gets the same
connect error and backoff it gets today, 2 s sooner. Rebinds over a live
control (lease rotation) are never refused.

* fix(relay): shed hellos only when a full pool queue has aged 1 s

Per review: require both an oldest waiter of at least 1 s and a pool's worth
of waiters, so one slow waiter cannot trip it, and state the real baseline
(p99 of 2 waiters on 10-07).
2026-10-07 21:18:26 -04:00
Jinwoo Hong 10bea3a8fa feat(relay): report lane service times, 503 causes, per-site hold p99 and a director-vs-cell lock-wait split (#25645)
* feat(relay): report lane service times, 503 causes, per-site hold p99 and lock-wait split

Adds the step-1 observability: drain-return and sticky slot service times,
every /v1/assign 503 by cause with a non-drain total, a p99 per lock hold
site (drain-return regional rows get their own site), a director-side
pg_stat_activity sample that splits lock waiters by director vs cell, and
a log line for each reserved_requests drift reconciliation corrects.
Log-based metrics for the new fields are declared in Terraform, not applied.

* fix(relay): classify a lock waiter by the first relay table its statement names

* fix(relay): attribute lock waiters to the root holder; document the director hold alert change

Every waiter after the first in a row-lock convoy is blocked by the first
waiter, so the sample now walks pg_blocking_pids to the root, reading it once
per waiter. The sampler is single-flight. Drain-return regional holds feeding
the non-paging director hold policy is documented as expected.

* docs(relay): note the lock-wait sample undercounts director waiters when the pool is full
2026-10-05 17:55:52 -04:00
Jinwoo Hong b82307937a fix(relay): admit drained hosts through their own lane and stagger their return (#24446)
* fix(relay): admit drained hosts through their own lane and stagger their return

A same-cap roll's drain sends every host on the isolated cell back to the
director at once. Those hosts reconnect through the sticky lane (one slot per
director), and each one's re-placement holds that slot for most of a second
behind the region-wide inventory lock, so ordinary reconnects time out behind
them and the drained hosts retry every 2 s: ~30k 503s per drain.

The reconnect verification read now also says whether the host's home cell is
isolated for a roll right now (same predicate re-placement uses). Those hosts
release the sticky slot after the read and take a separate drain-return lane
(1 per director, matching the store's per-director placement serialization).
When that lane is full the host gets a Retry-After that reserves the next
free service slot, paced by the measured re-placement time and capped at 300 s.

Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010

* fix(relay): keep drain returns inside the placement pool budget and their own slot

Review follow-ups for the drain-return lane:
- The lane now borrows placement permits (never placement's last, never ahead
  of a queued placement), so placement + sticky still bounds the database pool.
- A host's own early retry (row-busy redial, duplicate dial) gets the 2 s lane
  interval instead of a fresh slot behind the cohort, and a host that returns
  early to the same director keeps its reserved slot.
- Classification also excludes an open migration row whose lease counter
  lapsed, matching the re-placement rule.
- The load test now runs five directors behind random routing with a shared
  inventory lock.

Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
2026-10-01 15:34:23 -04:00
NeilandClaude 8da0ca52e7 perf(relay): release rejected first-frame connections [trade-off] (#23011)
* perf(relay): release rejected first-frame connections [trade-off]

* fix(relay): bound the director's rejected and redirected first-frame closes too

The parent PR routed four cell-side first-frame rejections through
closeRelayWebSocket but left two raw socket.close() calls in the same
handler. A real-socket probe shows both still pin a connection unit for
ws's full 30s close timer when the peer ignores the close frame:

- 'invalid invite' is reachable by an unauthenticated peer with a
  well-formed but bogus credential, so the exhaustion the parent PR
  claims to prevent stayed reachable on the director;
- 'connect to assigned cell' is the happy path for every phone's first
  director contact, so it is the highest-volume unbounded close here.

closeWithDrain gets the same treatment; host-session-registry already
closes the identical drain through the helper.

Also records that the bounded close is not a user-facing trade-off: the
close frame is written before the force-close timer can fire and TCP
delivers it ahead of the FIN, so an abandoned peer still reads code and
reason over a graceful close. The new regression asserts that, plus
exactly-once release across concurrent bursts and rejection racing the
peer's own disconnect (the ledger does not clamp at zero, so a double
release would surface as a negative count).

* refactor(relay): drop the unused closeWithDrain helper

`closeWithDrain` has no callers anywhere in the repo, and its `graceMs`
parameter promised a caller-supplied drain window that the body no longer
honours: routing it through `closeRelayWebSocket` force-terminates after 1s
regardless, so a future caller passing `graceMs: 30_000` would have had its
drain silently cut short while the signature still claimed otherwise.

The real drain path is `host-session-registry`, which sends the same
`resolve-director` drain and closes it there. Delete the dead duplicate
rather than bound a helper whose contract says "graceful".

Co-Authored-By: Claude <noreply@anthropic.com>

* docs(relay): call the bounded close's rejection delivery best effort, not guaranteed

The helper claimed the forced terminate costs an abandoned peer nothing it can
observe, and the test presented its fast-peer assertion as proof. Neither holds in
general: `ws` writes the close frame to the socket and `terminate()` destroys that
socket a second later, so under backpressure the frame — and any `relay-moved`
message queued ahead of it — can go unsent even to a peer that never stopped
reading.

Qualifies both comments to describe delivery as best effort and name the
backpressure case. The fast-peer assertion is valid and stays exactly as it was;
only its stated scope narrows, and the test is renamed to say which peer it speaks
for. What the bound actually buys — a stalled peer cannot hold admission — is now
stated on its own rather than resting on a delivery claim.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-09-27 00:28:52 -07:00
Jinwoo Hong 1043dc5e1d fix(relay): stop rehoming hosts off Asia cells until the lock fix lands (#22443)
The source cell runs the rehome commit. An Asia source pays a cross-ocean
round trip per statement while holding relay_cells row locks every cell
needs, which convoys the fleet database. Selection now drops source cells
outside the director's region before building the decision window, so the
incumbent_region filter shrinks while Asia cells stay valid targets. The
preview counts the same hosts as source-outside-director-region and the
poll summary reports skippedOffRegionSourceCells. Temporary stopgap.

Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
2026-09-23 03:15:52 -04:00
Jinwoo Hong fa0010e8d6 fix(relay): abort rehomes whose host never arrived, without disabling the switch (#21813)
A regional rehome whose host went offline right after accepting the move
left its migration row open forever: the target had registered it, the host
held nothing on the source, and the completion sweep could never finish it.
Eight such rows filled REGIONAL_REHOME_CONCURRENT_LIMIT and every later
candidate came back deferred, silently, for 21 hours.

The only sweep that touched them fires at 24 hours and also sets
enabled = 0 on the durable control, so the first leak to age out would have
turned rehoming off, repeatedly.

Adds a director sweep that rolls such an attempt back to its source after
one migration lease, with abort_reason = 'host_not_arrived', reusing the
existing rollback (assignment epoch bump back to the source, lease removal,
superseded target reservation release) and leaving the switch untouched.
The 24-hour sweep keeps its disable as a last-resort latch.

The source cell now names why it deferred, on a new optional response field,
and the director stops walking its candidate page on a deferral no later
candidate can pass. Each poll that dispatched logs one summary line.

Claude-Session: https://claude.ai/session/ced32ebb-7155-4413-adad-1eccd14c2010
2026-09-20 15:13:27 -04:00
Jinwoo Hong 0e7948fa6d feat(relay): pace the drain send during a same-cap cell roll (#21284)
* feat(relay): pace the drain send during a same-cap cell roll

A same-cap roll drains a cell with graceMs 0, which sends `drain` to all
~800 controls in one pass. Every desktop re-dials on receipt regardless of
graceMs, so the whole cell reconnects inside a second. On 2026-09-16 that
stampede hit a Cloud SQL stall: attaches timed out, each leaving 10 minutes
of late-arrival debt on connection headroom, and placement answered
relay_capacity_exhausted fleet-wide for ~13 minutes.

Spreading the sends spreads the re-dials. `HostSessionRegistry.drain` takes
an optional pacing window and schedules each session's send evenly across
it; admission is fenced for every session up front, and each host keeps its
own full grace after its own send. /v1/admin/drain accepts `paceWindowMs`
(<= 5 min) and echoes what it applied. The same-cap job asks for 120 s, and
the drain-completion wait grew by the same amount. A cell still on an older
image rejects the field, so the deploy script falls back to an unpaced
drain rather than failing the roll.

* fix(relay): scope the drain fence to the hosts already told

Review of the paced drain found two problems, both from treating "this cell
is draining" as one instant when pacing makes it a window.

Timers: the sends queued by a paced drain were neither cleared when a later
drain superseded them nor unref'd. A SIGTERM mid-window left up to 800 no-op
timers holding the event loop open until systemd escalated to SIGKILL. Drain
timers are now tracked, cleared on the next drain, and unref'd, so a retry
re-arms a session's teardown instead of stacking a second one.

Phones: the client fence read the global draining flag, so every phone was
refused for the whole window even though its own host had not been told yet
and was still serving. The director keeps pointing phones at this cell until
their host moves, so they would have looped for up to two minutes. A session
is now fenced when its drain is sent, not when the drain starts, and the
client paths key off that. New control connections and re-attaches stay
fenced globally: nothing new should land on a cell that is going away.
2026-09-17 16:45:25 -04:00
Jinwoo Hong 69787e763a fix(relay): serve readiness from last-known-good during auth or SQL blips (#21161)
* fix(relay): serve readiness from last-known-good during auth or SQL blips

The load balancer health check hits /ready, which re-probed the auth JWKS
endpoint and Postgres on every poll and reported not-ready on the first
failure. On 2026-09-16 an auth outage therefore took every cell out of the
load balancer within ~30s and dropped every connected host, even though the
token verifier caches keys in process and kept verifying tokens.

/ready now remembers when each dependency last answered and keeps reporting
ready while the failed one stays inside a grace window
(ORCA_RELAY_READINESS_GRACE_MS, default 15 minutes, 0 disables). A process
that has never succeeded still gates on the real dependencies, so cold boot
is unchanged. Grace answers carry degraded plus the failure reason on the
existing readiness observation, and entering or leaving grace logs once.

MIG autohealing still uses the dependency-free /health endpoint.

* fix(relay): split readiness grace per dependency and probe both every poll

Review follow-ups on the last-known-good readiness window.

An unset environment variable arrives as an empty string, which z.coerce
reads as 0, so the single ORCA_RELAY_READINESS_GRACE_MS would have switched
the window off instead of falling back to its default. The two replacement
variables preprocess '' to undefined.

JWKS and SQL now get separate windows and separate clocks:
ORCA_RELAY_READINESS_JWKS_GRACE_MS defaults to 15 minutes, and
ORCA_RELAY_READINESS_SQL_GRACE_MS to 3 minutes. Each cell is its own load
balancer backend, so failing readiness never re-routes a host, it only makes
that hostname unreachable, and a host that lands on a SQL-dead cell gets
WRONG_CELL and is re-placed by the director. Three minutes rides a Cloud SQL
failover without hiding a per-cell fault for a quarter of an hour.

Both dependencies are probed on every poll. A JWKS failure used to
short-circuit the SQL probe, which let the SQL clock age with no evidence
behind it. Grace transitions are emitted per dependency, so JWKS recovering
while SQL fails logs both sides instead of nothing.

/ready keeps its 200 and its {ok:true} body when healthy, and adds
degraded plus the dependency list when the answer comes from a window.
2026-09-17 01:46:33 -04:00
Jinwoo Hong cd9aa43a2c feat(relay): correct regional placement only when the source is idle (#20105)
* feat(relay): correct regional placement only at an idle source

* test(relay): lock source activity capacity semantics
2026-09-11 14:25:10 -04:00
Jinwoo Hong 91d7783f2b fix(relay): state pending-conn details to hosts that advertise the capability (cell side) (#19266)
The cell announces a connection with a single conn-open. When the desktop's
control socket dies mid-accept the phone waited out the 10s attach deadline and
was closed HOST_OFFLINE, even though the desktop was online. host-hello-ack
already restates those connections in pendingConns, but only by connId and
connTicket, which is not enough for the desktop to dial: kind and relayDeviceId
decide the pairing authority a connection carries and the E2EE device binding,
so neither may be guessed.

The cell now states kind and relayDeviceId on each pending entry, but only to a
host that advertised it can read them: a shipped host parses those entries
strictly, so an unannounced key fails the whole ack parse and kills a working
control. The advertisement rides the control upgrade as
x-orca-host-capabilities, not host-hello, because HostHelloSchema is strict on
the cell too and any new hello key is refused by every already-deployed cell.

The capability is keyed by socket, not by session: a rebind can land a successor
whose decoder is older or newer than the one that opened the session, and the
ack must follow the socket that will actually read it.

With no capable host in the fleet the emitted ack is byte-identical to today's.
The desktop half that consumes the new fields is #19238.
2026-09-07 05:06:38 -04:00
Jinwoo Hong 61b09b7a02 fix(relay): abandon dead client accepts, jitter and lengthen the control lease, fail direct probes fast (#18959)
* fix(relay): abandon a client accept once the phone hangs up; jitter the control lease

The accept runs several serialized Postgres calls behind the contended
cell-inventory lock, and phones bound their dial. Finishing that work for a
phone that had already left acquired (and leaked for 90s) an activity lease and
then failed at bind with host_data_reservation_already_bound. Check the client
socket between the DB steps and unwind what was taken, reporting the stage on
orca_relay_client_accept_abandoned.

Jitter the control lease grant so a cohort that reconnected in the same minute
(a cell recreate dumps hundreds at once) walks apart instead of rebinding
together every cycle.

On the phone, treat a probe session that enters 'reconnecting' as a failed
probe: it is the direct client's own backoff after a dead-LAN 1006, and waiting
it out held the supervisor's operation mutex for the full 12s bound.

* perf(relay): lengthen the control lease to 6h

The lease bounds how long a host lingers on a cell after a missed drain, and
rebinding it is the only passive rebalancing we have, so it stays finite. 6h
keeps both properties while cutting control-activation traffic on the contended
cell-inventory lock ~6x. The relay JWT (5 min, refreshed by the desktop) and the
75s silence watchdog are enforced separately, so the longer grant authorizes
nothing extra. The jitter widens with it, to +/-30 min.

* fix(relay): let one flap recover the direct probe; correct the leak window

'reconnecting' is published on any socket close, so rejecting on it outright
turned a single access-point flap into a booked direct failure and a 60s
cooldown. Give the first 'reconnecting' a 2s grace in which a 'connected'
transition still resolves; a dead LAN still fails in ~2s rather than holding the
supervisor's operation mutex for the 12s bound.

The abandoned accept held its activity lease for the 10s attach deadline, not
90s -- the attach timer is armed before bind throws and already unwinds it.

Also cover the assignment-stage check that guards reserveCredential, and drop a
spread assertion the two exact-value assertions above already imply.

* fix(relay): extend the probe grace once on a handshake; pin the lease band top

The redial fires at 500ms but 'connected' waits on the Noise handshake and a
capability RPC, so one 2s window is too tight for real work. A 'handshaking'
transition is evidence the peer answered, so extend the grace once; a stalled
handshake still fails at ~3.5s, far inside the 12s bound.

The longest-lease case only had an upper bound, which a jitter clamped to one
side would satisfy. Pin it to the exact top of the band instead, and assert the
assignment resolve ran so the third-guard test cannot pass vacuously.
2026-09-05 20:47:27 -04:00
Jinwoo Hong 11aace8dec fix(relay): reject malformed percent-escapes on upgrade instead of throwing (#18547)
decodeURIComponent on the /v1/connect/ and /v1/host/data/ path segments threw
URIError out of the http 'upgrade' listener, which is uncaught and kills the
relay process. Any client that sends GET /v1/connect/% could take down a cell
(and every connection on it) or a director instance. Pre-existing since the
splice landed (orca-cloud #20); not introduced by the import.

A malformed escape now takes the existing 4xx reject branch. The blackbox test
sends three malformed connect targets and one host-data target to the real
server and asserts no uncaughtException fires and a well-formed upgrade still
gets 101 afterwards; reverting either site fails it.
2026-09-03 21:31:53 -04:00
Jinwoo Hong 3eec77c11a chore(cloud): add the relay fence broker, ops console, Terraform root, scripts, and 24 cloud-* workflows (#18413)
Phase 6 of the relay split: the relay's deploy/operate surface moves under cloud/ with 24 cloud-* workflows gated on ORCA_CLOUD_OPERATIONS_ENABLED, the Cloud SQL rollout lease action, the relay Terraform root (dual-accept identities for both repositories), scripts, docs, CODEOWNERS, and a terraform validate job in Cloud Verify.
2026-09-03 06:55:14 -04:00