mirror of
https://github.com/stablyai/orca.git
synced 2026-11-16 10:58:10 +00:00
nwparker/github-cache-sqlite
44
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
5df115db15 |
fix(relay): runtime edge control on the director, safer skipped exports, faster evacuation (#39773)
- Directors with an edge read director/edge.json from the relay-control bucket through the shared flag channel: mode on/off/down and rolloutPermille (0..1000). Boot defaults are mode on and the env percent; every cohort decision reads the applied flags. Off empties the cohort and hosts leave normally; down also skips the export at once (reason mode_down). The permille bucket keeps every host of P percent in at P x 10 permille. A control change re-arms the evacuation sweep within 5 s. - A skipped export (120 s outage clock or mode down) removes the host's relay_devices rows before placing it, so revokes made on the edge are never undone; logs a count only. - Evacuation: 100 drains per tick with 10 in flight, each pass starting at a random point of the keys and wrapping around; pages stay 500 rows and an idle pass reads each edge row once. - Fence retries: 56 attempts (last about 25.5 h after the failure), 50 per sweep with 10 in flight; fence_abandoned now carries outcome, cell and epoch like fence_failed. |
||
|
|
340c917429 |
relay: edge export skip on outage, cohort-exit drain sweep, region-scoped account allow (#39707)
* relay: edge export skip on sustained outage, evacuation drain sweep, account allow scoped by region - Leaving hosts skip the edge export once it has been unavailable for 120 s (ORCA_RELAY_EDGE_EXPORT_SKIP_AFTER_MS), keeping Postgres devices and fencing best effort, instead of 503ing until the edge is back. - Directors drain edge hosts that left the cohort (POST /__admin/drain, 20 per jittered 60 s tick) so connected desktops re-ask and move off. - Account-allowlisted hosts must also be in the edge regions. - Tests for the invite-socket edge branch. * relay: export skip needs continuous failures; evacuation sweep runs soon after start, then every 10 min Failures more than 30 s apart restart the unavailable clock, so one old failure cannot skip an export. The sweep reads the edge rows (cell_id has no index): first pass 45-75 s after start (a cohort change is a new revision), every 45-75 s while hosts are queued, else every 7.5-12.5 min. |
||
|
|
568567f7ad |
relay: director edge cohort for the relay edge (dark by default) (#39362)
* feat(relay): director edge override for moving hosts to the relay edge (dark by default) Directors with ORCA_RELAY_EDGE_CELL_URL seed an existing-only 'edge' cell (registered with the admission selector so deploy workflows carry it) and answer a host cohort (hash bucket, allow/deny lists, region scope) on the edge before the reserve and database lanes. The epoch rises only on a cell change; a host still holding units on its old cell is told to retry. Phone resolve and the invite move answer edge rows without moving anything; old cells answer WRONG_CELL to phones of moved hosts. Unset URL keeps today's behaviour. * feat(relay): director pushes host data to the relay edge on a move and takes it back before any move away With ORCA_RELAY_EDGE_ADMIN_SIGNING_KEY set (its own key, never the assignment lease key), a move onto the edge pushes the host's relay_devices and a location hint under the row lock and commits only after the edge accepts; an edge holding a newer epoch is re-minted above once. Before any lane may re-place an edge row, the director exports (refused while the desktop is connected; the edge freezes the host), replaces the host's Postgres devices, places it as today, then fences the edge with the new cell and epoch. Idle regional rehome skips edge rows. Unset key: unchanged. * fix(relay): a deferred edge move keeps the host on today's path instead of a 503 loop A host still holding old-cell units (a pairing invite holds one for its 10 min TTL) or whose push the edge did not take now stays where it is and moves on a later ask. 503 stays for a held row lock and for a desktop still connected to the edge on the way out. * relay: director refuses an edge URL without the edge admin key The deployed edge keeps host data in its objects; without the key the director could not push it and hosts would land on an edge holding none of their devices. * relay: edge cohort by account (ORCA_RELAY_EDGE_ALLOW_USERS) Every host of a listed account (relay token sub) joins the edge cohort, in any region; a denied host still stays off. For the team's own accounts before any share rollout. * relay: edge modules follow main's relay module moves * relay: satisfy the changed-lines lint gate in the edge cohort code Braces on every if, T[] arrays, type aliases, one ternary, and a justified max-lines header on edge-relay.ts. No behaviour change: relay unit 845 passed. * relay: move edge cohort config and row helpers to edge-relay-cohort.ts Keeps edge-relay.ts under the 300-line limit without a max-lines disable. Code moved verbatim; edge-relay.ts re-exports what other modules import. Relay unit 845 passed. |
||
|
|
46fa97a6b4 |
refactor(relay): group route, forwarding and connection owners (#38374)
* refactor(relay): group route, forwarding and connection owners * refactor(relay): use explicit defining-owner type imports |
||
|
|
e74b40bd4e |
fix(relay): dead-man boot grace for a freshly booted reserve cell (#37225)
* fix(relay): give a freshly booted reserve cell a dead-man boot grace until its first director contact A reserve cell's dead-man starts its 60-120 s window when the switch file applies, but no placing director can reach a new process until its load balancer routes to it. On the 2026-11-01 production same-cap rolls that took up to about 110 s after the switch file applied; c10 (104 s window) and c16 (71 s) tripped 1-4 s before the first director poll landed, latched into db, and failed their roll after a clean drain and restart. Until the process's first contact (a reserver=1 feed poll or a real booking), the window is the hashed window plus 180 s, matching autohealing's initial delay. After the first contact nothing changes. A process that never hears from a placing director still falls back to db, up to three minutes later than before. * test(relay): assert the server wires the dead-man boot grace * docs(relay): state the boot gap behind the dead-man grace as inferred, not measured |
||
|
|
6dcbcd6be5 | refactor(cloud): colocate relay assignment store and tests (#37098) | ||
|
|
7d6603aa17 |
[comment audit] Complete repository-wide comment cleanup (#33249)
* [comment audit] Align legacy failure-read counts with cached display * [comment audit] Condense shared provider lifecycle and browser recovery comments [skip ci] * [comment audit] Condense shared generation, path and preview comments [skip ci] * [comment audit] Condense host, file and Git contract rationale [skip ci] * [comment audit] Tighten Git lifecycle and provider contract comments [skip ci] * [comment audit] Clarify shared host authority and launch isolation [skip ci] * [comment audit] simplify local access mobile and native chat contracts [skip ci] * [comment audit] simplify native chat presentation and state contracts [skip ci] * [comment audit] simplify native runtime and provider contracts [skip ci] * [comment audit] simplify migration and orchestration contracts [skip ci] * [comment audit] simplify plugin trust and process capture rationale [skip ci] * [comment audit] simplify terminal delivery and remote compatibility rationale [skip ci] * [comment audit] clarify remote recovery and launch wire rationale [skip ci] * [comment audit] shorten shared RPC compatibility and runtime evidence comments [skip ci] * [comment audit] simplify secure storage shell startup and sidebar rationale [skip ci] * [comment audit] clarify sidebar, skill and SSH invariants [skip ci] * [comment audit] condense structured chat delivery and timing rationale [skip ci] * [comment audit] condense terminal replay and telemetry invariants [skip ci] * [comment audit] clarify launch permission and terminal delivery rationale [skip ci] * [comment audit] clarify workspace ownership and layout contracts [skip ci] * [comment audit] wait for fitted and painted SSH terminal links * [comment audit] clarify layout persistence and session ownership [skip ci] * [comment audit] clarify host routing and workspace lifecycle contracts [skip ci] * [comment audit] clarify startup persistence and app lifetime comments [skip ci] * [comment audit] clarify recovery and activity boundaries [skip ci] * [comment audit] condense activity and artifact invariants [skip ci] * [comment audit] condense automation authority contracts [skip ci] * [comment audit] simplify automation scheduling and history contracts [skip ci] * [comment audit] clarify browser chrome and annotation lifecycles [skip ci] * [comment audit] clarify browser routing and retained guest ownership [skip ci] * [comment audit] clarify browser recovery and document ownership comments [skip ci] * [comment audit] restore hosted runners required by native IME fixtures * [comment audit] shorten preview, palette and dashboard invariants [skip ci] * [comment audit] clarify terminal preview, dashboard and retention comments [skip ci] * [comment audit] clarify diff-note recovery and editor lifecycle comments [skip ci] * [comment audit] shorten rich-text and combined-diff ownership comments [skip ci] * [comment audit] shorten autosave, draft recovery and model ownership comments [skip ci] * [comment audit] shorten save-queue, notebook and Markdown constraints [skip ci] * [comment audit] shorten Monaco, PDF and rich Markdown safeguards [skip ci] * [comment audit] clarify rich Markdown source and selection safeguards [skip ci] * [comment audit] clarify editor loading and emulator stream lifetimes [skip ci] * [comment audit] trim emulator and feature tour narration [skip ci] * [comment audit] clarify floating workspace and GitHub cache ownership [skip ci] * [comment audit] trim provider controls and Linear filter explanations [skip ci] * [comment audit] clarify pairing and chat resume lifetimes [skip ci] * [comment audit] clarify chat delivery and transcript lifetimes [skip ci] * [comment audit] clarify session options and host-owned status [skip ci] * [comment audit] clarify draft durability and live-status precedence [skip ci] * [comment audit] Trim chat send and model ownership explanations [skip ci] * [comment audit] Shorten transcript ownership and send recovery comments [skip ci] * [comment audit] Simplify native chat composer and transcript hook comments [skip ci] * [comment audit] Shorten native chat scroll and structured session explanations [skip ci] * [comment audit] Trim workspace picker and onboarding commentary [skip ci] * [comment audit] Simplify pet rendering, review cache and quick open comments [skip ci] * [comment audit] Trim sidebar and session history explanations [skip ci] * [comment audit] Condense session resume and review checks comments [skip ci] * [comment audit] Shorten checks state and file explorer refresh comments [skip ci] * [comment audit] Trim plugin bridge and source-control action rationale [skip ci] * [comment audit] Trim source-control listing and review rationale [skip ci] * [comment audit] Shorten sidebar lifecycle and explorer safeguards [skip ci] * [comment audit] Condense Settings account and appearance rationale [skip ci] * [comment audit] Trim pairing and repository Settings explanations [skip ci] * [comment audit] Condense Settings search SSH and terminal safeguards [skip ci] * [comment audit] Shorten Settings voice and provider ownership comments [skip ci] * [comment audit] Condense Settings draft persistence and host-state rationale [skip ci] * [comment audit] Trim Settings hooks and sidebar setup rationale [skip ci] * [comment audit] Condense sidebar deletion and board-filter safeguards [skip ci] * [comment audit] Condense worktree cards, host ownership and Markdown contracts [skip ci] * [comment audit] Condense sidebar drag, folder and filtering contracts [skip ci] * [comment audit] Condense sidebar parity, sleep and host ownership contracts [skip ci] * [comment audit] Prune focus-read baseline after comment cleanup * [comment audit] Clarify sidebar visibility, board drops and agent evidence [skip ci] * [comment audit] Tighten sidebar listing, drag and reveal contracts [skip ci] * [comment audit] Clarify viewport geometry and skill update safeguards [skip ci] * [comment audit] Tighten skill inventory, usage and host status comments [skip ci] * [comment audit] Clarify usage ownership and resource kill safeguards [skip ci] * [comment audit] Tighten tab selection, rename and search contracts [skip ci] * [comment audit] Tighten tab-strip geometry and drag ownership comments [skip ci] * [comment audit] Tighten Tasks recovery, pagination and mutation comments [skip ci] * [comment audit] Tighten provider loading and terminal recovery comments [skip ci] * [comment audit] Tighten completion, cadence and PTY ownership comments [skip ci] * [comment audit] Clarify terminal restore and parked-pane boundaries [skip ci] * [comment audit] Tighten PTY startup and replay explanations [skip ci] * [comment audit] Clarify PTY identity and delivery ownership [skip ci] * [comment audit] Shorten remote reconnect and input replay rationale [skip ci] * [comment audit] Condense terminal parking clipboard and file link comments [skip ci] * [comment audit] Clarify IME ownership and trim trace history [skip ci] * [comment audit] Shorten pane recovery and parked watcher explanations [skip ci] * [comment audit] Condense snapshot replay and visibility recovery rationale [skip ci] * [comment audit] Shorten terminal parking and keyboard ownership rationale [skip ci] * [comment audit] Trim renderer lifecycle and menu rationale [skip ci] * [comment audit] Condense workspace cleanup ownership and settlement notes [skip ci] * [comment audit] Clarify renderer hook evidence and source ownership [skip ci] * [comment audit] Tighten IPC ownership and reconnect explanations [skip ci] * [comment audit] Trim renderer focus, paste and attention comments [skip ci] * [comment audit] Shorten scroll, locale and agent lifecycle comments [skip ci] * [comment audit] Trim agent launch, resume and ownership commentary [skip ci] * [comment audit] Shorten account, browser and editor comments [skip ci] * [comment audit] Clarify host-mirror waits, previews and input ownership [skip ci] * [comment audit] Condense launch recovery and editor tokenization comments [skip ci] * [comment audit] Condense workspace selection and terminal pane rationale [skip ci] * [comment audit] Condense terminal fitting and rendering recovery comments [skip ci] * [comment audit] Condense terminal input and host routing explanations [skip ci] * [comment audit] Condense ownership and sleeping-session recovery rationale [skip ci] * [comment audit] Condense structured launch and terminal link explanations [skip ci] * [comment audit] Condense terminal ownership and workspace file routing rationale [skip ci] * [comment audit] Condense session persistence and workspace activation rationale [skip ci] * [comment audit] Condense workspace routing and browser ownership rationale [skip ci] * [comment audit] Condense mirror recovery and layout authority rationale [skip ci] * [comment audit] Condense runtime transport and file safety rationale [skip ci] * [comment audit] Condense runtime graph and structured session rationale [skip ci] * [comment audit] Condense browser creation and remote tab recovery rationale [skip ci] * [comment audit] Condense snapshot authority and publisher recovery rationale [skip ci] * [comment audit] Condense frame, startup and provider cache rationale [skip ci] * [comment audit] Condense host twin and agent lifecycle rationale [skip ci] * [comment audit] Condense browser authority and editor cleanup rationale [skip ci] * [comment audit] Condense editor buffer identity and review cache rationale [skip ci] * [comment audit] Condense provider lifetime and repository ownership rationale [skip ci] * [comment audit] Condense reconnect, hydration and terminal retirement rationale [skip ci] * [comment audit] Condense recovery, reconciliation and persistence rationale [skip ci] * [comment audit] Condense cleanup ownership and listing admission rationale [skip ci] * [comment audit] Condense teardown ordering and terminal ownership rationale [skip ci] * [comment audit] Condense recovery ledger and web host-boundary rationale [skip ci] * [comment audit] Condense web reconnect and credential lifecycle rationale [skip ci] * [comment audit] Condense cloud rollout and assignment safety rationale [skip ci] * [comment audit] Condense relay session and database recovery rationale [skip ci] * [comment audit] Condense relay placement and protocol compatibility rationale [skip ci] * [comment audit] Condense build and packaging constraints [skip ci] * [comment audit] Condense CI gates and validation isolation rationale [skip ci] * [comment audit] Condense catalog identity and browser validation rationale [skip ci] * [comment audit] Condense mobile browser and frame-budget rationale [skip ci] * [comment audit] Shorten mobile route and browser test explanations [skip ci] * [comment audit] Condense editor and session seam test rationale [skip ci] * [comment audit] Trim session and terminal closure history [skip ci] * [comment audit] Shorten terminal browser and census explanations [skip ci] * [comment audit] Preserve root-only E2E diagnostic baseline * [comment audit] Condense mobile seam and HTML preview test rationale [skip ci] * [comment audit] Condense preview instrumentation and render harness rationale [skip ci] * [comment audit] Condense bundle contracts and native-build constraints [skip ci] * [comment audit] Shorten native packaging and CI routing rationale [skip ci] * [comment audit] Condense release tooling and benchmark constraints [skip ci] * [comment audit] Condense packaging and platform gate rationale [skip ci] * [comment audit] Shorten mobile route and history rationale [skip ci] * [comment audit] Condense mobile browser and component rationale [skip ci] * [comment audit] Shorten drawer and workspace-form rationale [skip ci] * [comment audit] Trim mobile review and editor-document explanations [skip ci] * [comment audit] Condense editor lifecycle and Markdown explanations [skip ci] * [comment audit] Trim mobile file, diagnostics and dictation commentary [skip ci] * [comment audit] Simplify mobile host-list and workspace-view explanations [skip ci] * [comment audit] Clarify mobile shell lifetime and bridge delivery constraints [skip ci] * [comment audit] Condense bridge stream, native request and audio rationale [skip ci] * [comment audit] Reduce bridge schema and page-client commentary [skip ci] * [comment audit] Tighten mobile page bridge contracts [skip ci] * [comment audit] Clarify mobile cache and session constraints [skip ci] * [comment audit] Condense mobile route and lifecycle rationale [skip ci] * [comment audit] Tighten mobile shell hook timing rationale [skip ci] * [comment audit] Clarify route handoff and notification compatibility [skip ci] * [comment audit] Clarify push registration and dictation cleanup constraints [skip ci] * [comment audit] Condense keyboard geometry and native media ownership rationale [skip ci] * [comment audit] Tighten mobile chat scroll and composer concurrency rationale [skip ci] * [comment audit] Condense mobile reply compatibility and upload ownership rationale [skip ci] * [comment audit] Condense optimistic echo and scroll ownership rationale [skip ci] * [comment audit] Tighten stream replay and queued session ownership rationale [skip ci] * [comment audit] Condense mobile composer and reply-schema constraints [skip ci] * [comment audit] Clarify mobile transcript recovery and session ownership [skip ci] * [comment audit] Clarify mobile queue and source-control compatibility [skip ci] * [comment audit] Condense storage and task launch constraints [skip ci] * [comment audit] Clarify task schemas and terminal document ownership [skip ci] * [comment audit] Tighten terminal input and viewport rationale [skip ci] * [comment audit] Shorten terminal and RPC recorder contracts [skip ci] * [comment audit] Condense recorder engine and connection contracts [skip ci] * [comment audit] Shorten host credential and relay recovery comments [skip ci] * [comment audit] Tighten pairing recovery and bundle transport rationale [skip ci] * [comment audit] Clarify RPC delivery and workspace catalog boundaries [skip ci] * [comment audit] Condense catalog and relay entry contracts [skip ci] * [comment audit] Shorten shutdown and SSH relay delivery contracts [skip ci] * [comment audit] Condense WSL relay and E2E regression rationale [skip ci] * [comment audit] Shorten cross-version harness and renderer fixture explanations [skip ci] * [comment audit] Condense paired-host and E2E lifecycle rationale [skip ci] * [comment audit] Shorten layout oracle and regression explanations [skip ci] * [comment audit] Condense paired recovery and ownership regression contracts [skip ci] * [comment audit] Shorten restart, sidebar and SSH lifecycle contracts [skip ci] * [comment audit] Condense terminal rendering and IME regression rationale [skip ci] * [comment audit] Shorten terminal recovery and benchmark contracts [skip ci] * [comment audit] Condense tooling and Windows lifecycle proof rationale [skip ci] * [comment audit] Condense workflow routing and release rationale [skip ci] * [comment audit] Condense mobile security and installer contracts [skip ci] * [comment audit] Shorten native rationale and stylesheet workarounds [skip ci] * [comment audit] Shorten Windows visibility probe rationale [skip ci] * [comment audit] Condense embedded release and signing rationale [skip ci] * [comment audit] Retain current main test behavior in final reconciliation [skip ci] * [comment audit] Complete final comment review across current source inventory * [comment audit] Handle consolidated audit reports before final CI * [comment audit] Preserve credential storage census detail [skip ci] * [comment audit] Scale broad changed E2E selections with verified timing shards * [comment audit] Align E2E assertions with browser close and settled session metadata * [comment audit] Match existing E2Es to current resource and windowing contracts * [comment audit] Observe the shared server outage notice in reconnect coverage * [comment audit] Make existing renderer probes measure settled tab and PTY work * [comment audit] Align palette E2E selector with task search * [comment audit] Refresh remaining palette search selectors * [comment audit] Keep selector correction within selected audit specs * [comment audit] Await emoji fixture settings inside renderer evaluation * [comment audit] Cancel fixture fit retries between terminal tests --------- Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com> |
||
|
|
0ae42fe594 |
feat(relay): cap how many reserve cells a director outage trips to the database (#33415)
<!-- orca-pr-loc -->
<!-- Programmatic LoC summary. Do not edit by hand; rewritten on every commit. -->
| | Files | Added | Deleted | Net |
| :--- | ---: | ---: | ---: | ---: |
| Test | 4 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$406 | $\color{#cf222e}{\Huge{\mathbf{−}}}$4 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$402 |
| Prod | 6 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$165 | $\color{#cf222e}{\Huge{\mathbf{−}}}$10 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$155 |
<!-- /orca-pr-loc -->
## ELI5
Relay servers ("cells") normally get their seat bookings from the front door (the director). If the director goes quiet for a minute or two, each cell switches itself to an older mode that talks to the shared Postgres database for every connection (the "dead-man" trip). Today every cell switches at once, so the database budget must keep room for every cell's full connection pool. This PR adds a per-cell switch that caps how many cells may trip at the same time. A cell that cannot trip yet stays in its normal mode and keeps serving the desktops it already has.
## What Changed
- New cell flag `deadManTripSlots` (integer 0-64). Absent, which is every cell today, keeps the current behaviour: trip as soon as the dead-man window (60-120 s) passes.
- With the flag set, a cell whose window has passed first claims a slot in Postgres. The claim is one short transaction: it takes an advisory lock without waiting, counts the other reserve cells whose own `relay_cell_admit_effective` row says `db`, and writes this cell's row to `db` only if that count is below the flag. Only then does the cell trip (void bookings, re-register its hosts, run on its full pool).
- A cell that gets no slot is "held". It stays in reserve mode, logs `orca_relay_cell_reserve_dead_man_held` once, keeps writing `reserve` to its row, and asks again every 30 s.
- A slot is held while the holder's own row says `db`, its switch still says `reserve`, and the row was refreshed in the last 10 minutes. A holder that restarts or is re-armed writes `reserve` and frees its slot. One that died tripped frees it after 10 minutes. Heartbeats are not used: they reach Postgres through the director, so in a director outage they all go stale and every slot would read as free.
- A tripped cell skips its 15 s row refresh while its pool has waiters. So that a long brownout cannot age its row out of the count, a tripped cell's row older than 5 minutes is refreshed even when the pool is busy.
- A trip stays latched after the director returns, as today. The tripped cells keep their slots until an operator re-arms them with a new reserve write. Until then, a second outage trips nobody new: the other cells stay held. That is the intended bound, but re-arming after an outage now also frees the slots.
- The claim goes through `CellAdmitEffectiveWriter`, which already serialises every write of that row. A `reserve` write still in flight therefore cannot land after the claim and hide the slot from the next claimant.
- Cells keep their existing per-cell window spread (60-120 s by cell-id hash), so the earliest windows claim first.
What a held cell does during a director outage (unchanged code paths, now for longer than the window):
- Desktops already connected keep their sockets. Nothing is closed or re-registered.
- A desktop that drops and redials at the same epoch is admitted from memory (`seatedHereAt`). Any other hello falls through to the normal database check, as it does in reserve mode today.
- The database placement path places no new desktops on it, because its row says `reserve`. It also never re-places the held cell's desktops, because `liveReserveCell` keys on the switch (`relay_cell_admit_modes`), not on this row.
- New placements during the outage go to cells already in database mode (c32, c33 and c34 today) plus the tripped slot holders.
## Why
The database budget (Cloud SQL `orca-cloud-auth-db`, shared with auth, max_connections 500, 490 usable) currently reserves every relay cell's full pool. The only reason for that is the dead-man trip of the whole fleet at once. Spare is 35 on main and about 3 once C36/C37 land, which caps the number of Asia cells.
Budget arithmetic (`relay-cloud-sql-connection-budget.mjs` on orca-cloud main: directors 15, auth 20, API 50, rollout overlap 65, maintenance 5, explicit reserve 10):
| | Cells line | Operating | Spare |
|---|---:|---:|---:|
| Main today: 6 db/legacy cells at 10, 16 US at 8, 7 Asia at 16 | 300 | 455 | 35 |
| Plus C36/C37 at 16 | 332 | 487 | 3 |
| Same fleet, `deadManTripSlots=2`, Asia reserve cells budgeted at 12 (measured peak) | 76 + 128 + 8x12 + 2x4 = 308 | 463 | 27 |
| Plus two more Asia cells at 12 | 332 | 487 | 3 |
So with two slots, two more Asia cells fit in the room C36/C37 leave today. Two caveats keep this honest:
- The bound holds only once every reserve cell runs this image with the same flag value. An older image ignores the flag and trips freely, though its `db` row still takes a slot from the capped cells. With mixed values the cap is the largest value set.
- Counting reserve cells at their measured peak is a softer guarantee than counting their configured pool: the pool is still 16, and a reserve cell could burst past 12. Making it a hard bound needs a pool ceiling that depends on the mode (reserve-mode ceiling 12, the full 16 only for a slot holder or a deliberate db cell). That is a follow-up; this PR does not change pool sizes.
Alternatives considered:
- Longer or ranked trip delays only spread the trips out. A trip is latched until the switch is rewritten, so after the longest delay every cell is still in database mode. That bounds the burst, not the budget.
- A static per-cell "may trip" list in the switch file needs no database coordination, but a listed cell that is down or draining wastes its slot, and an operator has to keep the list current.
- Removing the fallback (cells never trip) is the target design, but it waits on the director stall fix and two weeks of clean running. `deadManTripSlots=0` gives exactly that per cell, so it is the same switch.
Deliberate db flips (for example, flipping reserve cells to db for an image roll) are outside the cap. Budget each Asia cell flipped that way at its full 16. With a two-cell Asia roll batch running during an outage, the slot row above is 316, which leaves room for one more Asia cell rather than two.
Rollout: the flag is dark. Setting it is an `Operate Relay Production Cell Flags` write, per cell, after every reserve cell runs this image.
## Linked Issue
N/A (internal relay capacity work).
## Visual Proof
N/A: relay server behaviour only, no UI.
## Testing
- `cell-reserve-dead-man.test.ts` (new): trips as before without the flag; held until a permit, then latched; slot 0 never trips; a permit for a silence that contact ended, or for an earlier switch generation, is dropped.
- `cell-admit-effective-writer.test.ts`: the claim waits for an in-flight reserve write; a refused claim keeps the reserve row fresh and retries after 30 s; no claim while the pool has waiters or the switch is off reserve; a tripped row is refreshed past a saturated pool after 5 minutes, a reserve row is not.
- `cell-reserve-dead-man-slots.blackbox.test.ts` (new, real cell on SQLite): a director outage with the only slot taken by another cell that writes nothing but its own row: the cell stays reserve across several claims, re-registers nothing, keeps its desktops connected, and lets a redialing desktop back in from memory; once the peer writes `reserve`, the cell trips and re-registers. Also: slot 0 rides out 30 minutes of outage without tripping; no flag trips as before.
- `cell-dead-man-trip-slot-postgres.test.ts` (new, PostgreSQL): eight cells claiming together get exactly two slots; a live holder keeps its slot; a stale one frees exactly one. Removing the advisory lock fails this test.
- Mutation checks: making the slot count always 0, or removing the lock, each fails a test.
- Full relay suite, including the PostgreSQL suites against a local PostgreSQL 17: 173 files, 1564 passed, 3 skipped. The pr.yml static-analysis steps pass locally.
- Two rounds of adversarial review. Round 1 found that the first version counted holders by director-written heartbeats (fixed). Round 2 found the brownout aging (fixed) and the latched slots after an outage (documented above).
- [x] I manually tested these changes locally
- [x] Automated tests added/updated, or explained why not below
## Review
## Agent skill upstream boundary
- [x] Not applicable, or this change copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.
## Notes
No new table or column. The claim reads `relay_cell_admit_effective` and `relay_cell_admit_modes` (one row per cell). Under a fleet-wide trip, held cells add one short transaction each per 30 s; the advisory lock is held for about one round trip.
## Checklist
- [x] This PR is small and focused
- [x] I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
- [x] Before/after screenshots or videos attached for UI changes, or `N/A` with reason
- [x] Self-reviewed for correctness, security, and performance
- [x] Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
- [x] `pnpm lint`, `pnpm typecheck`, `pnpm test`, and `pnpm build` pass (or CI will cover; local preferred)
|
||
|
|
e4d1c981a4 |
feat(relay): rehome reserve hosts into their own region on reconnect (off by default) (#32191)
Rehome on reconnect for reserve-mode cells: a desktop reconnecting on a reserve cell outside its preferred region is placed fresh in that region, only while the region has at least 10% of seats free, at most once a day per desktop, rate-limited, and switched off by default (ORCA_RELAY_RESERVE_REHOME_PERCENT=0); the regional rehome switch stops it. |
||
|
|
07a07a0326 |
fix(relay): stop reviving a session outside the cell's map, keep the seat feed on live sessions, and log demote outcomes (#32149)
* fix(relay): repair cell seat-log drift, and keep a new session's generation above a listed seat
Directors mirror each cell's seat feed, so the feed must list exactly the cell's live sessions.
The cell now sweeps its seat log against its sessions on a director's poll: a seat no session
backs is left (4409) and the live session listed, and a live session the feed lost is listed
again, each once two sweeps in a row find it. A demotion naming an unbacked seat repairs it at
once. A host's new session now starts above any generation the feed still lists, which closes
the drained-session reconnect race that removed a live seat.
Claude-Session: asia-drain-pulse
* fix(relay): refuse a rebind whose session left the map during activation, and keep a per-host generation floor
A rebind whose database activation outlived the orphan grace (or a drain) revived the removed
session outside the map: connected, still in the seat feed, and unreachable by phones or
demotion for as long as the host stayed up. Activation now rechecks the map after its awaits and
refuses with PEER_DROPPED so the host redials. Generations come from a per-host floor that
outlives the session, and an activation at a generation already used here is refused.
Claude-Session: asia-drain-pulse
* fix(relay): log each demote outcome on the director by host digest
The director now reads the cell's existing {v:1, outcome} demote reply and logs it; the cell's
feed stays the only source of its map. The supersede line now reaches the log as well.
Claude-Session: asia-drain-pulse
|
||
|
|
d667e54b02 | refactor: colocate relay assignment admission owners (#32142) | ||
|
|
e49236f1ec |
feat(relay): count and log phone-path failures, and every attach deadline (#30656)
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode Step 6 of the relay redesign keys every phone-path change on this switch. The cell reports the effective value in runtime status, and the flag tool refuses phonePath=memory unless the cell is reserve, and a reserve flip that would turn phones over unnamed. * feat(relay): resume paired phones from one device read on memory cells With phonePath=memory in reserve mode, a phone resume is admitted from one device read and the cell's own seat, and takes no activity leases. A refused device closes 4401, a failed read 4503, and only a missing device row falls through to the invite path. A phone whose host has no seat gets one assignment read: 4409 when the row names another cell or the read fails, otherwise 4404. * feat(relay): keep a memory resume's basis in the cell and confirm it from one read A phone resumed on the memory path keeps its connection basis in the cell, not in relay_connection_bases. Its confirm rereads the device row once, renews it only when less than 29 days remain, and always answers a current-version confirm renewed:true with the expiry the database holds. Confirms route by where their basis lives, so a flip between accept and confirm completes either way. Retries answer from a 10-min in-memory result map. * test(relay): roll a memory phone cell back with confirms in flight Flipping back by admitMode=db, phonePath=db or a dead-man trip leaves in-flight confirms to finish on their memory basis, writes no phone lease and leaks none, and evacuation still refuses the cell while Postgres records it as reserve. * feat(relay): batch a memory cell's resume-confirmed audit rows Memory confirms buffer their audit rows and the cell writes them as one multi-row insert every 5 s or 500 rows, idempotent by a per-event id. The buffer holds at most 5,000 rows and drops a row older than 10 minutes rather than retrying it, counting and logging both kinds of drop. * feat(relay): count and log phone-path failures, and every attach deadline The runtime metrics line gains phone refusals by close code, attach deadlines, accept database errors, resume confirm failures by code, completed accepts by phone path and the resume audit drop total. Every attach-deadline 4404 now logs what the cell knew of the desktop's control (pong age, ping outstanding, control age and round trip), so a half-open control can be told from a slow data dial. * fix(relay-ops): store phonePath=db as no key, so old images keep taking writes An explicit phonePath=db left in the switch file would make the flag tool refuse every later write after an image rollback, including a watcher's admitMode=db flip back. * fix(relay): log a failed no-seat read, recheck the switch after the device read A failed assignment read for a phone with no seat now logs a rate-limited line before its 4409. A switch that turned off during the device read sends the phone down today's path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy. * fix(relay): log a failed memory renewal; test flips, replays and basis cleanup A renewal write that fails still answers the stored expiry, but now logs a rate-limited line so a persistently failing update is visible before phones age toward re-pairing. * test(relay): assert a rolled-back memory splice issues no statement when it closes * fix(relay): flush resume audit early only on crossing the batch size, drop the shutdown flush A failing database no longer takes a 500-row insert per confirm; retries wait for the 5 s timer. The shutdown flush raced the pool closing, so it is gone and the at most 5 s loss is documented. A stopped buffer no longer re-arms its timer. |
||
|
|
1cf7472d1e |
feat(relay): batch a memory cell's resume-confirmed audit rows (#30655)
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode Step 6 of the relay redesign keys every phone-path change on this switch. The cell reports the effective value in runtime status, and the flag tool refuses phonePath=memory unless the cell is reserve, and a reserve flip that would turn phones over unnamed. * feat(relay): resume paired phones from one device read on memory cells With phonePath=memory in reserve mode, a phone resume is admitted from one device read and the cell's own seat, and takes no activity leases. A refused device closes 4401, a failed read 4503, and only a missing device row falls through to the invite path. A phone whose host has no seat gets one assignment read: 4409 when the row names another cell or the read fails, otherwise 4404. * feat(relay): keep a memory resume's basis in the cell and confirm it from one read A phone resumed on the memory path keeps its connection basis in the cell, not in relay_connection_bases. Its confirm rereads the device row once, renews it only when less than 29 days remain, and always answers a current-version confirm renewed:true with the expiry the database holds. Confirms route by where their basis lives, so a flip between accept and confirm completes either way. Retries answer from a 10-min in-memory result map. * test(relay): roll a memory phone cell back with confirms in flight Flipping back by admitMode=db, phonePath=db or a dead-man trip leaves in-flight confirms to finish on their memory basis, writes no phone lease and leaks none, and evacuation still refuses the cell while Postgres records it as reserve. * feat(relay): batch a memory cell's resume-confirmed audit rows Memory confirms buffer their audit rows and the cell writes them as one multi-row insert every 5 s or 500 rows, idempotent by a per-event id. The buffer holds at most 5,000 rows and drops a row older than 10 minutes rather than retrying it, counting and logging both kinds of drop. * fix(relay-ops): store phonePath=db as no key, so old images keep taking writes An explicit phonePath=db left in the switch file would make the flag tool refuse every later write after an image rollback, including a watcher's admitMode=db flip back. * fix(relay): log a failed no-seat read, recheck the switch after the device read A failed assignment read for a phone with no seat now logs a rate-limited line before its 4409. A switch that turned off during the device read sends the phone down today's path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy. * fix(relay): log a failed memory renewal; test flips, replays and basis cleanup A renewal write that fails still answers the stored expiry, but now logs a rate-limited line so a persistently failing update is visible before phones age toward re-pairing. * test(relay): assert a rolled-back memory splice issues no statement when it closes * fix(relay): flush resume audit early only on crossing the batch size, drop the shutdown flush A failing database no longer takes a 500-row insert per confirm; retries wait for the 5 s timer. The shutdown flush raced the pool closing, so it is gone and the at most 5 s loss is documented. A stopped buffer no longer re-arms its timer. |
||
|
|
a12c592f75 |
feat(relay): add the per-cell phonePath switch, effective only in reserve mode (#30629)
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode Step 6 of the relay redesign keys every phone-path change on this switch. The cell reports the effective value in runtime status, and the flag tool refuses phonePath=memory unless the cell is reserve, and a reserve flip that would turn phones over unnamed. * fix(relay-ops): store phonePath=db as no key, so old images keep taking writes An explicit phonePath=db left in the switch file would make the flag tool refuse every later write after an image rollback, including a watcher's admitMode=db flip back. |
||
|
|
88accb2e4a |
feat(relay): log where the sticky lane's slot time goes, per pooled connection (#28908)
* feat(relay): log what the sticky slot spends its hold on, per pooled connection * fix(relay): time every sticky round trip at the PostgreSQL client, pool wait apart * docs(relay): say which round trips the sticky slot tally covers |
||
|
|
2de7381204 |
fix(relay): supersede a reserve seat from the feed once a later, newer join shows (#28716)
* test(relay): check invariant 4 (one demotion winner) in the step-5 simulation Every demotion now names its winner (the newer booking, or the row). The model flags a director that names a different winner for a divergence another already named, and a host whose two seats are each demoted for the other. A fault where directors break level ties by their own cell order shows the check fires. Violations are capped per invariant so one flood cannot hide another class. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * test(relay): bound unseen duplicates by the silence close and assert all invariants on contested ties A reserve seat the booking director's map had not seen (joined, then half-open within a poll) is never superseded and has no lease to re-check, so it lasts until the cell's control silence close (75 s + one 15 s ping). Invariant 3 now uses that bound for such seats, and the level-divergence config asserts every invariant. Stale maps now judge a divergence another director already demoted, so the config's "contested" counter (divergences judged by two or more directors) is the case invariant 4 guards. The silence-close constant now matches production (90 s, not 100 s). Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * test(relay): raise the floor on map-judged demotes and count only level contests A reconcile whose map has not seen another director's demotion still demotes in production, which raises its epoch floor; the model now does too. The contested counter counts only same-epoch pairs, the only ones invariant 4 can split. The keep-this-seat rule is one helper. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * refactor(relay): name the simulation's fault-kept seat check for the fault Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): supersede a reserve seat from the feed once a later, newer join shows A half-open seat the booking director's map had not yet seen was never demoted and held its cell until the 75 s control silence close. Directors now demote it once both joins are in the map; the simulation holds invariant 3 at the supersede bound for it. Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): outrank only past cell clock skew, reuse demoteSeat, keep invariant 3's loose clock running Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * refactor(relay): share the outrank rule with the simulation, pin the skew boundary Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): count only confirmed level demotions and every second judgment, and bound duplicates by placement records alone Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * fix(relay): a drained seat keeps its drain grace under a demote, and the feed reports each seat once Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f * refactor(relay): name the outranked-report bound and the reconcile backstop Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f |
||
|
|
7ce15106f3 |
fix(relay): keep verifying host tokens from the last served keys while auth is down (#27490)
* fix(relay): keep verifying host tokens from the last served keys while auth is down jose refetches the relay JWKS once its cache is 10 min old and rejects the token when that fetch fails, so an auth outage turned every still-valid host token into a 4401. The cell now falls back to the last set auth served for up to an hour, skips refetching for 30 s after a failure, and still rejects any key that set never had. A fresh set replaces it as soon as auth answers, so a rotation takes effect at the first successful fetch. * fix(relay): bound stale relay keys at 30 min and document the emergency purge Thirty minutes still covers a full auth outage (no token the stale keys could accept outlives ~16 min of one) and halves how long a rotated-out key stays trusted on a cell kept from auth. A restarted process holds no set and refuses host tokens until a fetch succeeds; the ops doc names that as the purge after a signing-key leak, and a test pins the fail-closed start. * test(relay): pin the 30 min stale relay key bound (29 min accepts, 31 min rejects) |
||
|
|
735e6d6c5d |
fix(relay): let a host leave a dead reserve cell without an operator fence (#27458)
* fix(relay): let a host leave a dead reserve cell without an operator fence A host whose reserve home cell died got 503 relay_home_cell_unavailable until the cell came back: the database path demands a committed fence for every capped cell. On its own /v1/assign, a host now moves off a reserve cell (Postgres records it as reserve) when the cell's heartbeat is stale and this director's seat-feed polls have gone unanswered for 45 s, unless more than a third of the polled cells look the same. Sweeps, migration pins and capped db cells keep the fence. * fix(relay): guard the dead-cell verdict by region and sample size, and floor the move on the map - No verdict when half or more of the dead cell's own region fails with it (a cross-region partition), more than a third of the fleet does, or fewer than 6 cells are polled. A withheld region or fleet verdict logs orca_relay_dead_cell_verdict_withheld, which alerts. - The move needs the step-5 plan's epoch floor, so it mints above every seat the map knows; without the plan the fence is kept. A test revives the old cell's seat above the row and checks that the ledger never writes the row back. - The move's log line carries the fleet and region counts. * fix(relay): count only feed cells and young streaks in the dead-cell guards; gate the alert - Denominators hold only cells that served this director a feed page and have capacity: legacy rows answering 404 no longer fill the 6-cell sample or dilute the fleet and region ratios. - A peer counts as failing for the guards on any current streak (a failure in the last 5 s), so a staggered partition cannot let its first cell cross 45 s while its neighbours read healthy. - Every withheld verdict is logged, sample included (a cell missing from the feed cells is one). - Tests: ready=0 and enabled=0 with failing polls keep the fence (drain then death is fenced). - The withheld metric and alert are gated by relay_dead_cell_verdict_alert (default false, on in staging); the alert docs now describe poll failures, not heartbeats. * fix(relay): always sample the dead-cell candidate, so a restarted director can still judge it A director that restarted after a reserve cell died never received a feed page from it, so the cell was never in the sample and its verdict stayed withheld as 'sample'. The candidate now always counts when it has capacity (the store asks only about a cell Postgres records as reserve, and only feed images run reserve); peers still count only once they have served a feed. Tests: a restart during a lone outage decides, a partition begun before the restart withholds by region, and the capacity filter. The Terraform comments now say to apply the metric first and the policy minutes later. |
||
|
|
a18883130f |
fix(relay): database sweeps and ops reads skip reserve-mode cells (step-5 census, inert) (#27046)
Rebased onto main as one commit after its parent squash-merged. |
||
|
|
fda01facc2 |
feat(relay): the director that booked a join writes its row about a second later (off) (#27048)
Rebased onto main as one commit after its parent squash-merged. |
||
|
|
5493c5e815 |
feat(relay): directors place on reserve-mode cells from memory (off) (#27047)
Rebased onto main as one commit after its parent squash-merged. |
||
|
|
abff7ab86a |
feat(relay): flip a reserve-mode cell back without disconnecting anyone (E-mix) (#27043)
Rebased onto main as one commit after its parent squash-merged. |
||
|
|
d488ce5bc0 |
feat(relay): cell reserve mode: book, admit and demote without the database (off) (#27041)
Rebased onto main as one commit after its parent squash-merged. |
||
|
|
452e44464e |
feat(relay): read-only /v1/admin/host-whereabouts on directors (step 3) (#26816)
On-call can ask a director where a host is: its relay_assignments row and
open migration (primary-key reads only), the shadow map's seats and recent
leaves with close codes, the compare verdict (pure classification, judged as
/v1/resolve sees the host, recording nothing), and each named cell's poll
status and applied flags. shadow is {state:'off'} when the poller is off.
Deploy, monitor and fence identities may read it; capacity and asia-proof
may not. With no user id, the map names the user; an unknown host answers
empty.
|
||
|
|
404e54d7f6 |
feat(relay): dark shadow seat compare on sticky verify and /v1/resolve (step 3) (#26813)
After the database answers a reconnect verify or a phone resolve, the director compares that answer with the shadow seat map and counts the result by class, logging one count line per route and class a minute and a few rate-limited samples of unexplained disagreements (cell and epoch only, never raw ids). It runs only when the seat poller is on, inside try/catch, and never changes a response. A map-behind mismatch is explained only while this director first saw the newer epoch within 10 s (recorded only by the compare; classification is pure). A seat on a cell not polled within 10 s is cell-unpolled. A null answer for a seat on a cell placement's liveness rule calls dead is map-only-cell-unlive, or cell-unlive-pending when its heartbeat ran out after the last cell-list read. |
||
|
|
8848ca95d6 |
feat(relay): key id on assignment leases; cells shadow-check the lease (off) (#26815)
* feat(relay): key id on assignment leases; cells shadow-check the lease behind ticketCheck=shadow (off) Directors name the lease signing key with a kid. With the per-cell ticketCheck switch at shadow, a cell verifies the lease a host echoes in x-orca-relay-assignment-lease, compares it with the database answer that still decides, and logs per-class counts and the hello's database read time once a minute. * feat(relay): flat per-class totals on the lease shadow line, so log metrics can extract them * refactor(relay): the lease shadow line carries the flat classes and one database-refused count The per-class-and-verdict map duplicated the flat classes; disagree already names the valid-lease refusals, and dbRefused gives the total. |
||
|
|
f514cb2c66 |
feat(relay): dark shadow seat directory poller on directors (step 3) (#26810)
Directors can poll each cell's read-only seat feed (#26803) once a second and keep an in-memory map of which host is seated on which cell, plus a bounded 24 h recently-left memory. Off unless ORCA_RELAY_SHADOW_SEAT_FEED_CELLS names cells. Applies a change at or above the seat's generation, follows cut pages, resyncs on a gap or new incarnation, and checks the map against counts.seats. A 404 reads as no feed; timeouts and errors keep the last seats as unverifiable. The cell list is every cell (an existing-only cell still seats hosts), from RelayAssignmentStore.seatFeedCells with placement's liveness rule; completeness waits only on live cells, and a failed list read backs off. Nothing reads the map yet. |
||
|
|
de6d49d52f |
feat(relay): local /ready behind the per-cell readinessLocal switch (off) (#26812)
* feat(relay): local /ready behind the per-cell readinessLocal switch (off) With the switch on, a cell's /ready answers 200 while it listens and its host token verifier holds keys, and reports failing probes as degraded. The heartbeat keeps the SQL-gated verdict, so directors still stop placing on a cell that cannot reach the database. * refactor(relay): the app verifies relay tokens with the server's shared key set One cached JWKS per process now serves the host-control upgrade, HTTP token checks and local readiness, instead of a second set in the app. |
||
|
|
f0dd050a23 |
feat(relay): per-cell runtime switches read from Cloud Storage every 5 s (#26805)
* feat(relay): per-cell runtime switches read from Cloud Storage every 5 s Cells read cells/<cellId>.json from the <project-id>-relay-control bucket, start on all-off defaults, keep their applied values when the object cannot be read, and report what they applied in the seat feed. No consumer reads the switches yet. * test(relay): exempt the flag channel's poll from the floating-promise census; it catches every failure * fix(relay): release every switch-file reply the reader does not consume, and bound the read while streaming 404, 5xx, malformed-generation and lower-generation replies, and failed metadata replies, now cancel their bodies. The object read stops and cancels at the first byte past 64 KiB instead of buffering it whole. * feat(relay): runtime status reports the applied cell flags for the flag workflow's read-back * docs(relay): the switch channel reads per-cell objects only |
||
|
|
37d2154765 |
feat(relay): read-only seat feed on cells for the directors' shadow directory (#26803)
* feat(relay): read-only seat feed on cells for the directors' shadow directory Cells keep an in-memory log of host joins, leaves and drain notices and serve it at GET /v1/admin/cell-seats, authenticated with the directors' existing rehome identity. Nothing reads it yet. * test(relay): read the seat feed through its published shape so the tests typecheck * fix(relay): serve the seat feed's full snapshot from the change log's own seats counts.controls drops when a socket starts closing, but the leave lands only at its close event, so a full snapshot built from open sockets could disagree with a reader that applied every change. The log now keeps its own seats under the readers' generation rule and reports counts.seats. * fix(relay): report auth-expiry drain-only and its refresh as seat-feed changes The heartbeat's auth-expiry drain-only and an auth refresh back to active changed session state without a feed entry, so a reader's seat state drifted from the cell's. Both now go through the log; the refresh is a new 'active' kind. * refactor(relay): cut unread seat-feed fields; verify a feed poll once fixLevel, counts.splices, counts.enforcedUnits and counts.hardCap had no reader, so they leave before they become a cross-version contract. The admin middleware no longer verifies /v1/admin/cell-seats polls, which the route already gates on the rehome identity. |
||
|
|
80d7e2dded |
fix(relay): spread drain placement over a load band instead of one empty cell (#26802)
* fix(relay): spread placement over a load band so a drain does not land on one empty cell Placement took the single least-loaded cell, so a just-rolled cell took a whole drain and every follow-up serialized on its relay_cells row. Directors now pick at random among cells within 5% of capacity of the minimum, and skip a cell that already took over 40% of its pool's placements in the last 10 s (floor 4). It only reorders: the least-loaded cell is the fallback, the hard cap and region preference are unchanged. * fix(relay): size the load band's intake window for the 1,200 s drain pace At ~0.5 picks/s per director, a 10 s window with a floor of 4 let the floor decide and the empty cell took ~65% of a c27-sized drain. A 60 s window and a floor of 2 keep it at 40% at both the 300 s and 1,200 s paces across five directors. Adds a once-a-minute orca_relay_placement_load_band line (picks, budget skips, fallbacks, band size, picks per cell). * chore(relay): log the load-band summary at info, not warning |
||
|
|
3062b9d8d0 |
fix(relay): refuse host hellos fast when the database pool is timing them out; renewals jump the queue (#26362)
* fix(relay): refuse host hellos fast when the database pool is saturated; renewals jump the queue A reconnect herd (2026-10-06 18:34Z, 1,046 asia-east2 hosts dropped by the load balancer at once) queued hundreds of host hellos behind each cell's 16-connection pool. They timed out after 2 s, the desktops redialled into the same queue, and control-lease renewals starved behind them. - Host control upgrades get an immediate 503 (Retry-After: 2) once 32 callers wait for a pooled connection. Shipped desktops already treat that as a connect error and back off with jittered exponential retry. - General work now queues in PostgresPoolPressure instead of pg-pool, so renewal statements (priority lane) take the next released connection. - A renewal batch that never got a connection no longer fans out into one statement per host. - hostHellosShedDelta counts refusals in the runtime metrics. * fix(relay): shed host hellos on the pool's oldest wait, not its queue length asia-east2 queues of 50-196 waiters are routine and keep moving, so a 32-waiter limit would refuse hellos that were going to succeed. Shed only while the oldest pool waiter has waited 1.5 s of the 2 s acquire timeout, when a new hello would time out anyway; a refused desktop gets the same connect error and backoff it gets today, 2 s sooner. Rebinds over a live control (lease rotation) are never refused. * fix(relay): shed hellos only when a full pool queue has aged 1 s Per review: require both an oldest waiter of at least 1 s and a pool's worth of waiters, so one slow waiter cannot trip it, and state the real baseline (p99 of 2 waiters on 10-07). |
||
|
|
10bea3a8fa |
feat(relay): report lane service times, 503 causes, per-site hold p99 and a director-vs-cell lock-wait split (#25645)
* feat(relay): report lane service times, 503 causes, per-site hold p99 and lock-wait split Adds the step-1 observability: drain-return and sticky slot service times, every /v1/assign 503 by cause with a non-drain total, a p99 per lock hold site (drain-return regional rows get their own site), a director-side pg_stat_activity sample that splits lock waiters by director vs cell, and a log line for each reserved_requests drift reconciliation corrects. Log-based metrics for the new fields are declared in Terraform, not applied. * fix(relay): classify a lock waiter by the first relay table its statement names * fix(relay): attribute lock waiters to the root holder; document the director hold alert change Every waiter after the first in a row-lock convoy is blocked by the first waiter, so the sample now walks pg_blocking_pids to the root, reading it once per waiter. The sampler is single-flight. Drain-return regional holds feeding the non-paging director hold policy is documented as expected. * docs(relay): note the lock-wait sample undercounts director waiters when the pool is full |
||
|
|
b82307937a |
fix(relay): admit drained hosts through their own lane and stagger their return (#24446)
* fix(relay): admit drained hosts through their own lane and stagger their return A same-cap roll's drain sends every host on the isolated cell back to the director at once. Those hosts reconnect through the sticky lane (one slot per director), and each one's re-placement holds that slot for most of a second behind the region-wide inventory lock, so ordinary reconnects time out behind them and the drained hosts retry every 2 s: ~30k 503s per drain. The reconnect verification read now also says whether the host's home cell is isolated for a roll right now (same predicate re-placement uses). Those hosts release the sticky slot after the read and take a separate drain-return lane (1 per director, matching the store's per-director placement serialization). When that lane is full the host gets a Retry-After that reserves the next free service slot, paced by the measured re-placement time and capped at 300 s. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 * fix(relay): keep drain returns inside the placement pool budget and their own slot Review follow-ups for the drain-return lane: - The lane now borrows placement permits (never placement's last, never ahead of a queued placement), so placement + sticky still bounds the database pool. - A host's own early retry (row-busy redial, duplicate dial) gets the 2 s lane interval instead of a fresh slot behind the cohort, and a host that returns early to the same director keeps its reserved slot. - Classification also excludes an open migration row whose lease counter lapsed, matching the re-placement rule. - The load test now runs five directors behind random routing with a shared inventory lock. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 |
||
|
|
8da0ca52e7 |
perf(relay): release rejected first-frame connections [trade-off] (#23011)
* perf(relay): release rejected first-frame connections [trade-off] * fix(relay): bound the director's rejected and redirected first-frame closes too The parent PR routed four cell-side first-frame rejections through closeRelayWebSocket but left two raw socket.close() calls in the same handler. A real-socket probe shows both still pin a connection unit for ws's full 30s close timer when the peer ignores the close frame: - 'invalid invite' is reachable by an unauthenticated peer with a well-formed but bogus credential, so the exhaustion the parent PR claims to prevent stayed reachable on the director; - 'connect to assigned cell' is the happy path for every phone's first director contact, so it is the highest-volume unbounded close here. closeWithDrain gets the same treatment; host-session-registry already closes the identical drain through the helper. Also records that the bounded close is not a user-facing trade-off: the close frame is written before the force-close timer can fire and TCP delivers it ahead of the FIN, so an abandoned peer still reads code and reason over a graceful close. The new regression asserts that, plus exactly-once release across concurrent bursts and rejection racing the peer's own disconnect (the ledger does not clamp at zero, so a double release would surface as a negative count). * refactor(relay): drop the unused closeWithDrain helper `closeWithDrain` has no callers anywhere in the repo, and its `graceMs` parameter promised a caller-supplied drain window that the body no longer honours: routing it through `closeRelayWebSocket` force-terminates after 1s regardless, so a future caller passing `graceMs: 30_000` would have had its drain silently cut short while the signature still claimed otherwise. The real drain path is `host-session-registry`, which sends the same `resolve-director` drain and closes it there. Delete the dead duplicate rather than bound a helper whose contract says "graceful". Co-Authored-By: Claude <noreply@anthropic.com> * docs(relay): call the bounded close's rejection delivery best effort, not guaranteed The helper claimed the forced terminate costs an abandoned peer nothing it can observe, and the test presented its fast-peer assertion as proof. Neither holds in general: `ws` writes the close frame to the socket and `terminate()` destroys that socket a second later, so under backpressure the frame — and any `relay-moved` message queued ahead of it — can go unsent even to a peer that never stopped reading. Qualifies both comments to describe delivery as best effort and name the backpressure case. The fast-peer assertion is valid and stays exactly as it was; only its stated scope narrows, and the test is renamed to say which peer it speaks for. What the bound actually buys — a stalled peer cannot hold admission — is now stated on its own rather than resting on a delivery claim. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
1043dc5e1d |
fix(relay): stop rehoming hosts off Asia cells until the lock fix lands (#22443)
The source cell runs the rehome commit. An Asia source pays a cross-ocean round trip per statement while holding relay_cells row locks every cell needs, which convoys the fleet database. Selection now drops source cells outside the director's region before building the decision window, so the incumbent_region filter shrinks while Asia cells stay valid targets. The preview counts the same hosts as source-outside-director-region and the poll summary reports skippedOffRegionSourceCells. Temporary stopgap. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 |
||
|
|
fa0010e8d6 |
fix(relay): abort rehomes whose host never arrived, without disabling the switch (#21813)
A regional rehome whose host went offline right after accepting the move left its migration row open forever: the target had registered it, the host held nothing on the source, and the completion sweep could never finish it. Eight such rows filled REGIONAL_REHOME_CONCURRENT_LIMIT and every later candidate came back deferred, silently, for 21 hours. The only sweep that touched them fires at 24 hours and also sets enabled = 0 on the durable control, so the first leak to age out would have turned rehoming off, repeatedly. Adds a director sweep that rolls such an attempt back to its source after one migration lease, with abort_reason = 'host_not_arrived', reusing the existing rollback (assignment epoch bump back to the source, lease removal, superseded target reservation release) and leaving the switch untouched. The 24-hour sweep keeps its disable as a last-resort latch. The source cell now names why it deferred, on a new optional response field, and the director stops walking its candidate page on a deferral no later candidate can pass. Each poll that dispatched logs one summary line. Claude-Session: https://claude.ai/session/ced32ebb-7155-4413-adad-1eccd14c2010 |
||
|
|
0e7948fa6d |
feat(relay): pace the drain send during a same-cap cell roll (#21284)
* feat(relay): pace the drain send during a same-cap cell roll A same-cap roll drains a cell with graceMs 0, which sends `drain` to all ~800 controls in one pass. Every desktop re-dials on receipt regardless of graceMs, so the whole cell reconnects inside a second. On 2026-09-16 that stampede hit a Cloud SQL stall: attaches timed out, each leaving 10 minutes of late-arrival debt on connection headroom, and placement answered relay_capacity_exhausted fleet-wide for ~13 minutes. Spreading the sends spreads the re-dials. `HostSessionRegistry.drain` takes an optional pacing window and schedules each session's send evenly across it; admission is fenced for every session up front, and each host keeps its own full grace after its own send. /v1/admin/drain accepts `paceWindowMs` (<= 5 min) and echoes what it applied. The same-cap job asks for 120 s, and the drain-completion wait grew by the same amount. A cell still on an older image rejects the field, so the deploy script falls back to an unpaced drain rather than failing the roll. * fix(relay): scope the drain fence to the hosts already told Review of the paced drain found two problems, both from treating "this cell is draining" as one instant when pacing makes it a window. Timers: the sends queued by a paced drain were neither cleared when a later drain superseded them nor unref'd. A SIGTERM mid-window left up to 800 no-op timers holding the event loop open until systemd escalated to SIGKILL. Drain timers are now tracked, cleared on the next drain, and unref'd, so a retry re-arms a session's teardown instead of stacking a second one. Phones: the client fence read the global draining flag, so every phone was refused for the whole window even though its own host had not been told yet and was still serving. The director keeps pointing phones at this cell until their host moves, so they would have looped for up to two minutes. A session is now fenced when its drain is sent, not when the drain starts, and the client paths key off that. New control connections and re-attaches stay fenced globally: nothing new should land on a cell that is going away. |
||
|
|
69787e763a |
fix(relay): serve readiness from last-known-good during auth or SQL blips (#21161)
* fix(relay): serve readiness from last-known-good during auth or SQL blips
The load balancer health check hits /ready, which re-probed the auth JWKS
endpoint and Postgres on every poll and reported not-ready on the first
failure. On 2026-09-16 an auth outage therefore took every cell out of the
load balancer within ~30s and dropped every connected host, even though the
token verifier caches keys in process and kept verifying tokens.
/ready now remembers when each dependency last answered and keeps reporting
ready while the failed one stays inside a grace window
(ORCA_RELAY_READINESS_GRACE_MS, default 15 minutes, 0 disables). A process
that has never succeeded still gates on the real dependencies, so cold boot
is unchanged. Grace answers carry degraded plus the failure reason on the
existing readiness observation, and entering or leaving grace logs once.
MIG autohealing still uses the dependency-free /health endpoint.
* fix(relay): split readiness grace per dependency and probe both every poll
Review follow-ups on the last-known-good readiness window.
An unset environment variable arrives as an empty string, which z.coerce
reads as 0, so the single ORCA_RELAY_READINESS_GRACE_MS would have switched
the window off instead of falling back to its default. The two replacement
variables preprocess '' to undefined.
JWKS and SQL now get separate windows and separate clocks:
ORCA_RELAY_READINESS_JWKS_GRACE_MS defaults to 15 minutes, and
ORCA_RELAY_READINESS_SQL_GRACE_MS to 3 minutes. Each cell is its own load
balancer backend, so failing readiness never re-routes a host, it only makes
that hostname unreachable, and a host that lands on a SQL-dead cell gets
WRONG_CELL and is re-placed by the director. Three minutes rides a Cloud SQL
failover without hiding a per-cell fault for a quarter of an hour.
Both dependencies are probed on every poll. A JWKS failure used to
short-circuit the SQL probe, which let the SQL clock age with no evidence
behind it. Grace transitions are emitted per dependency, so JWKS recovering
while SQL fails logs both sides instead of nothing.
/ready keeps its 200 and its {ok:true} body when healthy, and adds
degraded plus the dependency list when the answer comes from a window.
|
||
|
|
cd9aa43a2c |
feat(relay): correct regional placement only when the source is idle (#20105)
* feat(relay): correct regional placement only at an idle source * test(relay): lock source activity capacity semantics |
||
|
|
91d7783f2b |
fix(relay): state pending-conn details to hosts that advertise the capability (cell side) (#19266)
The cell announces a connection with a single conn-open. When the desktop's control socket dies mid-accept the phone waited out the 10s attach deadline and was closed HOST_OFFLINE, even though the desktop was online. host-hello-ack already restates those connections in pendingConns, but only by connId and connTicket, which is not enough for the desktop to dial: kind and relayDeviceId decide the pairing authority a connection carries and the E2EE device binding, so neither may be guessed. The cell now states kind and relayDeviceId on each pending entry, but only to a host that advertised it can read them: a shipped host parses those entries strictly, so an unannounced key fails the whole ack parse and kills a working control. The advertisement rides the control upgrade as x-orca-host-capabilities, not host-hello, because HostHelloSchema is strict on the cell too and any new hello key is refused by every already-deployed cell. The capability is keyed by socket, not by session: a rebind can land a successor whose decoder is older or newer than the one that opened the session, and the ack must follow the socket that will actually read it. With no capable host in the fleet the emitted ack is byte-identical to today's. The desktop half that consumes the new fields is #19238. |
||
|
|
61b09b7a02 |
fix(relay): abandon dead client accepts, jitter and lengthen the control lease, fail direct probes fast (#18959)
* fix(relay): abandon a client accept once the phone hangs up; jitter the control lease The accept runs several serialized Postgres calls behind the contended cell-inventory lock, and phones bound their dial. Finishing that work for a phone that had already left acquired (and leaked for 90s) an activity lease and then failed at bind with host_data_reservation_already_bound. Check the client socket between the DB steps and unwind what was taken, reporting the stage on orca_relay_client_accept_abandoned. Jitter the control lease grant so a cohort that reconnected in the same minute (a cell recreate dumps hundreds at once) walks apart instead of rebinding together every cycle. On the phone, treat a probe session that enters 'reconnecting' as a failed probe: it is the direct client's own backoff after a dead-LAN 1006, and waiting it out held the supervisor's operation mutex for the full 12s bound. * perf(relay): lengthen the control lease to 6h The lease bounds how long a host lingers on a cell after a missed drain, and rebinding it is the only passive rebalancing we have, so it stays finite. 6h keeps both properties while cutting control-activation traffic on the contended cell-inventory lock ~6x. The relay JWT (5 min, refreshed by the desktop) and the 75s silence watchdog are enforced separately, so the longer grant authorizes nothing extra. The jitter widens with it, to +/-30 min. * fix(relay): let one flap recover the direct probe; correct the leak window 'reconnecting' is published on any socket close, so rejecting on it outright turned a single access-point flap into a booked direct failure and a 60s cooldown. Give the first 'reconnecting' a 2s grace in which a 'connected' transition still resolves; a dead LAN still fails in ~2s rather than holding the supervisor's operation mutex for the 12s bound. The abandoned accept held its activity lease for the 10s attach deadline, not 90s -- the attach timer is armed before bind throws and already unwinds it. Also cover the assignment-stage check that guards reserveCredential, and drop a spread assertion the two exact-value assertions above already imply. * fix(relay): extend the probe grace once on a handshake; pin the lease band top The redial fires at 500ms but 'connected' waits on the Noise handshake and a capability RPC, so one 2s window is too tight for real work. A 'handshaking' transition is evidence the peer answered, so extend the grace once; a stalled handshake still fails at ~3.5s, far inside the 12s bound. The longest-lease case only had an upper bound, which a jitter clamped to one side would satisfy. Pin it to the exact top of the band instead, and assert the assignment resolve ran so the third-guard test cannot pass vacuously. |
||
|
|
11aace8dec |
fix(relay): reject malformed percent-escapes on upgrade instead of throwing (#18547)
decodeURIComponent on the /v1/connect/ and /v1/host/data/ path segments threw URIError out of the http 'upgrade' listener, which is uncaught and kills the relay process. Any client that sends GET /v1/connect/% could take down a cell (and every connection on it) or a director instance. Pre-existing since the splice landed (orca-cloud #20); not introduced by the import. A malformed escape now takes the existing 4xx reject branch. The blackbox test sends three malformed connect targets and one host-data target to the real server and asserts no uncaughtException fires and a well-formed upgrade still gets 101 afterwards; reverting either site fails it. |
||
|
|
3eec77c11a |
chore(cloud): add the relay fence broker, ops console, Terraform root, scripts, and 24 cloud-* workflows (#18413)
Phase 6 of the relay split: the relay's deploy/operate surface moves under cloud/ with 24 cloud-* workflows gated on ORCA_CLOUD_OPERATIONS_ENABLED, the Cloud SQL rollout lease action, the relay Terraform root (dual-accept identities for both repositories), scripts, docs, CODEOWNERS, and a terraform validate job in Cloud Verify. |