54 Commits

Include renames

Author	SHA1	Message	Date
OrcaWinandOrcaWin	7d6603aa17	[comment audit] Complete repository-wide comment cleanup (#33249) ... * [comment audit] Align legacy failure-read counts with cached display * [comment audit] Condense shared provider lifecycle and browser recovery comments [skip ci] * [comment audit] Condense shared generation, path and preview comments [skip ci] * [comment audit] Condense host, file and Git contract rationale [skip ci] * [comment audit] Tighten Git lifecycle and provider contract comments [skip ci] * [comment audit] Clarify shared host authority and launch isolation [skip ci] * [comment audit] simplify local access mobile and native chat contracts [skip ci] * [comment audit] simplify native chat presentation and state contracts [skip ci] * [comment audit] simplify native runtime and provider contracts [skip ci] * [comment audit] simplify migration and orchestration contracts [skip ci] * [comment audit] simplify plugin trust and process capture rationale [skip ci] * [comment audit] simplify terminal delivery and remote compatibility rationale [skip ci] * [comment audit] clarify remote recovery and launch wire rationale [skip ci] * [comment audit] shorten shared RPC compatibility and runtime evidence comments [skip ci] * [comment audit] simplify secure storage shell startup and sidebar rationale [skip ci] * [comment audit] clarify sidebar, skill and SSH invariants [skip ci] * [comment audit] condense structured chat delivery and timing rationale [skip ci] * [comment audit] condense terminal replay and telemetry invariants [skip ci] * [comment audit] clarify launch permission and terminal delivery rationale [skip ci] * [comment audit] clarify workspace ownership and layout contracts [skip ci] * [comment audit] wait for fitted and painted SSH terminal links * [comment audit] clarify layout persistence and session ownership [skip ci] * [comment audit] clarify host routing and workspace lifecycle contracts [skip ci] * [comment audit] clarify startup persistence and app lifetime comments [skip ci] * [comment audit] clarify recovery and activity boundaries [skip ci] * [comment audit] condense activity and artifact invariants [skip ci] * [comment audit] condense automation authority contracts [skip ci] * [comment audit] simplify automation scheduling and history contracts [skip ci] * [comment audit] clarify browser chrome and annotation lifecycles [skip ci] * [comment audit] clarify browser routing and retained guest ownership [skip ci] * [comment audit] clarify browser recovery and document ownership comments [skip ci] * [comment audit] restore hosted runners required by native IME fixtures * [comment audit] shorten preview, palette and dashboard invariants [skip ci] * [comment audit] clarify terminal preview, dashboard and retention comments [skip ci] * [comment audit] clarify diff-note recovery and editor lifecycle comments [skip ci] * [comment audit] shorten rich-text and combined-diff ownership comments [skip ci] * [comment audit] shorten autosave, draft recovery and model ownership comments [skip ci] * [comment audit] shorten save-queue, notebook and Markdown constraints [skip ci] * [comment audit] shorten Monaco, PDF and rich Markdown safeguards [skip ci] * [comment audit] clarify rich Markdown source and selection safeguards [skip ci] * [comment audit] clarify editor loading and emulator stream lifetimes [skip ci] * [comment audit] trim emulator and feature tour narration [skip ci] * [comment audit] clarify floating workspace and GitHub cache ownership [skip ci] * [comment audit] trim provider controls and Linear filter explanations [skip ci] * [comment audit] clarify pairing and chat resume lifetimes [skip ci] * [comment audit] clarify chat delivery and transcript lifetimes [skip ci] * [comment audit] clarify session options and host-owned status [skip ci] * [comment audit] clarify draft durability and live-status precedence [skip ci] * [comment audit] Trim chat send and model ownership explanations [skip ci] * [comment audit] Shorten transcript ownership and send recovery comments [skip ci] * [comment audit] Simplify native chat composer and transcript hook comments [skip ci] * [comment audit] Shorten native chat scroll and structured session explanations [skip ci] * [comment audit] Trim workspace picker and onboarding commentary [skip ci] * [comment audit] Simplify pet rendering, review cache and quick open comments [skip ci] * [comment audit] Trim sidebar and session history explanations [skip ci] * [comment audit] Condense session resume and review checks comments [skip ci] * [comment audit] Shorten checks state and file explorer refresh comments [skip ci] * [comment audit] Trim plugin bridge and source-control action rationale [skip ci] * [comment audit] Trim source-control listing and review rationale [skip ci] * [comment audit] Shorten sidebar lifecycle and explorer safeguards [skip ci] * [comment audit] Condense Settings account and appearance rationale [skip ci] * [comment audit] Trim pairing and repository Settings explanations [skip ci] * [comment audit] Condense Settings search SSH and terminal safeguards [skip ci] * [comment audit] Shorten Settings voice and provider ownership comments [skip ci] * [comment audit] Condense Settings draft persistence and host-state rationale [skip ci] * [comment audit] Trim Settings hooks and sidebar setup rationale [skip ci] * [comment audit] Condense sidebar deletion and board-filter safeguards [skip ci] * [comment audit] Condense worktree cards, host ownership and Markdown contracts [skip ci] * [comment audit] Condense sidebar drag, folder and filtering contracts [skip ci] * [comment audit] Condense sidebar parity, sleep and host ownership contracts [skip ci] * [comment audit] Prune focus-read baseline after comment cleanup * [comment audit] Clarify sidebar visibility, board drops and agent evidence [skip ci] * [comment audit] Tighten sidebar listing, drag and reveal contracts [skip ci] * [comment audit] Clarify viewport geometry and skill update safeguards [skip ci] * [comment audit] Tighten skill inventory, usage and host status comments [skip ci] * [comment audit] Clarify usage ownership and resource kill safeguards [skip ci] * [comment audit] Tighten tab selection, rename and search contracts [skip ci] * [comment audit] Tighten tab-strip geometry and drag ownership comments [skip ci] * [comment audit] Tighten Tasks recovery, pagination and mutation comments [skip ci] * [comment audit] Tighten provider loading and terminal recovery comments [skip ci] * [comment audit] Tighten completion, cadence and PTY ownership comments [skip ci] * [comment audit] Clarify terminal restore and parked-pane boundaries [skip ci] * [comment audit] Tighten PTY startup and replay explanations [skip ci] * [comment audit] Clarify PTY identity and delivery ownership [skip ci] * [comment audit] Shorten remote reconnect and input replay rationale [skip ci] * [comment audit] Condense terminal parking clipboard and file link comments [skip ci] * [comment audit] Clarify IME ownership and trim trace history [skip ci] * [comment audit] Shorten pane recovery and parked watcher explanations [skip ci] * [comment audit] Condense snapshot replay and visibility recovery rationale [skip ci] * [comment audit] Shorten terminal parking and keyboard ownership rationale [skip ci] * [comment audit] Trim renderer lifecycle and menu rationale [skip ci] * [comment audit] Condense workspace cleanup ownership and settlement notes [skip ci] * [comment audit] Clarify renderer hook evidence and source ownership [skip ci] * [comment audit] Tighten IPC ownership and reconnect explanations [skip ci] * [comment audit] Trim renderer focus, paste and attention comments [skip ci] * [comment audit] Shorten scroll, locale and agent lifecycle comments [skip ci] * [comment audit] Trim agent launch, resume and ownership commentary [skip ci] * [comment audit] Shorten account, browser and editor comments [skip ci] * [comment audit] Clarify host-mirror waits, previews and input ownership [skip ci] * [comment audit] Condense launch recovery and editor tokenization comments [skip ci] * [comment audit] Condense workspace selection and terminal pane rationale [skip ci] * [comment audit] Condense terminal fitting and rendering recovery comments [skip ci] * [comment audit] Condense terminal input and host routing explanations [skip ci] * [comment audit] Condense ownership and sleeping-session recovery rationale [skip ci] * [comment audit] Condense structured launch and terminal link explanations [skip ci] * [comment audit] Condense terminal ownership and workspace file routing rationale [skip ci] * [comment audit] Condense session persistence and workspace activation rationale [skip ci] * [comment audit] Condense workspace routing and browser ownership rationale [skip ci] * [comment audit] Condense mirror recovery and layout authority rationale [skip ci] * [comment audit] Condense runtime transport and file safety rationale [skip ci] * [comment audit] Condense runtime graph and structured session rationale [skip ci] * [comment audit] Condense browser creation and remote tab recovery rationale [skip ci] * [comment audit] Condense snapshot authority and publisher recovery rationale [skip ci] * [comment audit] Condense frame, startup and provider cache rationale [skip ci] * [comment audit] Condense host twin and agent lifecycle rationale [skip ci] * [comment audit] Condense browser authority and editor cleanup rationale [skip ci] * [comment audit] Condense editor buffer identity and review cache rationale [skip ci] * [comment audit] Condense provider lifetime and repository ownership rationale [skip ci] * [comment audit] Condense reconnect, hydration and terminal retirement rationale [skip ci] * [comment audit] Condense recovery, reconciliation and persistence rationale [skip ci] * [comment audit] Condense cleanup ownership and listing admission rationale [skip ci] * [comment audit] Condense teardown ordering and terminal ownership rationale [skip ci] * [comment audit] Condense recovery ledger and web host-boundary rationale [skip ci] * [comment audit] Condense web reconnect and credential lifecycle rationale [skip ci] * [comment audit] Condense cloud rollout and assignment safety rationale [skip ci] * [comment audit] Condense relay session and database recovery rationale [skip ci] * [comment audit] Condense relay placement and protocol compatibility rationale [skip ci] * [comment audit] Condense build and packaging constraints [skip ci] * [comment audit] Condense CI gates and validation isolation rationale [skip ci] * [comment audit] Condense catalog identity and browser validation rationale [skip ci] * [comment audit] Condense mobile browser and frame-budget rationale [skip ci] * [comment audit] Shorten mobile route and browser test explanations [skip ci] * [comment audit] Condense editor and session seam test rationale [skip ci] * [comment audit] Trim session and terminal closure history [skip ci] * [comment audit] Shorten terminal browser and census explanations [skip ci] * [comment audit] Preserve root-only E2E diagnostic baseline * [comment audit] Condense mobile seam and HTML preview test rationale [skip ci] * [comment audit] Condense preview instrumentation and render harness rationale [skip ci] * [comment audit] Condense bundle contracts and native-build constraints [skip ci] * [comment audit] Shorten native packaging and CI routing rationale [skip ci] * [comment audit] Condense release tooling and benchmark constraints [skip ci] * [comment audit] Condense packaging and platform gate rationale [skip ci] * [comment audit] Shorten mobile route and history rationale [skip ci] * [comment audit] Condense mobile browser and component rationale [skip ci] * [comment audit] Shorten drawer and workspace-form rationale [skip ci] * [comment audit] Trim mobile review and editor-document explanations [skip ci] * [comment audit] Condense editor lifecycle and Markdown explanations [skip ci] * [comment audit] Trim mobile file, diagnostics and dictation commentary [skip ci] * [comment audit] Simplify mobile host-list and workspace-view explanations [skip ci] * [comment audit] Clarify mobile shell lifetime and bridge delivery constraints [skip ci] * [comment audit] Condense bridge stream, native request and audio rationale [skip ci] * [comment audit] Reduce bridge schema and page-client commentary [skip ci] * [comment audit] Tighten mobile page bridge contracts [skip ci] * [comment audit] Clarify mobile cache and session constraints [skip ci] * [comment audit] Condense mobile route and lifecycle rationale [skip ci] * [comment audit] Tighten mobile shell hook timing rationale [skip ci] * [comment audit] Clarify route handoff and notification compatibility [skip ci] * [comment audit] Clarify push registration and dictation cleanup constraints [skip ci] * [comment audit] Condense keyboard geometry and native media ownership rationale [skip ci] * [comment audit] Tighten mobile chat scroll and composer concurrency rationale [skip ci] * [comment audit] Condense mobile reply compatibility and upload ownership rationale [skip ci] * [comment audit] Condense optimistic echo and scroll ownership rationale [skip ci] * [comment audit] Tighten stream replay and queued session ownership rationale [skip ci] * [comment audit] Condense mobile composer and reply-schema constraints [skip ci] * [comment audit] Clarify mobile transcript recovery and session ownership [skip ci] * [comment audit] Clarify mobile queue and source-control compatibility [skip ci] * [comment audit] Condense storage and task launch constraints [skip ci] * [comment audit] Clarify task schemas and terminal document ownership [skip ci] * [comment audit] Tighten terminal input and viewport rationale [skip ci] * [comment audit] Shorten terminal and RPC recorder contracts [skip ci] * [comment audit] Condense recorder engine and connection contracts [skip ci] * [comment audit] Shorten host credential and relay recovery comments [skip ci] * [comment audit] Tighten pairing recovery and bundle transport rationale [skip ci] * [comment audit] Clarify RPC delivery and workspace catalog boundaries [skip ci] * [comment audit] Condense catalog and relay entry contracts [skip ci] * [comment audit] Shorten shutdown and SSH relay delivery contracts [skip ci] * [comment audit] Condense WSL relay and E2E regression rationale [skip ci] * [comment audit] Shorten cross-version harness and renderer fixture explanations [skip ci] * [comment audit] Condense paired-host and E2E lifecycle rationale [skip ci] * [comment audit] Shorten layout oracle and regression explanations [skip ci] * [comment audit] Condense paired recovery and ownership regression contracts [skip ci] * [comment audit] Shorten restart, sidebar and SSH lifecycle contracts [skip ci] * [comment audit] Condense terminal rendering and IME regression rationale [skip ci] * [comment audit] Shorten terminal recovery and benchmark contracts [skip ci] * [comment audit] Condense tooling and Windows lifecycle proof rationale [skip ci] * [comment audit] Condense workflow routing and release rationale [skip ci] * [comment audit] Condense mobile security and installer contracts [skip ci] * [comment audit] Shorten native rationale and stylesheet workarounds [skip ci] * [comment audit] Shorten Windows visibility probe rationale [skip ci] * [comment audit] Condense embedded release and signing rationale [skip ci] * [comment audit] Retain current main test behavior in final reconciliation [skip ci] * [comment audit] Complete final comment review across current source inventory * [comment audit] Handle consolidated audit reports before final CI * [comment audit] Preserve credential storage census detail [skip ci] * [comment audit] Scale broad changed E2E selections with verified timing shards * [comment audit] Align E2E assertions with browser close and settled session metadata * [comment audit] Match existing E2Es to current resource and windowing contracts * [comment audit] Observe the shared server outage notice in reconnect coverage * [comment audit] Make existing renderer probes measure settled tab and PTY work * [comment audit] Align palette E2E selector with task search * [comment audit] Refresh remaining palette search selectors * [comment audit] Keep selector correction within selected audit specs * [comment audit] Await emoji fixture settings inside renderer evaluation * [comment audit] Cancel fixture fit retries between terminal tests --------- Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>	2026-10-25 04:05:46 -07:00
Jinwoo Hong	0ae42fe594	feat(relay): cap how many reserve cells a director outage trips to the database (#33415) ... <!-- orca-pr-loc --> <!-- Programmatic LoC summary. Do not edit by hand; rewritten on every commit. --> | | Files | Added | Deleted | Net | | :--- | ---: | ---: | ---: | ---: | | Test | 4 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​406 | $\color{#cf222e}{\Huge{\mathbf{−}}}$​4 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​402 | | Prod | 6 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​165 | $\color{#cf222e}{\Huge{\mathbf{−}}}$​10 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​155 | <!-- /orca-pr-loc --> ## ELI5 Relay servers ("cells") normally get their seat bookings from the front door (the director). If the director goes quiet for a minute or two, each cell switches itself to an older mode that talks to the shared Postgres database for every connection (the "dead-man" trip). Today every cell switches at once, so the database budget must keep room for every cell's full connection pool. This PR adds a per-cell switch that caps how many cells may trip at the same time. A cell that cannot trip yet stays in its normal mode and keeps serving the desktops it already has. ## What Changed - New cell flag `deadManTripSlots` (integer 0-64). Absent, which is every cell today, keeps the current behaviour: trip as soon as the dead-man window (60-120 s) passes. - With the flag set, a cell whose window has passed first claims a slot in Postgres. The claim is one short transaction: it takes an advisory lock without waiting, counts the other reserve cells whose own `relay_cell_admit_effective` row says `db`, and writes this cell's row to `db` only if that count is below the flag. Only then does the cell trip (void bookings, re-register its hosts, run on its full pool). - A cell that gets no slot is "held". It stays in reserve mode, logs `orca_relay_cell_reserve_dead_man_held` once, keeps writing `reserve` to its row, and asks again every 30 s. - A slot is held while the holder's own row says `db`, its switch still says `reserve`, and the row was refreshed in the last 10 minutes. A holder that restarts or is re-armed writes `reserve` and frees its slot. One that died tripped frees it after 10 minutes. Heartbeats are not used: they reach Postgres through the director, so in a director outage they all go stale and every slot would read as free. - A tripped cell skips its 15 s row refresh while its pool has waiters. So that a long brownout cannot age its row out of the count, a tripped cell's row older than 5 minutes is refreshed even when the pool is busy. - A trip stays latched after the director returns, as today. The tripped cells keep their slots until an operator re-arms them with a new reserve write. Until then, a second outage trips nobody new: the other cells stay held. That is the intended bound, but re-arming after an outage now also frees the slots. - The claim goes through `CellAdmitEffectiveWriter`, which already serialises every write of that row. A `reserve` write still in flight therefore cannot land after the claim and hide the slot from the next claimant. - Cells keep their existing per-cell window spread (60-120 s by cell-id hash), so the earliest windows claim first. What a held cell does during a director outage (unchanged code paths, now for longer than the window): - Desktops already connected keep their sockets. Nothing is closed or re-registered. - A desktop that drops and redials at the same epoch is admitted from memory (`seatedHereAt`). Any other hello falls through to the normal database check, as it does in reserve mode today. - The database placement path places no new desktops on it, because its row says `reserve`. It also never re-places the held cell's desktops, because `liveReserveCell` keys on the switch (`relay_cell_admit_modes`), not on this row. - New placements during the outage go to cells already in database mode (c32, c33 and c34 today) plus the tripped slot holders. ## Why The database budget (Cloud SQL `orca-cloud-auth-db`, shared with auth, max_connections 500, 490 usable) currently reserves every relay cell's full pool. The only reason for that is the dead-man trip of the whole fleet at once. Spare is 35 on main and about 3 once C36/C37 land, which caps the number of Asia cells. Budget arithmetic (`relay-cloud-sql-connection-budget.mjs` on orca-cloud main: directors 15, auth 20, API 50, rollout overlap 65, maintenance 5, explicit reserve 10): | | Cells line | Operating | Spare | |---|---:|---:|---:| | Main today: 6 db/legacy cells at 10, 16 US at 8, 7 Asia at 16 | 300 | 455 | 35 | | Plus C36/C37 at 16 | 332 | 487 | 3 | | Same fleet, `deadManTripSlots=2`, Asia reserve cells budgeted at 12 (measured peak) | 76 + 128 + 8x12 + 2x4 = 308 | 463 | 27 | | Plus two more Asia cells at 12 | 332 | 487 | 3 | So with two slots, two more Asia cells fit in the room C36/C37 leave today. Two caveats keep this honest: - The bound holds only once every reserve cell runs this image with the same flag value. An older image ignores the flag and trips freely, though its `db` row still takes a slot from the capped cells. With mixed values the cap is the largest value set. - Counting reserve cells at their measured peak is a softer guarantee than counting their configured pool: the pool is still 16, and a reserve cell could burst past 12. Making it a hard bound needs a pool ceiling that depends on the mode (reserve-mode ceiling 12, the full 16 only for a slot holder or a deliberate db cell). That is a follow-up; this PR does not change pool sizes. Alternatives considered: - Longer or ranked trip delays only spread the trips out. A trip is latched until the switch is rewritten, so after the longest delay every cell is still in database mode. That bounds the burst, not the budget. - A static per-cell "may trip" list in the switch file needs no database coordination, but a listed cell that is down or draining wastes its slot, and an operator has to keep the list current. - Removing the fallback (cells never trip) is the target design, but it waits on the director stall fix and two weeks of clean running. `deadManTripSlots=0` gives exactly that per cell, so it is the same switch. Deliberate db flips (for example, flipping reserve cells to db for an image roll) are outside the cap. Budget each Asia cell flipped that way at its full 16. With a two-cell Asia roll batch running during an outage, the slot row above is 316, which leaves room for one more Asia cell rather than two. Rollout: the flag is dark. Setting it is an `Operate Relay Production Cell Flags` write, per cell, after every reserve cell runs this image. ## Linked Issue N/A (internal relay capacity work). ## Visual Proof N/A: relay server behaviour only, no UI. ## Testing - `cell-reserve-dead-man.test.ts` (new): trips as before without the flag; held until a permit, then latched; slot 0 never trips; a permit for a silence that contact ended, or for an earlier switch generation, is dropped. - `cell-admit-effective-writer.test.ts`: the claim waits for an in-flight reserve write; a refused claim keeps the reserve row fresh and retries after 30 s; no claim while the pool has waiters or the switch is off reserve; a tripped row is refreshed past a saturated pool after 5 minutes, a reserve row is not. - `cell-reserve-dead-man-slots.blackbox.test.ts` (new, real cell on SQLite): a director outage with the only slot taken by another cell that writes nothing but its own row: the cell stays reserve across several claims, re-registers nothing, keeps its desktops connected, and lets a redialing desktop back in from memory; once the peer writes `reserve`, the cell trips and re-registers. Also: slot 0 rides out 30 minutes of outage without tripping; no flag trips as before. - `cell-dead-man-trip-slot-postgres.test.ts` (new, PostgreSQL): eight cells claiming together get exactly two slots; a live holder keeps its slot; a stale one frees exactly one. Removing the advisory lock fails this test. - Mutation checks: making the slot count always 0, or removing the lock, each fails a test. - Full relay suite, including the PostgreSQL suites against a local PostgreSQL 17: 173 files, 1564 passed, 3 skipped. The pr.yml static-analysis steps pass locally. - Two rounds of adversarial review. Round 1 found that the first version counted holders by director-written heartbeats (fixed). Round 2 found the brownout aging (fixed) and the latched slots after an outage (documented above). - [x] I manually tested these changes locally - [x] Automated tests added/updated, or explained why not below ## Review ## Agent skill upstream boundary - [x] Not applicable, or this change copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation. ## Notes No new table or column. The claim reads `relay_cell_admit_effective` and `relay_cell_admit_modes` (one row per cell). Under a fleet-wide trip, held cells add one short transaction each per 30 s; the advisory lock is held for about one round trip. ## Checklist - [x] This PR is small and focused - [x] I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives) - [x] Before/after screenshots or videos attached for UI changes, or `N/A` with reason - [x] Self-reviewed for correctness, security, and performance - [x] Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A) - [x] `pnpm lint`, `pnpm typecheck`, `pnpm test`, and `pnpm build` pass (or CI will cover; local preferred)	2026-10-25 01:46:23 -04:00
Jinwoo Hong	7ffcb7c7b4	fix(relay): re-place hosts off a roll-isolated cell while its replacement process boots (#33312) ... <!-- orca-pr-loc --> <!-- Programmatic LoC summary. Do not edit by hand; rewritten on every commit. --> | | Files | Added | Deleted | Net | | :--- | ---: | ---: | ---: | ---: | | Test | 1 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​279 | 0 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​279 | | Prod | 1 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​58 | $\color{#cf222e}{\Huge{\mathbf{−}}}$​13 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​45 | <!-- /orca-pr-loc --> ## ELI5 When we restart a relay server during a routine rollout, computers that were offline during the rollout still have that server written down as "their" server. If one of them comes back during the ~45 seconds while the replacement server is starting up, the director currently tells it "your server is unavailable, try again" over and over, instead of just sending it to another server. This change lets the director send it elsewhere, but only when it has proof that the old server process is really gone. ## What Changed **Before (production, 2026-10-24 23:43Z, cell production-gce-c15):** during a same-capacity roll the cell is roll-isolated, drained, then its single VM is replaced. A desktop host that was offline during the drain reconnected in the gap between the old process stopping and the new one being ready. The director refused it for ~26 s (`assignment rejected route=assign lane=placement hinted=true reason=relay_home_cell_unavailable cause=not_ready`, x5, plus `route=resolve ... cause=unheard`, x2). It got a grant somewhere else only once the new process was live. **After:** while the replacement process is starting up (`not_ready`), the same host is moved to another cell on its first try. A host on a roll-isolated cell can be moved off even though the cell is not live, as long as the process heartbeating for the cell *started after the roll isolated it* (`relay_cell_runtime.started_at > relay_cell_admission.roll_isolated_at`), its heartbeat is fresh, and (for connection-capped cells) its latest heartbeat reports zero connections. Both lanes that move an isolated host apply it: the narrowed isolated-replacement path (`isolatedIncumbentMayMove`) and the all-rows path it falls back to. Together these serve assign and resolve, because resolve hands a non-live cell to `assign`. "Roll-isolated" is the existing predicate: `migration-only` with a roll stamp younger than 2 h. **Still refused (residual):** the `unheard` window, when the old process's heartbeat is past the 45 s TTL and the new process has not been heard yet. On c15 that was about 5-10 s. Nothing proves the drained sockets are gone there, so the refusal stays, and the client's retry recovers once the new process heartbeats. Also unchanged: an ordinary readiness dip, a `migration-only` cell without a roll stamp, a stale (over 2 h) stamp, and a new process that later goes unheard. Fence handling is untouched. ## Why The refusal exists because moving a host off a cell we cannot fully trust could leave the host's old socket served by the old process while its clients go to the new cell. Until now, the only proof the code accepted was an operator-written fence. This adds a second proof that costs no operator step: - **A new incarnation means the drained process is gone.** Each process mints a fresh incarnation UUID and `startedAt` at boot (`cell-heartbeat-client.ts`), and the director rejects a heartbeat with an older `startedAt` once a newer one is recorded. Each cell VM runs exactly one relay container (`docker run --name orca-relay --restart always`, with stale containers force-removed at boot). The cell MIG uses `replacement_method = RECREATE` with `max_surge = 0` (pinned by a Terraform assert), so the old VM is deleted before the new one is created. A same-VM container restart also produces a new incarnation and is just as safe: Docker restarts only after the old process exits, and that exit closes all of its sockets. - **Whatever the new process holds, it can enforce.** It is heartbeating (fresh), and on capped cells it reports no connections at all. A host that attaches between two heartbeats carries the old epoch, so the epoch bump this move writes makes the cell refuse it or close it with 4409. That check is the same live `verifyCellAssignment` / control-renewal read the existing live-isolated path relies on. - **Clock skew** between the cell's `startedAt` and the director's stamp is covered by the zero-connections check on capped cells only: a drained process that only *looks* post-stamp still holds its sockets, so it is refused. Uncapped cells report no counts and get no such guard; they already move off without a fence on the dead-cell path today, so this adds no new exposure there. Alternatives considered: (a) adding an incarnation column to `relay_assignments`: a schema change for no extra proof. (b) Honouring any not-live roll-isolated cell: that drops the proof. (c) Having the roll write a fence for the `unheard` window: the only fence driver refuses enabled cells, a fence is a MIG resize to 0 that would fight the roll's own apply, and the window is a few seconds the client retry already absorbs. ## Linked Issue N/A (internal production incident, 2026-10-24 c15 roll). ## Visual Proof N/A: director placement logic only, no UI. ## Testing - [x] Automated tests added: `cloud/apps/relay/src/assignment-roll-replaced-cell.test.ts` (18 cases; the redial cases run for both hinted assign and resolve): - not_ready with an incarnation started after isolation: re-placed, epoch bumped. - Refused as before: not_ready with the pre-isolation incarnation, with reported connections, after the new process goes unheard, with a stale stamp, on an ordinary unstamped readiness dip, and on an unstamped migration-only cell. Also refused: the unheard window between the old and new process. - One case per lane. The narrowed path's cross-region move fails if only the narrowed-path change is reverted. The all-rows path's move, forced by units no lease backs, fails if only the all-rows change is reverted. - [x] Full relay vitest suite, `--maxWorkers=2`: 130 files / 1344 tests passed, 39 files (201 tests) skipped. The Postgres-backed suites skip locally (no `ORCA_RELAY_TEST_POSTGRES_URL`); CI runs them. - [x] `pnpm typecheck` in `cloud/apps/relay`. ## Review ## Agent skill upstream boundary - [x] Not applicable, or this change copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation. ## Notes Backwards compatibility: no schema, wire or RPC change; old directors keep refusing as today. Performance: the new query runs only when a host's home cell is roll-isolated and not live; the general sticky hot path is unchanged. ## Checklist - [x] This PR is small and focused - [x] I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives) - [x] Before/after screenshots or videos attached for UI changes, or `N/A` with reason - [x] Self-reviewed for correctness, security, and performance - [x] Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A) - [x] `pnpm lint`, `pnpm typecheck`, `pnpm test`, and `pnpm build` pass (or CI will cover; local preferred)	2026-10-24 20:33:39 -04:00
Jinwoo Hong	d96dca39da	fix(relay): place against min(configured, cell-reported) connection cap (#33039) ... The director now accepts a cell heartbeat whose reported connection cap is at or below its configured cap, and places with the lower of the two; a reported cap above the configured one is still refused. A cap-only change keeps the cell's stored capacity rows. This lets the US cells' configured cap be raised once, ahead of their rolls, so an old process at the lower cap stays live until it is replaced, instead of desktops pointed at it being refused during a resize.	2026-10-24 12:51:08 -04:00
Jinwoo Hong	6e3f205749	fix(relay): let roll tooling read a reserve-mode cell and count its drain (#32178) ... Reserve-aware cell status for the director: reserve cells report their own runtime and seat feed, drain causes are attributed per cell, and paced 503s on a host leaving a draining cell are recorded as the drain's.	2026-10-23 19:00:57 -04:00
Jinwoo Hong	e4d1c981a4	feat(relay): rehome reserve hosts into their own region on reconnect (off by default) (#32191) ... Rehome on reconnect for reserve-mode cells: a desktop reconnecting on a reserve cell outside its preferred region is placed fresh in that region, only while the region has at least 10% of seats free, at most once a day per desktop, rate-limited, and switched off by default (ORCA_RELAY_RESERVE_REHOME_PERCENT=0); the regional rehome switch stops it.	2026-10-23 18:48:35 -04:00
Jinwoo Hong	567ae1249b	fix(relay): spill onto a reserve seat in either region before relay_capacity_exhausted (#32188) ... Before refusing a placement with relay_capacity_exhausted, the director tries a reserve seat in the desktop's own region, then the other region, through the cell's own reserve API; if every region paces it answers the shortest Retry-After.	2026-10-23 18:41:55 -04:00
Jinwoo HongandClaude	6b24bf596c	fix(relay): count region selection on every assign lane, fallback only for new placements (#32186) ... * fix(relay): count region selection on every assign lane, fallback only for new placements Co-Authored-By: Claude <noreply@anthropic.com> * test(relay): pin placed marker on isolated re-place and kept pins; drain-return fallback Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-10-23 18:35:53 -04:00
Neil	d667e54b02	refactor: colocate relay assignment admission owners (#32142)	2026-10-22 18:39:05 -07:00
Jinwoo Hong	54f1417af7	fix(relay): give region-correction reconnects and roll re-placements the in-region reserve fallback (#31042) ... * fix(relay): give region-correction reconnects and roll re-placements the in-region reserve fallback Every desktop session open carries a region-correction request, and those skipped the reserve plan entirely, so with the region's database cells full they still spilled abroad while an in-region reserve cell had room. The plan now runs for them in a mode that never answers itself (the database lane must reply so the window exchange follows) and offers only the in-region reserve fallback. A host re-placed off a roll-isolated cell went straight to the other-regions tier when its own region's tier was full. With an in-region reserve fallback it now books that first, and the roll canary event is emitted for the reserve booking. * fix(relay): never pace a roll drain on reserve intake, and count every reserve move off an isolated cell A host leaving a roll-isolated cell whose region was full could get a paced 503 from a throttled reserve cell, stretching the drain and re-running the failed in-region transaction on each retry. For those hosts a pace is now no room, and they spill as before. Sticky now says when it declined an isolated pin, so a share-draw booking off that cell also emits the roll canary event.	2026-10-20 03:33:13 -04:00
Jinwoo Hong	0059d062ea	fix(relay): keep a host in its region when the database lane is full and a reserve cell has room (#30974) ... * fix(relay): keep a host in its region when the database lane is full and a reserve cell has room A host that lost the reserve share draw went to the database lane, which spills to another region when its own region's database cells are full, even while an in-region reserve cell had free seats. The database lane now requires the region when an in-region reserve cell has room; if the region is full it books that reserve cell, and only spills when neither has room. * fix(relay): pace a throttled in-region reserve cell instead of spilling, and decide reserve room at plan time With the region's database cells full, a reserve cell with seats but no intake budget fell through the placer's database-cells shortcut and the host went abroad. That placement now answers pace (503 with Retry-After). regionReserve is a single callback, present only while an in-region reserve cell has a seat. * fix(relay): pace only on a reserve cell that lacked budget, not one that refused A cell that refused this placement (full, unreachable, seated-newer) got its token back and drove a 1 s Retry-After forever while the map still showed a seat. Pacing now skips cells refused for anything but intake, so those placements fall to the database lane's cross-region fallback as before.	2026-10-20 01:16:26 -04:00
Jinwoo Hong	cb82bc80da	fix(relay): never call a reserve cell's hosts stranded when it goes existing-only (#30609) ... A reserve cell keeps no activity rows and its ledger stamps last_activity_at, so the stranded rule read its connected hosts as unserved and re-placed them.	2026-10-19 03:33:15 -04:00
Jinwoo Hong	735e6d6c5d	fix(relay): let a host leave a dead reserve cell without an operator fence (#27458) ... * fix(relay): let a host leave a dead reserve cell without an operator fence A host whose reserve home cell died got 503 relay_home_cell_unavailable until the cell came back: the database path demands a committed fence for every capped cell. On its own /v1/assign, a host now moves off a reserve cell (Postgres records it as reserve) when the cell's heartbeat is stale and this director's seat-feed polls have gone unanswered for 45 s, unless more than a third of the polled cells look the same. Sweeps, migration pins and capped db cells keep the fence. * fix(relay): guard the dead-cell verdict by region and sample size, and floor the move on the map - No verdict when half or more of the dead cell's own region fails with it (a cross-region partition), more than a third of the fleet does, or fewer than 6 cells are polled. A withheld region or fleet verdict logs orca_relay_dead_cell_verdict_withheld, which alerts. - The move needs the step-5 plan's epoch floor, so it mints above every seat the map knows; without the plan the fence is kept. A test revives the old cell's seat above the row and checks that the ledger never writes the row back. - The move's log line carries the fleet and region counts. * fix(relay): count only feed cells and young streaks in the dead-cell guards; gate the alert - Denominators hold only cells that served this director a feed page and have capacity: legacy rows answering 404 no longer fill the 6-cell sample or dilute the fleet and region ratios. - A peer counts as failing for the guards on any current streak (a failure in the last 5 s), so a staggered partition cannot let its first cell cross 45 s while its neighbours read healthy. - Every withheld verdict is logged, sample included (a cell missing from the feed cells is one). - Tests: ready=0 and enabled=0 with failing polls keep the fence (drain then death is fenced). - The withheld metric and alert are gated by relay_dead_cell_verdict_alert (default false, on in staging); the alert docs now describe poll failures, not heartbeats. * fix(relay): always sample the dead-cell candidate, so a restarted director can still judge it A director that restarted after a reserve cell died never received a feed page from it, so the cell was never in the sample and its verdict stayed withheld as 'sample'. The candidate now always counts when it has capacity (the store asks only about a cell Postgres records as reserve, and only feed images run reserve); peers still count only once they have served a feed. Tests: a restart during a lone outage decides, a partition begun before the restart withholds by region, and the capacity filter. The Terraform comments now say to apply the metric first and the policy minutes later.	2026-10-11 14:24:52 -04:00
Jinwoo Hong	a18883130f	fix(relay): database sweeps and ops reads skip reserve-mode cells (step-5 census, inert) (#27046) ... Rebased onto main as one commit after its parent squash-merged.	2026-10-10 19:22:11 -04:00
Jinwoo Hong	3f7943c411	fix(relay): cells without a seat feed stay out of shadow coverage and switch state (#27058) ... Rebased onto main as one commit after its parent squash-merged.	2026-10-10 18:54:30 -04:00
Jinwoo Hong	5493c5e815	feat(relay): directors place on reserve-mode cells from memory (off) (#27047) ... Rebased onto main as one commit after its parent squash-merged.	2026-10-10 02:26:12 -04:00
Jinwoo Hong	abff7ab86a	feat(relay): flip a reserve-mode cell back without disconnecting anyone (E-mix) (#27043) ... Rebased onto main as one commit after its parent squash-merged.	2026-10-10 02:18:37 -04:00
Jinwoo Hong	452e44464e	feat(relay): read-only /v1/admin/host-whereabouts on directors (step 3) (#26816) ... On-call can ask a director where a host is: its relay_assignments row and open migration (primary-key reads only), the shadow map's seats and recent leaves with close codes, the compare verdict (pure classification, judged as /v1/resolve sees the host, recording nothing), and each named cell's poll status and applied flags. shadow is {state:'off'} when the poller is off. Deploy, monitor and fence identities may read it; capacity and asia-proof may not. With no user id, the map names the user; an unknown host answers empty.	2026-10-09 18:58:10 -04:00
Jinwoo Hong	f514cb2c66	feat(relay): dark shadow seat directory poller on directors (step 3) (#26810) ... Directors can poll each cell's read-only seat feed (#26803) once a second and keep an in-memory map of which host is seated on which cell, plus a bounded 24 h recently-left memory. Off unless ORCA_RELAY_SHADOW_SEAT_FEED_CELLS names cells. Applies a change at or above the seat's generation, follows cut pages, resyncs on a gap or new incarnation, and checks the map against counts.seats. A 404 reads as no feed; timeouts and errors keep the last seats as unverifiable. The cell list is every cell (an existing-only cell still seats hosts), from RelayAssignmentStore.seatFeedCells with placement's liveness rule; completeness waits only on live cells, and a failed list read backs off. Nothing reads the map yet.	2026-10-09 18:33:32 -04:00
Jinwoo Hong	e72044a936	fix(relay): a lost database reply ends its connection within statement timeout + 10 s (#26829) ... * fix(relay): a lost database reply ends its connection within statement timeout + 2 s The request pool sets query_timeout to the statement timeout plus 2 s. A read timeout poisons its transaction: later statements fail at once, COMMIT never runs (even when the operation caught the error), no ROLLBACK queues behind the dead statement, and the client is released with the error so pg-pool destroys it. Autocommit statements destroy their client too. The server bounds an orphaned idle backend with idle_session_timeout 30 s. A read timeout is answered 503 and never retried, and a renewal batch that loses its reply fails whole instead of fanning out. * fix(relay): read timeout margin 10 s, configurable, so routine 5-7 s database stalls never trip it A whole-VM stall delays replies the server already timed. With a 2 s margin the read timeout fired inside routine stalls, destroying healthy clients and failing requests that succeed today. The margin is now 10 s (ORCA_RELAY_POSTGRES_READ_TIMEOUT_MARGIN_MS), and a real-server test holds a reply 6.5 s and requires it delivered on the same connection. The fix level rises to 2 here, with the lost-reply fix. * docs(relay): name the startup retry that can see a read timeout, and why it is safe Boot open retries only an unreachable database. The director's startup cell reconcile does retry a lost reply, which is safe: it re-runs upserts and limit-change deletes that converge on the same rows. * fix(relay): answer 503, not 500, when Postgres ends the session mid-request Staging B1: pg_terminate_backend during /v1/assign returned 500 29 times. 57P01/57P02 and 08000/08003 join the transient SQLSTATEs; pg's 'Connection terminated' and ECONNRESET count only when the database layer threw them. transaction() still never retries them. * fix(relay): answer 503 when the pool acquire fails on a dead socket Staging B1 re-test: a Cloud SQL restart failed PostgresDatabase.transaction's acquire with 'write EPIPE', answered 500. An acquire failure with EPIPE, ECONNRESET, ECONNREFUSED, ETIMEDOUT or EHOSTUNREACH is now a pool connect failure (no statement ran), and acquire failures count as database-layer errors. EPIPE mid-statement is treated like ECONNRESET.	2026-10-09 18:00:10 -04:00
Jinwoo Hong	80d7e2dded	fix(relay): spread drain placement over a load band instead of one empty cell (#26802) ... * fix(relay): spread placement over a load band so a drain does not land on one empty cell Placement took the single least-loaded cell, so a just-rolled cell took a whole drain and every follow-up serialized on its relay_cells row. Directors now pick at random among cells within 5% of capacity of the minimum, and skip a cell that already took over 40% of its pool's placements in the last 10 s (floor 4). It only reorders: the least-loaded cell is the fallback, the hard cap and region preference are unchanged. * fix(relay): size the load band's intake window for the 1,200 s drain pace At ~0.5 picks/s per director, a 10 s window with a floor of 4 let the floor decide and the empty cell took ~65% of a c27-sized drain. A 60 s window and a floor of 2 keep it at 40% at both the 300 s and 1,200 s paces across five directors. Adds a once-a-minute orca_relay_placement_load_band line (picks, budget skips, fallbacks, band size, picks per cell). * chore(relay): log the load-band summary at info, not warning	2026-10-09 16:58:07 -04:00
Jinwoo Hong	3062b9d8d0	fix(relay): refuse host hellos fast when the database pool is timing them out; renewals jump the queue (#26362) ... * fix(relay): refuse host hellos fast when the database pool is saturated; renewals jump the queue A reconnect herd (2026-10-06 18:34Z, 1,046 asia-east2 hosts dropped by the load balancer at once) queued hundreds of host hellos behind each cell's 16-connection pool. They timed out after 2 s, the desktops redialled into the same queue, and control-lease renewals starved behind them. - Host control upgrades get an immediate 503 (Retry-After: 2) once 32 callers wait for a pooled connection. Shipped desktops already treat that as a connect error and back off with jittered exponential retry. - General work now queues in PostgresPoolPressure instead of pg-pool, so renewal statements (priority lane) take the next released connection. - A renewal batch that never got a connection no longer fans out into one statement per host. - hostHellosShedDelta counts refusals in the runtime metrics. * fix(relay): shed host hellos on the pool's oldest wait, not its queue length asia-east2 queues of 50-196 waiters are routine and keep moving, so a 32-waiter limit would refuse hellos that were going to succeed. Shed only while the oldest pool waiter has waited 1.5 s of the 2 s acquire timeout, when a new hello would time out anyway; a refused desktop gets the same connect error and backoff it gets today, 2 s sooner. Rebinds over a live control (lease rotation) are never refused. * fix(relay): shed hellos only when a full pool queue has aged 1 s Per review: require both an oldest waiter of at least 1 s and a pool's worth of waiters, so one slow waiter cannot trip it, and state the real baseline (p99 of 2 waiters on 10-07).	2026-10-07 21:18:26 -04:00
Jinwoo Hong	bac9e0c2bf	fix(relay): latch rehome off only on a sql failure breach that outlasts a DB stall (#26371) ... A single 5-7 s stall of the shared database pushed the fleet sql failure sum to 396 (bar 250) on 2026-10-06 and a source cell latched regional rehome off fleet-wide. Source cells now only pause the claim on a sql-only breach; the director's regular poll latches once the breach has been watched for 150 s. Other safety reasons still latch at once on the cell.	2026-10-07 20:55:21 -04:00
Jinwoo Hong	54ded3bc18	fix(relay): commit the cell counter in one round trip; cells boot without the database (#25765) ... * fix(relay): commit the cell counter in one round trip; cells boot without the DB Step 2 (option B) cell image: - One-round-trip counter commit at acquireActivity, releaseActivity and activateControl: the final counter UPDATE and COMMIT go as one simple-query message. Server errors mean COMMIT never ran (retry as today; 22012 = no row, rolled back and disambiguated outside the transaction); a lost connection is never retried. - Cells skip the schema apply and region backfill, so they listen while the database is down and turn ready on their first successful query. - G13: rehome target connection headroom folded into the existing NOWAIT UPDATE, excluding the host's own reservation by key. - fixLevel on every runtime metrics line, plus declared (not applied) cell fix-level metrics and alert. - Per-desktop drain disconnect-gap measurement from existing log lines. - Census test that fails on floating database promises; fixes two shutdown sites. Lock-wait sample keeps the combined role. * fix(relay): make the outdated-image alert creatable: one PromQL condition, 1 h lookback, fixed floor A PromQL condition must be the only condition in its policy, and alerts on log-based metrics may look back at most 25 h. Replace the 6-day/7-day design with relay_cell_min_fix_level (tfvars, raised by a targeted apply after each wave) and one query: a serving cell below the floor or reporting no level, sustained 6 h. Drops the separate without-level metric. * fix(relay): review fixes: gap-script ordering, wider promise census, fused-path guard, row-busy as scheduled - Drain gap script: sort closes by time (gcloud exports newest first) and refuse an invalid drain start. - Census: any floating promise in relay src, including callback-discarded and never-read ones, with a reviewed never-rejects list. - Test the fused counter commit through the store the server builds, so a wrapper that stops forwarding commitWithFinal fails CI. - Same-cap shadow gate: a row-busy refusal (the host's own release still holds its row) is a scheduled 503, like an own early retry. No client change. * test(relay): judge drain redials by no host refused twice, not a refusal count The row-busy count tracks how many releases are still in flight at the dial (80 of 180 every run at 1 s, against a bar of 90). What matters is that the release has finished by the next dial: assert no host is refused twice, keep the time-to-placed p95 bound. * fix(relay): cap row-busy as scheduled at the drain-return admissions; bound the gap script's window Shadow gate: a row-busy refusal of a drained host follows its drain-return lane admission, so per minute only that many (plus a rounding margin of 2) are scheduled; the rest stay non-drain, so row contention the drain does not explain still fails the budget. Gap script: --drain-ended-at excludes the new container's closes after the roll; later grants still close a gap.	2026-10-06 00:53:51 -04:00
Jinwoo Hong	10bea3a8fa	feat(relay): report lane service times, 503 causes, per-site hold p99 and a director-vs-cell lock-wait split (#25645) ... * feat(relay): report lane service times, 503 causes, per-site hold p99 and lock-wait split Adds the step-1 observability: drain-return and sticky slot service times, every /v1/assign 503 by cause with a non-drain total, a p99 per lock hold site (drain-return regional rows get their own site), a director-side pg_stat_activity sample that splits lock waiters by director vs cell, and a log line for each reserved_requests drift reconciliation corrects. Log-based metrics for the new fields are declared in Terraform, not applied. * fix(relay): classify a lock waiter by the first relay table its statement names * fix(relay): attribute lock waiters to the root holder; document the director hold alert change Every waiter after the first in a row-lock convoy is blocked by the first waiter, so the sample now walks pg_blocking_pids to the root, reading it once per waiter. The sampler is single-flight. Drain-return regional holds feeding the non-paging director hold policy is documented as expected. * docs(relay): note the lock-wait sample undercounts director waiters when the pool is full	2026-10-05 17:55:52 -04:00
Jinwoo Hong	17ecee6337	fix(relay): read reconcile's counter units after the cell lock (#25638) ... reconcileReservationAccounting computed each cell's absolute reserved_requests from leases read before lockCellRows, so a host placed in that gap lost its unit: the counter went to 0 with one lease on the cell.	2026-10-05 15:54:06 -04:00
Jinwoo Hong	e9daf9b746	fix(relay): prune released control-connection reservations in bounded batches (#25352) ... relay_control_connection_reservations was never deleted: 13.8M rows and 9.1 GB in production, 99.999% of them in state 'released' and ~400k more a day. Nothing reads a released row (every reader filters it out by state), but each placement still locks all of its host's rows, ~160 on average. A new director sweep step deletes released rows older than a day. It walks the heap in TID ranges of 16 pages, because without an index on released_at a `LIMIT n` delete plans as a sequential scan from page 0 that gets slower as the head of the heap empties (production EXPLAIN). Each statement selects its rows FOR UPDATE SKIP LOCKED, so a row a request holds is skipped rather than waited on, and deletes them by `ctid = ANY(ARRAY(...))` so the delete is always a TID scan. A tick stops at 400 rows, 128 pages or 250 ms, which drains the backlog over about three days at five directors.	2026-10-04 22:38:24 -04:00
Jinwoo Hong	b9b0f29172	fix(relay): skip the dead-cell sweep's host scan when no cell qualifies (#25350) ... evacuateDeadCells ran one query that joined every assignment to its cell's liveness and fence state, ordered by primary key with LIMIT 100. In production no row ever matched (the only dead cells are stale existing-only cells without a committed fence), so the planner walked the whole relay_assignments primary key every call: 551 ms mean over 85k calls, ~12% of relay database time. A cell-level pre-check now evaluates the predicate's cell-only half over the ~33-row cell tables (3 ms in production). It is a superset of the cells the host query can act on, so the sweep returns early when it is empty and otherwise restricts the host query to those cells. The host predicate is unchanged.	2026-10-04 22:38:16 -04:00
Jinwoo Hong	b82307937a	fix(relay): admit drained hosts through their own lane and stagger their return (#24446) ... * fix(relay): admit drained hosts through their own lane and stagger their return A same-cap roll's drain sends every host on the isolated cell back to the director at once. Those hosts reconnect through the sticky lane (one slot per director), and each one's re-placement holds that slot for most of a second behind the region-wide inventory lock, so ordinary reconnects time out behind them and the drained hosts retry every 2 s: ~30k 503s per drain. The reconnect verification read now also says whether the host's home cell is isolated for a roll right now (same predicate re-placement uses). Those hosts release the sticky slot after the read and take a separate drain-return lane (1 per director, matching the store's per-director placement serialization). When that lane is full the host gets a Retry-After that reserves the next free service slot, paced by the measured re-placement time and capped at 300 s. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 * fix(relay): keep drain returns inside the placement pool budget and their own slot Review follow-ups for the drain-return lane: - The lane now borrows placement permits (never placement's last, never ahead of a queued placement), so placement + sticky still bounds the database pool. - A host's own early retry (row-busy redial, duplicate dial) gets the 2 s lane interval instead of a fresh slot behind the cohort, and a host that returns early to the same director keeps its reserved slot. - Classification also excludes an open migration row whose lease counter lapsed, matching the re-placement rule. - The load test now runs five directors behind random routing with a shared inventory lock. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010	2026-10-01 15:34:23 -04:00
Jinwoo Hong	84f58a1fc7	fix(relay): refuse a redial at once while the host's own release holds its row (#24225) ... * fix(relay): refuse a redial at once while the host's own release holds its row During an Asia drain the host whose socket closes is the one that redials. Its release on the draining cell locks its assignment row first, then waits on the cell's busy row for up to the lock timeout. The director's sticky and placement paths waited on that assignment row inside the single sticky slot, and the sticky path then locked the busy cell row itself before it checked isolation. The slot backed up and dials timed out fleet-wide. Both paths now take the host's assignment row NOWAIT and throw RelayAssignmentRowBusyError when it is held. /v1/assign answers that with 503, Retry-After 1 and error assignment_row_busy, logged with its own reason. The sticky path decides isolation before it touches the pinned cell row. An isolated retry keeps its own tier as its retry scope, so a busy lock inside it no longer falls to the all-rows path. The local drain arm drops its zero-release-failures bar, which the Asia arm never had. The drain harness gains a departing-host arm: each host releases its own lease, then redials after 150, 400 or 1000 ms on the desktop client's 5-5.5 s pacing. At 400 ms, main rejected 83 of 180 first dials by sticky wait timeout, placed 11.6/s with 3.1 director backends lock-waiting, and took 11.2 s at p95 from release to placed. Now: 16 fast refusals, 18/s, no lock waits, 5.7 s at p95. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 * fix(relay): wait briefly for a calm host's row and keep the dead-cell sweep going The dead-cell sweep treated RelayAssignmentRowBusyError as fatal, so one busy host ended the sweep for every later host each tick. It now skips that host and carries on. The sticky path refused a busy row at once for every host. A calm host redialling after its own clean close often meets its own short release, and a refusal costs it the client's 5 s assign gate. When the pinned cell is general and live, the sticky path now waits up to 1 s for the row before refusing. A roll-isolated, parked or dead cell still gets the immediate refusal. Placement keeps NOWAIT, because it holds cell rows while it would wait. A resume refused for a busy row now carries Retry-After 1 as well. The departing-host harness arm now bounds the busy refusals at 20% of hosts and the p95 at 8 s, and counts unexpected errors apart from retryable refusals. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 * fix(relay): never wait on a host's row while the sticky retry holds its cell row The inventory-first sticky retry takes the pinned cell row before the assignment row. With the calm-host bounded wait it could then wait up to 1 s on the assignment row while holding the cell row, the reverse of the ranked lock order, against this host's own release, which holds its row and wants the cell's. The bounded wait now applies only when no cell row is held; the retry stays NOWAIT. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010	2026-09-30 17:58:30 -04:00
Jinwoo Hong	ed462b2caa	fix(relay): re-place hosts off a draining cell without locking its row (#24216) ... * test(relay): reproduce drain-release contention against director placement Adds a Postgres harness that drives releases from an isolated cell over a 171 ms per-statement pool while five directors re-place reconnecting hosts through the sticky lane. At 18 releases/s placements fall from 20/s to about 5/s and every active director backend is blocked on relay_cells. Moves the per-statement delay pool into a shared test fixture so the rehome target-row test and this harness use one implementation. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 * fix(relay): re-place hosts off a draining cell without locking its row Sticky re-placement of a host whose cell is isolated for a roll locked every relay_cells row. During an Asia drain the source row is held by the cell's own releases for a round trip each, so the placement waited on it, the single sticky slot backed up, and /v1/assign returned 503 fleet-wide. The isolation decision now comes from an unlocked read, and the placement locks only the same-region general rows it can move to. It no longer writes the source row: the host's source leases stay, and each one's own release or expiry takes its units back off the source. The assignment keeps its activity counters and adds one control instead of resetting them. With no same-region headroom the path falls back to the all-rows lock, as before. Dormant hosts hold no units, so their placement also locks only the general rows and skips the zero write to their old cell. The drain harness now asserts the after picture: 20 placements/s at Asia latency with no director lock waits, against 9.2/s and 4.8/s before. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 * fix(relay): take a moved host's units off the cell that holds them After a narrowed re-placement a host keeps leases on its old cell while its assignment row names the new one. Two paths charged the row's whole counted total to the row's cell: aggregate expiry, and the lease deletion in dead-cell and stranded re-placement. Both over-charged the new cell and left the old cell's units stranded. Aggregate expiry now skips hosts that still hold any lease; the lease sweep takes each lease's units off its own cell. Placement frees each deleted lease's units on that lease's cell, charges the old cell only for units no lease backs, and sets the counters from the leases it keeps plus the new control. The narrowed path runs only when the counters already match the leases, so it never needs to write the old cell's row. The all-rows re-placement off an isolated cell follows the same rule, so it no longer decrements the source at placement either. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 * fix(relay): try other regions before the all-rows lock when re-placing off a roll With every same-region neighbour at its connection cap, the narrowed path found no target and fell back to the all-rows lock behind the busy source row, which is the drain brownout again. It now tries a second tier, general cells in every other region, in its own transaction over one ordered lockCellRows, still never the source row. Only when no general cell in any region has room does it fall back to the all-rows path, which keeps the pin. This changes the policy from #21911, which refused to re-place an isolated host across a region. The unit tests that encoded that rule now assert the tier order instead. The drain harness gains a US cell and an arm with every Asia neighbour capped: 200 of 200 dials placed cross-region at 20/s with no lock waits, against 0 placed and 144 sticky rejections on the previous head. Its pass bars are now the rejection share and the lock-waiting share, not the placement rate a slow runner's pacing can move. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 * fix(relay): take the narrowed path for hosts whose counters sit below their leases Main's old placement reset a moved host's counters while keeping its source leases, and those leases' releases floored the counters at zero. Such hosts hold fewer counted units than lease units, and requiring equality sent them down the all-rows path behind the busy source row. The narrowed path now requires only that the host holds no units no lease backs, the one case that needs a write to the old row. Its placement already rebuilds the counters from the kept leases plus the new control, so a drifted host is healed by its next re-placement. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010 * test(relay): judge the local drain arm on completion and lock waits, not rate The local-latency arm asserted at least 18 placements/s at a 20/s dial rate, which a slow runner's pacing alone can miss. It now asserts what the Asia arm does: no dial failures, sticky rejections under 10% of dials, every other dial placed, and director lock-waiting under half a backend. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010	2026-09-30 17:00:03 -04:00
Neil	6e1b7e7fa3	test: remove junk tests that assert source text instead of behavior (#23815) ... Deletes 101 test files and trims 112 more, all matching documented junk patterns: exact source/import/string greps, copied inventories and export lists, duplicate invocations of a contract another test already owns, typeof-shape checks TypeScript already enforces, and self-comparisons. The largest group read a production `.ts` file and asserted on its text — for example a TaskPage test that required the source to contain `selectedRepos.find((r) => r.id === newIssueRepoId) ?? selectedRepos[0] ?? null`. Any behavior-preserving rename broke it; no behavior change ever did. Production-side follow-through: exports that only these tests imported are de-exported or deleted, stale comments pointing at removed censuses are dropped, and the reliability-gate registry, `cloud/package.json` test lists, and orphaned source-reading helpers are updated so nothing references a deleted file. Two files kept their real coverage and lost only the census scaffolding: `agent-status-producer-census.test.ts` now drives all five producers end to end instead of grepping the source tree, and `config-toml-trust-stale-writes` replaces an export-list parity check.	2026-09-29 01:21:53 -07:00
Neil	71344de12f	Scope relay capacity checks to the requested cell (#23036) ... * perf: scope relay capacity checks to the requested cell * fix: restore catalog entries required by the current CI baseline * fix(i18n): make AI the recipient of diff notes	2026-09-26 14:17:47 -07:00
Jinwoo Hong	ae3d380b23	fix(relay): lock only the target cell row, last and NOWAIT, in the rehome commit (#22449) ... * fix(relay): lock only the target cell row, last and NOWAIT, in the rehome commit The idle-rehome commit runs on the source cell. From an Asia cell each statement is a cross-region round trip, and the transaction locked every relay_cells row plus every runtime, capability and safety row before about twenty more statements, so each Asia-source rehome held the whole fleet's cell rows for ~3.6 s and every reconnect, renewal and placement queued or timed out behind it. The commit now reads the cell inventory and the runtime, capability and safety tables unlocked, keeps the control and worker rows locked (now NOWAIT), and takes one cell lock: the target row, in a single statement that locks it NOWAIT, re-checks enabled, general admission and capacity, and reserves the units, issued as the last statement before COMMIT. A target that changed admission, filled up, or is locked by another writer rolls the whole commit back and answers deferred (candidate-ineligible). The hold is sampled under a site label, so cellInventoryHoldMsMax still sees rehome holds and rehomeTargetRowHoldMsMax reports them apart. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(relay): fail closed on the rehome target-row lock clause The target-row statement now carries FOR UPDATE ... NOWAIT unless the dialect is explicitly SQLite, so a wrapper that omits the optional dialect can no longer run the reservation unlocked. Test wrappers and the fault injection entry forward the dialect they wrap. The latency test also probes the admission and region tables at every round trip; only the target's admission row may be locked, and only before COMMIT. The runbook notes that an Asia-sourced commit holds the rehome control row for about 6.5 s, so a pause that fails once is retried. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb	2026-09-23 03:45:47 -04:00
Jinwoo Hong	1043dc5e1d	fix(relay): stop rehoming hosts off Asia cells until the lock fix lands (#22443) ... The source cell runs the rehome commit. An Asia source pays a cross-ocean round trip per statement while holding relay_cells row locks every cell needs, which convoys the fleet database. Selection now drops source cells outside the director's region before building the decision window, so the incumbent_region filter shrinks while Asia cells stay valid targets. The preview counts the same hosts as source-outside-director-region and the poll summary reports skippedOffRegionSourceCells. Temporary stopgap. Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010	2026-09-23 03:15:52 -04:00
Jinwoo Hong	c8a5580659	fix(relay): re-place hosts off a cell isolated for a roll (#21911) ... * fix(relay): re-place hosts off a cell isolated for a roll A roll isolates a cell by moving it out of the 'general' admission class; the cell then refuses every attach with 4503. The director never noticed, because the only liveness test it applies to a host's current cell reads `relay_cell_runtime.ready` and the heartbeat, and an isolated cell keeps heartbeating ready=1 for the whole drain. So every host on that cell was handed its own dead cell, closed, and handed it back — 500-1,900 hosts looping for 13-16 minutes per cell roll, at ~6 dials each per minute, with no neighbour absorbing anything. The sticky lane now treats a live incumbent whose admission is 'migration-only' — the state a roll's isolate step writes — the same way it treats a dead one: it returns null, which means "fall through to placement". The placement lane had the identical hole eleven lines further down, so it takes the same predicate; without that second swap the sticky change is inert, because placement would hand the pin straight back (a draining cell has more headroom than anyone). An isolated incumbent skips the dead-cell fence branch: that branch exists to prove an unreachable cell stopped serving a host, and this one is reachable and enforces the epoch itself. 'existing-only' is deliberately untouched — those cells serve the hosts they already hold, and only `assignmentStrandedOnUnservedCell` may release that pin. A host with an open `relay_assignment_migrations` row keeps its pin too, so this stays disjoint from the migration machinery. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(relay): gate re-placement on a roll-isolation marker, not on admission Review of the first commit found the predicate wrong. `migration-only` is an admission class, not a drain signal: an Asia `--mode rollback`, an evacuation or forward-recovery target awaiting a separate promote dispatch, a failed same-cap wave's re-isolate, an abandoned migration retired on its target and a rehome settlement all park loaded cells there durably, with no migration lease and no open migration row. All five were indistinguishable from a roll's isolate, so the first commit would have converted `operate-relay-asia-admission --mode rollback` from a reversible admission flip into a mass move of ~4,000 hosts — and, because `leastLoadedCell` treated region as a preference, into us-central1. The signal is now an explicit stamp. `relay_cell_admission` gains a nullable `roll_isolated_at`, added through the shared schema runner's catalog pre-check so a migrated database takes no relation lock on boot and an un-migrated one gets a catalog-only rewrite. The same-cap isolate step is its only writer, via a new optional `rollIsolatedCells` on the selector apply; the same UPDATE that writes the state clears the stamp whenever a cell leaves 'migration-only', so a restore cannot leave one behind and a failed wave's re-isolate keeps the one it has. Every other admission writer omits the field, so its cells stay unmarked and their hosts stay pinned. Old directors ignore the field; old callers never send it. Region is now a constraint rather than a preference on this path only: a re-placement must find a general, live cell with connection headroom in the host's own region, or the pin is kept and one `orca_relay_sticky_replacement_deferred` event is logged. Cross-region spill is no longer reachable here. The fence bypass is narrowed to a live incumbent. It was always a no-op for the intended case, and for a stamped cell that stops heartbeating while still holding sockets it reopened split-brain; that cell now takes the dead-cell path unchanged. Also: the hot-path admission reader no longer throws on an unrecognised state — it sits on every sticky dial and the rule it feeds is "move the host", so an unreadable row has to mean "don't". And the sticky lane reads the admission row once for both the stranded rule and the stamp instead of twice. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(relay): emit the re-placement events after the transaction commits CodeRabbit on assignment-store.ts:1075. Both events were written where they are decided, which is inside assignOnce's transaction. A reservation or lease write failing after that point rolls the placement back, but a line already on stdout cannot be rolled back with it — so the canary this PR asks an operator to read would count re-placements that never happened, and a Postgres transaction retry could leave a stale line behind as well. The transaction now returns its events alongside the RelayAssignment and the caller flushes them once it has resolved. Returning them rather than setting a variable in the enclosing scope is what makes the retry case safe too: only the attempt that committed can carry its events out. assign()'s signature is unchanged; the extra shape lives entirely inside assignOnce. orca_relay_sticky_replacement_deferred was moved the same way. It cost one more push into the array that already existed, and it is decided inside the same transaction, so leaving it behind would have been the odd case rather than the cheap one. The new test injects a failure on the first write after the decision, asserts no event is emitted, and asserts the assignment is still on its original cell — without that second assertion the absence would only prove the emit was early, not that it would have been wrong. A control dial with nothing injected emits exactly one event, so the case cannot pass on a broken harness. With the emit put back inside the transaction, it fails. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(relay): expire the roll stamp, correct the wire note, assert the stamp landed Delta review findings B, D and E. A (the deferral path's cost) is deliberately not implemented; it is now written up under Follow-ups in the PR body as required before any Asia roll, because it cannot fire in a US canary. B, which also closes C: the stamp was written, carried and never compared to anything. A roll isolates and restores one cell inside ~15 minutes, so a stamp older than two hours is not a roll in progress. It is a failed wave whose failsafe re-isolated a possibly healthy cell and is waiting on an operator — the postmortem in this tree records gaps of hours — or an orphan left by a director rollback whose restore wrote 'general' without the clause that clears the stamp, which the selector's 'keep' branch would then preserve until some later park reactivated it. Both want the same answer and it is the pre-existing one: keep the pin. One comparison against a value already on the row. The bound takes the caller's `now` rather than reading the clock again, so one assign reasons about one instant; the stamp's age is now a thing that decides whether a host moves, and two clock reads could disagree across it. D: the comment beside the new request field claimed an updated caller reaching an older director "is simply ignored". The schema is .strict(), so it is a 400. That fails closed — the isolate aborts before MUTATION_STARTED is set and nothing is written — but it is a deploy ordering constraint, and it was undocumented. The comment now says so and the PR body's rollout notes carry it. E: nothing read the `rollIsolated` the script already prints, so an older script against a newer director would silently produce today's behaviour and the canary would read as "the fix did nothing" with no way to tell that from a wrong premise. Both isolate steps now assert it, beside the generation they already parse. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb	2026-09-21 04:18:57 -04:00
Jinwoo Hong	fa0010e8d6	fix(relay): abort rehomes whose host never arrived, without disabling the switch (#21813) ... A regional rehome whose host went offline right after accepting the move left its migration row open forever: the target had registered it, the host held nothing on the source, and the completion sweep could never finish it. Eight such rows filled REGIONAL_REHOME_CONCURRENT_LIMIT and every later candidate came back deferred, silently, for 21 hours. The only sweep that touched them fires at 24 hours and also sets enabled = 0 on the durable control, so the first leak to age out would have turned rehoming off, repeatedly. Adds a director sweep that rolls such an attempt back to its source after one migration lease, with abort_reason = 'host_not_arrived', reusing the existing rollback (assignment epoch bump back to the source, lease removal, superseded target reservation release) and leaving the switch untouched. The 24-hour sweep keeps its disable as a last-resort latch. The source cell now names why it deferred, on a new optional response field, and the director stops walking its candidate page on a deferral no later candidate can pass. Each poll that dispatched logs one summary line. Claude-Session: https://claude.ai/session/ced32ebb-7155-4413-adad-1eccd14c2010	2026-09-20 15:13:27 -04:00
Jinwoo Hong	030a1e0c77	fix(relay): stop holding a cell row across the whole control accept (#21563) ... * fix(relay): stop holding a cell row across the whole control accept The cell accept path took the host's cell row FOR UPDATE at its first supersession statement and held it to COMMIT across a dozen round trips, which capped a cell far from Postgres at a couple of accepts a second. Fold every cell-row change on the path into one conditional delta write issued last, so the contended row is held only across the commit. * fix(relay): give relay_cells one global row lock order, taken last Moving the accept's cell-row write to the end of its transaction put it after the host's relay_control_connection_reservations rows, while every director path that reads the inventory took those rows the other way round. Pin one order for both roles -- host rows, then the shared cell row -- by locking the host's reservation rows before the inventory in the nine director paths that take both, document the tiers next to CellInventoryLockMode, and add a census that fails on a new path taking relay_cells first.	2026-09-18 23:16:15 -04:00
Jinwoo HongandClaude	7080eb0604	fix(relay): bound the idle-rehome candidate poll to a window of decisions (#21557) ... * fix(relay): bound the idle-rehome candidate poll to a window of decisions The director's idle-regional-rehome poll built every (eligible host x target cell in its preferred region) pair, applied the cohort predicate downstream of that fan-out, sorted the lot, and took LIMIT 100 OFFSET n. Its cost was set by the size of the fleet and the width of the cohort, so raising the cohort from 10% to 100% pushed it past the serving pool's 5 s statement_timeout and the rollout stalled at 0.37 hosts/min. The poll now resolves the cell inventory once (tens of rows), takes a bounded window of decision rows in primary-key order from a keyset cursor with the cohort, freshness and cross-region predicates applied first, verifies only that window against the host-side gates, and ranks targets in the process. Same candidates in the same priority order; the work per poll no longer depends on the cohort or the fleet. Adds a once-a-minute aggregated poll summary so an operator can tell a poll gated by the dispatch budget from one that found nobody to move. Co-Authored-By: Claude <noreply@anthropic.com> * fix(relay): pin the rehome verification to the window's exact keys The window read and the verification read take separate snapshots. The verification repeated the window's predicate with its own LIMIT, so a decision that turned eligible between the two reads shifted that LIMIT and pushed the window's last host out of it -- while the cursor still advanced past that host, skipping it for a whole sweep. The verification now names the keys the window returned. Its LIMIT stays as the optimisation fence that stops Postgres flattening the subquery, but can no longer truncate a key set that is at most one window long. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-09-18 21:16:19 -04:00
Jinwoo Hong	164c7140fd	fix(relay): stop reporting an unavailable home cell as exhausted capacity (#21518) ... A host whose home cell is not live — readiness false, drained, or inside a boot window — is refused by the committed-fence branch in assignOnce() without any capacity being consulted. It answered relay_capacity_exhausted, so every cell boot and every readiness dip printed capacity rejections at 17% fleet utilisation and sent an investigation after headroom that was never short. The branch now raises RelayHomeCellUnavailableError, which carries the cell id and which of cellIsLive()'s conditions failed (draining / booting / unheard / not_ready). The director logs reason, cause and cell, and returns the new reason in the same retryable 503. Nothing on the wire reads the body: the desktop client discards it unread and branches on status only, and no log-based metric or alert parses the reason. The load harness, the only body-reading consumer, gets its own bucket so a home-cell rejection no longer inflates the capacity count. Hinted grants are now logged on whichever lane served them, so a host that failed sticky verification and was rehomed by placement leaves a record of where it landed. Unhinted placement grants stay silent.	2026-09-18 17:55:35 -04:00
Jinwoo Hong	3467e5f6b5	fix(relay): check the rehome dispatch budget before planning the candidate join (#21517) ... * fix(relay): check the rehome dispatch budget before planning the candidate join `selectIdleRegionalRehomeCandidates` read the enable control and the fleet safety snapshot, then ran the twenty-table candidate join, then handed every row to the worker, which POSTed each one to its source cell. Only there — in `commitIdleRegionalRehome`, three statements into a write transaction that takes `FOR UPDATE` on two global single-row tables — was the durable dispatch budget consulted. The budget is ten moves a minute (`next_dispatch_at = now + 6s`), and five directors poll every six seconds, so most of that work was spent to be told the budget was closed. A five-minute `paused_until` made every poll in the window do it. The gate is a single-row primary-key read, so it goes in front. An absent row means the budget has never been spent and opens the gate, matching the INSERT ... ON CONFLICT DO NOTHING the commit path already relies on. * test(relay): assign the closed budget field once so the case runs on Postgres The two gate cases zeroed both `next_dispatch_at` and `paused_until` and then set the one under test, which names that column twice in a single `SET`. SQLite accepts it; Postgres raises "multiple assignments to same column", so both cases failed whenever `ORCA_IDLE_REHOME_POSTGRES_URL` pointed the suite at a real server -- exactly the backend the gate has to hold on. Setup already leaves both fields at 0, so naming the other one bought nothing.	2026-09-18 17:55:27 -04:00
Jinwoo Hong	91ade4b82a	perf(relay): batch control lease renewals per cell instead of one write transaction per host (#21303) ... * perf(relay): batch control lease renewals per cell instead of one write transaction per host Every connected desktop renewed its own control lease with its own single-row write transaction every 30s. At ~14,000 hosts that is ~470 write transactions per second fleet-wide, each with its own transaction id, all updating the same few heap pages of relay_assignments and relay_assignment_activity_leases. Sampling three onsets at 250ms showed no lock queue and no slow statement: 60-144 backends piled into LWLock:BufferContent and Timeout/SpinDelay inside that one statement, and Query Insights attributed 152 of 157 seconds of lightweight-lock wait in the onset minute to it. The heartbeat now enqueues a due renewal instead of issuing it. A cell flushes its queue once per second, or as soon as 500 rows are waiting, through one statement that unnests the parameter arrays and applies the same CTE row-wise. Concurrent writers drop from the host count to the cell count, and transaction ids with them. Measured against a 20,000-row table: 1 row 4.1ms, 12 rows 3.3ms, 100 rows 6.6ms, 500 rows 22.7ms. Per-session semantics are unchanged. Each enqueue still resolves on a renewal and rejects with the outcome as its message, so the completed-attempt counter, the staleness guard, and every close path route exactly as before, and one outcome per row is recorded against the flush latency. Lock order is (user_id, relay_host_id), the primary key of relay_assignments, applied in JavaScript and repeated as the statement's ORDER BY. EXPLAIN confirms LockRows sits above that Sort, so a batch acquires its assignment rows in one global order. Every writer in the store locks a host's assignment row before its migration or lease rows and only ever touches one host, so a batch can only wait on a row a single-host writer holds, never the reverse. One statement also means one contended row could fail the whole batch, so a failed batch degrades to the per-host statements it replaced rather than costing every other host on the cell its renewal. * perf(relay): batch control lease renewals per cell instead of one write transaction per host Every connected desktop renewed its own control lease with its own single-row write transaction every 30s. At ~14,000 hosts that is ~470 write transactions per second fleet-wide, each with its own transaction id, all updating the same few heap pages of relay_assignments and relay_assignment_activity_leases. Sampling three onsets at 250ms showed no lock queue and no slow statement: 60-144 backends piled into LWLock:BufferContent and Timeout/SpinDelay inside that one statement, and Query Insights attributed 152 of 157 seconds of lightweight-lock wait in the onset minute to it. The heartbeat now enqueues a due renewal instead of issuing it. A cell flushes its queue every second, or as soon as 200 rows are waiting, through one statement that unnests the parameter arrays and applies the same CTE row-wise. Concurrent writers drop from the host count to the cell count, and transaction ids with them. Per-host buffer traffic is unchanged: 30 hits for one row, 25.5 per host at 12 rows, 30.1 per host at 200, against the 28.7 the single-row statement reports in production. Per-session semantics are unchanged. Each enqueue still resolves on a renewal and rejects with the outcome as its message, so the completed-attempt counter, the staleness guard, and every close path route as before, and one outcome per row is recorded against the flush latency. Row locks live until the statement commits, so a batch that waited on a contended row would hold every other row's lock for that whole wait. The assignment pass therefore takes its locks with SKIP LOCKED and reports a contended host as assignment_lock_unavailable, which the registry retries on the next tick instead of closing the control. That keeps the hold to the statement's own execution: 11.5ms for 200 rows against a 20,000-row table, and 9.4ms with a host wedged in a per-host transaction, where a blocking FOR UPDATE spends the pool's whole 1s lock_timeout and then fails every row in the flush. An unlocked present_assignment probe separates a host with no assignment row from one the skip passed over, so a skipped row can never be mistaken for a missing assignment and close a live desktop. With no wait on the assignment pass the lock order is only needed for the two later passes, and it holds: every writer takes a host's assignment row before that host's lease rows, and a host whose assignment row is held was skipped, so the batch never reaches its lease. markMigrationTargetRegistered is the one writer that locks a migration row first, and it takes no further locks. * fix(relay): answer every row of a control-renewal batch from its own lease update Review findings on the batched renewal. Two control leases on one host in one batch made the second report control_activity_not_found although both were renewed: the assignment UPDATE is offered the same target row twice, applies one source row and returns one, so the other row_index never came back. The verdict now reads renewed_lease, which has a row per input row, and the assignment UPDATE groups per host so it also stops taking an arbitrary one of the two expiries instead of the later one. The queue now partitions per (userId, relayHostId) rather than per activity, so a second control activity for one host opens the next flush instead of sharing this one. Belt to the statement fix, not a substitute: the store API has to be right for the rows it accepts. renewControlActivities recorded no outcome for a one-row flush that threw, and none at all when every row failed validation, where a mixed batch recorded its invalid_* rows. Both now record in a finally, the way the single-row path's finally always did, and the error-to-outcome mapping both paths share is one function. The four flush fields the runtime metrics event emits had no log-based metric, so add them next to the existing controlRenewalLatencyMs* entries. Applying the Terraform is a separate manual step. controlRenewalLatencyMsP50/P95/Max now measure a batched row's flush duration rather than its own statement latency. Left named as they are for history, with a line at the emit site recording that the meaning changed here.	2026-09-17 19:25:54 -04:00
Neil	0537c6eb3b	perf: index drain migration inventories by assignment identity (#20344)	2026-09-12 18:12:20 -07:00
Jinwoo Hong	f7238ce469	fix(relay): bound idle rehome polling and skip disabled scans (#20203) ... * fix(relay): bound idle rehome polling and skip disabled scans * test(relay): align sweep jitter expectation with polling budget	2026-09-11 23:58:24 -04:00
Jinwoo Hong	1d7bb47a11	test(relay): harden regional rehome race coverage (#20136)	2026-09-11 11:42:03 -07:00
Jinwoo Hong	cd9aa43a2c	feat(relay): correct regional placement only when the source is idle (#20105) ... * feat(relay): correct regional placement only at an idle source * test(relay): lock source activity capacity semantics	2026-09-11 14:25:10 -04:00
Jinwoo Hong	a6e6de93c4	fix(relay): keep failed rehome polls out of the durable failure budget (#19915) ... * fix(relay): keep failed rehome polls out of the durable failure budget The regional rehome worker polls claimRegionalRehome about once a second. Any error thrown before an attempt was claimed - in practice a director pool timeout on the pre-claim control read, 52-74 a day against a pool of 3 - was charged to relay_region_rehome_worker_state.consecutive_failures, which durably disables the control at three. That counter only ever resets on a drain receipt, so while the control is disabled it never resets: production sits at 1068 and still climbing. Enabling the control leaves the stale counter in place, so the next pool timeout latches it straight back off. That is what ended the 2026-08-28 enable after ten minutes. - A poll that never claimed an attempt drained nothing, so it no longer feeds the dispatch-failure budget and logs .._poll_failed instead of .._dispatch_failed. recordRegionalRehomeWorkerFailure had no other caller and is removed. - Enabling the control clears consecutive_failures and paused_until, so a budget spent under a previous enable cannot kill a fresh one. The dispatch interval in next_dispatch_at is deliberately left alone. - The budget's auto-disable now emits orca_relay_regional_rehome_failure_budget_disabled, matching the existing .._safety_disabled precedent. It wrote no event before, which is why this went unnoticed for two weeks. No change to region selection, the candidate query, or host eligibility. * fix(relay): serialize rehome failure accounting with control updates	2026-09-10 16:04:25 -04:00
Jinwoo Hong	f5be177e44	fix(relay): rehome hosts to their preferred region in either direction (#19241) ... * fix(relay): rehome hosts to their preferred region in either direction The regional-rehome worker only moved hosts from a us-central1 cell to an asia-east2 one, so a host whose desktop later records us-central1 stays where it was put. Rehoming now compares the fresh preference against the region of the cell the host is on and moves it to a general cell in the preferred region either way, through the same drain, migrate, safety, and rate-limit machinery. - relay_region_rehome_attempts.preferred_region accepts both regions; existing databases are upgraded in place by an idempotent named-constraint swap that is safe when several directors start at once. - A target must carry the drain protocol too: moving a host onto a cell it can never be drained off again is the trap this change exists to undo. The fleet whose health gates a rehome is now every general drainable cell, which is exactly the set of legal sources and targets. - The trust probe accepts a source cell in any region. No wire change, and no behaviour change while the durable control is off. * fix(relay): bound bidirectional rehoming with a per-host cooldown Moving hosts in both directions removed the property that made the old one-way worker self-terminating: a desktop whose region probe flips would be dragged back and forth, one full drain and migrate per flip, because the preference age never expires while the host keeps reconnecting. - relay_region_rehome_control gains host_cooldown_ms, an operator input plumbed like preference_max_age_ms (workflow, ops script, admin route, durable row) and defaulted to seven days. A host with any attempt row inside the window, whichever way that move went, is not a candidate; the claim re-reads it under lock so an attempt landing between scan and claim cannot start a second move. Skips are named host_cooldown, and the lookup rides a new index on (user_id, relay_host_id, created_at). - The candidate scan now also requires the target cell to be enabled, so it mirrors the claim-time filter exactly and stops spending batch slots on candidates that are certain to be skipped. - Region CHECK lists are rendered from the shared region list instead of being written out four times. - The operations runbook states that cells without the drain protocol are neither sources, targets, nor members of the safety gate. * fix(relay): keep rehome reads and brakes working across the cooldown rollout The ops script validated hostCooldownMs on every inspected control, so against any director image predating the field inspect, pause, disable, and failed-enable recovery all threw client-side. The workflow always runs from main while the director image is operator-supplied, so that window opened at merge and reopened on every rollback: the operator lost read-only visibility and both emergency brakes while the worker could still be enabled. The field is now validated only when the director reports it, and every apply body that echoes an inspected control omits the key when that control lacks it, so a legacy director never sees an unknown key. The write path stays fail-closed the other way: enable refuses up front, before any mutation, when the director does not report a cooldown it could honour. Also replaces two bare 'us-central1' defaults with RELAY_DEFAULT_REGION.	2026-09-07 04:40:37 -04:00
Jinwoo Hong	b6f453df06	perf(relay): per-cell inventory locks, delta counters, and a pool statement timeout (#18722) ... The sticky refresh path and reservation reconciliation both took the fleet-wide `relay_cells ... FOR UPDATE` scan to mutate one or two rows, so one busy cell queued unrelated reconnects and migration completions behind it. Both now lock only the rows they touch, in the same ascending cell_id order, and the sticky grant moves its counter by a delta instead of writing back a snapshot value. Placement keeps the ordered inventory lock: choosing the least-loaded cell is a genuinely fleet-wide decision, and dynamically locking only the selected target is what allowed cross-cell cycles before. The pool's statement_timeout becomes env-configurable and a 57014 now reaches the bounded transaction retry instead of surfacing as a terminal failure. Schema DDL moves to its own `max: 1`, statement_timeout-free pool that is ended before the serving pool opens, so a slow CREATE INDEX cannot inherit a request deadline it will never fit inside.	2026-09-04 18:18:46 -04:00
Jinwoo Hong	7b108abf71	fix(relay): stop taking the fleet-wide cell inventory lock on per-connection paths (#18606) ... * fix(relay): stop taking the fleet-wide cell inventory lock on per-connection paths activateControl, acquireActivity, changeActivity and removeSupersededSameCellControls each adjust exactly one cell's reservation, yet took SELECT * FROM relay_cells FOR UPDATE, so every desktop rebind and phone reconnect in the fleet queued behind every other one and behind placement. They now use the single-row atomic update (or lock only their own cell row), leaving the inventory lock to placement and sweeps. Fleet-wide 55P03 retries ran p50 430 / p99 1320 per five minutes on 2026-09-03, every cell pinned sqlLatencyMsMax at the lock timeout, and the old cell image crashed on the resulting pool timeouts ~every 15 minutes. A real-Postgres test holds another cell's row and asserts a rebind proceeds; re-adding the inventory lock fails it. * fix(relay): lock the touched cell rows in order on cross-cell activity moves Review found that acquireActivity's existing-lease branch could lock the old lease's cell row (via removeActivityLease) before the new cell's row, which cycles with placement's ascending inventory lock; reproduced on real Postgres as paired 55P03 retries. lockCellRows now takes the one or two rows a per-connection path touches in cell_id order with the 500 ms request bound, and the census fails on any inline relay_cells FOR UPDATE outside the named lock helpers. A three-cell Postgres test moves an activity from the highest cell to a lower one while the target row is held and asserts the mover holds nothing else; five revert-mutants (inventory lock on each path, dropped ordering, dropped ORDER BY) fail it. * test(relay): make the inline relay_cells lock census scan whole statements Review showed two evasions: a FOR UPDATE inside query() and a queryLocked whose FROM relay_cells sat past a fixed line window. The guard now matches every query()/queryLocked() template statement in full; both evasions fail it. Also clears relay_cell_connection_snapshots in the connection- headroom Postgres suite so an aborted run does not poison the next.	2026-09-04 04:34:22 -04:00