<!-- orca-pr-loc -->
<!-- Programmatic LoC summary. Do not edit by hand; rewritten on every commit. -->
| | Files | Added | Deleted | Net |
| :--- | ---: | ---: | ---: | ---: |
| Test | 4 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$406 | $\color{#cf222e}{\Huge{\mathbf{−}}}$4 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$402 |
| Prod | 6 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$165 | $\color{#cf222e}{\Huge{\mathbf{−}}}$10 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$155 |
<!-- /orca-pr-loc -->
## ELI5
Relay servers ("cells") normally get their seat bookings from the front door (the director). If the director goes quiet for a minute or two, each cell switches itself to an older mode that talks to the shared Postgres database for every connection (the "dead-man" trip). Today every cell switches at once, so the database budget must keep room for every cell's full connection pool. This PR adds a per-cell switch that caps how many cells may trip at the same time. A cell that cannot trip yet stays in its normal mode and keeps serving the desktops it already has.
## What Changed
- New cell flag `deadManTripSlots` (integer 0-64). Absent, which is every cell today, keeps the current behaviour: trip as soon as the dead-man window (60-120 s) passes.
- With the flag set, a cell whose window has passed first claims a slot in Postgres. The claim is one short transaction: it takes an advisory lock without waiting, counts the other reserve cells whose own `relay_cell_admit_effective` row says `db`, and writes this cell's row to `db` only if that count is below the flag. Only then does the cell trip (void bookings, re-register its hosts, run on its full pool).
- A cell that gets no slot is "held". It stays in reserve mode, logs `orca_relay_cell_reserve_dead_man_held` once, keeps writing `reserve` to its row, and asks again every 30 s.
- A slot is held while the holder's own row says `db`, its switch still says `reserve`, and the row was refreshed in the last 10 minutes. A holder that restarts or is re-armed writes `reserve` and frees its slot. One that died tripped frees it after 10 minutes. Heartbeats are not used: they reach Postgres through the director, so in a director outage they all go stale and every slot would read as free.
- A tripped cell skips its 15 s row refresh while its pool has waiters. So that a long brownout cannot age its row out of the count, a tripped cell's row older than 5 minutes is refreshed even when the pool is busy.
- A trip stays latched after the director returns, as today. The tripped cells keep their slots until an operator re-arms them with a new reserve write. Until then, a second outage trips nobody new: the other cells stay held. That is the intended bound, but re-arming after an outage now also frees the slots.
- The claim goes through `CellAdmitEffectiveWriter`, which already serialises every write of that row. A `reserve` write still in flight therefore cannot land after the claim and hide the slot from the next claimant.
- Cells keep their existing per-cell window spread (60-120 s by cell-id hash), so the earliest windows claim first.
What a held cell does during a director outage (unchanged code paths, now for longer than the window):
- Desktops already connected keep their sockets. Nothing is closed or re-registered.
- A desktop that drops and redials at the same epoch is admitted from memory (`seatedHereAt`). Any other hello falls through to the normal database check, as it does in reserve mode today.
- The database placement path places no new desktops on it, because its row says `reserve`. It also never re-places the held cell's desktops, because `liveReserveCell` keys on the switch (`relay_cell_admit_modes`), not on this row.
- New placements during the outage go to cells already in database mode (c32, c33 and c34 today) plus the tripped slot holders.
## Why
The database budget (Cloud SQL `orca-cloud-auth-db`, shared with auth, max_connections 500, 490 usable) currently reserves every relay cell's full pool. The only reason for that is the dead-man trip of the whole fleet at once. Spare is 35 on main and about 3 once C36/C37 land, which caps the number of Asia cells.
Budget arithmetic (`relay-cloud-sql-connection-budget.mjs` on orca-cloud main: directors 15, auth 20, API 50, rollout overlap 65, maintenance 5, explicit reserve 10):
| | Cells line | Operating | Spare |
|---|---:|---:|---:|
| Main today: 6 db/legacy cells at 10, 16 US at 8, 7 Asia at 16 | 300 | 455 | 35 |
| Plus C36/C37 at 16 | 332 | 487 | 3 |
| Same fleet, `deadManTripSlots=2`, Asia reserve cells budgeted at 12 (measured peak) | 76 + 128 + 8x12 + 2x4 = 308 | 463 | 27 |
| Plus two more Asia cells at 12 | 332 | 487 | 3 |
So with two slots, two more Asia cells fit in the room C36/C37 leave today. Two caveats keep this honest:
- The bound holds only once every reserve cell runs this image with the same flag value. An older image ignores the flag and trips freely, though its `db` row still takes a slot from the capped cells. With mixed values the cap is the largest value set.
- Counting reserve cells at their measured peak is a softer guarantee than counting their configured pool: the pool is still 16, and a reserve cell could burst past 12. Making it a hard bound needs a pool ceiling that depends on the mode (reserve-mode ceiling 12, the full 16 only for a slot holder or a deliberate db cell). That is a follow-up; this PR does not change pool sizes.
Alternatives considered:
- Longer or ranked trip delays only spread the trips out. A trip is latched until the switch is rewritten, so after the longest delay every cell is still in database mode. That bounds the burst, not the budget.
- A static per-cell "may trip" list in the switch file needs no database coordination, but a listed cell that is down or draining wastes its slot, and an operator has to keep the list current.
- Removing the fallback (cells never trip) is the target design, but it waits on the director stall fix and two weeks of clean running. `deadManTripSlots=0` gives exactly that per cell, so it is the same switch.
Deliberate db flips (for example, flipping reserve cells to db for an image roll) are outside the cap. Budget each Asia cell flipped that way at its full 16. With a two-cell Asia roll batch running during an outage, the slot row above is 316, which leaves room for one more Asia cell rather than two.
Rollout: the flag is dark. Setting it is an `Operate Relay Production Cell Flags` write, per cell, after every reserve cell runs this image.
## Linked Issue
N/A (internal relay capacity work).
## Visual Proof
N/A: relay server behaviour only, no UI.
## Testing
- `cell-reserve-dead-man.test.ts` (new): trips as before without the flag; held until a permit, then latched; slot 0 never trips; a permit for a silence that contact ended, or for an earlier switch generation, is dropped.
- `cell-admit-effective-writer.test.ts`: the claim waits for an in-flight reserve write; a refused claim keeps the reserve row fresh and retries after 30 s; no claim while the pool has waiters or the switch is off reserve; a tripped row is refreshed past a saturated pool after 5 minutes, a reserve row is not.
- `cell-reserve-dead-man-slots.blackbox.test.ts` (new, real cell on SQLite): a director outage with the only slot taken by another cell that writes nothing but its own row: the cell stays reserve across several claims, re-registers nothing, keeps its desktops connected, and lets a redialing desktop back in from memory; once the peer writes `reserve`, the cell trips and re-registers. Also: slot 0 rides out 30 minutes of outage without tripping; no flag trips as before.
- `cell-dead-man-trip-slot-postgres.test.ts` (new, PostgreSQL): eight cells claiming together get exactly two slots; a live holder keeps its slot; a stale one frees exactly one. Removing the advisory lock fails this test.
- Mutation checks: making the slot count always 0, or removing the lock, each fails a test.
- Full relay suite, including the PostgreSQL suites against a local PostgreSQL 17: 173 files, 1564 passed, 3 skipped. The pr.yml static-analysis steps pass locally.
- Two rounds of adversarial review. Round 1 found that the first version counted holders by director-written heartbeats (fixed). Round 2 found the brownout aging (fixed) and the latched slots after an outage (documented above).
- [x] I manually tested these changes locally
- [x] Automated tests added/updated, or explained why not below
## Review
## Agent skill upstream boundary
- [x] Not applicable, or this change copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.
## Notes
No new table or column. The claim reads `relay_cell_admit_effective` and `relay_cell_admit_modes` (one row per cell). Under a fleet-wide trip, held cells add one short transaction each per 30 s; the advisory lock is held for about one round trip.
## Checklist
- [x] This PR is small and focused
- [x] I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
- [x] Before/after screenshots or videos attached for UI changes, or `N/A` with reason
- [x] Self-reviewed for correctness, security, and performance
- [x] Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
- [x] `pnpm lint`, `pnpm typecheck`, `pnpm test`, and `pnpm build` pass (or CI will cover; local preferred)
<!-- orca-pr-loc -->
<!-- Programmatic LoC summary. Do not edit by hand; rewritten on every commit. -->
| | Files | Added | Deleted | Net |
| :--- | ---: | ---: | ---: | ---: |
| Test | 3 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$45 | 0 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$45 |
| Prod | 3 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$48 | 0 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$48 |
<!-- /orca-pr-loc -->
## ELI5
Some relay director instances talk to the database about twice as slowly as the rest. Those slow ones are where almost all the single-instance stalls happen. Today the logs can't tell us where an instance is running, so we can't work out why some are slow. This PR adds two numbers to the log line every director already writes every 30 seconds: the zone the instance runs in, and the instance's typical database round-trip time.
## What Changed
- **Before:** the director's `orca_relay_runtime_metrics` line had per-connection statement timings but nothing about where the instance runs. Getting one per-instance speed number meant pooling those connection timings by hand.
- **After:** two new fields on that line.
- `instanceZone`: read once at boot from the Cloud Run metadata server (`instance/zone`). It's only read when `K_SERVICE` is set, so GCE cells and local runs skip it. If the read fails, the field is simply left off.
- `stickyStatementMsP50`: the median duration of every sticky-lane database statement in the window, across all connections. It comes from a new window-wide sample of up to 1024 statements, the same kind already kept per connection.
- **Unchanged:** behaviour. The zone read is fire-and-forget with a 3 s timeout and never blocks or fails boot. No request path reads either field.
## Why
In 7 days of director logs (10-17 to 10-25), each instance's median per-statement time sat at either about 1.7 ms or about 3.6 ms. An instance sometimes switched between the two mid-life, with no deploy. 74 of the 81 single-instance stalls happened while the instance was at the slow speed: about 15x the rate at the fast speed.
- CPU, the event loop, the pg pool and database locks were all ruled out. That leaves the network path between the instance and Cloud SQL, and we can't confirm it without knowing each instance's zone.
- With zone and speed on the same line, a couple of days of logs will show whether "slow" follows the zone or the host. That decides between the next fixes: changing the Cloud Run sandbox generation, using a private IP, or cutting round trips per assignment.
- **Alternative considered:** a boot-time-only zone log line. It would force every analysis to join two log streams, and the field on each line costs a few bytes.
- **Alternative considered:** logging a "fast/slow" label instead of the raw median. That would bake today's thresholds into the image.
## Linked Issue
N/A (internal investigation of director single-instance stalls).
## Visual Proof
N/A: log-only change on a backend service, with no UI.
## Testing
- `pnpm typecheck` for `@orca-cloud/relay`.
- The full relay vitest suite: 1348 passed, 201 skipped (postgres-backed suites skip without a database).
- New tests cover:
- parsing the zone from the metadata answer, and returning nothing on a refused, unreachable or malformed answer;
- the zone appearing only once it is known;
- the pooled statement median.
- The floating-promise census lists the zone read as settling its own failures.
- [ ] I manually tested these changes locally
- [x] Automated tests added/updated, or explained why not below
## AI Disclosure
## Review
## Agent skill upstream boundary
- [x] Not applicable, or this change copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.
## Notes
- **Wire compatibility:** none affected. These are new optional fields on a log line, and the existing watchers read named fields only.
- **SSH, mobile, desktop:** not affected.
- **Rollout:** rides the next director image. No pin or deploy is part of this PR.
## Checklist
- [x] This PR is small and focused
- [x] I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
- [x] Before/after screenshots or videos attached for UI changes, or `N/A` with reason
- [x] Self-reviewed for correctness, security, and performance
- [x] Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
- [x] `pnpm lint`, `pnpm typecheck`, `pnpm test`, and `pnpm build` pass (or CI will cover; local preferred)
<!-- orca-pr-loc -->
<!-- Programmatic LoC summary. Do not edit by hand; rewritten on every commit. -->
| | Files | Added | Deleted | Net |
| :--- | ---: | ---: | ---: | ---: |
| Test | 1 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$279 | 0 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$279 |
| Prod | 1 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$58 | $\color{#cf222e}{\Huge{\mathbf{−}}}$13 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$45 |
<!-- /orca-pr-loc -->
## ELI5
When we restart a relay server during a routine rollout, computers that were offline during the rollout still have that server written down as "their" server. If one of them comes back during the ~45 seconds while the replacement server is starting up, the director currently tells it "your server is unavailable, try again" over and over, instead of just sending it to another server. This change lets the director send it elsewhere, but only when it has proof that the old server process is really gone.
## What Changed
**Before (production, 2026-10-24 23:43Z, cell production-gce-c15):** during a same-capacity roll the cell is roll-isolated, drained, then its single VM is replaced. A desktop host that was offline during the drain reconnected in the gap between the old process stopping and the new one being ready. The director refused it for ~26 s (`assignment rejected route=assign lane=placement hinted=true reason=relay_home_cell_unavailable cause=not_ready`, x5, plus `route=resolve ... cause=unheard`, x2). It got a grant somewhere else only once the new process was live.
**After:** while the replacement process is starting up (`not_ready`), the same host is moved to another cell on its first try. A host on a roll-isolated cell can be moved off even though the cell is not live, as long as the process heartbeating for the cell *started after the roll isolated it* (`relay_cell_runtime.started_at > relay_cell_admission.roll_isolated_at`), its heartbeat is fresh, and (for connection-capped cells) its latest heartbeat reports zero connections. Both lanes that move an isolated host apply it: the narrowed isolated-replacement path (`isolatedIncumbentMayMove`) and the all-rows path it falls back to. Together these serve assign and resolve, because resolve hands a non-live cell to `assign`. "Roll-isolated" is the existing predicate: `migration-only` with a roll stamp younger than 2 h.
**Still refused (residual):** the `unheard` window, when the old process's heartbeat is past the 45 s TTL and the new process has not been heard yet. On c15 that was about 5-10 s. Nothing proves the drained sockets are gone there, so the refusal stays, and the client's retry recovers once the new process heartbeats. Also unchanged: an ordinary readiness dip, a `migration-only` cell without a roll stamp, a stale (over 2 h) stamp, and a new process that later goes unheard. Fence handling is untouched.
## Why
The refusal exists because moving a host off a cell we cannot fully trust could leave the host's old socket served by the old process while its clients go to the new cell. Until now, the only proof the code accepted was an operator-written fence. This adds a second proof that costs no operator step:
- **A new incarnation means the drained process is gone.** Each process mints a fresh incarnation UUID and `startedAt` at boot (`cell-heartbeat-client.ts`), and the director rejects a heartbeat with an older `startedAt` once a newer one is recorded. Each cell VM runs exactly one relay container (`docker run --name orca-relay --restart always`, with stale containers force-removed at boot). The cell MIG uses `replacement_method = RECREATE` with `max_surge = 0` (pinned by a Terraform assert), so the old VM is deleted before the new one is created. A same-VM container restart also produces a new incarnation and is just as safe: Docker restarts only after the old process exits, and that exit closes all of its sockets.
- **Whatever the new process holds, it can enforce.** It is heartbeating (fresh), and on capped cells it reports no connections at all. A host that attaches between two heartbeats carries the old epoch, so the epoch bump this move writes makes the cell refuse it or close it with 4409. That check is the same live `verifyCellAssignment` / control-renewal read the existing live-isolated path relies on.
- **Clock skew** between the cell's `startedAt` and the director's stamp is covered by the zero-connections check on capped cells only: a drained process that only *looks* post-stamp still holds its sockets, so it is refused. Uncapped cells report no counts and get no such guard; they already move off without a fence on the dead-cell path today, so this adds no new exposure there.
Alternatives considered: (a) adding an incarnation column to `relay_assignments`: a schema change for no extra proof. (b) Honouring any not-live roll-isolated cell: that drops the proof. (c) Having the roll write a fence for the `unheard` window: the only fence driver refuses enabled cells, a fence is a MIG resize to 0 that would fight the roll's own apply, and the window is a few seconds the client retry already absorbs.
## Linked Issue
N/A (internal production incident, 2026-10-24 c15 roll).
## Visual Proof
N/A: director placement logic only, no UI.
## Testing
- [x] Automated tests added: `cloud/apps/relay/src/assignment-roll-replaced-cell.test.ts` (18 cases; the redial cases run for both hinted assign and resolve):
- not_ready with an incarnation started after isolation: re-placed, epoch bumped.
- Refused as before: not_ready with the pre-isolation incarnation, with reported connections, after the new process goes unheard, with a stale stamp, on an ordinary unstamped readiness dip, and on an unstamped migration-only cell. Also refused: the unheard window between the old and new process.
- One case per lane. The narrowed path's cross-region move fails if only the narrowed-path change is reverted. The all-rows path's move, forced by units no lease backs, fails if only the all-rows change is reverted.
- [x] Full relay vitest suite, `--maxWorkers=2`: 130 files / 1344 tests passed, 39 files (201 tests) skipped. The Postgres-backed suites skip locally (no `ORCA_RELAY_TEST_POSTGRES_URL`); CI runs them.
- [x] `pnpm typecheck` in `cloud/apps/relay`.
## Review
## Agent skill upstream boundary
- [x] Not applicable, or this change copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.
## Notes
Backwards compatibility: no schema, wire or RPC change; old directors keep refusing as today. Performance: the new query runs only when a host's home cell is roll-isolated and not live; the general sticky hot path is unchanged.
## Checklist
- [x] This PR is small and focused
- [x] I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
- [x] Before/after screenshots or videos attached for UI changes, or `N/A` with reason
- [x] Self-reviewed for correctness, security, and performance
- [x] Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
- [x] `pnpm lint`, `pnpm typecheck`, `pnpm test`, and `pnpm build` pass (or CI will cover; local preferred)
The director now accepts a cell heartbeat whose reported connection cap is at or below its configured cap, and places with the lower of the two; a reported cap above the configured one is still refused. A cap-only change keeps the cell's stored capacity rows. This lets the US cells' configured cap be raised once, ahead of their rolls, so an old process at the lower cap stays live until it is replaced, instead of desktops pointed at it being refused during a resize.
Reserve-aware cell status for the director: reserve cells report their own runtime and seat feed, drain causes are attributed per cell, and paced 503s on a host leaving a draining cell are recorded as the drain's.
Rehome on reconnect for reserve-mode cells: a desktop reconnecting on a reserve cell outside its preferred region is placed fresh in that region, only while the region has at least 10% of seats free, at most once a day per desktop, rate-limited, and switched off by default (ORCA_RELAY_RESERVE_REHOME_PERCENT=0); the regional rehome switch stops it.
Before refusing a placement with relay_capacity_exhausted, the director tries a reserve seat in the desktop's own region, then the other region, through the cell's own reserve API; if every region paces it answers the shortest Retry-After.
* fix(relay): count region selection on every assign lane, fallback only for new placements
Co-Authored-By: Claude <noreply@anthropic.com>
* test(relay): pin placed marker on isolated re-place and kept pins; drain-return fallback
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* test: remove redundant cases and repair behavioral assertions
* docs: reconcile audit records with current main
* fix(deps): apply security updates from #32101
Apply the six-file DOMPurify 3.4.17 and Next.js 16.4.1 patch from PR #32101 to resolve the dependency security gate for this test audit. Original patch by Brennan Benson.
* test: satisfy changed-line quality gates
Add required braces and replace filter indexing with find. Rerun both CI quality gates and the 27 affected relay tests; refresh audit hashes and record the corrections.
---------
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
Co-authored-by: Brennan Benson <79079362+brennanb2025@users.noreply.github.com>
* fix(relay): repair cell seat-log drift, and keep a new session's generation above a listed seat
Directors mirror each cell's seat feed, so the feed must list exactly the cell's live sessions.
The cell now sweeps its seat log against its sessions on a director's poll: a seat no session
backs is left (4409) and the live session listed, and a live session the feed lost is listed
again, each once two sweeps in a row find it. A demotion naming an unbacked seat repairs it at
once. A host's new session now starts above any generation the feed still lists, which closes
the drained-session reconnect race that removed a live seat.
Claude-Session: asia-drain-pulse
* fix(relay): refuse a rebind whose session left the map during activation, and keep a per-host generation floor
A rebind whose database activation outlived the orphan grace (or a drain) revived the removed
session outside the map: connected, still in the seat feed, and unreachable by phones or
demotion for as long as the host stayed up. Activation now rechecks the map after its awaits and
refuses with PEER_DROPPED so the host redials. Generations come from a per-host floor that
outlives the session, and an activation at a generation already used here is refused.
Claude-Session: asia-drain-pulse
* fix(relay): log each demote outcome on the director by host digest
The director now reads the cell's existing {v:1, outcome} demote reply and logs it; the cell's
feed stays the only source of its map. The supersede line now reaches the log as well.
Claude-Session: asia-drain-pulse
* fix(ci): recover verified pnpm setup from registry metadata outages
* fix(ci): prepare native pnpm shims after fallback verification
* fix(ci): provide Node for pnpm fallback in bare containers
* chore(cloud): update relay runtimes and security scanning
* docs(cloud): record Go binary advisory reachability
* chore(cloud): rerun security checks after resolving main conflict
* fix(relay): release a drained host before telling it to leave
A zero-grace drain sent `drain` and closed with 4503 in the same tick, then
released the host's control lease. The desktop redials the director as soon
as it reads either, and from asia-east2 the release holds the host's
assignment row for ~1 s, so the redial met it and was refused row_busy.
The release now runs first and the host is told once it lands, bounded at
2 s; a failed or stuck release still closes. Wire frames are unchanged.
* fix(relay): give a drained host's release 3 s and log when it runs out
One host's releases run one after another, so 3 s covers a control lease
plus one open phone connection from asia-east2.
* style(relay): brace the drain release branches
* fix(relay): give region-correction reconnects and roll re-placements the in-region reserve fallback
Every desktop session open carries a region-correction request, and those
skipped the reserve plan entirely, so with the region's database cells full they
still spilled abroad while an in-region reserve cell had room. The plan now runs
for them in a mode that never answers itself (the database lane must reply so the
window exchange follows) and offers only the in-region reserve fallback.
A host re-placed off a roll-isolated cell went straight to the other-regions tier
when its own region's tier was full. With an in-region reserve fallback it now
books that first, and the roll canary event is emitted for the reserve booking.
* fix(relay): never pace a roll drain on reserve intake, and count every reserve move off an isolated cell
A host leaving a roll-isolated cell whose region was full could get a paced 503
from a throttled reserve cell, stretching the drain and re-running the failed
in-region transaction on each retry. For those hosts a pace is now no room, and
they spill as before. Sticky now says when it declined an isolated pin, so a
share-draw booking off that cell also emits the roll canary event.
* fix(relay): keep a host in its region when the database lane is full and a reserve cell has room
A host that lost the reserve share draw went to the database lane, which
spills to another region when its own region's database cells are full,
even while an in-region reserve cell had free seats. The database lane now
requires the region when an in-region reserve cell has room; if the region
is full it books that reserve cell, and only spills when neither has room.
* fix(relay): pace a throttled in-region reserve cell instead of spilling, and decide reserve room at plan time
With the region's database cells full, a reserve cell with seats but no intake
budget fell through the placer's database-cells shortcut and the host went
abroad. That placement now answers pace (503 with Retry-After). regionReserve
is a single callback, present only while an in-region reserve cell has a seat.
* fix(relay): pace only on a reserve cell that lacked budget, not one that refused
A cell that refused this placement (full, unreachable, seated-newer) got its
token back and drove a 1 s Retry-After forever while the map still showed a
seat. Pacing now skips cells refused for anything but intake, so those
placements fall to the database lane's cross-region fallback as before.
* perf(push): wake delivery worker after notifications are queued
* perf(push): wake one drain per send so bursts cost O(sends + drains) claims
A wake started all twelve drains, so each send arriving at an idle worker cost
~13 claim transactions. A wake now starts one drain; the poll tops up to the
full drain set.
A host control request that failed for an unexpected reason sent the thrown
error's message to the desktop as the error code. During a staging database
outage the desktop received Postgres's own text, and node-pg messages can
carry the database connection URL.
The relay now sends store codes and this codebase's snake_case invariant
slugs unchanged, and maps everything else to database_transient or
control_operation_failed. The original is logged through the existing
redaction (now shared with the background-operation logger), so only the
slug shape and SQLSTATE leave the process.
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay): resume paired phones from one device read on memory cells
With phonePath=memory in reserve mode, a phone resume is admitted from one device
read and the cell's own seat, and takes no activity leases. A refused device closes
4401, a failed read 4503, and only a missing device row falls through to the invite
path. A phone whose host has no seat gets one assignment read: 4409 when the row
names another cell or the read fails, otherwise 4404.
* feat(relay): keep a memory resume's basis in the cell and confirm it from one read
A phone resumed on the memory path keeps its connection basis in the cell, not in
relay_connection_bases. Its confirm rereads the device row once, renews it only when
less than 29 days remain, and always answers a current-version confirm renewed:true
with the expiry the database holds. Confirms route by where their basis lives, so a
flip between accept and confirm completes either way. Retries answer from a 10-min
in-memory result map.
* test(relay): roll a memory phone cell back with confirms in flight
Flipping back by admitMode=db, phonePath=db or a dead-man trip leaves in-flight
confirms to finish on their memory basis, writes no phone lease and leaks none, and
evacuation still refuses the cell while Postgres records it as reserve.
* feat(relay): batch a memory cell's resume-confirmed audit rows
Memory confirms buffer their audit rows and the cell writes them as one multi-row
insert every 5 s or 500 rows, idempotent by a per-event id. The buffer holds at most
5,000 rows and drops a row older than 10 minutes rather than retrying it, counting
and logging both kinds of drop.
* feat(relay): count and log phone-path failures, and every attach deadline
The runtime metrics line gains phone refusals by close code, attach deadlines, accept
database errors, resume confirm failures by code, completed accepts by phone path and
the resume audit drop total. Every attach-deadline 4404 now logs what the cell knew of
the desktop's control (pong age, ping outstanding, control age and round trip), so a
half-open control can be told from a slow data dial.
* feat(relay): close an attach-deadline 4404 with the attach-timeout reason
The phone close reason from #30607 now rides the close frame of the 4404 a cell
sends when the desktop's data leg misses the attach deadline, so OTA phones can
redial the same cell once instead of waiting out the host-offline delay. The failed
relay-hello frame is unchanged, and old phones ignore the reason.
* test(relay): drill a memory cell through a database outage, and pin phone renewal
A blocked database leaves splices forwarding, closes new resumes 4503 and fails
confirms cleanly, lets an in-flight confirm finish on its memory basis after a flip
back, and drops audit rows past 10 minutes instead of writing them late. On the phone,
a relay-only session lives 31 days under the once-a-day renewal rule, and would not
if a not-due confirm answered renewed:false.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay): log a failed no-seat read, recheck the switch after the device read
A failed assignment read for a phone with no seat now logs a rate-limited line before
its 4409. A switch that turned off during the device read sends the phone down today's
path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy.
* fix(relay): log a failed memory renewal; test flips, replays and basis cleanup
A renewal write that fails still answers the stored expiry, but now logs a rate-limited
line so a persistently failing update is visible before phones age toward re-pairing.
* test(relay): assert a rolled-back memory splice issues no statement when it closes
* fix(relay): flush resume audit early only on crossing the batch size, drop the shutdown flush
A failing database no longer takes a 500-row insert per confirm; retries wait for the
5 s timer. The shutdown flush raced the pool closing, so it is gone and the at most 5 s
loss is documented. A stopped buffer no longer re-arms its timer.
* test(relay): assert the aged audit rows were dropped, not written
---------
Co-authored-by: E2E Test <e2e@test.local>
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay): resume paired phones from one device read on memory cells
With phonePath=memory in reserve mode, a phone resume is admitted from one device
read and the cell's own seat, and takes no activity leases. A refused device closes
4401, a failed read 4503, and only a missing device row falls through to the invite
path. A phone whose host has no seat gets one assignment read: 4409 when the row
names another cell or the read fails, otherwise 4404.
* feat(relay): keep a memory resume's basis in the cell and confirm it from one read
A phone resumed on the memory path keeps its connection basis in the cell, not in
relay_connection_bases. Its confirm rereads the device row once, renews it only when
less than 29 days remain, and always answers a current-version confirm renewed:true
with the expiry the database holds. Confirms route by where their basis lives, so a
flip between accept and confirm completes either way. Retries answer from a 10-min
in-memory result map.
* test(relay): roll a memory phone cell back with confirms in flight
Flipping back by admitMode=db, phonePath=db or a dead-man trip leaves in-flight
confirms to finish on their memory basis, writes no phone lease and leaks none, and
evacuation still refuses the cell while Postgres records it as reserve.
* feat(relay): batch a memory cell's resume-confirmed audit rows
Memory confirms buffer their audit rows and the cell writes them as one multi-row
insert every 5 s or 500 rows, idempotent by a per-event id. The buffer holds at most
5,000 rows and drops a row older than 10 minutes rather than retrying it, counting
and logging both kinds of drop.
* feat(relay): count and log phone-path failures, and every attach deadline
The runtime metrics line gains phone refusals by close code, attach deadlines, accept
database errors, resume confirm failures by code, completed accepts by phone path and
the resume audit drop total. Every attach-deadline 4404 now logs what the cell knew of
the desktop's control (pong age, ping outstanding, control age and round trip), so a
half-open control can be told from a slow data dial.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay): log a failed no-seat read, recheck the switch after the device read
A failed assignment read for a phone with no seat now logs a rate-limited line before
its 4409. A switch that turned off during the device read sends the phone down today's
path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy.
* fix(relay): log a failed memory renewal; test flips, replays and basis cleanup
A renewal write that fails still answers the stored expiry, but now logs a rate-limited
line so a persistently failing update is visible before phones age toward re-pairing.
* test(relay): assert a rolled-back memory splice issues no statement when it closes
* fix(relay): flush resume audit early only on crossing the batch size, drop the shutdown flush
A failing database no longer takes a 500-row insert per confirm; retries wait for the
5 s timer. The shutdown flush raced the pool closing, so it is gone and the at most 5 s
loss is documented. A stopped buffer no longer re-arms its timer.
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay): resume paired phones from one device read on memory cells
With phonePath=memory in reserve mode, a phone resume is admitted from one device
read and the cell's own seat, and takes no activity leases. A refused device closes
4401, a failed read 4503, and only a missing device row falls through to the invite
path. A phone whose host has no seat gets one assignment read: 4409 when the row
names another cell or the read fails, otherwise 4404.
* feat(relay): keep a memory resume's basis in the cell and confirm it from one read
A phone resumed on the memory path keeps its connection basis in the cell, not in
relay_connection_bases. Its confirm rereads the device row once, renews it only when
less than 29 days remain, and always answers a current-version confirm renewed:true
with the expiry the database holds. Confirms route by where their basis lives, so a
flip between accept and confirm completes either way. Retries answer from a 10-min
in-memory result map.
* test(relay): roll a memory phone cell back with confirms in flight
Flipping back by admitMode=db, phonePath=db or a dead-man trip leaves in-flight
confirms to finish on their memory basis, writes no phone lease and leaks none, and
evacuation still refuses the cell while Postgres records it as reserve.
* feat(relay): batch a memory cell's resume-confirmed audit rows
Memory confirms buffer their audit rows and the cell writes them as one multi-row
insert every 5 s or 500 rows, idempotent by a per-event id. The buffer holds at most
5,000 rows and drops a row older than 10 minutes rather than retrying it, counting
and logging both kinds of drop.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay): log a failed no-seat read, recheck the switch after the device read
A failed assignment read for a phone with no seat now logs a rate-limited line before
its 4409. A switch that turned off during the device read sends the phone down today's
path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy.
* fix(relay): log a failed memory renewal; test flips, replays and basis cleanup
A renewal write that fails still answers the stored expiry, but now logs a rate-limited
line so a persistently failing update is visible before phones age toward re-pairing.
* test(relay): assert a rolled-back memory splice issues no statement when it closes
* fix(relay): flush resume audit early only on crossing the batch size, drop the shutdown flush
A failing database no longer takes a 500-row insert per confirm; retries wait for the
5 s timer. The shutdown flush raced the pool closing, so it is gone and the at most 5 s
loss is documented. A stopped buffer no longer re-arms its timer.
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay): resume paired phones from one device read on memory cells
With phonePath=memory in reserve mode, a phone resume is admitted from one device
read and the cell's own seat, and takes no activity leases. A refused device closes
4401, a failed read 4503, and only a missing device row falls through to the invite
path. A phone whose host has no seat gets one assignment read: 4409 when the row
names another cell or the read fails, otherwise 4404.
* feat(relay): keep a memory resume's basis in the cell and confirm it from one read
A phone resumed on the memory path keeps its connection basis in the cell, not in
relay_connection_bases. Its confirm rereads the device row once, renews it only when
less than 29 days remain, and always answers a current-version confirm renewed:true
with the expiry the database holds. Confirms route by where their basis lives, so a
flip between accept and confirm completes either way. Retries answer from a 10-min
in-memory result map.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay): log a failed no-seat read, recheck the switch after the device read
A failed assignment read for a phone with no seat now logs a rate-limited line before
its 4409. A switch that turned off during the device read sends the phone down today's
path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy.
* fix(relay): log a failed memory renewal; test flips, replays and basis cleanup
A renewal write that fails still answers the stored expiry, but now logs a rate-limited
line so a persistently failing update is visible before phones age toward re-pairing.
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay): resume paired phones from one device read on memory cells
With phonePath=memory in reserve mode, a phone resume is admitted from one device
read and the cell's own seat, and takes no activity leases. A refused device closes
4401, a failed read 4503, and only a missing device row falls through to the invite
path. A phone whose host has no seat gets one assignment read: 4409 when the row
names another cell or the read fails, otherwise 4404.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay): log a failed no-seat read, recheck the switch after the device read
A failed assignment read for a phone with no seat now logs a rate-limited line before
its 4409. A switch that turned off during the device read sends the phone down today's
path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy.
* test(relay): census every activity-lease reader for phones without leases
Step 6 lets a reserve cell accept phones without writing confirmation or
splice leases. This ratchet classifies every reader of the lease table by why
that is safe, and pins the guards that keep completion and idle rehome off a
reserve cell whose host holds no source lease.
* test(relay): scan subdirectories and USING reads, and ban a bare lease table name
The ratchet now walks every production file under src, counts USING as a read, and
fails on a bare table-name value, which could reach an interpolated FROM it cannot see.
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
A reserve cell's cell-status read is refused, so it carried no
connection_hard_cap and the evaluator fell back to the 500 default:
run 39225932592 froze on reserve c7 at 731 connections under its
1000 cap. Read connection_hard_cap from the durable Terraform cells
relay-ops already parses and emit it for reserve cells.
The director's cell-status refuses a step-5 reserve cell with a 404
{error: cell_in_reserve_mode}, so every monitor sample failed while any
cell was reserve. Classify that exact refusal as reserve-not-probed, skip
the refused migration reads, and judge the cell on probes, logs and the
selector instead. Any other 404 still fails the sample.
A reserve cell keeps no activity rows and its ledger stamps last_activity_at, so the
stranded rule read its connected hosts as unserved and re-placed them.
* fix(relay): log controls a cell closes after host proof but before wiring
* fix(relay): log a pre-refusal hang-up as the hang-up and share printableCloseReason
* refactor(relay): move printableCloseReason into its own module to break the import cycle
* test(relay): assemble the fake pg URL at runtime so the secret scan passes
The folder is the cloud relay that reaches phones; under its old name it read as the legacy SSH
relay. git mv plus import and path-literal rewrites only.
* fix(relay): destroy a pooled client whose session the server ended, and mark work that never ran
pg keeps calling a client queryable until its socket's end arrives, so a client released after
57P01 (pg_terminate_backend, shutdown) or a dropped socket went back to the pool and failed the
next caller's statement with 'Connection terminated unexpectedly'. Such clients are now released
with the error, so pg-pool discards them.
Also marks failures where no statement of the operation reached the server (acquire, a failed
BEGIN, a client dead at checkout) so a caller can tell work safe to re-run.
* fix(relay): skip ROLLBACK on an ended session, drop unused nothing-ran marking
Adds 57P04/57P05 to the session-ended codes and covers 08xxx, a dropped
socket, an ordinary 23505, and a real pg_terminate_backend.
Claude-Session: f2fe8cb5
OrcaWinandOrcaWin
Jinwoo Hong
Neil
