<!-- orca-pr-loc -->
<!-- Programmatic LoC summary. Do not edit by hand; rewritten on every commit. -->
| | Files | Added | Deleted | Net |
| :--- | ---: | ---: | ---: | ---: |
| Test | 1 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$279 | 0 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$279 |
| Prod | 1 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$58 | $\color{#cf222e}{\Huge{\mathbf{−}}}$13 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$45 |
<!-- /orca-pr-loc -->
## ELI5
When we restart a relay server during a routine rollout, computers that were offline during the rollout still have that server written down as "their" server. If one of them comes back during the ~45 seconds while the replacement server is starting up, the director currently tells it "your server is unavailable, try again" over and over, instead of just sending it to another server. This change lets the director send it elsewhere, but only when it has proof that the old server process is really gone.
## What Changed
**Before (production, 2026-10-24 23:43Z, cell production-gce-c15):** during a same-capacity roll the cell is roll-isolated, drained, then its single VM is replaced. A desktop host that was offline during the drain reconnected in the gap between the old process stopping and the new one being ready. The director refused it for ~26 s (`assignment rejected route=assign lane=placement hinted=true reason=relay_home_cell_unavailable cause=not_ready`, x5, plus `route=resolve ... cause=unheard`, x2). It got a grant somewhere else only once the new process was live.
**After:** while the replacement process is starting up (`not_ready`), the same host is moved to another cell on its first try. A host on a roll-isolated cell can be moved off even though the cell is not live, as long as the process heartbeating for the cell *started after the roll isolated it* (`relay_cell_runtime.started_at > relay_cell_admission.roll_isolated_at`), its heartbeat is fresh, and (for connection-capped cells) its latest heartbeat reports zero connections. Both lanes that move an isolated host apply it: the narrowed isolated-replacement path (`isolatedIncumbentMayMove`) and the all-rows path it falls back to. Together these serve assign and resolve, because resolve hands a non-live cell to `assign`. "Roll-isolated" is the existing predicate: `migration-only` with a roll stamp younger than 2 h.
**Still refused (residual):** the `unheard` window, when the old process's heartbeat is past the 45 s TTL and the new process has not been heard yet. On c15 that was about 5-10 s. Nothing proves the drained sockets are gone there, so the refusal stays, and the client's retry recovers once the new process heartbeats. Also unchanged: an ordinary readiness dip, a `migration-only` cell without a roll stamp, a stale (over 2 h) stamp, and a new process that later goes unheard. Fence handling is untouched.
## Why
The refusal exists because moving a host off a cell we cannot fully trust could leave the host's old socket served by the old process while its clients go to the new cell. Until now, the only proof the code accepted was an operator-written fence. This adds a second proof that costs no operator step:
- **A new incarnation means the drained process is gone.** Each process mints a fresh incarnation UUID and `startedAt` at boot (`cell-heartbeat-client.ts`), and the director rejects a heartbeat with an older `startedAt` once a newer one is recorded. Each cell VM runs exactly one relay container (`docker run --name orca-relay --restart always`, with stale containers force-removed at boot). The cell MIG uses `replacement_method = RECREATE` with `max_surge = 0` (pinned by a Terraform assert), so the old VM is deleted before the new one is created. A same-VM container restart also produces a new incarnation and is just as safe: Docker restarts only after the old process exits, and that exit closes all of its sockets.
- **Whatever the new process holds, it can enforce.** It is heartbeating (fresh), and on capped cells it reports no connections at all. A host that attaches between two heartbeats carries the old epoch, so the epoch bump this move writes makes the cell refuse it or close it with 4409. That check is the same live `verifyCellAssignment` / control-renewal read the existing live-isolated path relies on.
- **Clock skew** between the cell's `startedAt` and the director's stamp is covered by the zero-connections check on capped cells only: a drained process that only *looks* post-stamp still holds its sockets, so it is refused. Uncapped cells report no counts and get no such guard; they already move off without a fence on the dead-cell path today, so this adds no new exposure there.
Alternatives considered: (a) adding an incarnation column to `relay_assignments`: a schema change for no extra proof. (b) Honouring any not-live roll-isolated cell: that drops the proof. (c) Having the roll write a fence for the `unheard` window: the only fence driver refuses enabled cells, a fence is a MIG resize to 0 that would fight the roll's own apply, and the window is a few seconds the client retry already absorbs.
## Linked Issue
N/A (internal production incident, 2026-10-24 c15 roll).
## Visual Proof
N/A: director placement logic only, no UI.
## Testing
- [x] Automated tests added: `cloud/apps/relay/src/assignment-roll-replaced-cell.test.ts` (18 cases; the redial cases run for both hinted assign and resolve):
- not_ready with an incarnation started after isolation: re-placed, epoch bumped.
- Refused as before: not_ready with the pre-isolation incarnation, with reported connections, after the new process goes unheard, with a stale stamp, on an ordinary unstamped readiness dip, and on an unstamped migration-only cell. Also refused: the unheard window between the old and new process.
- One case per lane. The narrowed path's cross-region move fails if only the narrowed-path change is reverted. The all-rows path's move, forced by units no lease backs, fails if only the all-rows change is reverted.
- [x] Full relay vitest suite, `--maxWorkers=2`: 130 files / 1344 tests passed, 39 files (201 tests) skipped. The Postgres-backed suites skip locally (no `ORCA_RELAY_TEST_POSTGRES_URL`); CI runs them.
- [x] `pnpm typecheck` in `cloud/apps/relay`.
## Review
## Agent skill upstream boundary
- [x] Not applicable, or this change copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.
## Notes
Backwards compatibility: no schema, wire or RPC change; old directors keep refusing as today. Performance: the new query runs only when a host's home cell is roll-isolated and not live; the general sticky hot path is unchanged.
## Checklist
- [x] This PR is small and focused
- [x] I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
- [x] Before/after screenshots or videos attached for UI changes, or `N/A` with reason
- [x] Self-reviewed for correctness, security, and performance
- [x] Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
- [x] `pnpm lint`, `pnpm typecheck`, `pnpm test`, and `pnpm build` pass (or CI will cover; local preferred)
The director now accepts a cell heartbeat whose reported connection cap is at or below its configured cap, and places with the lower of the two; a reported cap above the configured one is still refused. A cap-only change keeps the cell's stored capacity rows. This lets the US cells' configured cap be raised once, ahead of their rolls, so an old process at the lower cap stays live until it is replaced, instead of desktops pointed at it being refused during a resize.
Reserve-aware cell status for the director: reserve cells report their own runtime and seat feed, drain causes are attributed per cell, and paced 503s on a host leaving a draining cell are recorded as the drain's.
Rehome on reconnect for reserve-mode cells: a desktop reconnecting on a reserve cell outside its preferred region is placed fresh in that region, only while the region has at least 10% of seats free, at most once a day per desktop, rate-limited, and switched off by default (ORCA_RELAY_RESERVE_REHOME_PERCENT=0); the regional rehome switch stops it.
Before refusing a placement with relay_capacity_exhausted, the director tries a reserve seat in the desktop's own region, then the other region, through the cell's own reserve API; if every region paces it answers the shortest Retry-After.
* fix(relay): count region selection on every assign lane, fallback only for new placements
Co-Authored-By: Claude <noreply@anthropic.com>
* test(relay): pin placed marker on isolated re-place and kept pins; drain-return fallback
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* test: remove redundant cases and repair behavioral assertions
* docs: reconcile audit records with current main
* fix(deps): apply security updates from #32101
Apply the six-file DOMPurify 3.4.17 and Next.js 16.4.1 patch from PR #32101 to resolve the dependency security gate for this test audit. Original patch by Brennan Benson.
* test: satisfy changed-line quality gates
Add required braces and replace filter indexing with find. Rerun both CI quality gates and the 27 affected relay tests; refresh audit hashes and record the corrections.
---------
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
Co-authored-by: Brennan Benson <79079362+brennanb2025@users.noreply.github.com>
* fix(relay): repair cell seat-log drift, and keep a new session's generation above a listed seat
Directors mirror each cell's seat feed, so the feed must list exactly the cell's live sessions.
The cell now sweeps its seat log against its sessions on a director's poll: a seat no session
backs is left (4409) and the live session listed, and a live session the feed lost is listed
again, each once two sweeps in a row find it. A demotion naming an unbacked seat repairs it at
once. A host's new session now starts above any generation the feed still lists, which closes
the drained-session reconnect race that removed a live seat.
Claude-Session: asia-drain-pulse
* fix(relay): refuse a rebind whose session left the map during activation, and keep a per-host generation floor
A rebind whose database activation outlived the orphan grace (or a drain) revived the removed
session outside the map: connected, still in the seat feed, and unreachable by phones or
demotion for as long as the host stayed up. Activation now rechecks the map after its awaits and
refuses with PEER_DROPPED so the host redials. Generations come from a per-host floor that
outlives the session, and an activation at a generation already used here is refused.
Claude-Session: asia-drain-pulse
* fix(relay): log each demote outcome on the director by host digest
The director now reads the cell's existing {v:1, outcome} demote reply and logs it; the cell's
feed stays the only source of its map. The supersede line now reaches the log as well.
Claude-Session: asia-drain-pulse
* fix(ci): recover verified pnpm setup from registry metadata outages
* fix(ci): prepare native pnpm shims after fallback verification
* fix(ci): provide Node for pnpm fallback in bare containers
* chore(cloud): update relay runtimes and security scanning
* docs(cloud): record Go binary advisory reachability
* chore(cloud): rerun security checks after resolving main conflict
* fix(relay): release a drained host before telling it to leave
A zero-grace drain sent `drain` and closed with 4503 in the same tick, then
released the host's control lease. The desktop redials the director as soon
as it reads either, and from asia-east2 the release holds the host's
assignment row for ~1 s, so the redial met it and was refused row_busy.
The release now runs first and the host is told once it lands, bounded at
2 s; a failed or stuck release still closes. Wire frames are unchanged.
* fix(relay): give a drained host's release 3 s and log when it runs out
One host's releases run one after another, so 3 s covers a control lease
plus one open phone connection from asia-east2.
* style(relay): brace the drain release branches
* fix(relay): give region-correction reconnects and roll re-placements the in-region reserve fallback
Every desktop session open carries a region-correction request, and those
skipped the reserve plan entirely, so with the region's database cells full they
still spilled abroad while an in-region reserve cell had room. The plan now runs
for them in a mode that never answers itself (the database lane must reply so the
window exchange follows) and offers only the in-region reserve fallback.
A host re-placed off a roll-isolated cell went straight to the other-regions tier
when its own region's tier was full. With an in-region reserve fallback it now
books that first, and the roll canary event is emitted for the reserve booking.
* fix(relay): never pace a roll drain on reserve intake, and count every reserve move off an isolated cell
A host leaving a roll-isolated cell whose region was full could get a paced 503
from a throttled reserve cell, stretching the drain and re-running the failed
in-region transaction on each retry. For those hosts a pace is now no room, and
they spill as before. Sticky now says when it declined an isolated pin, so a
share-draw booking off that cell also emits the roll canary event.
* fix(relay): keep a host in its region when the database lane is full and a reserve cell has room
A host that lost the reserve share draw went to the database lane, which
spills to another region when its own region's database cells are full,
even while an in-region reserve cell had free seats. The database lane now
requires the region when an in-region reserve cell has room; if the region
is full it books that reserve cell, and only spills when neither has room.
* fix(relay): pace a throttled in-region reserve cell instead of spilling, and decide reserve room at plan time
With the region's database cells full, a reserve cell with seats but no intake
budget fell through the placer's database-cells shortcut and the host went
abroad. That placement now answers pace (503 with Retry-After). regionReserve
is a single callback, present only while an in-region reserve cell has a seat.
* fix(relay): pace only on a reserve cell that lacked budget, not one that refused
A cell that refused this placement (full, unreachable, seated-newer) got its
token back and drove a 1 s Retry-After forever while the map still showed a
seat. Pacing now skips cells refused for anything but intake, so those
placements fall to the database lane's cross-region fallback as before.
* perf(push): wake delivery worker after notifications are queued
* perf(push): wake one drain per send so bursts cost O(sends + drains) claims
A wake started all twelve drains, so each send arriving at an idle worker cost
~13 claim transactions. A wake now starts one drain; the poll tops up to the
full drain set.
A host control request that failed for an unexpected reason sent the thrown
error's message to the desktop as the error code. During a staging database
outage the desktop received Postgres's own text, and node-pg messages can
carry the database connection URL.
The relay now sends store codes and this codebase's snake_case invariant
slugs unchanged, and maps everything else to database_transient or
control_operation_failed. The original is logged through the existing
redaction (now shared with the background-operation logger), so only the
slug shape and SQLSTATE leave the process.
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay): resume paired phones from one device read on memory cells
With phonePath=memory in reserve mode, a phone resume is admitted from one device
read and the cell's own seat, and takes no activity leases. A refused device closes
4401, a failed read 4503, and only a missing device row falls through to the invite
path. A phone whose host has no seat gets one assignment read: 4409 when the row
names another cell or the read fails, otherwise 4404.
* feat(relay): keep a memory resume's basis in the cell and confirm it from one read
A phone resumed on the memory path keeps its connection basis in the cell, not in
relay_connection_bases. Its confirm rereads the device row once, renews it only when
less than 29 days remain, and always answers a current-version confirm renewed:true
with the expiry the database holds. Confirms route by where their basis lives, so a
flip between accept and confirm completes either way. Retries answer from a 10-min
in-memory result map.
* test(relay): roll a memory phone cell back with confirms in flight
Flipping back by admitMode=db, phonePath=db or a dead-man trip leaves in-flight
confirms to finish on their memory basis, writes no phone lease and leaks none, and
evacuation still refuses the cell while Postgres records it as reserve.
* feat(relay): batch a memory cell's resume-confirmed audit rows
Memory confirms buffer their audit rows and the cell writes them as one multi-row
insert every 5 s or 500 rows, idempotent by a per-event id. The buffer holds at most
5,000 rows and drops a row older than 10 minutes rather than retrying it, counting
and logging both kinds of drop.
* feat(relay): count and log phone-path failures, and every attach deadline
The runtime metrics line gains phone refusals by close code, attach deadlines, accept
database errors, resume confirm failures by code, completed accepts by phone path and
the resume audit drop total. Every attach-deadline 4404 now logs what the cell knew of
the desktop's control (pong age, ping outstanding, control age and round trip), so a
half-open control can be told from a slow data dial.
* feat(relay): close an attach-deadline 4404 with the attach-timeout reason
The phone close reason from #30607 now rides the close frame of the 4404 a cell
sends when the desktop's data leg misses the attach deadline, so OTA phones can
redial the same cell once instead of waiting out the host-offline delay. The failed
relay-hello frame is unchanged, and old phones ignore the reason.
* test(relay): drill a memory cell through a database outage, and pin phone renewal
A blocked database leaves splices forwarding, closes new resumes 4503 and fails
confirms cleanly, lets an in-flight confirm finish on its memory basis after a flip
back, and drops audit rows past 10 minutes instead of writing them late. On the phone,
a relay-only session lives 31 days under the once-a-day renewal rule, and would not
if a not-due confirm answered renewed:false.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay): log a failed no-seat read, recheck the switch after the device read
A failed assignment read for a phone with no seat now logs a rate-limited line before
its 4409. A switch that turned off during the device read sends the phone down today's
path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy.
* fix(relay): log a failed memory renewal; test flips, replays and basis cleanup
A renewal write that fails still answers the stored expiry, but now logs a rate-limited
line so a persistently failing update is visible before phones age toward re-pairing.
* test(relay): assert a rolled-back memory splice issues no statement when it closes
* fix(relay): flush resume audit early only on crossing the batch size, drop the shutdown flush
A failing database no longer takes a 500-row insert per confirm; retries wait for the
5 s timer. The shutdown flush raced the pool closing, so it is gone and the at most 5 s
loss is documented. A stopped buffer no longer re-arms its timer.
* test(relay): assert the aged audit rows were dropped, not written
---------
Co-authored-by: E2E Test <e2e@test.local>
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay): resume paired phones from one device read on memory cells
With phonePath=memory in reserve mode, a phone resume is admitted from one device
read and the cell's own seat, and takes no activity leases. A refused device closes
4401, a failed read 4503, and only a missing device row falls through to the invite
path. A phone whose host has no seat gets one assignment read: 4409 when the row
names another cell or the read fails, otherwise 4404.
* feat(relay): keep a memory resume's basis in the cell and confirm it from one read
A phone resumed on the memory path keeps its connection basis in the cell, not in
relay_connection_bases. Its confirm rereads the device row once, renews it only when
less than 29 days remain, and always answers a current-version confirm renewed:true
with the expiry the database holds. Confirms route by where their basis lives, so a
flip between accept and confirm completes either way. Retries answer from a 10-min
in-memory result map.
* test(relay): roll a memory phone cell back with confirms in flight
Flipping back by admitMode=db, phonePath=db or a dead-man trip leaves in-flight
confirms to finish on their memory basis, writes no phone lease and leaks none, and
evacuation still refuses the cell while Postgres records it as reserve.
* feat(relay): batch a memory cell's resume-confirmed audit rows
Memory confirms buffer their audit rows and the cell writes them as one multi-row
insert every 5 s or 500 rows, idempotent by a per-event id. The buffer holds at most
5,000 rows and drops a row older than 10 minutes rather than retrying it, counting
and logging both kinds of drop.
* feat(relay): count and log phone-path failures, and every attach deadline
The runtime metrics line gains phone refusals by close code, attach deadlines, accept
database errors, resume confirm failures by code, completed accepts by phone path and
the resume audit drop total. Every attach-deadline 4404 now logs what the cell knew of
the desktop's control (pong age, ping outstanding, control age and round trip), so a
half-open control can be told from a slow data dial.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay): log a failed no-seat read, recheck the switch after the device read
A failed assignment read for a phone with no seat now logs a rate-limited line before
its 4409. A switch that turned off during the device read sends the phone down today's
path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy.
* fix(relay): log a failed memory renewal; test flips, replays and basis cleanup
A renewal write that fails still answers the stored expiry, but now logs a rate-limited
line so a persistently failing update is visible before phones age toward re-pairing.
* test(relay): assert a rolled-back memory splice issues no statement when it closes
* fix(relay): flush resume audit early only on crossing the batch size, drop the shutdown flush
A failing database no longer takes a 500-row insert per confirm; retries wait for the
5 s timer. The shutdown flush raced the pool closing, so it is gone and the at most 5 s
loss is documented. A stopped buffer no longer re-arms its timer.
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay): resume paired phones from one device read on memory cells
With phonePath=memory in reserve mode, a phone resume is admitted from one device
read and the cell's own seat, and takes no activity leases. A refused device closes
4401, a failed read 4503, and only a missing device row falls through to the invite
path. A phone whose host has no seat gets one assignment read: 4409 when the row
names another cell or the read fails, otherwise 4404.
* feat(relay): keep a memory resume's basis in the cell and confirm it from one read
A phone resumed on the memory path keeps its connection basis in the cell, not in
relay_connection_bases. Its confirm rereads the device row once, renews it only when
less than 29 days remain, and always answers a current-version confirm renewed:true
with the expiry the database holds. Confirms route by where their basis lives, so a
flip between accept and confirm completes either way. Retries answer from a 10-min
in-memory result map.
* test(relay): roll a memory phone cell back with confirms in flight
Flipping back by admitMode=db, phonePath=db or a dead-man trip leaves in-flight
confirms to finish on their memory basis, writes no phone lease and leaks none, and
evacuation still refuses the cell while Postgres records it as reserve.
* feat(relay): batch a memory cell's resume-confirmed audit rows
Memory confirms buffer their audit rows and the cell writes them as one multi-row
insert every 5 s or 500 rows, idempotent by a per-event id. The buffer holds at most
5,000 rows and drops a row older than 10 minutes rather than retrying it, counting
and logging both kinds of drop.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay): log a failed no-seat read, recheck the switch after the device read
A failed assignment read for a phone with no seat now logs a rate-limited line before
its 4409. A switch that turned off during the device read sends the phone down today's
path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy.
* fix(relay): log a failed memory renewal; test flips, replays and basis cleanup
A renewal write that fails still answers the stored expiry, but now logs a rate-limited
line so a persistently failing update is visible before phones age toward re-pairing.
* test(relay): assert a rolled-back memory splice issues no statement when it closes
* fix(relay): flush resume audit early only on crossing the batch size, drop the shutdown flush
A failing database no longer takes a 500-row insert per confirm; retries wait for the
5 s timer. The shutdown flush raced the pool closing, so it is gone and the at most 5 s
loss is documented. A stopped buffer no longer re-arms its timer.
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay): resume paired phones from one device read on memory cells
With phonePath=memory in reserve mode, a phone resume is admitted from one device
read and the cell's own seat, and takes no activity leases. A refused device closes
4401, a failed read 4503, and only a missing device row falls through to the invite
path. A phone whose host has no seat gets one assignment read: 4409 when the row
names another cell or the read fails, otherwise 4404.
* feat(relay): keep a memory resume's basis in the cell and confirm it from one read
A phone resumed on the memory path keeps its connection basis in the cell, not in
relay_connection_bases. Its confirm rereads the device row once, renews it only when
less than 29 days remain, and always answers a current-version confirm renewed:true
with the expiry the database holds. Confirms route by where their basis lives, so a
flip between accept and confirm completes either way. Retries answer from a 10-min
in-memory result map.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay): log a failed no-seat read, recheck the switch after the device read
A failed assignment read for a phone with no seat now logs a rate-limited line before
its 4409. A switch that turned off during the device read sends the phone down today's
path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy.
* fix(relay): log a failed memory renewal; test flips, replays and basis cleanup
A renewal write that fails still answers the stored expiry, but now logs a rate-limited
line so a persistently failing update is visible before phones age toward re-pairing.
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay): resume paired phones from one device read on memory cells
With phonePath=memory in reserve mode, a phone resume is admitted from one device
read and the cell's own seat, and takes no activity leases. A refused device closes
4401, a failed read 4503, and only a missing device row falls through to the invite
path. A phone whose host has no seat gets one assignment read: 4409 when the row
names another cell or the read fails, otherwise 4404.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay): log a failed no-seat read, recheck the switch after the device read
A failed assignment read for a phone with no seat now logs a rate-limited line before
its 4409. A switch that turned off during the device read sends the phone down today's
path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy.
* test(relay): census every activity-lease reader for phones without leases
Step 6 lets a reserve cell accept phones without writing confirmation or
splice leases. This ratchet classifies every reader of the lease table by why
that is safe, and pins the guards that keep completion and idle rehome off a
reserve cell whose host holds no source lease.
* test(relay): scan subdirectories and USING reads, and ban a bare lease table name
The ratchet now walks every production file under src, counts USING as a read, and
fails on a bare table-name value, which could reach an interpolated FROM it cannot see.
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
A reserve cell's cell-status read is refused, so it carried no
connection_hard_cap and the evaluator fell back to the 500 default:
run 39225932592 froze on reserve c7 at 731 connections under its
1000 cap. Read connection_hard_cap from the durable Terraform cells
relay-ops already parses and emit it for reserve cells.
The director's cell-status refuses a step-5 reserve cell with a 404
{error: cell_in_reserve_mode}, so every monitor sample failed while any
cell was reserve. Classify that exact refusal as reserve-not-probed, skip
the refused migration reads, and judge the cell on probes, logs and the
selector instead. Any other 404 still fails the sample.
A reserve cell keeps no activity rows and its ledger stamps last_activity_at, so the
stranded rule read its connected hosts as unserved and re-placed them.
* fix(relay): log controls a cell closes after host proof but before wiring
* fix(relay): log a pre-refusal hang-up as the hang-up and share printableCloseReason
* refactor(relay): move printableCloseReason into its own module to break the import cycle
* test(relay): assemble the fake pg URL at runtime so the secret scan passes
The folder is the cloud relay that reaches phones; under its old name it read as the legacy SSH
relay. git mv plus import and path-literal rewrites only.
* fix(relay): destroy a pooled client whose session the server ended, and mark work that never ran
pg keeps calling a client queryable until its socket's end arrives, so a client released after
57P01 (pg_terminate_backend, shutdown) or a dropped socket went back to the pool and failed the
next caller's statement with 'Connection terminated unexpectedly'. Such clients are now released
with the error, so pg-pool discards them.
Also marks failures where no statement of the operation reached the server (acquire, a failed
BEGIN, a client dead at checkout) so a caller can tell work safe to re-run.
* fix(relay): skip ROLLBACK on an ended session, drop unused nothing-ran marking
Adds 57P04/57P05 to the session-ended codes and covers 08xxx, a dropped
socket, an ordinary 23505, and a real pg_terminate_backend.
Claude-Session: f2fe8cb5
* feat(relay): log what the sticky slot spends its hold on, per pooled connection
* fix(relay): time every sticky round trip at the PostgreSQL client, pool wait apart
* docs(relay): say which round trips the sticky slot tally covers
* Avoid unused module imports and trim cosmetic test coverage
* Capture hidden popup frames and attach CDP before guest creation
* Overlap relay test projects within the existing worker budget
* Consolidate native preparation and add an opt-in transform cache comparison
* Release inherited debugger sockets during final E2E teardown
* Refresh unit scheduling weights from all five qualified ARM shards
* Avoid retaining unasserted RPC call history in the delivery model
* Use the supported unconditional background wait syntax
* Overlap unconditional output upload with native preparation
* Verify relay integration stays enabled for ordinary unit callers
* feat(relay): split step-5 shadow compare and dry-run logs per cell
The 24 h per-cell shadow gate needs per-cell sticky-compare counts, per-cell
unexplained counts, and dry-run `full` refusals while the map showed room.
Both summary lines gain additive per-cell fields; existing fields are unchanged.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count a cell-mismatch on both cells and bound per-cell maps by the fleet
- countByCell (was cells) counts a compare on every cell it involves: the
database's cell, every seat's cell when there is no row, and both cells
for a cell-mismatch. Drop the 64-cell cap and cellsOverflow.
- Add ReservePlacer.free and use it for the dry-run free share.
- Rename the room bound to RESERVE_DRY_RUN_ROOM_BOUND.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count a duplicate-seat compare on every seat cell
An unexplained duplicate seat counted only on the database's cell, so a cell
whose stale feed kept a second seat could pass its own step-5 check. Count it
once on every seat cell, the same as a cell-mismatch.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count explained cell-mismatch and duplicate-seat on the database cell only
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): key per-cell counts on prototype-free records
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): explain a duplicate seat left by a just-superseded half-open control
When the database names the newest seat exactly, every other seat has a
strictly lower epoch, and the newest seat joined within the cell's silence
close (timeout + one ping interval + the map lag bound), the older seat is
a half-open control awaiting its 75 s pong-silence close, not a fault.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): bound the superseded duplicate-seat rule on the move's age
A same-epoch rejoin resets joinedAt, so a stale older seat could stay
explained forever; the director-clock sighting of the epoch moves only
when the epoch rises.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): leave a superseded duplicate seat unexplained without an epoch sighting
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): explain a superseded duplicate seat only from a prior sighting
compare() recorded the epoch before classifying, so a first-ever sighting
(restart or eviction) read as a fresh move. Classify first and take the
move's age from the record that existed before this compare.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): also bound a superseded duplicate seat on its join age
A long-ago lower-epoch sighting dates the move to this compare, so the
current seat's join age bounds it too.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count an ageless superseded duplicate seat as its own class
A superseded-shaped duplicate-seat with no prior sighting of the host (the
first compare since a restart) has no move age, so it is now
`superseded-unknown-age` (explained: false) rather than a plain duplicate.
The count line also carries `uptimeMs`, so the gate can discount that
class in an instance's first 100 s.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): limit superseded-unknown-age to an instance's first 100 s
Flushes are lazy, so a window has no upper bound and an uptime field on the
count line cannot date it. The compare now records when it started and
emits superseded-unknown-age only within the superseded close bound of
that; later, a missing sighting means another instance served the host or
the sighting was evicted, so it is a plain unexplained duplicate-seat.
Drops the uptimeMs field.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): say where a missing sighting comes from after the first 100 s
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* perf(ci): reduce test overhead and reuse expensive build work
* test(ci): keep worker startup outside failure-case deadlines
* test: isolate push fixture advisory locks
* test(ci): preserve fixed drain retry allowance
* Align CI contracts with current triggers and guide content
* Stabilize consolidated reveal scenarios and retain behavioral CI contracts
* Fix upstream lint blockers while retaining behavior and assertions
* Classify global Jira preferences in the WSL migration inventory
* Align Jira test fixtures with startup and build safety checks
* Include package-test routing dependency in CI checkouts and fixtures
* Keep native stream classes intact in transcript dependency tests
* test(relay): check invariant 4 (one demotion winner) in the step-5 simulation
Every demotion now names its winner (the newer booking, or the row). The model
flags a director that names a different winner for a divergence another already
named, and a host whose two seats are each demoted for the other. A fault where
directors break level ties by their own cell order shows the check fires.
Violations are capped per invariant so one flood cannot hide another class.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* test(relay): bound unseen duplicates by the silence close and assert all invariants on contested ties
A reserve seat the booking director's map had not seen (joined, then half-open within a poll)
is never superseded and has no lease to re-check, so it lasts until the cell's control silence
close (75 s + one 15 s ping). Invariant 3 now uses that bound for such seats, and the
level-divergence config asserts every invariant.
Stale maps now judge a divergence another director already demoted, so the config's
"contested" counter (divergences judged by two or more directors) is the case invariant 4
guards. The silence-close constant now matches production (90 s, not 100 s).
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* test(relay): raise the floor on map-judged demotes and count only level contests
A reconcile whose map has not seen another director's demotion still demotes in
production, which raises its epoch floor; the model now does too. The contested
counter counts only same-epoch pairs, the only ones invariant 4 can split. The
keep-this-seat rule is one helper.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* refactor(relay): name the simulation's fault-kept seat check for the fault
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): supersede a reserve seat from the feed once a later, newer join shows
A half-open seat the booking director's map had not yet seen was never demoted and held its
cell until the 75 s control silence close. Directors now demote it once both joins are in the
map; the simulation holds invariant 3 at the supersede bound for it.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): outrank only past cell clock skew, reuse demoteSeat, keep invariant 3's loose clock running
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* refactor(relay): share the outrank rule with the simulation, pin the skew boundary
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count only confirmed level demotions and every second judgment, and bound duplicates by placement records alone
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): a drained seat keeps its drain grace under a demote, and the feed reports each seat once
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* refactor(relay): name the outranked-report bound and the reconcile backstop
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
Jinwoo Hong
OrcaWinandOrcaWin
Neil
