* fix(ci): recover verified pnpm setup from registry metadata outages
* fix(ci): prepare native pnpm shims after fallback verification
* fix(ci): provide Node for pnpm fallback in bare containers
* chore(cloud): update relay runtimes and security scanning
* docs(cloud): record Go binary advisory reachability
* chore(cloud): rerun security checks after resolving main conflict
* fix(relay): release a drained host before telling it to leave
A zero-grace drain sent `drain` and closed with 4503 in the same tick, then
released the host's control lease. The desktop redials the director as soon
as it reads either, and from asia-east2 the release holds the host's
assignment row for ~1 s, so the redial met it and was refused row_busy.
The release now runs first and the host is told once it lands, bounded at
2 s; a failed or stuck release still closes. Wire frames are unchanged.
* fix(relay): give a drained host's release 3 s and log when it runs out
One host's releases run one after another, so 3 s covers a control lease
plus one open phone connection from asia-east2.
* style(relay): brace the drain release branches
* fix(relay): give region-correction reconnects and roll re-placements the in-region reserve fallback
Every desktop session open carries a region-correction request, and those
skipped the reserve plan entirely, so with the region's database cells full they
still spilled abroad while an in-region reserve cell had room. The plan now runs
for them in a mode that never answers itself (the database lane must reply so the
window exchange follows) and offers only the in-region reserve fallback.
A host re-placed off a roll-isolated cell went straight to the other-regions tier
when its own region's tier was full. With an in-region reserve fallback it now
books that first, and the roll canary event is emitted for the reserve booking.
* fix(relay): never pace a roll drain on reserve intake, and count every reserve move off an isolated cell
A host leaving a roll-isolated cell whose region was full could get a paced 503
from a throttled reserve cell, stretching the drain and re-running the failed
in-region transaction on each retry. For those hosts a pace is now no room, and
they spill as before. Sticky now says when it declined an isolated pin, so a
share-draw booking off that cell also emits the roll canary event.
* fix(relay): keep a host in its region when the database lane is full and a reserve cell has room
A host that lost the reserve share draw went to the database lane, which
spills to another region when its own region's database cells are full,
even while an in-region reserve cell had free seats. The database lane now
requires the region when an in-region reserve cell has room; if the region
is full it books that reserve cell, and only spills when neither has room.
* fix(relay): pace a throttled in-region reserve cell instead of spilling, and decide reserve room at plan time
With the region's database cells full, a reserve cell with seats but no intake
budget fell through the placer's database-cells shortcut and the host went
abroad. That placement now answers pace (503 with Retry-After). regionReserve
is a single callback, present only while an in-region reserve cell has a seat.
* fix(relay): pace only on a reserve cell that lacked budget, not one that refused
A cell that refused this placement (full, unreachable, seated-newer) got its
token back and drove a 1 s Retry-After forever while the map still showed a
seat. Pacing now skips cells refused for anything but intake, so those
placements fall to the database lane's cross-region fallback as before.
* perf(push): wake delivery worker after notifications are queued
* perf(push): wake one drain per send so bursts cost O(sends + drains) claims
A wake started all twelve drains, so each send arriving at an idle worker cost
~13 claim transactions. A wake now starts one drain; the poll tops up to the
full drain set.
A host control request that failed for an unexpected reason sent the thrown
error's message to the desktop as the error code. During a staging database
outage the desktop received Postgres's own text, and node-pg messages can
carry the database connection URL.
The relay now sends store codes and this codebase's snake_case invariant
slugs unchanged, and maps everything else to database_transient or
control_operation_failed. The original is logged through the existing
redaction (now shared with the background-operation logger), so only the
slug shape and SQLSTATE leave the process.
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay): resume paired phones from one device read on memory cells
With phonePath=memory in reserve mode, a phone resume is admitted from one device
read and the cell's own seat, and takes no activity leases. A refused device closes
4401, a failed read 4503, and only a missing device row falls through to the invite
path. A phone whose host has no seat gets one assignment read: 4409 when the row
names another cell or the read fails, otherwise 4404.
* feat(relay): keep a memory resume's basis in the cell and confirm it from one read
A phone resumed on the memory path keeps its connection basis in the cell, not in
relay_connection_bases. Its confirm rereads the device row once, renews it only when
less than 29 days remain, and always answers a current-version confirm renewed:true
with the expiry the database holds. Confirms route by where their basis lives, so a
flip between accept and confirm completes either way. Retries answer from a 10-min
in-memory result map.
* test(relay): roll a memory phone cell back with confirms in flight
Flipping back by admitMode=db, phonePath=db or a dead-man trip leaves in-flight
confirms to finish on their memory basis, writes no phone lease and leaks none, and
evacuation still refuses the cell while Postgres records it as reserve.
* feat(relay): batch a memory cell's resume-confirmed audit rows
Memory confirms buffer their audit rows and the cell writes them as one multi-row
insert every 5 s or 500 rows, idempotent by a per-event id. The buffer holds at most
5,000 rows and drops a row older than 10 minutes rather than retrying it, counting
and logging both kinds of drop.
* feat(relay): count and log phone-path failures, and every attach deadline
The runtime metrics line gains phone refusals by close code, attach deadlines, accept
database errors, resume confirm failures by code, completed accepts by phone path and
the resume audit drop total. Every attach-deadline 4404 now logs what the cell knew of
the desktop's control (pong age, ping outstanding, control age and round trip), so a
half-open control can be told from a slow data dial.
* feat(relay): close an attach-deadline 4404 with the attach-timeout reason
The phone close reason from #30607 now rides the close frame of the 4404 a cell
sends when the desktop's data leg misses the attach deadline, so OTA phones can
redial the same cell once instead of waiting out the host-offline delay. The failed
relay-hello frame is unchanged, and old phones ignore the reason.
* test(relay): drill a memory cell through a database outage, and pin phone renewal
A blocked database leaves splices forwarding, closes new resumes 4503 and fails
confirms cleanly, lets an in-flight confirm finish on its memory basis after a flip
back, and drops audit rows past 10 minutes instead of writing them late. On the phone,
a relay-only session lives 31 days under the once-a-day renewal rule, and would not
if a not-due confirm answered renewed:false.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay): log a failed no-seat read, recheck the switch after the device read
A failed assignment read for a phone with no seat now logs a rate-limited line before
its 4409. A switch that turned off during the device read sends the phone down today's
path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy.
* fix(relay): log a failed memory renewal; test flips, replays and basis cleanup
A renewal write that fails still answers the stored expiry, but now logs a rate-limited
line so a persistently failing update is visible before phones age toward re-pairing.
* test(relay): assert a rolled-back memory splice issues no statement when it closes
* fix(relay): flush resume audit early only on crossing the batch size, drop the shutdown flush
A failing database no longer takes a 500-row insert per confirm; retries wait for the
5 s timer. The shutdown flush raced the pool closing, so it is gone and the at most 5 s
loss is documented. A stopped buffer no longer re-arms its timer.
* test(relay): assert the aged audit rows were dropped, not written
---------
Co-authored-by: E2E Test <e2e@test.local>
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay): resume paired phones from one device read on memory cells
With phonePath=memory in reserve mode, a phone resume is admitted from one device
read and the cell's own seat, and takes no activity leases. A refused device closes
4401, a failed read 4503, and only a missing device row falls through to the invite
path. A phone whose host has no seat gets one assignment read: 4409 when the row
names another cell or the read fails, otherwise 4404.
* feat(relay): keep a memory resume's basis in the cell and confirm it from one read
A phone resumed on the memory path keeps its connection basis in the cell, not in
relay_connection_bases. Its confirm rereads the device row once, renews it only when
less than 29 days remain, and always answers a current-version confirm renewed:true
with the expiry the database holds. Confirms route by where their basis lives, so a
flip between accept and confirm completes either way. Retries answer from a 10-min
in-memory result map.
* test(relay): roll a memory phone cell back with confirms in flight
Flipping back by admitMode=db, phonePath=db or a dead-man trip leaves in-flight
confirms to finish on their memory basis, writes no phone lease and leaks none, and
evacuation still refuses the cell while Postgres records it as reserve.
* feat(relay): batch a memory cell's resume-confirmed audit rows
Memory confirms buffer their audit rows and the cell writes them as one multi-row
insert every 5 s or 500 rows, idempotent by a per-event id. The buffer holds at most
5,000 rows and drops a row older than 10 minutes rather than retrying it, counting
and logging both kinds of drop.
* feat(relay): count and log phone-path failures, and every attach deadline
The runtime metrics line gains phone refusals by close code, attach deadlines, accept
database errors, resume confirm failures by code, completed accepts by phone path and
the resume audit drop total. Every attach-deadline 4404 now logs what the cell knew of
the desktop's control (pong age, ping outstanding, control age and round trip), so a
half-open control can be told from a slow data dial.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay): log a failed no-seat read, recheck the switch after the device read
A failed assignment read for a phone with no seat now logs a rate-limited line before
its 4409. A switch that turned off during the device read sends the phone down today's
path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy.
* fix(relay): log a failed memory renewal; test flips, replays and basis cleanup
A renewal write that fails still answers the stored expiry, but now logs a rate-limited
line so a persistently failing update is visible before phones age toward re-pairing.
* test(relay): assert a rolled-back memory splice issues no statement when it closes
* fix(relay): flush resume audit early only on crossing the batch size, drop the shutdown flush
A failing database no longer takes a 500-row insert per confirm; retries wait for the
5 s timer. The shutdown flush raced the pool closing, so it is gone and the at most 5 s
loss is documented. A stopped buffer no longer re-arms its timer.
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay): resume paired phones from one device read on memory cells
With phonePath=memory in reserve mode, a phone resume is admitted from one device
read and the cell's own seat, and takes no activity leases. A refused device closes
4401, a failed read 4503, and only a missing device row falls through to the invite
path. A phone whose host has no seat gets one assignment read: 4409 when the row
names another cell or the read fails, otherwise 4404.
* feat(relay): keep a memory resume's basis in the cell and confirm it from one read
A phone resumed on the memory path keeps its connection basis in the cell, not in
relay_connection_bases. Its confirm rereads the device row once, renews it only when
less than 29 days remain, and always answers a current-version confirm renewed:true
with the expiry the database holds. Confirms route by where their basis lives, so a
flip between accept and confirm completes either way. Retries answer from a 10-min
in-memory result map.
* test(relay): roll a memory phone cell back with confirms in flight
Flipping back by admitMode=db, phonePath=db or a dead-man trip leaves in-flight
confirms to finish on their memory basis, writes no phone lease and leaks none, and
evacuation still refuses the cell while Postgres records it as reserve.
* feat(relay): batch a memory cell's resume-confirmed audit rows
Memory confirms buffer their audit rows and the cell writes them as one multi-row
insert every 5 s or 500 rows, idempotent by a per-event id. The buffer holds at most
5,000 rows and drops a row older than 10 minutes rather than retrying it, counting
and logging both kinds of drop.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay): log a failed no-seat read, recheck the switch after the device read
A failed assignment read for a phone with no seat now logs a rate-limited line before
its 4409. A switch that turned off during the device read sends the phone down today's
path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy.
* fix(relay): log a failed memory renewal; test flips, replays and basis cleanup
A renewal write that fails still answers the stored expiry, but now logs a rate-limited
line so a persistently failing update is visible before phones age toward re-pairing.
* test(relay): assert a rolled-back memory splice issues no statement when it closes
* fix(relay): flush resume audit early only on crossing the batch size, drop the shutdown flush
A failing database no longer takes a 500-row insert per confirm; retries wait for the
5 s timer. The shutdown flush raced the pool closing, so it is gone and the at most 5 s
loss is documented. A stopped buffer no longer re-arms its timer.
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay): resume paired phones from one device read on memory cells
With phonePath=memory in reserve mode, a phone resume is admitted from one device
read and the cell's own seat, and takes no activity leases. A refused device closes
4401, a failed read 4503, and only a missing device row falls through to the invite
path. A phone whose host has no seat gets one assignment read: 4409 when the row
names another cell or the read fails, otherwise 4404.
* feat(relay): keep a memory resume's basis in the cell and confirm it from one read
A phone resumed on the memory path keeps its connection basis in the cell, not in
relay_connection_bases. Its confirm rereads the device row once, renews it only when
less than 29 days remain, and always answers a current-version confirm renewed:true
with the expiry the database holds. Confirms route by where their basis lives, so a
flip between accept and confirm completes either way. Retries answer from a 10-min
in-memory result map.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay): log a failed no-seat read, recheck the switch after the device read
A failed assignment read for a phone with no seat now logs a rate-limited line before
its 4409. A switch that turned off during the device read sends the phone down today's
path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy.
* fix(relay): log a failed memory renewal; test flips, replays and basis cleanup
A renewal write that fails still answers the stored expiry, but now logs a rate-limited
line so a persistently failing update is visible before phones age toward re-pairing.
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay): resume paired phones from one device read on memory cells
With phonePath=memory in reserve mode, a phone resume is admitted from one device
read and the cell's own seat, and takes no activity leases. A refused device closes
4401, a failed read 4503, and only a missing device row falls through to the invite
path. A phone whose host has no seat gets one assignment read: 4409 when the row
names another cell or the read fails, otherwise 4404.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay): log a failed no-seat read, recheck the switch after the device read
A failed assignment read for a phone with no seat now logs a rate-limited line before
its 4409. A switch that turned off during the device read sends the phone down today's
path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy.
* test(relay): census every activity-lease reader for phones without leases
Step 6 lets a reserve cell accept phones without writing confirmation or
splice leases. This ratchet classifies every reader of the lease table by why
that is safe, and pins the guards that keep completion and idle rehome off a
reserve cell whose host holds no source lease.
* test(relay): scan subdirectories and USING reads, and ban a bare lease table name
The ratchet now walks every production file under src, counts USING as a read, and
fails on a bare table-name value, which could reach an interpolated FROM it cannot see.
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
A reserve cell's cell-status read is refused, so it carried no
connection_hard_cap and the evaluator fell back to the 500 default:
run 39225932592 froze on reserve c7 at 731 connections under its
1000 cap. Read connection_hard_cap from the durable Terraform cells
relay-ops already parses and emit it for reserve cells.
The director's cell-status refuses a step-5 reserve cell with a 404
{error: cell_in_reserve_mode}, so every monitor sample failed while any
cell was reserve. Classify that exact refusal as reserve-not-probed, skip
the refused migration reads, and judge the cell on probes, logs and the
selector instead. Any other 404 still fails the sample.
A reserve cell keeps no activity rows and its ledger stamps last_activity_at, so the
stranded rule read its connected hosts as unserved and re-placed them.
* fix(relay): log controls a cell closes after host proof but before wiring
* fix(relay): log a pre-refusal hang-up as the hang-up and share printableCloseReason
* refactor(relay): move printableCloseReason into its own module to break the import cycle
* test(relay): assemble the fake pg URL at runtime so the secret scan passes
The folder is the cloud relay that reaches phones; under its old name it read as the legacy SSH
relay. git mv plus import and path-literal rewrites only.
* fix(relay): destroy a pooled client whose session the server ended, and mark work that never ran
pg keeps calling a client queryable until its socket's end arrives, so a client released after
57P01 (pg_terminate_backend, shutdown) or a dropped socket went back to the pool and failed the
next caller's statement with 'Connection terminated unexpectedly'. Such clients are now released
with the error, so pg-pool discards them.
Also marks failures where no statement of the operation reached the server (acquire, a failed
BEGIN, a client dead at checkout) so a caller can tell work safe to re-run.
* fix(relay): skip ROLLBACK on an ended session, drop unused nothing-ran marking
Adds 57P04/57P05 to the session-ended codes and covers 08xxx, a dropped
socket, an ordinary 23505, and a real pg_terminate_backend.
Claude-Session: f2fe8cb5
* feat(relay): log what the sticky slot spends its hold on, per pooled connection
* fix(relay): time every sticky round trip at the PostgreSQL client, pool wait apart
* docs(relay): say which round trips the sticky slot tally covers
* Avoid unused module imports and trim cosmetic test coverage
* Capture hidden popup frames and attach CDP before guest creation
* Overlap relay test projects within the existing worker budget
* Consolidate native preparation and add an opt-in transform cache comparison
* Release inherited debugger sockets during final E2E teardown
* Refresh unit scheduling weights from all five qualified ARM shards
* Avoid retaining unasserted RPC call history in the delivery model
* Use the supported unconditional background wait syntax
* Overlap unconditional output upload with native preparation
* Verify relay integration stays enabled for ordinary unit callers
* feat(relay): split step-5 shadow compare and dry-run logs per cell
The 24 h per-cell shadow gate needs per-cell sticky-compare counts, per-cell
unexplained counts, and dry-run `full` refusals while the map showed room.
Both summary lines gain additive per-cell fields; existing fields are unchanged.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count a cell-mismatch on both cells and bound per-cell maps by the fleet
- countByCell (was cells) counts a compare on every cell it involves: the
database's cell, every seat's cell when there is no row, and both cells
for a cell-mismatch. Drop the 64-cell cap and cellsOverflow.
- Add ReservePlacer.free and use it for the dry-run free share.
- Rename the room bound to RESERVE_DRY_RUN_ROOM_BOUND.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count a duplicate-seat compare on every seat cell
An unexplained duplicate seat counted only on the database's cell, so a cell
whose stale feed kept a second seat could pass its own step-5 check. Count it
once on every seat cell, the same as a cell-mismatch.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count explained cell-mismatch and duplicate-seat on the database cell only
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): key per-cell counts on prototype-free records
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): explain a duplicate seat left by a just-superseded half-open control
When the database names the newest seat exactly, every other seat has a
strictly lower epoch, and the newest seat joined within the cell's silence
close (timeout + one ping interval + the map lag bound), the older seat is
a half-open control awaiting its 75 s pong-silence close, not a fault.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): bound the superseded duplicate-seat rule on the move's age
A same-epoch rejoin resets joinedAt, so a stale older seat could stay
explained forever; the director-clock sighting of the epoch moves only
when the epoch rises.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): leave a superseded duplicate seat unexplained without an epoch sighting
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): explain a superseded duplicate seat only from a prior sighting
compare() recorded the epoch before classifying, so a first-ever sighting
(restart or eviction) read as a fresh move. Classify first and take the
move's age from the record that existed before this compare.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): also bound a superseded duplicate seat on its join age
A long-ago lower-epoch sighting dates the move to this compare, so the
current seat's join age bounds it too.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count an ageless superseded duplicate seat as its own class
A superseded-shaped duplicate-seat with no prior sighting of the host (the
first compare since a restart) has no move age, so it is now
`superseded-unknown-age` (explained: false) rather than a plain duplicate.
The count line also carries `uptimeMs`, so the gate can discount that
class in an instance's first 100 s.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): limit superseded-unknown-age to an instance's first 100 s
Flushes are lazy, so a window has no upper bound and an uptime field on the
count line cannot date it. The compare now records when it started and
emits superseded-unknown-age only within the superseded close bound of
that; later, a missing sighting means another instance served the host or
the sighting was evicted, so it is a plain unexplained duplicate-seat.
Drops the uptimeMs field.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): say where a missing sighting comes from after the first 100 s
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* perf(ci): reduce test overhead and reuse expensive build work
* test(ci): keep worker startup outside failure-case deadlines
* test: isolate push fixture advisory locks
* test(ci): preserve fixed drain retry allowance
* Align CI contracts with current triggers and guide content
* Stabilize consolidated reveal scenarios and retain behavioral CI contracts
* Fix upstream lint blockers while retaining behavior and assertions
* Classify global Jira preferences in the WSL migration inventory
* Align Jira test fixtures with startup and build safety checks
* Include package-test routing dependency in CI checkouts and fixtures
* Keep native stream classes intact in transcript dependency tests
* test(relay): check invariant 4 (one demotion winner) in the step-5 simulation
Every demotion now names its winner (the newer booking, or the row). The model
flags a director that names a different winner for a divergence another already
named, and a host whose two seats are each demoted for the other. A fault where
directors break level ties by their own cell order shows the check fires.
Violations are capped per invariant so one flood cannot hide another class.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* test(relay): bound unseen duplicates by the silence close and assert all invariants on contested ties
A reserve seat the booking director's map had not seen (joined, then half-open within a poll)
is never superseded and has no lease to re-check, so it lasts until the cell's control silence
close (75 s + one 15 s ping). Invariant 3 now uses that bound for such seats, and the
level-divergence config asserts every invariant.
Stale maps now judge a divergence another director already demoted, so the config's
"contested" counter (divergences judged by two or more directors) is the case invariant 4
guards. The silence-close constant now matches production (90 s, not 100 s).
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* test(relay): raise the floor on map-judged demotes and count only level contests
A reconcile whose map has not seen another director's demotion still demotes in
production, which raises its epoch floor; the model now does too. The contested
counter counts only same-epoch pairs, the only ones invariant 4 can split. The
keep-this-seat rule is one helper.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* refactor(relay): name the simulation's fault-kept seat check for the fault
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): supersede a reserve seat from the feed once a later, newer join shows
A half-open seat the booking director's map had not yet seen was never demoted and held its
cell until the 75 s control silence close. Directors now demote it once both joins are in the
map; the simulation holds invariant 3 at the supersede bound for it.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): outrank only past cell clock skew, reuse demoteSeat, keep invariant 3's loose clock running
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* refactor(relay): share the outrank rule with the simulation, pin the skew boundary
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count only confirmed level demotions and every second judgment, and bound duplicates by placement records alone
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): a drained seat keeps its drain grace under a demote, and the feed reports each seat once
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* refactor(relay): name the outranked-report bound and the reconcile backstop
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* feat(relay): measure each host hello's pre-auth time on the cell
Adds preAuthLatencyMs to the per-cell orca_relay_runtime_metrics line: per
admit path (db, reserve) and outcome (admitted, refused), a count plus
p50/p95/p99/max for the window. Each sample runs from the cell accepting the
host control WebSocket to the hello being admitted (proof verified) or
refused. Samples are kept in a bounded reservoir, shared with the control
round-trip metric. No behaviour, wire, or flag change.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count abandoned hellos and publish mergeable pre-auth buckets
- Record a third outcome, abandoned, for a host that hangs up before a decision or before
the cell acts on one, so the slowest hellos no longer drop out of the window.
- Add non-cumulative bucket counts per group (bounds 50..5000 ms and +Inf, 450 exact) so
the share at or under 450 ms sums exactly across windows and cells.
- Split refusals that end before an admission check answers into a beforeCheck path.
- One refuse() closure per phase records the metric, and the path is passed to settle.
- Drop the half-used LatencyReservoir type; the shared reservoir function stays.
- Replace test casts with a typed accessor or SAFETY comments.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): note pre-auth path labels for late throws and the combined role
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* test(relay): check invariant 4 (one demotion winner) in the step-5 simulation
Every demotion now names its winner (the newer booking, or the row). The model
flags a director that names a different winner for a divergence another already
named, and a host whose two seats are each demoted for the other. A fault where
directors break level ties by their own cell order shows the check fires.
Violations are capped per invariant so one flood cannot hide another class.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* test(relay): bound unseen duplicates by the silence close and assert all invariants on contested ties
A reserve seat the booking director's map had not seen (joined, then half-open within a poll)
is never superseded and has no lease to re-check, so it lasts until the cell's control silence
close (75 s + one 15 s ping). Invariant 3 now uses that bound for such seats, and the
level-divergence config asserts every invariant.
Stale maps now judge a divergence another director already demoted, so the config's
"contested" counter (divergences judged by two or more directors) is the case invariant 4
guards. The silence-close constant now matches production (90 s, not 100 s).
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* test(relay): raise the floor on map-judged demotes and count only level contests
A reconcile whose map has not seen another director's demotion still demotes in
production, which raises its epoch floor; the model now does too. The contested
counter counts only same-epoch pairs, the only ones invariant 4 can split. The
keep-this-seat rule is one helper.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* refactor(relay): name the simulation's fault-kept seat check for the fault
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count only confirmed level demotions and every second judgment, and bound duplicates by placement records alone
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): keep verifying host tokens from the last served keys while auth is down
jose refetches the relay JWKS once its cache is 10 min old and rejects the token when that
fetch fails, so an auth outage turned every still-valid host token into a 4401. The cell now
falls back to the last set auth served for up to an hour, skips refetching for 30 s after a
failure, and still rejects any key that set never had. A fresh set replaces it as soon as auth
answers, so a rotation takes effect at the first successful fetch.
* fix(relay): bound stale relay keys at 30 min and document the emergency purge
Thirty minutes still covers a full auth outage (no token the stale keys could accept outlives
~16 min of one) and halves how long a rotated-out key stays trusted on a cell kept from auth.
A restarted process holds no set and refuses host tokens until a fetch succeeds; the ops doc
names that as the purge after a signing-key leak, and a test pins the fail-closed start.
* test(relay): pin the 30 min stale relay key bound (29 min accepts, 31 min rejects)
* fix(relay): a resumed host control adopts the token its hello proved
A rebind replaced the socket, lease and epoch but kept the predecessor's token claims, so a
host that resumed after its token lapsed (auth down past expiry plus grace) was closed 4401
"relay authorization expired" at the next heartbeat, 15 s later, and every resume repeated it.
The rebind now takes the hello's verified claims when they carry a later expiry, and a hello
that speaks for another profile or organization starts a new generation instead of resuming.
* fix(relay): a refresh never lowers the held expiry and closes only its own socket
An auth refresh now keeps the later of the held and refreshed expiry, the rule a resume uses.
A refused refresh, or one that meets an idle cutover, closes the socket it arrived on: after a
rebind, session.socket is the successor, which proved itself and must not be closed for its
predecessor's late frame. Adds an organization-mismatch resume test beside the profile one.
* test(relay): a superseded socket's refresh during an idle cutover closes only that socket
* fix(relay): let a host leave a dead reserve cell without an operator fence
A host whose reserve home cell died got 503 relay_home_cell_unavailable until the
cell came back: the database path demands a committed fence for every capped cell.
On its own /v1/assign, a host now moves off a reserve cell (Postgres records it as
reserve) when the cell's heartbeat is stale and this director's seat-feed polls have
gone unanswered for 45 s, unless more than a third of the polled cells look the same.
Sweeps, migration pins and capped db cells keep the fence.
* fix(relay): guard the dead-cell verdict by region and sample size, and floor the move on the map
- No verdict when half or more of the dead cell's own region fails with it (a cross-region
partition), more than a third of the fleet does, or fewer than 6 cells are polled. A withheld
region or fleet verdict logs orca_relay_dead_cell_verdict_withheld, which alerts.
- The move needs the step-5 plan's epoch floor, so it mints above every seat the map knows;
without the plan the fence is kept. A test revives the old cell's seat above the row and checks
that the ledger never writes the row back.
- The move's log line carries the fleet and region counts.
* fix(relay): count only feed cells and young streaks in the dead-cell guards; gate the alert
- Denominators hold only cells that served this director a feed page and have capacity: legacy
rows answering 404 no longer fill the 6-cell sample or dilute the fleet and region ratios.
- A peer counts as failing for the guards on any current streak (a failure in the last 5 s), so a
staggered partition cannot let its first cell cross 45 s while its neighbours read healthy.
- Every withheld verdict is logged, sample included (a cell missing from the feed cells is one).
- Tests: ready=0 and enabled=0 with failing polls keep the fence (drain then death is fenced).
- The withheld metric and alert are gated by relay_dead_cell_verdict_alert (default false, on in
staging); the alert docs now describe poll failures, not heartbeats.
* fix(relay): always sample the dead-cell candidate, so a restarted director can still judge it
A director that restarted after a reserve cell died never received a feed page from it, so the
cell was never in the sample and its verdict stayed withheld as 'sample'. The candidate now always
counts when it has capacity (the store asks only about a cell Postgres records as reserve, and only
feed images run reserve); peers still count only once they have served a feed. Tests: a restart
during a lone outage decides, a partition begun before the restart withholds by region, and the
capacity filter. The Terraform comments now say to apply the metric first and the policy minutes later.
* fix(relay): keep seat polls off the cell-list read and report event-loop freezes
The seat poller no longer waits on its database cell-list read before polling
cells once it has a list, so a stalled read cannot starve the polls that feed a
reserve cell's dead-man. Runtime metrics add eventLoopDelayMsMax: a single long
freeze is one sample, so p99 read 68 ms across a 134 s staging freeze.
Adds a pool test against a database that accepts and never answers, and a
real-Postgres stop/start recovery test.
* fix(relay): state the seat-poller gain precisely, justify the AddressInfo cast
A desktop relays alerts for agents on its paired servers, but the push named
only the server's worktree, so a tap opened it on the desktop and bounced.
The gateway accepts an optional runtime execution host and forwards it.
Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb
* feat(relay): per-cell switches for the rejection fence and the lost-reply margin, and report ignored keys
rejectionFence (default on) turns the unhandled-rejection fence off without a roll.
readTimeoutMarginMs (1-60 s) replaces the boot margin per query through pg's query_timeout.
The applied log line and flagsApplied name keys this image does not know.
* feat(relay): runtime-status names the switches and values this image accepts
The flag tool checks a write against it, so it can refuse ticketCheck=enforce (or any key)
on an image that would void or drop it.
The flag channel recorded a generation as seen before reading its body, so a
read that failed mid-body sent ifGenerationNotMatch for it on every later poll
and got 304 forever: that write was stranded until the object changed again.
The generation now counts as seen only once its body has been read and parsed.
On-call can ask a director where a host is: its relay_assignments row and
open migration (primary-key reads only), the shadow map's seats and recent
leaves with close codes, the compare verdict (pure classification, judged as
/v1/resolve sees the host, recording nothing), and each named cell's poll
status and applied flags. shadow is {state:'off'} when the poller is off.
Deploy, monitor and fence identities may read it; capacity and asia-proof
may not. With no user id, the map names the user; an unknown host answers
empty.
After the database answers a reconnect verify or a phone resolve, the
director compares that answer with the shadow seat map and counts the
result by class, logging one count line per route and class a minute and a
few rate-limited samples of unexplained disagreements (cell and epoch only,
never raw ids). It runs only when the seat poller is on, inside try/catch,
and never changes a response.
A map-behind mismatch is explained only while this director first saw the
newer epoch within 10 s (recorded only by the compare; classification is
pure). A seat on a cell not polled within 10 s is cell-unpolled. A null
answer for a seat on a cell placement's liveness rule calls dead is
map-only-cell-unlive, or cell-unlive-pending when its heartbeat ran out
after the last cell-list read.
* feat(relay): key id on assignment leases; cells shadow-check the lease behind ticketCheck=shadow (off)
Directors name the lease signing key with a kid. With the per-cell
ticketCheck switch at shadow, a cell verifies the lease a host echoes in
x-orca-relay-assignment-lease, compares it with the database answer that
still decides, and logs per-class counts and the hello's database read
time once a minute.
* feat(relay): flat per-class totals on the lease shadow line, so log metrics can extract them
* refactor(relay): the lease shadow line carries the flat classes and one database-refused count
The per-class-and-verdict map duplicated the flat classes; disagree
already names the valid-lease refusals, and dbRefused gives the total.
Neil
Jinwoo Hong
