* fix(relay): repair cell seat-log drift, and keep a new session's generation above a listed seat
Directors mirror each cell's seat feed, so the feed must list exactly the cell's live sessions.
The cell now sweeps its seat log against its sessions on a director's poll: a seat no session
backs is left (4409) and the live session listed, and a live session the feed lost is listed
again, each once two sweeps in a row find it. A demotion naming an unbacked seat repairs it at
once. A host's new session now starts above any generation the feed still lists, which closes
the drained-session reconnect race that removed a live seat.
Claude-Session: asia-drain-pulse
* fix(relay): refuse a rebind whose session left the map during activation, and keep a per-host generation floor
A rebind whose database activation outlived the orphan grace (or a drain) revived the removed
session outside the map: connected, still in the seat feed, and unreachable by phones or
demotion for as long as the host stayed up. Activation now rechecks the map after its awaits and
refuses with PEER_DROPPED so the host redials. Generations come from a per-host floor that
outlives the session, and an activation at a generation already used here is refused.
Claude-Session: asia-drain-pulse
* fix(relay): log each demote outcome on the director by host digest
The director now reads the cell's existing {v:1, outcome} demote reply and logs it; the cell's
feed stays the only source of its map. The supersede line now reaches the log as well.
Claude-Session: asia-drain-pulse
* fix(ci): recover verified pnpm setup from registry metadata outages
* fix(ci): prepare native pnpm shims after fallback verification
* fix(ci): provide Node for pnpm fallback in bare containers
* chore(cloud): update relay runtimes and security scanning
* docs(cloud): record Go binary advisory reachability
* chore(cloud): rerun security checks after resolving main conflict
* fix(relay): release a drained host before telling it to leave
A zero-grace drain sent `drain` and closed with 4503 in the same tick, then
released the host's control lease. The desktop redials the director as soon
as it reads either, and from asia-east2 the release holds the host's
assignment row for ~1 s, so the redial met it and was refused row_busy.
The release now runs first and the host is told once it lands, bounded at
2 s; a failed or stuck release still closes. Wire frames are unchanged.
* fix(relay): give a drained host's release 3 s and log when it runs out
One host's releases run one after another, so 3 s covers a control lease
plus one open phone connection from asia-east2.
* style(relay): brace the drain release branches
* fix(relay-ops): make each reserve-watch sample one Cloud Logging read
The project allows 60 ListLogEntries a minute shared by every watch and tool. Each watch sample
made four to six reads (two director runtime-metrics sub-windows, the ledger summaries, the
seat-behind-row lines, the cell's phone metrics, and since #31484 the 503 request log), so on
10-21 18:39Z twelve watches drew 429s until c21 and c22 tripped unverifiable.
A sample now sends its queries as one OR filter and splits the lines back client-side. The
per-instance 503 split counts each director's own logged /v1/assign 503s from the runtime
metrics it already reads, with the rest of the fleet's count as one unattributed instance,
instead of reading the request log.
* fix(relay-ops): judge unattributed 503s on their 3-minute sum against the whole fleet line
* docs(relay-ops): state that an instance that stops logging trips the unattributed rule
* fix(relay-ops): keep one stalled director instance from tripping the reserve watch's 503 rule
The reserve watch's director503 rule counted fleet-wide non-drain 503s. On 10-21 07:55Z one
director instance's database calls stalled for 3 minutes; its 503s alone carried the fleet over
the line and all four watched cells flipped back to db.
Over the fleet line the rule now also needs at least 2 director instances each above a fifth of
it for the same 3 minutes, counted from Cloud Run's request log by labels.instanceId less each
instance's scheduled 503s. Above max(10x background, 200) for 3 minutes it trips with no split.
* fix(relay-ops): size the instance share to the directors seen and count platform 503s
* fix(relay-ops): size the instance share per judged minute and refuse a missing count
The deploy passed `--max`, which is gcloud's service-wide cap. Cloud Run divides
that cap among the revisions taking traffic, so during the 20/80 stage a
candidate under a cap of 5 got one instance while the serving floor kept four,
and the spread check could never see the full floor (deploy M, 1 of 5 for six
minutes). Candidates now pin `--max-instances` per revision, and the deploy
resets a service cap below two full revisions to Cloud Run's default before it
creates any candidate. The rollout overlap of two full directors is already in
the checked Cloud SQL connection budget.
A freshly restarted cell can be ready while the global load balancer still
answers 503 for it. The probe retried once after 2 s, so c10's 2026-10-20
roll failed after its drain and was left isolated. Retry wrapped source
5xx every 10 s for up to 2 minutes.
Co-authored-by: E2E Test <e2e@test.local>
* fix(relay): move director traffic in two steps so a deploy does not land on one instance
Cloud Run routes a new director revision's floor instances their first request
roughly 60-110 s after they start, while a traffic move reaches clients in
15-30 s. The one-step 100% move therefore put every assignment on the single
instance started with the revision, which sits at its own admission limit at
peak; the 2026-10-20 deploy returned ~550 assign 503s in two minutes that way.
With a floor above one, the deploy now moves 20% first, waits until Cloud Run's
request log shows every floor instance serving (six-minute deadline), then
moves the rest. A timeout or failure at that stage returns all traffic to the
previous serving revision, which still holds its warm floor. The staging deploy
identity gains log read access for the same check.
* fix(relay): bound the director spread log read, log the instances seen at timeout, and document split recovery
A hung gcloud log read could hold the 20/80 split past the six-minute deadline;
each read now times out after 30 s and counts as an unread poll. A timeout logs
the distinct instances seen, so a partial share packed onto fewer instances than
the floor shows up. The runbook covers restoring a split left by a cancelled job
and the missing selector-rollback tag after a timed-out deploy.
* fix(relay): give region-correction reconnects and roll re-placements the in-region reserve fallback
Every desktop session open carries a region-correction request, and those
skipped the reserve plan entirely, so with the region's database cells full they
still spilled abroad while an in-region reserve cell had room. The plan now runs
for them in a mode that never answers itself (the database lane must reply so the
window exchange follows) and offers only the in-region reserve fallback.
A host re-placed off a roll-isolated cell went straight to the other-regions tier
when its own region's tier was full. With an in-region reserve fallback it now
books that first, and the roll canary event is emitted for the reserve booking.
* fix(relay): never pace a roll drain on reserve intake, and count every reserve move off an isolated cell
A host leaving a roll-isolated cell whose region was full could get a paced 503
from a throttled reserve cell, stretching the drain and re-running the failed
in-region transaction on each retry. For those hosts a pace is now no room, and
they spill as before. Sticky now says when it declined an isolated pin, so a
share-draw booking off that cell also emits the roll canary event.
* fix(relay): keep a host in its region when the database lane is full and a reserve cell has room
A host that lost the reserve share draw went to the database lane, which
spills to another region when its own region's database cells are full,
even while an in-region reserve cell had free seats. The database lane now
requires the region when an in-region reserve cell has room; if the region
is full it books that reserve cell, and only spills when neither has room.
* fix(relay): pace a throttled in-region reserve cell instead of spilling, and decide reserve room at plan time
With the region's database cells full, a reserve cell with seats but no intake
budget fell through the placer's database-cells shortcut and the host went
abroad. That placement now answers pace (503 with Retry-After). regionReserve
is a single callback, present only while an in-region reserve cell has a seat.
* fix(relay): pace only on a reserve cell that lacked budget, not one that refused
A cell that refused this placement (full, unreachable, seated-newer) got its
token back and drove a 1 s Retry-After forever while the map still showed a
seat. Pacing now skips cells refused for anything but intake, so those
placements fall to the database lane's cross-region fallback as before.
evaluateWatchSample's two early returns (waiting for reserve to apply, and
a cell flipped back to db by hand) carried no pages, so watchSegment threw
'result.pages is not iterable' and the run ended as failed instead of
stopped (c29, 10-20 02:25Z, run 39354895994).
* perf(push): wake delivery worker after notifications are queued
* perf(push): wake one drain per send so bursts cost O(sends + drains) claims
A wake started all twelve drains, so each send arriving at an idle worker cost
~13 claim transactions. A wake now starts one drain; the poll tops up to the
full drain set.
A host control request that failed for an unexpected reason sent the thrown
error's message to the desktop as the error code. During a staging database
outage the desktop received Postgres's own text, and node-pg messages can
carry the database connection URL.
The relay now sends store codes and this codebase's snake_case invariant
slugs unchanged, and maps everything else to database_transient or
control_operation_failed. The original is logged through the existing
redaction (now shared with the background-operation logger), so only the
slug shape and SQLSTATE leave the process.
The load generator only opened invite splices, which never reach the device-resume accept or the
desktop's resume confirm. --resume-phones N installs one device credential per phone over an
invite splice, then reconnects it every --resume-interval-seconds: accept, device-resume-confirm,
and a held splice with echoes. Each attempt is a relay_load_resume_attempt line with accept and
confirm ms; failures are recorded, never fatal, so a database-block drill can run under it.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
Co-authored-by: E2E Test <e2e@test.local>
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay): resume paired phones from one device read on memory cells
With phonePath=memory in reserve mode, a phone resume is admitted from one device
read and the cell's own seat, and takes no activity leases. A refused device closes
4401, a failed read 4503, and only a missing device row falls through to the invite
path. A phone whose host has no seat gets one assignment read: 4409 when the row
names another cell or the read fails, otherwise 4404.
* feat(relay): keep a memory resume's basis in the cell and confirm it from one read
A phone resumed on the memory path keeps its connection basis in the cell, not in
relay_connection_bases. Its confirm rereads the device row once, renews it only when
less than 29 days remain, and always answers a current-version confirm renewed:true
with the expiry the database holds. Confirms route by where their basis lives, so a
flip between accept and confirm completes either way. Retries answer from a 10-min
in-memory result map.
* test(relay): roll a memory phone cell back with confirms in flight
Flipping back by admitMode=db, phonePath=db or a dead-man trip leaves in-flight
confirms to finish on their memory basis, writes no phone lease and leaks none, and
evacuation still refuses the cell while Postgres records it as reserve.
* feat(relay): batch a memory cell's resume-confirmed audit rows
Memory confirms buffer their audit rows and the cell writes them as one multi-row
insert every 5 s or 500 rows, idempotent by a per-event id. The buffer holds at most
5,000 rows and drops a row older than 10 minutes rather than retrying it, counting
and logging both kinds of drop.
* feat(relay): count and log phone-path failures, and every attach deadline
The runtime metrics line gains phone refusals by close code, attach deadlines, accept
database errors, resume confirm failures by code, completed accepts by phone path and
the resume audit drop total. Every attach-deadline 4404 now logs what the cell knew of
the desktop's control (pong age, ping outstanding, control age and round trip), so a
half-open control can be told from a slow data dial.
* feat(relay): close an attach-deadline 4404 with the attach-timeout reason
The phone close reason from #30607 now rides the close frame of the 4404 a cell
sends when the desktop's data leg misses the attach deadline, so OTA phones can
redial the same cell once instead of waiting out the host-offline delay. The failed
relay-hello frame is unchanged, and old phones ignore the reason.
* test(relay): drill a memory cell through a database outage, and pin phone renewal
A blocked database leaves splices forwarding, closes new resumes 4503 and fails
confirms cleanly, lets an in-flight confirm finish on its memory basis after a flip
back, and drops audit rows past 10 minutes instead of writing them late. On the phone,
a relay-only session lives 31 days under the once-a-day renewal rule, and would not
if a not-due confirm answered renewed:false.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay): log a failed no-seat read, recheck the switch after the device read
A failed assignment read for a phone with no seat now logs a rate-limited line before
its 4409. A switch that turned off during the device read sends the phone down today's
path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy.
* fix(relay): log a failed memory renewal; test flips, replays and basis cleanup
A renewal write that fails still answers the stored expiry, but now logs a rate-limited
line so a persistently failing update is visible before phones age toward re-pairing.
* test(relay): assert a rolled-back memory splice issues no statement when it closes
* fix(relay): flush resume audit early only on crossing the batch size, drop the shutdown flush
A failing database no longer takes a 500-row insert per confirm; retries wait for the
5 s timer. The shutdown flush raced the pool closing, so it is gone and the at most 5 s
loss is documented. A stopped buffer no longer re-arms its timer.
* test(relay): assert the aged audit rows were dropped, not written
---------
Co-authored-by: E2E Test <e2e@test.local>
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay): resume paired phones from one device read on memory cells
With phonePath=memory in reserve mode, a phone resume is admitted from one device
read and the cell's own seat, and takes no activity leases. A refused device closes
4401, a failed read 4503, and only a missing device row falls through to the invite
path. A phone whose host has no seat gets one assignment read: 4409 when the row
names another cell or the read fails, otherwise 4404.
* feat(relay): keep a memory resume's basis in the cell and confirm it from one read
A phone resumed on the memory path keeps its connection basis in the cell, not in
relay_connection_bases. Its confirm rereads the device row once, renews it only when
less than 29 days remain, and always answers a current-version confirm renewed:true
with the expiry the database holds. Confirms route by where their basis lives, so a
flip between accept and confirm completes either way. Retries answer from a 10-min
in-memory result map.
* test(relay): roll a memory phone cell back with confirms in flight
Flipping back by admitMode=db, phonePath=db or a dead-man trip leaves in-flight
confirms to finish on their memory basis, writes no phone lease and leaks none, and
evacuation still refuses the cell while Postgres records it as reserve.
* feat(relay): batch a memory cell's resume-confirmed audit rows
Memory confirms buffer their audit rows and the cell writes them as one multi-row
insert every 5 s or 500 rows, idempotent by a per-event id. The buffer holds at most
5,000 rows and drops a row older than 10 minutes rather than retrying it, counting
and logging both kinds of drop.
* feat(relay): count and log phone-path failures, and every attach deadline
The runtime metrics line gains phone refusals by close code, attach deadlines, accept
database errors, resume confirm failures by code, completed accepts by phone path and
the resume audit drop total. Every attach-deadline 4404 now logs what the cell knew of
the desktop's control (pong age, ping outstanding, control age and round trip), so a
half-open control can be told from a slow data dial.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay): log a failed no-seat read, recheck the switch after the device read
A failed assignment read for a phone with no seat now logs a rate-limited line before
its 4409. A switch that turned off during the device read sends the phone down today's
path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy.
* fix(relay): log a failed memory renewal; test flips, replays and basis cleanup
A renewal write that fails still answers the stored expiry, but now logs a rate-limited
line so a persistently failing update is visible before phones age toward re-pairing.
* test(relay): assert a rolled-back memory splice issues no statement when it closes
* fix(relay): flush resume audit early only on crossing the batch size, drop the shutdown flush
A failing database no longer takes a 500-row insert per confirm; retries wait for the
5 s timer. The shutdown flush raced the pool closing, so it is gone and the at most 5 s
loss is documented. A stopped buffer no longer re-arms its timer.
* fix(relay-ops): let same-cap rolls run while another cell is reserve; refuse a reserve roll target by name
The director refuses cell-status for a step-5 reserve cell with a 404
{error: cell_in_reserve_mode}. The same-cap headroom check read it for
every general cell, so any roll failed while c7 was reserve. A reserve
cell that is not being rolled now counts as no free slots (its seats are
not in the database) and is listed with reserve: true. A reserve roll
target, in the headroom check, the same-cap job's first cell-status read
and the candidate tool, now fails with the design's procedure: flip it to
admitMode=db first, roll, then flip it back. Any other 404 still fails.
* fix(relay-ops): pin the headroom gate's imports as evidence code; name both cells in a two-cell reserve refusal
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay): resume paired phones from one device read on memory cells
With phonePath=memory in reserve mode, a phone resume is admitted from one device
read and the cell's own seat, and takes no activity leases. A refused device closes
4401, a failed read 4503, and only a missing device row falls through to the invite
path. A phone whose host has no seat gets one assignment read: 4409 when the row
names another cell or the read fails, otherwise 4404.
* feat(relay): keep a memory resume's basis in the cell and confirm it from one read
A phone resumed on the memory path keeps its connection basis in the cell, not in
relay_connection_bases. Its confirm rereads the device row once, renews it only when
less than 29 days remain, and always answers a current-version confirm renewed:true
with the expiry the database holds. Confirms route by where their basis lives, so a
flip between accept and confirm completes either way. Retries answer from a 10-min
in-memory result map.
* test(relay): roll a memory phone cell back with confirms in flight
Flipping back by admitMode=db, phonePath=db or a dead-man trip leaves in-flight
confirms to finish on their memory basis, writes no phone lease and leaks none, and
evacuation still refuses the cell while Postgres records it as reserve.
* feat(relay): batch a memory cell's resume-confirmed audit rows
Memory confirms buffer their audit rows and the cell writes them as one multi-row
insert every 5 s or 500 rows, idempotent by a per-event id. The buffer holds at most
5,000 rows and drops a row older than 10 minutes rather than retrying it, counting
and logging both kinds of drop.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay): log a failed no-seat read, recheck the switch after the device read
A failed assignment read for a phone with no seat now logs a rate-limited line before
its 4409. A switch that turned off during the device read sends the phone down today's
path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy.
* fix(relay): log a failed memory renewal; test flips, replays and basis cleanup
A renewal write that fails still answers the stored expiry, but now logs a rate-limited
line so a persistently failing update is visible before phones age toward re-pairing.
* test(relay): assert a rolled-back memory splice issues no statement when it closes
* fix(relay): flush resume audit early only on crossing the batch size, drop the shutdown flush
A failing database no longer takes a 500-row insert per confirm; retries wait for the
5 s timer. The shutdown flush raced the pool closing, so it is gone and the at most 5 s
loss is documented. A stopped buffer no longer re-arms its timer.
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay): resume paired phones from one device read on memory cells
With phonePath=memory in reserve mode, a phone resume is admitted from one device
read and the cell's own seat, and takes no activity leases. A refused device closes
4401, a failed read 4503, and only a missing device row falls through to the invite
path. A phone whose host has no seat gets one assignment read: 4409 when the row
names another cell or the read fails, otherwise 4404.
* feat(relay): keep a memory resume's basis in the cell and confirm it from one read
A phone resumed on the memory path keeps its connection basis in the cell, not in
relay_connection_bases. Its confirm rereads the device row once, renews it only when
less than 29 days remain, and always answers a current-version confirm renewed:true
with the expiry the database holds. Confirms route by where their basis lives, so a
flip between accept and confirm completes either way. Retries answer from a 10-min
in-memory result map.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay): log a failed no-seat read, recheck the switch after the device read
A failed assignment read for a phone with no seat now logs a rate-limited line before
its 4409. A switch that turned off during the device read sends the phone down today's
path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy.
* fix(relay): log a failed memory renewal; test flips, replays and basis cleanup
A renewal write that fails still answers the stored expiry, but now logs a rate-limited
line so a persistently failing update is visible before phones age toward re-pairing.
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay-ops): watch a memory phone cell and turn its phone path off on a breach
The reserve watch now also judges five phone rules while the switch file says
phonePath=memory, from the cell's own runtime metrics against the 10 minutes before
flipped-at: 4401s, confirm failures, attach-deadline 4404s and accept database errors
over their lines for 3 minutes, and completed accepts under half of background. A
phone-rule trip writes phonePath=db only; any other trip still flips the cell back
whole. Resume audit drops page after 5 minutes and flip nothing.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay-ops): keep phone watch rules off traffic noise, and keep watching after a phone trip
Completed accepts are no longer judged. Rate lines use the background mean plus
max(margin, share, 4 Poisson deviations), with a margin of 3 for attach deadlines. A
phone trip writes phonePath=db mid-watch and the reserve rules go on; both flip backs
remove the phonePath key so a cell rolled back to an image without it takes the write.
The phone background starts where phonePath went memory, and a background broken by a
restart refuses the watch. Audit drops page only while rising; failing renewals page too.
* test(relay-ops): replay the phone rules on 26 recorded hours of c7 and c29
Per-minute counts from the cells' runtime metrics (read-only, 10-18/19) replace the
synthetic quiet window. c29's bursty auth failures tripped the attach-deadline rule at a
margin of 3 in 29% of 300-minute watches, so its margin is now 5 like the others.
* fix(relay): judge the phone rules as a share of attempts, retry a failed phonePath write
The count rule false-tripped busy c28 (33/121 five-hour watches). The share rule (over half of
the minute's attempts, 0.25 over the background, capped at 0.6) trips none of the recorded c7,
c28 and c29 watches and catches two refusals per accept within 3 minutes. A failed mid-watch
phonePath=db write now keeps watching, retries each sample and pages after 3 failures; break
glass also drops phonePath.
Co-Authored-By: Claude <noreply@anthropic.com>
* docs(relay): cite the replayed c28 count-rule false trips
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(relay): judge phone confirm failures against completed accepts
Each completed accept makes one confirm, so failures/(failures + accepts) could never pass half
and a total confirm outage never tripped. Confirm failures are now a share of accepts.
Co-Authored-By: Claude <noreply@anthropic.com>
* test(relay): note why the confirm replay tests only trips
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay): resume paired phones from one device read on memory cells
With phonePath=memory in reserve mode, a phone resume is admitted from one device
read and the cell's own seat, and takes no activity leases. A refused device closes
4401, a failed read 4503, and only a missing device row falls through to the invite
path. A phone whose host has no seat gets one assignment read: 4409 when the row
names another cell or the read fails, otherwise 4404.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay): log a failed no-seat read, recheck the switch after the device read
A failed assignment read for a phone with no seat now logs a rate-limited line before
its 4409. A switch that turned off during the device read sends the phone down today's
path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy.
* test(relay): census every activity-lease reader for phones without leases
Step 6 lets a reserve cell accept phones without writing confirmation or
splice leases. This ratchet classifies every reader of the lease table by why
that is safe, and pins the guards that keep completion and idle rehome off a
reserve cell whose host holds no source lease.
* test(relay): scan subdirectories and USING reads, and ban a bare lease table name
The ratchet now walks every production file under src, counts USING as a read, and
fails on a bare table-name value, which could reach an interpolated FROM it cannot see.
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
A reserve cell's cell-status read is refused, so it carried no
connection_hard_cap and the evaluator fell back to the 500 default:
run 39225932592 froze on reserve c7 at 731 connections under its
1000 cap. Read connection_hard_cap from the durable Terraform cells
relay-ops already parses and emit it for reserve cells.
The director's cell-status refuses a step-5 reserve cell with a 404
{error: cell_in_reserve_mode}, so every monitor sample failed while any
cell was reserve. Classify that exact refusal as reserve-not-probed, skip
the refused migration reads, and judge the cell on probes, logs and the
selector instead. Any other 404 still fails the sample.
A reserve cell keeps no activity rows and its ledger stamps last_activity_at, so the
stranded rule read its connected hosts as unserved and re-placed them.
* fix(mobile): recover a relay that died during an app switch in one short probe, and retry an attach timeout through the director once
Refs STA-11130.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(mobile): redial the same cell once on attach-timeout instead of re-resolving
Aligns with the relay step 6 design: reason string attach-timeout, no director
re-resolve (the director names the cell the host is seated on).
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(mobile): probe decisively only on a real background return, 4s, and jitter the attach-timeout redial
Co-Authored-By: Claude <noreply@anthropic.com>
* test(mobile): replace type assertions with a typed fake in the relay probe tests
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
The root moved to pnpm 12 in #17156 but these two isolated workspaces were
left on pnpm 10, so CI and deploys carried a second pnpm and pnpm/action-setup.
- Pin the root's integrity-checked pnpm 12.14.0 and regenerate both
lockfiles (only the pnpm 12 package-manager document is added; the
dependency graph is unchanged).
- Move settings pnpm 12 ignores out of .npmrc and package.json#pnpm into
each pnpm-workspace.yaml; onlyBuiltDependencies becomes allowBuilds, and
cloud keeps esbuild's script skipped as pnpm 10 did. package-manager-strict
is gone in pnpm 12, whose own version pinning supersedes it.
- docs/site drops --ignore-workspace: under pnpm 12 it skips the workspace
file, losing the overrides and failing the frozen install. vercel.json now
pins install/build so a stale dashboard command cannot reintroduce it.
- Cloud and docs workflows use pnpm/setup@v2 like the root; the relay
unit-test step uses the root's pnpm instead of npx pnpm@10.
* fix(relay): log controls a cell closes after host proof but before wiring
* fix(relay): log a pre-refusal hang-up as the hang-up and share printableCloseReason
* refactor(relay): move printableCloseReason into its own module to break the import cycle
* test(relay): assemble the fake pg URL at runtime so the secret scan passes
* fix(relay): judge the reserve watch's 503 rule on non-drain 503s, not a host's own re-dials
The watch counted every director 503, and on 2026-10-18 ~98% were host-rate-limited re-dials that
grow with traffic, so it flipped production-gce-c7 back twice with no fault. It now uses the roll
ladder's non-drain measure (request_count 503s less drain-return deferrals and host-rate-limited /
host-in-flight re-dials from the director's runtime metrics) for both the background and the live
samples. A judged minute no runtime-metrics line covers is unverifiable, not zero.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(relay): state the reserve watch's metrics coverage honestly and share the metrics read limit
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
The folder is the cloud relay that reaches phones; under its old name it read as the legacy SSH
relay. git mv plus import and path-literal rewrites only.
* fix(relay): destroy a pooled client whose session the server ended, and mark work that never ran
pg keeps calling a client queryable until its socket's end arrives, so a client released after
57P01 (pg_terminate_backend, shutdown) or a dropped socket went back to the pool and failed the
next caller's statement with 'Connection terminated unexpectedly'. Such clients are now released
with the error, so pg-pool discards them.
Also marks failures where no statement of the operation reached the server (acquire, a failed
BEGIN, a client dead at checkout) so a caller can tell work safe to re-run.
* fix(relay): skip ROLLBACK on an ended session, drop unused nothing-ran marking
Adds 57P04/57P05 to the session-ended codes and covers 08xxx, a dropped
socket, an ordinary 23505, and a real pg_terminate_backend.
Claude-Session: f2fe8cb5
* Monitor clock skew on every relay cell in production.tfvars
* test(cloud): run the clock-skew monitor workflow test in Cloud Verify
---------
Co-authored-by: Neil <neil@stably.ai>
* feat(relay): log what the sticky slot spends its hold on, per pooled connection
* fix(relay): time every sticky round trip at the PostgreSQL client, pool wait apart
* docs(relay): say which round trips the sticky slot tally covers
* feat(relay): watch a reserve cell and flip it back on the first step 5 stop rule
A manual, inert-until-dispatched workflow samples one reserve cell every minute for up to
300 minutes against the step 5 §9 stop rules, logs one JSON line per sample, and on the
first trip (or N unreadable samples in a row) runs the flag tool's own admitMode=db write
under the flag workflow's concurrency group. dry-run logs the flag tool's plan instead.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): keep the reserve watch's flip back out of the flag group, and baseline 503s before the flip
- The flip back holds no concurrency group: GitHub keeps one pending run per group and a
newer one cancels it. The flag tool's CAS orders it instead; a generation conflict is
re-read and retried (5 tries, backoff) while the switch file still says reserve.
- The 503 background comes from the 10 minutes before a required flipped-at, or an explicit
value; neither is refused.
- A failed or cancelled write-mode watch flips back too. An unreachable cell fails with the
exact break-glass inputs.
- Missing 503 data (empty series, unpublished minute, unparseable time) is unverifiable.
- Seat-behind-row key adds the director instance and cell; six segments share one command.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): read the reserve watch's 503 publishing front from every response code
A quiet fleet has no 503 series, so the watch read it as unverifiable and flipped a healthy
cell back within minutes. The 503 read now groups request_count by response code in one
query; the newest minute any code has a point for is how far the series is published, so a
503 minute at or before it with no point is zero and minutes past it are left out.
Also: the flip-back job's steps run on always(), flipped-at ages against the watch's start
in later segments, a pending pre-flip background no longer adds to the unverifiable streak,
the flip back retries runtime-status before calling for break glass, and the concurrency
rationale lives once in the docs.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): state what a cancel during the flip-back lease wait can leave, and the 503 publish-skew assumption
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): fail a segment closed on a corrupt watch state file, and keep no checkout token on disk
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): alarm on a failed flip back, refuse a watch under a rollout lease, wait for reserve to apply, and control the duplicate read
- A flip back whose write or plan never finished (no ID token, a failed login, a silent cell)
raises an ::error:: annotation and a FLIP BACK FAILED summary, as the clock-skew monitor does.
- The first segment reads the Cloud SQL rollout lease through the lease action's own reader and
refuses to start while any run holds it, or when it cannot be read.
- A cell not yet reporting reserve is waited on for up to 10 minutes, judging nothing, before the
watch ends refused without a write.
- The duplicate-seat read takes the per-minute ledger summary in the same query and window; with
no fresh summary in it the sample is unverifiable, not zero duplicates.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): page on a refused reserve watch or a failed flip back
A refused watch can leave a cell in reserve and unwatched, the same as a flip back that did not
land. Both now raise an escaped ::error:: annotation through one `alarm` command, and in write
mode write orca_relay_reserve_watch_alarm with gcloud logging write. A new log-based metric and
alert policy page on any such line, per cell, through relay_alert_notification_channels, and the
deploy identity gets roles/logging.logWriter in production to write it.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): name the log-writer risk, check the alarm's project, and say a failed login cannot page
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): note a force-cancel can skip the reserve-watch flip-back write
* Avoid unused module imports and trim cosmetic test coverage
* Capture hidden popup frames and attach CDP before guest creation
* Overlap relay test projects within the existing worker budget
* Consolidate native preparation and add an opt-in transform cache comparison
* Release inherited debugger sockets during final E2E teardown
* Refresh unit scheduling weights from all five qualified ARM shards
* Avoid retaining unasserted RPC call history in the delivery model
* Use the supported unconditional background wait syntax
* Overlap unconditional output upload with native preparation
* Verify relay integration stays enabled for ordinary unit callers
* feat(relay): split step-5 shadow compare and dry-run logs per cell
The 24 h per-cell shadow gate needs per-cell sticky-compare counts, per-cell
unexplained counts, and dry-run `full` refusals while the map showed room.
Both summary lines gain additive per-cell fields; existing fields are unchanged.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count a cell-mismatch on both cells and bound per-cell maps by the fleet
- countByCell (was cells) counts a compare on every cell it involves: the
database's cell, every seat's cell when there is no row, and both cells
for a cell-mismatch. Drop the 64-cell cap and cellsOverflow.
- Add ReservePlacer.free and use it for the dry-run free share.
- Rename the room bound to RESERVE_DRY_RUN_ROOM_BOUND.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count a duplicate-seat compare on every seat cell
An unexplained duplicate seat counted only on the database's cell, so a cell
whose stale feed kept a second seat could pass its own step-5 check. Count it
once on every seat cell, the same as a cell-mismatch.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count explained cell-mismatch and duplicate-seat on the database cell only
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): key per-cell counts on prototype-free records
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): explain a duplicate seat left by a just-superseded half-open control
When the database names the newest seat exactly, every other seat has a
strictly lower epoch, and the newest seat joined within the cell's silence
close (timeout + one ping interval + the map lag bound), the older seat is
a half-open control awaiting its 75 s pong-silence close, not a fault.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): bound the superseded duplicate-seat rule on the move's age
A same-epoch rejoin resets joinedAt, so a stale older seat could stay
explained forever; the director-clock sighting of the epoch moves only
when the epoch rises.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): leave a superseded duplicate seat unexplained without an epoch sighting
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): explain a superseded duplicate seat only from a prior sighting
compare() recorded the epoch before classifying, so a first-ever sighting
(restart or eviction) read as a fresh move. Classify first and take the
move's age from the record that existed before this compare.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): also bound a superseded duplicate seat on its join age
A long-ago lower-epoch sighting dates the move to this compare, so the
current seat's join age bounds it too.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count an ageless superseded duplicate seat as its own class
A superseded-shaped duplicate-seat with no prior sighting of the host (the
first compare since a restart) has no move age, so it is now
`superseded-unknown-age` (explained: false) rather than a plain duplicate.
The count line also carries `uptimeMs`, so the gate can discount that
class in an instance's first 100 s.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): limit superseded-unknown-age to an instance's first 100 s
Flushes are lazy, so a window has no upper bound and an uptime field on the
count line cannot date it. The compare now records when it started and
emits superseded-unknown-age only within the superseded close bound of
that; later, a missing sighting means another instance served the host or
the sighting was evicted, so it is a plain unexplained duplicate-seat.
Drops the uptimeMs field.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): say where a missing sighting comes from after the first 100 s
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* perf(ci): reduce test overhead and reuse expensive build work
* test(ci): keep worker startup outside failure-case deadlines
* test: isolate push fixture advisory locks
* test(ci): preserve fixed drain retry allowance
* Align CI contracts with current triggers and guide content
* Stabilize consolidated reveal scenarios and retain behavioral CI contracts
* Fix upstream lint blockers while retaining behavior and assertions
* Classify global Jira preferences in the WSL migration inventory
* Align Jira test fixtures with startup and build safety checks
* Include package-test routing dependency in CI checkouts and fixtures
* Keep native stream classes intact in transcript dependency tests
Jinwoo Hong
Neil
Žan MlakarandNeil