Reserve-aware cell status for the director: reserve cells report their own runtime and seat feed, drain causes are attributed per cell, and paced 503s on a host leaving a draining cell are recorded as the drain's.
Rehome on reconnect for reserve-mode cells: a desktop reconnecting on a reserve cell outside its preferred region is placed fresh in that region, only while the region has at least 10% of seats free, at most once a day per desktop, rate-limited, and switched off by default (ORCA_RELAY_RESERVE_REHOME_PERCENT=0); the regional rehome switch stops it.
Before refusing a placement with relay_capacity_exhausted, the director tries a reserve seat in the desktop's own region, then the other region, through the cell's own reserve API; if every region paces it answers the shortest Retry-After.
* fix(relay): count region selection on every assign lane, fallback only for new placements
Co-Authored-By: Claude <noreply@anthropic.com>
* test(relay): pin placed marker on isolated re-place and kept pins; drain-return fallback
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* test: remove redundant cases and repair behavioral assertions
* docs: reconcile audit records with current main
* fix(deps): apply security updates from #32101
Apply the six-file DOMPurify 3.4.17 and Next.js 16.4.1 patch from PR #32101 to resolve the dependency security gate for this test audit. Original patch by Brennan Benson.
* test: satisfy changed-line quality gates
Add required braces and replace filter indexing with find. Rerun both CI quality gates and the 27 affected relay tests; refresh audit hashes and record the corrections.
---------
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
Co-authored-by: Brennan Benson <79079362+brennanb2025@users.noreply.github.com>
* fix(relay): repair cell seat-log drift, and keep a new session's generation above a listed seat
Directors mirror each cell's seat feed, so the feed must list exactly the cell's live sessions.
The cell now sweeps its seat log against its sessions on a director's poll: a seat no session
backs is left (4409) and the live session listed, and a live session the feed lost is listed
again, each once two sweeps in a row find it. A demotion naming an unbacked seat repairs it at
once. A host's new session now starts above any generation the feed still lists, which closes
the drained-session reconnect race that removed a live seat.
Claude-Session: asia-drain-pulse
* fix(relay): refuse a rebind whose session left the map during activation, and keep a per-host generation floor
A rebind whose database activation outlived the orphan grace (or a drain) revived the removed
session outside the map: connected, still in the seat feed, and unreachable by phones or
demotion for as long as the host stayed up. Activation now rechecks the map after its awaits and
refuses with PEER_DROPPED so the host redials. Generations come from a per-host floor that
outlives the session, and an activation at a generation already used here is refused.
Claude-Session: asia-drain-pulse
* fix(relay): log each demote outcome on the director by host digest
The director now reads the cell's existing {v:1, outcome} demote reply and logs it; the cell's
feed stays the only source of its map. The supersede line now reaches the log as well.
Claude-Session: asia-drain-pulse
* fix(ci): recover verified pnpm setup from registry metadata outages
* fix(ci): prepare native pnpm shims after fallback verification
* fix(ci): provide Node for pnpm fallback in bare containers
* chore(cloud): update relay runtimes and security scanning
* docs(cloud): record Go binary advisory reachability
* chore(cloud): rerun security checks after resolving main conflict
* fix(relay): release a drained host before telling it to leave
A zero-grace drain sent `drain` and closed with 4503 in the same tick, then
released the host's control lease. The desktop redials the director as soon
as it reads either, and from asia-east2 the release holds the host's
assignment row for ~1 s, so the redial met it and was refused row_busy.
The release now runs first and the host is told once it lands, bounded at
2 s; a failed or stuck release still closes. Wire frames are unchanged.
* fix(relay): give a drained host's release 3 s and log when it runs out
One host's releases run one after another, so 3 s covers a control lease
plus one open phone connection from asia-east2.
* style(relay): brace the drain release branches
* fix(relay-ops): make each reserve-watch sample one Cloud Logging read
The project allows 60 ListLogEntries a minute shared by every watch and tool. Each watch sample
made four to six reads (two director runtime-metrics sub-windows, the ledger summaries, the
seat-behind-row lines, the cell's phone metrics, and since #31484 the 503 request log), so on
10-21 18:39Z twelve watches drew 429s until c21 and c22 tripped unverifiable.
A sample now sends its queries as one OR filter and splits the lines back client-side. The
per-instance 503 split counts each director's own logged /v1/assign 503s from the runtime
metrics it already reads, with the rest of the fleet's count as one unattributed instance,
instead of reading the request log.
* fix(relay-ops): judge unattributed 503s on their 3-minute sum against the whole fleet line
* docs(relay-ops): state that an instance that stops logging trips the unattributed rule
* fix(relay-ops): keep one stalled director instance from tripping the reserve watch's 503 rule
The reserve watch's director503 rule counted fleet-wide non-drain 503s. On 10-21 07:55Z one
director instance's database calls stalled for 3 minutes; its 503s alone carried the fleet over
the line and all four watched cells flipped back to db.
Over the fleet line the rule now also needs at least 2 director instances each above a fifth of
it for the same 3 minutes, counted from Cloud Run's request log by labels.instanceId less each
instance's scheduled 503s. Above max(10x background, 200) for 3 minutes it trips with no split.
* fix(relay-ops): size the instance share to the directors seen and count platform 503s
* fix(relay-ops): size the instance share per judged minute and refuse a missing count
The deploy passed `--max`, which is gcloud's service-wide cap. Cloud Run divides
that cap among the revisions taking traffic, so during the 20/80 stage a
candidate under a cap of 5 got one instance while the serving floor kept four,
and the spread check could never see the full floor (deploy M, 1 of 5 for six
minutes). Candidates now pin `--max-instances` per revision, and the deploy
resets a service cap below two full revisions to Cloud Run's default before it
creates any candidate. The rollout overlap of two full directors is already in
the checked Cloud SQL connection budget.
A freshly restarted cell can be ready while the global load balancer still
answers 503 for it. The probe retried once after 2 s, so c10's 2026-10-20
roll failed after its drain and was left isolated. Retry wrapped source
5xx every 10 s for up to 2 minutes.
Co-authored-by: E2E Test <e2e@test.local>
* fix(relay): move director traffic in two steps so a deploy does not land on one instance
Cloud Run routes a new director revision's floor instances their first request
roughly 60-110 s after they start, while a traffic move reaches clients in
15-30 s. The one-step 100% move therefore put every assignment on the single
instance started with the revision, which sits at its own admission limit at
peak; the 2026-10-20 deploy returned ~550 assign 503s in two minutes that way.
With a floor above one, the deploy now moves 20% first, waits until Cloud Run's
request log shows every floor instance serving (six-minute deadline), then
moves the rest. A timeout or failure at that stage returns all traffic to the
previous serving revision, which still holds its warm floor. The staging deploy
identity gains log read access for the same check.
* fix(relay): bound the director spread log read, log the instances seen at timeout, and document split recovery
A hung gcloud log read could hold the 20/80 split past the six-minute deadline;
each read now times out after 30 s and counts as an unread poll. A timeout logs
the distinct instances seen, so a partial share packed onto fewer instances than
the floor shows up. The runbook covers restoring a split left by a cancelled job
and the missing selector-rollback tag after a timed-out deploy.
* fix(relay): give region-correction reconnects and roll re-placements the in-region reserve fallback
Every desktop session open carries a region-correction request, and those
skipped the reserve plan entirely, so with the region's database cells full they
still spilled abroad while an in-region reserve cell had room. The plan now runs
for them in a mode that never answers itself (the database lane must reply so the
window exchange follows) and offers only the in-region reserve fallback.
A host re-placed off a roll-isolated cell went straight to the other-regions tier
when its own region's tier was full. With an in-region reserve fallback it now
books that first, and the roll canary event is emitted for the reserve booking.
* fix(relay): never pace a roll drain on reserve intake, and count every reserve move off an isolated cell
A host leaving a roll-isolated cell whose region was full could get a paced 503
from a throttled reserve cell, stretching the drain and re-running the failed
in-region transaction on each retry. For those hosts a pace is now no room, and
they spill as before. Sticky now says when it declined an isolated pin, so a
share-draw booking off that cell also emits the roll canary event.
* fix(relay): keep a host in its region when the database lane is full and a reserve cell has room
A host that lost the reserve share draw went to the database lane, which
spills to another region when its own region's database cells are full,
even while an in-region reserve cell had free seats. The database lane now
requires the region when an in-region reserve cell has room; if the region
is full it books that reserve cell, and only spills when neither has room.
* fix(relay): pace a throttled in-region reserve cell instead of spilling, and decide reserve room at plan time
With the region's database cells full, a reserve cell with seats but no intake
budget fell through the placer's database-cells shortcut and the host went
abroad. That placement now answers pace (503 with Retry-After). regionReserve
is a single callback, present only while an in-region reserve cell has a seat.
* fix(relay): pace only on a reserve cell that lacked budget, not one that refused
A cell that refused this placement (full, unreachable, seated-newer) got its
token back and drove a 1 s Retry-After forever while the map still showed a
seat. Pacing now skips cells refused for anything but intake, so those
placements fall to the database lane's cross-region fallback as before.
evaluateWatchSample's two early returns (waiting for reserve to apply, and
a cell flipped back to db by hand) carried no pages, so watchSegment threw
'result.pages is not iterable' and the run ended as failed instead of
stopped (c29, 10-20 02:25Z, run 39354895994).
* perf(push): wake delivery worker after notifications are queued
* perf(push): wake one drain per send so bursts cost O(sends + drains) claims
A wake started all twelve drains, so each send arriving at an idle worker cost
~13 claim transactions. A wake now starts one drain; the poll tops up to the
full drain set.
A host control request that failed for an unexpected reason sent the thrown
error's message to the desktop as the error code. During a staging database
outage the desktop received Postgres's own text, and node-pg messages can
carry the database connection URL.
The relay now sends store codes and this codebase's snake_case invariant
slugs unchanged, and maps everything else to database_transient or
control_operation_failed. The original is logged through the existing
redaction (now shared with the background-operation logger), so only the
slug shape and SQLSTATE leave the process.
The load generator only opened invite splices, which never reach the device-resume accept or the
desktop's resume confirm. --resume-phones N installs one device credential per phone over an
invite splice, then reconnects it every --resume-interval-seconds: accept, device-resume-confirm,
and a held splice with echoes. Each attempt is a relay_load_resume_attempt line with accept and
confirm ms; failures are recorded, never fatal, so a database-block drill can run under it.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
Co-authored-by: E2E Test <e2e@test.local>
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay): resume paired phones from one device read on memory cells
With phonePath=memory in reserve mode, a phone resume is admitted from one device
read and the cell's own seat, and takes no activity leases. A refused device closes
4401, a failed read 4503, and only a missing device row falls through to the invite
path. A phone whose host has no seat gets one assignment read: 4409 when the row
names another cell or the read fails, otherwise 4404.
* feat(relay): keep a memory resume's basis in the cell and confirm it from one read
A phone resumed on the memory path keeps its connection basis in the cell, not in
relay_connection_bases. Its confirm rereads the device row once, renews it only when
less than 29 days remain, and always answers a current-version confirm renewed:true
with the expiry the database holds. Confirms route by where their basis lives, so a
flip between accept and confirm completes either way. Retries answer from a 10-min
in-memory result map.
* test(relay): roll a memory phone cell back with confirms in flight
Flipping back by admitMode=db, phonePath=db or a dead-man trip leaves in-flight
confirms to finish on their memory basis, writes no phone lease and leaks none, and
evacuation still refuses the cell while Postgres records it as reserve.
* feat(relay): batch a memory cell's resume-confirmed audit rows
Memory confirms buffer their audit rows and the cell writes them as one multi-row
insert every 5 s or 500 rows, idempotent by a per-event id. The buffer holds at most
5,000 rows and drops a row older than 10 minutes rather than retrying it, counting
and logging both kinds of drop.
* feat(relay): count and log phone-path failures, and every attach deadline
The runtime metrics line gains phone refusals by close code, attach deadlines, accept
database errors, resume confirm failures by code, completed accepts by phone path and
the resume audit drop total. Every attach-deadline 4404 now logs what the cell knew of
the desktop's control (pong age, ping outstanding, control age and round trip), so a
half-open control can be told from a slow data dial.
* feat(relay): close an attach-deadline 4404 with the attach-timeout reason
The phone close reason from #30607 now rides the close frame of the 4404 a cell
sends when the desktop's data leg misses the attach deadline, so OTA phones can
redial the same cell once instead of waiting out the host-offline delay. The failed
relay-hello frame is unchanged, and old phones ignore the reason.
* test(relay): drill a memory cell through a database outage, and pin phone renewal
A blocked database leaves splices forwarding, closes new resumes 4503 and fails
confirms cleanly, lets an in-flight confirm finish on its memory basis after a flip
back, and drops audit rows past 10 minutes instead of writing them late. On the phone,
a relay-only session lives 31 days under the once-a-day renewal rule, and would not
if a not-due confirm answered renewed:false.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay): log a failed no-seat read, recheck the switch after the device read
A failed assignment read for a phone with no seat now logs a rate-limited line before
its 4409. A switch that turned off during the device read sends the phone down today's
path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy.
* fix(relay): log a failed memory renewal; test flips, replays and basis cleanup
A renewal write that fails still answers the stored expiry, but now logs a rate-limited
line so a persistently failing update is visible before phones age toward re-pairing.
* test(relay): assert a rolled-back memory splice issues no statement when it closes
* fix(relay): flush resume audit early only on crossing the batch size, drop the shutdown flush
A failing database no longer takes a 500-row insert per confirm; retries wait for the
5 s timer. The shutdown flush raced the pool closing, so it is gone and the at most 5 s
loss is documented. A stopped buffer no longer re-arms its timer.
* test(relay): assert the aged audit rows were dropped, not written
---------
Co-authored-by: E2E Test <e2e@test.local>
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay): resume paired phones from one device read on memory cells
With phonePath=memory in reserve mode, a phone resume is admitted from one device
read and the cell's own seat, and takes no activity leases. A refused device closes
4401, a failed read 4503, and only a missing device row falls through to the invite
path. A phone whose host has no seat gets one assignment read: 4409 when the row
names another cell or the read fails, otherwise 4404.
* feat(relay): keep a memory resume's basis in the cell and confirm it from one read
A phone resumed on the memory path keeps its connection basis in the cell, not in
relay_connection_bases. Its confirm rereads the device row once, renews it only when
less than 29 days remain, and always answers a current-version confirm renewed:true
with the expiry the database holds. Confirms route by where their basis lives, so a
flip between accept and confirm completes either way. Retries answer from a 10-min
in-memory result map.
* test(relay): roll a memory phone cell back with confirms in flight
Flipping back by admitMode=db, phonePath=db or a dead-man trip leaves in-flight
confirms to finish on their memory basis, writes no phone lease and leaks none, and
evacuation still refuses the cell while Postgres records it as reserve.
* feat(relay): batch a memory cell's resume-confirmed audit rows
Memory confirms buffer their audit rows and the cell writes them as one multi-row
insert every 5 s or 500 rows, idempotent by a per-event id. The buffer holds at most
5,000 rows and drops a row older than 10 minutes rather than retrying it, counting
and logging both kinds of drop.
* feat(relay): count and log phone-path failures, and every attach deadline
The runtime metrics line gains phone refusals by close code, attach deadlines, accept
database errors, resume confirm failures by code, completed accepts by phone path and
the resume audit drop total. Every attach-deadline 4404 now logs what the cell knew of
the desktop's control (pong age, ping outstanding, control age and round trip), so a
half-open control can be told from a slow data dial.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay): log a failed no-seat read, recheck the switch after the device read
A failed assignment read for a phone with no seat now logs a rate-limited line before
its 4409. A switch that turned off during the device read sends the phone down today's
path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy.
* fix(relay): log a failed memory renewal; test flips, replays and basis cleanup
A renewal write that fails still answers the stored expiry, but now logs a rate-limited
line so a persistently failing update is visible before phones age toward re-pairing.
* test(relay): assert a rolled-back memory splice issues no statement when it closes
* fix(relay): flush resume audit early only on crossing the batch size, drop the shutdown flush
A failing database no longer takes a 500-row insert per confirm; retries wait for the
5 s timer. The shutdown flush raced the pool closing, so it is gone and the at most 5 s
loss is documented. A stopped buffer no longer re-arms its timer.
* fix(relay-ops): let same-cap rolls run while another cell is reserve; refuse a reserve roll target by name
The director refuses cell-status for a step-5 reserve cell with a 404
{error: cell_in_reserve_mode}. The same-cap headroom check read it for
every general cell, so any roll failed while c7 was reserve. A reserve
cell that is not being rolled now counts as no free slots (its seats are
not in the database) and is listed with reserve: true. A reserve roll
target, in the headroom check, the same-cap job's first cell-status read
and the candidate tool, now fails with the design's procedure: flip it to
admitMode=db first, roll, then flip it back. Any other 404 still fails.
* fix(relay-ops): pin the headroom gate's imports as evidence code; name both cells in a two-cell reserve refusal
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay): resume paired phones from one device read on memory cells
With phonePath=memory in reserve mode, a phone resume is admitted from one device
read and the cell's own seat, and takes no activity leases. A refused device closes
4401, a failed read 4503, and only a missing device row falls through to the invite
path. A phone whose host has no seat gets one assignment read: 4409 when the row
names another cell or the read fails, otherwise 4404.
* feat(relay): keep a memory resume's basis in the cell and confirm it from one read
A phone resumed on the memory path keeps its connection basis in the cell, not in
relay_connection_bases. Its confirm rereads the device row once, renews it only when
less than 29 days remain, and always answers a current-version confirm renewed:true
with the expiry the database holds. Confirms route by where their basis lives, so a
flip between accept and confirm completes either way. Retries answer from a 10-min
in-memory result map.
* test(relay): roll a memory phone cell back with confirms in flight
Flipping back by admitMode=db, phonePath=db or a dead-man trip leaves in-flight
confirms to finish on their memory basis, writes no phone lease and leaks none, and
evacuation still refuses the cell while Postgres records it as reserve.
* feat(relay): batch a memory cell's resume-confirmed audit rows
Memory confirms buffer their audit rows and the cell writes them as one multi-row
insert every 5 s or 500 rows, idempotent by a per-event id. The buffer holds at most
5,000 rows and drops a row older than 10 minutes rather than retrying it, counting
and logging both kinds of drop.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay): log a failed no-seat read, recheck the switch after the device read
A failed assignment read for a phone with no seat now logs a rate-limited line before
its 4409. A switch that turned off during the device read sends the phone down today's
path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy.
* fix(relay): log a failed memory renewal; test flips, replays and basis cleanup
A renewal write that fails still answers the stored expiry, but now logs a rate-limited
line so a persistently failing update is visible before phones age toward re-pairing.
* test(relay): assert a rolled-back memory splice issues no statement when it closes
* fix(relay): flush resume audit early only on crossing the batch size, drop the shutdown flush
A failing database no longer takes a 500-row insert per confirm; retries wait for the
5 s timer. The shutdown flush raced the pool closing, so it is gone and the at most 5 s
loss is documented. A stopped buffer no longer re-arms its timer.
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay): resume paired phones from one device read on memory cells
With phonePath=memory in reserve mode, a phone resume is admitted from one device
read and the cell's own seat, and takes no activity leases. A refused device closes
4401, a failed read 4503, and only a missing device row falls through to the invite
path. A phone whose host has no seat gets one assignment read: 4409 when the row
names another cell or the read fails, otherwise 4404.
* feat(relay): keep a memory resume's basis in the cell and confirm it from one read
A phone resumed on the memory path keeps its connection basis in the cell, not in
relay_connection_bases. Its confirm rereads the device row once, renews it only when
less than 29 days remain, and always answers a current-version confirm renewed:true
with the expiry the database holds. Confirms route by where their basis lives, so a
flip between accept and confirm completes either way. Retries answer from a 10-min
in-memory result map.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay): log a failed no-seat read, recheck the switch after the device read
A failed assignment read for a phone with no seat now logs a rate-limited line before
its 4409. A switch that turned off during the device read sends the phone down today's
path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy.
* fix(relay): log a failed memory renewal; test flips, replays and basis cleanup
A renewal write that fails still answers the stored expiry, but now logs a rate-limited
line so a persistently failing update is visible before phones age toward re-pairing.
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay-ops): watch a memory phone cell and turn its phone path off on a breach
The reserve watch now also judges five phone rules while the switch file says
phonePath=memory, from the cell's own runtime metrics against the 10 minutes before
flipped-at: 4401s, confirm failures, attach-deadline 4404s and accept database errors
over their lines for 3 minutes, and completed accepts under half of background. A
phone-rule trip writes phonePath=db only; any other trip still flips the cell back
whole. Resume audit drops page after 5 minutes and flip nothing.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay-ops): keep phone watch rules off traffic noise, and keep watching after a phone trip
Completed accepts are no longer judged. Rate lines use the background mean plus
max(margin, share, 4 Poisson deviations), with a margin of 3 for attach deadlines. A
phone trip writes phonePath=db mid-watch and the reserve rules go on; both flip backs
remove the phonePath key so a cell rolled back to an image without it takes the write.
The phone background starts where phonePath went memory, and a background broken by a
restart refuses the watch. Audit drops page only while rising; failing renewals page too.
* test(relay-ops): replay the phone rules on 26 recorded hours of c7 and c29
Per-minute counts from the cells' runtime metrics (read-only, 10-18/19) replace the
synthetic quiet window. c29's bursty auth failures tripped the attach-deadline rule at a
margin of 3 in 29% of 300-minute watches, so its margin is now 5 like the others.
* fix(relay): judge the phone rules as a share of attempts, retry a failed phonePath write
The count rule false-tripped busy c28 (33/121 five-hour watches). The share rule (over half of
the minute's attempts, 0.25 over the background, capped at 0.6) trips none of the recorded c7,
c28 and c29 watches and catches two refusals per accept within 3 minutes. A failed mid-watch
phonePath=db write now keeps watching, retries each sample and pages after 3 failures; break
glass also drops phonePath.
Co-Authored-By: Claude <noreply@anthropic.com>
* docs(relay): cite the replayed c28 count-rule false trips
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(relay): judge phone confirm failures against completed accepts
Each completed accept makes one confirm, so failures/(failures + accepts) could never pass half
and a total confirm outage never tripped. Confirm failures are now a share of accepts.
Co-Authored-By: Claude <noreply@anthropic.com>
* test(relay): note why the confirm replay tests only trips
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay): resume paired phones from one device read on memory cells
With phonePath=memory in reserve mode, a phone resume is admitted from one device
read and the cell's own seat, and takes no activity leases. A refused device closes
4401, a failed read 4503, and only a missing device row falls through to the invite
path. A phone whose host has no seat gets one assignment read: 4409 when the row
names another cell or the read fails, otherwise 4404.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay): log a failed no-seat read, recheck the switch after the device read
A failed assignment read for a phone with no seat now logs a rate-limited line before
its 4409. A switch that turned off during the device read sends the phone down today's
path. resolveResume reuses the one-read lookup, so the expiry and grace rules have one copy.
* test(relay): census every activity-lease reader for phones without leases
Step 6 lets a reserve cell accept phones without writing confirmation or
splice leases. This ratchet classifies every reader of the lease table by why
that is safe, and pins the guards that keep completion and idle rehome off a
reserve cell whose host holds no source lease.
* test(relay): scan subdirectories and USING reads, and ban a bare lease table name
The ratchet now walks every production file under src, counts USING as a read, and
fails on a bare table-name value, which could reach an interpolated FROM it cannot see.
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
A reserve cell's cell-status read is refused, so it carried no
connection_hard_cap and the evaluator fell back to the 500 default:
run 39225932592 froze on reserve c7 at 731 connections under its
1000 cap. Read connection_hard_cap from the durable Terraform cells
relay-ops already parses and emit it for reserve cells.
The director's cell-status refuses a step-5 reserve cell with a 404
{error: cell_in_reserve_mode}, so every monitor sample failed while any
cell was reserve. Classify that exact refusal as reserve-not-probed, skip
the refused migration reads, and judge the cell on probes, logs and the
selector instead. Any other 404 still fails the sample.
A reserve cell keeps no activity rows and its ledger stamps last_activity_at, so the
stranded rule read its connected hosts as unserved and re-placed them.
* fix(mobile): recover a relay that died during an app switch in one short probe, and retry an attach timeout through the director once
Refs STA-11130.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(mobile): redial the same cell once on attach-timeout instead of re-resolving
Aligns with the relay step 6 design: reason string attach-timeout, no director
re-resolve (the director names the cell the host is seated on).
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(mobile): probe decisively only on a real background return, 4s, and jitter the attach-timeout redial
Co-Authored-By: Claude <noreply@anthropic.com>
* test(mobile): replace type assertions with a typed fake in the relay probe tests
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
OrcaWinandOrcaWin
Jinwoo Hong
Neil
