* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* feat(relay-ops): watch a memory phone cell and turn its phone path off on a breach
The reserve watch now also judges five phone rules while the switch file says
phonePath=memory, from the cell's own runtime metrics against the 10 minutes before
flipped-at: 4401s, confirm failures, attach-deadline 4404s and accept database errors
over their lines for 3 minutes, and completed accepts under half of background. A
phone-rule trip writes phonePath=db only; any other trip still flips the cell back
whole. Resume audit drops page after 5 minutes and flip nothing.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
* fix(relay-ops): keep phone watch rules off traffic noise, and keep watching after a phone trip
Completed accepts are no longer judged. Rate lines use the background mean plus
max(margin, share, 4 Poisson deviations), with a margin of 3 for attach deadlines. A
phone trip writes phonePath=db mid-watch and the reserve rules go on; both flip backs
remove the phonePath key so a cell rolled back to an image without it takes the write.
The phone background starts where phonePath went memory, and a background broken by a
restart refuses the watch. Audit drops page only while rising; failing renewals page too.
* test(relay-ops): replay the phone rules on 26 recorded hours of c7 and c29
Per-minute counts from the cells' runtime metrics (read-only, 10-18/19) replace the
synthetic quiet window. c29's bursty auth failures tripped the attach-deadline rule at a
margin of 3 in 29% of 300-minute watches, so its margin is now 5 like the others.
* fix(relay): judge the phone rules as a share of attempts, retry a failed phonePath write
The count rule false-tripped busy c28 (33/121 five-hour watches). The share rule (over half of
the minute's attempts, 0.25 over the background, capped at 0.6) trips none of the recorded c7,
c28 and c29 watches and catches two refusals per accept within 3 minutes. A failed mid-watch
phonePath=db write now keeps watching, retries each sample and pages after 3 failures; break
glass also drops phonePath.
Co-Authored-By: Claude <noreply@anthropic.com>
* docs(relay): cite the replayed c28 count-rule false trips
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(relay): judge phone confirm failures against completed accepts
Each completed accept makes one confirm, so failures/(failures + accepts) could never pass half
and a total confirm outage never tripped. Confirm failures are now a share of accepts.
Co-Authored-By: Claude <noreply@anthropic.com>
* test(relay): note why the confirm replay tests only trips
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* feat(relay): add the per-cell phonePath switch, effective only in reserve mode
Step 6 of the relay redesign keys every phone-path change on this switch. The cell
reports the effective value in runtime status, and the flag tool refuses phonePath=memory
unless the cell is reserve, and a reserve flip that would turn phones over unnamed.
* fix(relay-ops): store phonePath=db as no key, so old images keep taking writes
An explicit phonePath=db left in the switch file would make the flag tool refuse every
later write after an image rollback, including a watcher's admitMode=db flip back.
A reserve cell's dead-man is fed only by director polls through the load balancer. With SQL
readiness the balancer drops the cell during a database outage, the polls get 503
failed_to_pick_backend, and the cell trips (staging drill 2026-10-11, c2 at 11:47:59Z).
admitMode=reserve now needs readinessLocal=true in the object being written, already there
or set in the same write, and a reserve cell refuses readinessLocal=false unless the same
write flips it back to db.
* fix(relay): the cell switch tool changes only the named switches and checks every key on read-back
A cell resets a key missing from its switch file to its default, so the tool now reads the
current object and changes only what --set names. The read-back compares every known key and
fails when the cell reports ignoredKeys.
* fix(relay): the cell switch tool refuses a switch the cell's image does not support
It reads supportedFlags from runtime-status (#27057). An image from before that field takes only
readinessLocal and ticketCheck off/shadow, so ticketCheck=enforce cannot void an old cell's object.
* fix(relay): the switch tool validates the whole object, refuses voided files, and records admitMode in Postgres
Every key and value of the object to be written is checked against what the image accepts (an
image without the list: only the keys it applies, ticketCheck off/shadow). A file the cell did
not apply is refused. admitMode=reserve needs a typed RESERVE <cell>; Postgres records reserve
before the file is written, and db only after the cell reports every control leased again.
* feat(relay): the switch tool knows reregisterInFlight (1-16), the flip-back pace
* fix(relay): the switch tool closes the cache gap before reserve, undoes a refused write, and has a break-glass db flip
After recording reserve in Postgres it waits past every director's 5 s cache and records it
again (re-running the open-flow check) before writing the file. A file write refused outright
(412 or 4xx) puts Postgres back to db; after a 5xx it stays reserve. An object the cell did not
apply is repaired when the new object passes the image check. Break glass (typed BREAK GLASS
<cell>) records db for a dead or unreachable reserve cell without its read-back.
* fix(relay): a director deploy refuses a traffic move across reserve-mode support while any cell is reserve
* fix(relay): the flip-back wait scales with the cell's controls and pace, and says how to finish
* fix(relay): the switch tool refuses to re-arm a tripped cell, breaks glass only on a silent one, and undoes reserve if the second post is refused
* feat(relay): director deploys set reserve placement, and refuse turning it off while any cell is reserve
* fix(relay): Terraform keeps the deploy-owned placement and shadow cells, and the deploy guard reads placement as built and refuses dropping a reserve cell from the shadow list
* fix(relay): director deploys take --admin-audience alone, name a drifted deploy identity, and carry the shadow seat-feed cells
Staging passes --admin-audience for the reserve guard without durable rehome
verification, which the parser refused. Only the generation and control
origin now travel together, and they still need the audience.
The guard decodes the caller from its token and refuses, by name, a revision
whose ORCA_RELAY_DEPLOY_SERVICE_ACCOUNT is another account, instead of
reading into a 401.
--shadow-seat-feed-cells (preserve, all, or a cell list) is a workflow input
on both director deploys; preserve keeps the serving value. A contract test
parses every workflow's real argv for both deploy scripts.
* fix(relay): the flag tool paces a flip back from the pool size the cell reports
An image from before the field falls back to the deployed sizes.
* feat(relay): one-cell flag write workflow with dry run, compare-and-swap and read-back
Operate Relay Production Cell Flags writes cells/<cellId>.json for one
cell: it checks the origin answers as that cell with a flag channel,
refuses a stale expected generation, writes with ifGenerationMatch,
stamps the actor and run on the object, and waits up to 15 s for the
cell's runtime status to report the new generation applied. The deploy
Workload Identity condition admits the new workflow.
* fix(relay): flag workflow retries a transient 5xx, shows the plan before the generation check, and names the failure
Runtime-status reads go through the shared one-retry admin helper, and a
failed read-back poll no longer ends the wait early. The plan line is
logged before the generation check, a failure is logged as one JSON line
the job summary carries, and the object and summary name the triggering
actor.
Jinwoo HongandClaude