* test(relay): check invariant 4 (one demotion winner) in the step-5 simulation
Every demotion now names its winner (the newer booking, or the row). The model
flags a director that names a different winner for a divergence another already
named, and a host whose two seats are each demoted for the other. A fault where
directors break level ties by their own cell order shows the check fires.
Violations are capped per invariant so one flood cannot hide another class.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* test(relay): bound unseen duplicates by the silence close and assert all invariants on contested ties
A reserve seat the booking director's map had not seen (joined, then half-open within a poll)
is never superseded and has no lease to re-check, so it lasts until the cell's control silence
close (75 s + one 15 s ping). Invariant 3 now uses that bound for such seats, and the
level-divergence config asserts every invariant.
Stale maps now judge a divergence another director already demoted, so the config's
"contested" counter (divergences judged by two or more directors) is the case invariant 4
guards. The silence-close constant now matches production (90 s, not 100 s).
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* test(relay): raise the floor on map-judged demotes and count only level contests
A reconcile whose map has not seen another director's demotion still demotes in
production, which raises its epoch floor; the model now does too. The contested
counter counts only same-epoch pairs, the only ones invariant 4 can split. The
keep-this-seat rule is one helper.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* refactor(relay): name the simulation's fault-kept seat check for the fault
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): count only confirmed level demotions and every second judgment, and bound duplicates by placement records alone
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* feat(relay): declare Asia spare cell c35 as migration-only
Adds a seventh asia-east2 cell at the C34 shape (cap 3000, 6000 request
units, pool 16, e2-standard-4) in asia-east2-a, pinned to image F. It gets
its own topology and registration wave but no promotion wave, so the
admission script and workflow refuse to promote it; it stays a
migration-only spare and out of the fleet pool list. Mirrors #25336.
* test(relay): accept the c35 spare under the same-cap capacity allowlist
Staging proved both on image F (2026-10-11): the sidecar keeps the director's
readiness through a Cloud SQL outage, and the event-loop stall alert reads
eventLoopDelayMsMax. Production adopts the shape on its serving image first.
* chore(relay): move the staging cells onto image F
Staging c1, c2 and c4-c7 run image F (main 3c1141f2), so the director and
every staging cell share one digest for the c5-c7 promotion and the
six-cell dead-cell rerun. c3 is unchanged.
* test(relay): pin the staging C4 digest to image F
* fix(relay): keep verifying host tokens from the last served keys while auth is down
jose refetches the relay JWKS once its cache is 10 min old and rejects the token when that
fetch fails, so an auth outage turned every still-valid host token into a 4401. The cell now
falls back to the last set auth served for up to an hour, skips refetching for 30 s after a
failure, and still rejects any key that set never had. A fresh set replaces it as soon as auth
answers, so a rotation takes effect at the first successful fetch.
* fix(relay): bound stale relay keys at 30 min and document the emergency purge
Thirty minutes still covers a full auth outage (no token the stale keys could accept outlives
~16 min of one) and halves how long a rotated-out key stays trusted on a cell kept from auth.
A restarted process holds no set and refuses host tokens until a fetch succeeds; the ops doc
names that as the purge after a signing-key leak, and a test pins the fail-closed start.
* test(relay): pin the 30 min stale relay key bound (29 min accepts, 31 min rejects)
* fix(relay): a resumed host control adopts the token its hello proved
A rebind replaced the socket, lease and epoch but kept the predecessor's token claims, so a
host that resumed after its token lapsed (auth down past expiry plus grace) was closed 4401
"relay authorization expired" at the next heartbeat, 15 s later, and every resume repeated it.
The rebind now takes the hello's verified claims when they carry a later expiry, and a hello
that speaks for another profile or organization starts a new generation instead of resuming.
* fix(relay): a refresh never lowers the held expiry and closes only its own socket
An auth refresh now keeps the later of the held and refreshed expiry, the rule a resume uses.
A refused refresh, or one that meets an idle cutover, closes the socket it arrived on: after a
rebind, session.socket is the successor, which proved itself and must not be closed for its
predecessor's late frame. Adds an organization-mismatch resume test beside the profile one.
* test(relay): a superseded socket's refresh during an idle cutover closes only that socket
* fix(relay): let a host leave a dead reserve cell without an operator fence
A host whose reserve home cell died got 503 relay_home_cell_unavailable until the
cell came back: the database path demands a committed fence for every capped cell.
On its own /v1/assign, a host now moves off a reserve cell (Postgres records it as
reserve) when the cell's heartbeat is stale and this director's seat-feed polls have
gone unanswered for 45 s, unless more than a third of the polled cells look the same.
Sweeps, migration pins and capped db cells keep the fence.
* fix(relay): guard the dead-cell verdict by region and sample size, and floor the move on the map
- No verdict when half or more of the dead cell's own region fails with it (a cross-region
partition), more than a third of the fleet does, or fewer than 6 cells are polled. A withheld
region or fleet verdict logs orca_relay_dead_cell_verdict_withheld, which alerts.
- The move needs the step-5 plan's epoch floor, so it mints above every seat the map knows;
without the plan the fence is kept. A test revives the old cell's seat above the row and checks
that the ledger never writes the row back.
- The move's log line carries the fleet and region counts.
* fix(relay): count only feed cells and young streaks in the dead-cell guards; gate the alert
- Denominators hold only cells that served this director a feed page and have capacity: legacy
rows answering 404 no longer fill the 6-cell sample or dilute the fleet and region ratios.
- A peer counts as failing for the guards on any current streak (a failure in the last 5 s), so a
staggered partition cannot let its first cell cross 45 s while its neighbours read healthy.
- Every withheld verdict is logged, sample included (a cell missing from the feed cells is one).
- Tests: ready=0 and enabled=0 with failing polls keep the fence (drain then death is fenced).
- The withheld metric and alert are gated by relay_dead_cell_verdict_alert (default false, on in
staging); the alert docs now describe poll failures, not heartbeats.
* fix(relay): always sample the dead-cell candidate, so a restarted director can still judge it
A director that restarted after a reserve cell died never received a feed page from it, so the
cell was never in the sample and its verdict stayed withheld as 'sample'. The candidate now always
counts when it has capacity (the store asks only about a cell Postgres records as reserve, and only
feed images run reserve); peers still count only once they have served a feed. Tests: a restart
during a lone outage decides, a partition begun before the restart withholds by region, and the
capacity filter. The Terraform comments now say to apply the metric first and the policy minutes later.
The staging admission config hard-wired staging-gce-c4, so new staging cells
had no reviewed path into the admission selector. Add c5-c7 as a second
staging wave at their US shape; production cell shapes are unchanged.
* docs(relay): director Cloud SQL sidecar enablement and rollback, as run on staging
The 0% apply revision is not started by Cloud Run, so the first sidecar boot is the deploy's
rollback candidate. A new log-based metric can take minutes to appear, so its alert policy
can fail the first apply. Both observed on staging 2026-10-11.
* docs(relay): production sidecar enablement needs the tfvars PR, a passed drill and the rollout lease
A reserve cell's dead-man is fed only by director polls through the load balancer. With SQL
readiness the balancer drops the cell during a database outage, the polls get 503
failed_to_pick_backend, and the cell trips (staging drill 2026-10-11, c2 at 11:47:59Z).
admitMode=reserve now needs readinessLocal=true in the object being written, already there
or set in the same write, and a reserve cell refuses readinessLocal=false unless the same
write flips it back to db.
Six feed cells, five in us-central1, so one dead US reserve cell is 1/6 of the fleet and 1/5 of its
region and the dead-cell guards in #27458 can decide. Staging only; nothing is applied by merging.
The staging director deploy binds its digest to staging-gce-c4's checked-in image. E is main
2bdb55350d (#27405 event-loop max, #27406 sidecar), already mirrored to staging. C4 itself
changes only through its own reviewed refresh workflow.
* fix(relay): director reaches Cloud SQL through a proxy sidecar (staging), gated event-loop stall alert
On Cloud Run's built-in /cloudsql socket a connect() blocks the director's whole
event loop until the connector's upstream dial ends, 10 s per attempt while the
instance starts. Staging froze both directors for 134 s on 2026-10-11 and
production four directors for 40-50 s on 2026-09-28.
relay_director_cloud_sql_proxy_sidecar runs the Cloud SQL Auth Proxy beside the
director on 127.0.0.1:5432, as the GCE cells already do, with its own TCP
database URL secret. relay_event_loop_stall_alert adds the event_loop_ms_max
metric and a 5 s stall alert. Both default off, so production plans unchanged;
staging turns both on.
* fix(relay): adopt a reviewed Terraform director shape on deploy; counter-based stall alert; cheaper probed sidecar
- deploy-relay-blue-green: --shape-baseline-revision names the 0% revision a Terraform
apply created; the deploy preserves that shape instead of the serving one. It must be the
latest created, unserved revision on the serving image. Wired into the staging and
production director workflows behind ADOPT_RELAY_DIRECTOR_TERRAFORM_SHAPE.
- Event-loop stall alert is now a gated counter metric on eventLoopDelayMsMax>=5000.
- Sidecar gets a /liveness probe and 0.25 vCPU (Cloud Run rejects only a total under 1 vCPU
with concurrency above 1; checked with validateOnly against the API).
- staging.tfvars pins the serving director's concurrency 1000 and timeout 3600 s, so the
adopted apply shape does not fall to the 80 / 30 s defaults.
* fix(relay): shape adoption refuses identity, topology, pruning and image changes; GOMAXPROCS=1 on the sidecar
parseArguments now refuses --shape-baseline-revision with a runtime identity bootstrap, a
director topology change or revision pruning, so every caller is covered, and the deploy
refuses a shape baseline whose image differs from the serving one. Both director workflows
also refuse the combinations up front. The proxy sidecar runs with GOMAXPROCS=1 to match its
0.25 vCPU quota. The long-hold alert comment is back above its resource.
* fix(relay): keep seat polls off the cell-list read and report event-loop freezes
The seat poller no longer waits on its database cell-list read before polling
cells once it has a list, so a stalled read cannot starve the polls that feed a
reserve cell's dead-man. Runtime metrics add eventLoopDelayMsMax: a single long
freeze is one sample, so p99 read 68 ms across a 134 s staging freeze.
Adds a pool test against a database that accepts and never answers, and a
real-Postgres stop/start recovery test.
* fix(relay): state the seat-poller gain precisely, justify the AddressInfo cast
Staging auth mints principal sets for shards 0-3, so the second hosted runner's shards 4-7 were
refused. Their hosts stay distinct by peer index; each set carries the same host count as the
proven 4 x 710 run.
* test(relay): load generator can reopen a closed phone splice instead of failing the run (--splice-retry-max-seconds)
* fix(relay-load): declare a reconnect after an established control, as desktops do
Without it the director's sticky lane is never reached, so a cell restart under load
cannot exercise the sticky re-booking.
* fix(relay-load): close a control after 75 s of silence, as desktops do
The load balancer keeps a dead backend's client websocket open, so a stopped cell's
simulated hosts never noticed and never re-assigned.
A desktop relays alerts for agents on its paired servers, but the push named
only the server's worktree, so a tap opened it on the desktop and bounced.
The gateway accepts an optional runtime execution host and forwards it.
Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb
* fix(relay): staging sleep runs under the admission selector instead of refusing
The selector generation is monotonic and staging is past 0, so the nightly sleep could
never run again. Under a selector, admission belongs to the selector and the selector-era
wake restores it, so the sleep proves every cell empty and not reserve, never writes a
cell's admission, and refuses if the selector moved during the sleep.
* fix(relay): staging sleep re-reads reserve cells just before stopping the MIGs
The cell-flags workflow runs outside the staging lock, so a reserve flip can land
while Cloud Run scales down.
Staging auth mints relay load principals for one workflow file only, so the step-5
staging run's load lives as a mode of that workflow. Load runs take their own
concurrency group and no Cloud SQL lease, so deploys and switch writes can run
under load. Origins, region, principals and sharding are pinned by the workflow.
* infra(relay): staging c1 gets a 600/60 cap and c1/c4 get rehome trust for the step-5 e2e
The reserve and demote routes answer only on cells with rehome trust, and Terraform
requires a rehome source to carry explicit connection limits. Staging only.
* infra(relay): pin staging c1 to c2's cap-capable image with its new cap
The legacy 2d0f6e6 image cannot report capped connection telemetry, so main holding it
with a 600/60 cap would leave any c1 apply failing its heartbeats for good.
* fix(relay): the cell switch tool changes only the named switches and checks every key on read-back
A cell resets a key missing from its switch file to its default, so the tool now reads the
current object and changes only what --set names. The read-back compares every known key and
fails when the cell reports ignoredKeys.
* fix(relay): the cell switch tool refuses a switch the cell's image does not support
It reads supportedFlags from runtime-status (#27057). An image from before that field takes only
readinessLocal and ticketCheck off/shadow, so ticketCheck=enforce cannot void an old cell's object.
* fix(relay): the switch tool validates the whole object, refuses voided files, and records admitMode in Postgres
Every key and value of the object to be written is checked against what the image accepts (an
image without the list: only the keys it applies, ticketCheck off/shadow). A file the cell did
not apply is refused. admitMode=reserve needs a typed RESERVE <cell>; Postgres records reserve
before the file is written, and db only after the cell reports every control leased again.
* feat(relay): the switch tool knows reregisterInFlight (1-16), the flip-back pace
* fix(relay): the switch tool closes the cache gap before reserve, undoes a refused write, and has a break-glass db flip
After recording reserve in Postgres it waits past every director's 5 s cache and records it
again (re-running the open-flow check) before writing the file. A file write refused outright
(412 or 4xx) puts Postgres back to db; after a 5xx it stays reserve. An object the cell did not
apply is repaired when the new object passes the image check. Break glass (typed BREAK GLASS
<cell>) records db for a dead or unreachable reserve cell without its read-back.
* fix(relay): a director deploy refuses a traffic move across reserve-mode support while any cell is reserve
* fix(relay): the flip-back wait scales with the cell's controls and pace, and says how to finish
* fix(relay): the switch tool refuses to re-arm a tripped cell, breaks glass only on a silent one, and undoes reserve if the second post is refused
* feat(relay): director deploys set reserve placement, and refuse turning it off while any cell is reserve
* fix(relay): Terraform keeps the deploy-owned placement and shadow cells, and the deploy guard reads placement as built and refuses dropping a reserve cell from the shadow list
* fix(relay): director deploys take --admin-audience alone, name a drifted deploy identity, and carry the shadow seat-feed cells
Staging passes --admin-audience for the reserve guard without durable rehome
verification, which the parser refused. Only the generation and control
origin now travel together, and they still need the audience.
The guard decodes the caller from its token and refuses, by name, a revision
whose ORCA_RELAY_DEPLOY_SERVICE_ACCOUNT is another account, instead of
reading into a 401.
--shadow-seat-feed-cells (preserve, all, or a cell list) is a workflow input
on both director deploys; preserve keeps the serving value. A contract test
parses every workflow's real argv for both deploy scripts.
* fix(relay): the flag tool paces a flip back from the pool size the cell reports
An image from before the field falls back to the deployed sizes.
* feat(relay): per-cell switches for the rejection fence and the lost-reply margin, and report ignored keys
rejectionFence (default on) turns the unhandled-rejection fence off without a roll.
readTimeoutMarginMs (1-60 s) replaces the boot margin per query through pg's query_timeout.
The applied log line and flagsApplied name keys this image does not know.
* feat(relay): runtime-status names the switches and values this image accepts
The flag tool checks a write against it, so it can refuse ticketCheck=enforce (or any key)
on an image that would void or drop it.
The flag channel recorded a generation as seen before reading its body, so a
read that failed mid-body sent ifGenerationNotMatch for it on every later poll
and got 304 forever: that write was stranded until the object changed again.
The generation now counts as seen only once its body has been read and parsed.
On-call can ask a director where a host is: its relay_assignments row and
open migration (primary-key reads only), the shadow map's seats and recent
leaves with close codes, the compare verdict (pure classification, judged as
/v1/resolve sees the host, recording nothing), and each named cell's poll
status and applied flags. shadow is {state:'off'} when the poller is off.
Deploy, monitor and fence identities may read it; capacity and asia-proof
may not. With no user id, the map names the user; an unknown host answers
empty.
After the database answers a reconnect verify or a phone resolve, the
director compares that answer with the shadow seat map and counts the
result by class, logging one count line per route and class a minute and a
few rate-limited samples of unexplained disagreements (cell and epoch only,
never raw ids). It runs only when the seat poller is on, inside try/catch,
and never changes a response.
A map-behind mismatch is explained only while this director first saw the
newer epoch within 10 s (recorded only by the compare; classification is
pure). A seat on a cell not polled within 10 s is cell-unpolled. A null
answer for a seat on a cell placement's liveness rule calls dead is
map-only-cell-unlive, or cell-unlive-pending when its heartbeat ran out
after the last cell-list read.
* feat(relay): key id on assignment leases; cells shadow-check the lease behind ticketCheck=shadow (off)
Directors name the lease signing key with a kid. With the per-cell
ticketCheck switch at shadow, a cell verifies the lease a host echoes in
x-orca-relay-assignment-lease, compares it with the database answer that
still decides, and logs per-class counts and the hello's database read
time once a minute.
* feat(relay): flat per-class totals on the lease shadow line, so log metrics can extract them
* refactor(relay): the lease shadow line carries the flat classes and one database-refused count
The per-class-and-verdict map duplicated the flat classes; disagree
already names the valid-lease refusals, and dbRefused gives the total.
Directors can poll each cell's read-only seat feed (#26803) once a second
and keep an in-memory map of which host is seated on which cell, plus a
bounded 24 h recently-left memory. Off unless ORCA_RELAY_SHADOW_SEAT_FEED_CELLS
names cells. Applies a change at or above the seat's generation, follows
cut pages, resyncs on a gap or new incarnation, and checks the map against
counts.seats. A 404 reads as no feed; timeouts and errors keep the last
seats as unverifiable. The cell list is every cell (an existing-only cell
still seats hosts), from RelayAssignmentStore.seatFeedCells with
placement's liveness rule; completeness waits only on live cells, and a
failed list read backs off. Nothing reads the map yet.
* feat(relay): local /ready behind the per-cell readinessLocal switch (off)
With the switch on, a cell's /ready answers 200 while it listens and its
host token verifier holds keys, and reports failing probes as degraded.
The heartbeat keeps the SQL-gated verdict, so directors still stop
placing on a cell that cannot reach the database.
* refactor(relay): the app verifies relay tokens with the server's shared key set
One cached JWKS per process now serves the host-control upgrade, HTTP
token checks and local readiness, instead of a second set in the app.
Jinwoo Hong
NeilandMatheus Breguêz