* chore(relay): move the staging cells onto image F
Staging c1, c2 and c4-c7 run image F (main 3c1141f2), so the director and
every staging cell share one digest for the c5-c7 promotion and the
six-cell dead-cell rerun. c3 is unchanged.
* test(relay): pin the staging C4 digest to image F
* fix(relay): let a host leave a dead reserve cell without an operator fence
A host whose reserve home cell died got 503 relay_home_cell_unavailable until the
cell came back: the database path demands a committed fence for every capped cell.
On its own /v1/assign, a host now moves off a reserve cell (Postgres records it as
reserve) when the cell's heartbeat is stale and this director's seat-feed polls have
gone unanswered for 45 s, unless more than a third of the polled cells look the same.
Sweeps, migration pins and capped db cells keep the fence.
* fix(relay): guard the dead-cell verdict by region and sample size, and floor the move on the map
- No verdict when half or more of the dead cell's own region fails with it (a cross-region
partition), more than a third of the fleet does, or fewer than 6 cells are polled. A withheld
region or fleet verdict logs orca_relay_dead_cell_verdict_withheld, which alerts.
- The move needs the step-5 plan's epoch floor, so it mints above every seat the map knows;
without the plan the fence is kept. A test revives the old cell's seat above the row and checks
that the ledger never writes the row back.
- The move's log line carries the fleet and region counts.
* fix(relay): count only feed cells and young streaks in the dead-cell guards; gate the alert
- Denominators hold only cells that served this director a feed page and have capacity: legacy
rows answering 404 no longer fill the 6-cell sample or dilute the fleet and region ratios.
- A peer counts as failing for the guards on any current streak (a failure in the last 5 s), so a
staggered partition cannot let its first cell cross 45 s while its neighbours read healthy.
- Every withheld verdict is logged, sample included (a cell missing from the feed cells is one).
- Tests: ready=0 and enabled=0 with failing polls keep the fence (drain then death is fenced).
- The withheld metric and alert are gated by relay_dead_cell_verdict_alert (default false, on in
staging); the alert docs now describe poll failures, not heartbeats.
* fix(relay): always sample the dead-cell candidate, so a restarted director can still judge it
A director that restarted after a reserve cell died never received a feed page from it, so the
cell was never in the sample and its verdict stayed withheld as 'sample'. The candidate now always
counts when it has capacity (the store asks only about a cell Postgres records as reserve, and only
feed images run reserve); peers still count only once they have served a feed. Tests: a restart
during a lone outage decides, a partition begun before the restart withholds by region, and the
capacity filter. The Terraform comments now say to apply the metric first and the policy minutes later.
The staging admission config hard-wired staging-gce-c4, so new staging cells
had no reviewed path into the admission selector. Add c5-c7 as a second
staging wave at their US shape; production cell shapes are unchanged.
A reserve cell's dead-man is fed only by director polls through the load balancer. With SQL
readiness the balancer drops the cell during a database outage, the polls get 503
failed_to_pick_backend, and the cell trips (staging drill 2026-10-11, c2 at 11:47:59Z).
admitMode=reserve now needs readinessLocal=true in the object being written, already there
or set in the same write, and a reserve cell refuses readinessLocal=false unless the same
write flips it back to db.
The staging director deploy binds its digest to staging-gce-c4's checked-in image. E is main
2bdb55350d (#27405 event-loop max, #27406 sidecar), already mirrored to staging. C4 itself
changes only through its own reviewed refresh workflow.
* fix(relay): director reaches Cloud SQL through a proxy sidecar (staging), gated event-loop stall alert
On Cloud Run's built-in /cloudsql socket a connect() blocks the director's whole
event loop until the connector's upstream dial ends, 10 s per attempt while the
instance starts. Staging froze both directors for 134 s on 2026-10-11 and
production four directors for 40-50 s on 2026-09-28.
relay_director_cloud_sql_proxy_sidecar runs the Cloud SQL Auth Proxy beside the
director on 127.0.0.1:5432, as the GCE cells already do, with its own TCP
database URL secret. relay_event_loop_stall_alert adds the event_loop_ms_max
metric and a 5 s stall alert. Both default off, so production plans unchanged;
staging turns both on.
* fix(relay): adopt a reviewed Terraform director shape on deploy; counter-based stall alert; cheaper probed sidecar
- deploy-relay-blue-green: --shape-baseline-revision names the 0% revision a Terraform
apply created; the deploy preserves that shape instead of the serving one. It must be the
latest created, unserved revision on the serving image. Wired into the staging and
production director workflows behind ADOPT_RELAY_DIRECTOR_TERRAFORM_SHAPE.
- Event-loop stall alert is now a gated counter metric on eventLoopDelayMsMax>=5000.
- Sidecar gets a /liveness probe and 0.25 vCPU (Cloud Run rejects only a total under 1 vCPU
with concurrency above 1; checked with validateOnly against the API).
- staging.tfvars pins the serving director's concurrency 1000 and timeout 3600 s, so the
adopted apply shape does not fall to the 80 / 30 s defaults.
* fix(relay): shape adoption refuses identity, topology, pruning and image changes; GOMAXPROCS=1 on the sidecar
parseArguments now refuses --shape-baseline-revision with a runtime identity bootstrap, a
director topology change or revision pruning, so every caller is covered, and the deploy
refuses a shape baseline whose image differs from the serving one. Both director workflows
also refuse the combinations up front. The proxy sidecar runs with GOMAXPROCS=1 to match its
0.25 vCPU quota. The long-hold alert comment is back above its resource.
Staging auth mints principal sets for shards 0-3, so the second hosted runner's shards 4-7 were
refused. Their hosts stay distinct by peer index; each set carries the same host count as the
proven 4 x 710 run.
* test(relay): load generator can reopen a closed phone splice instead of failing the run (--splice-retry-max-seconds)
* fix(relay-load): declare a reconnect after an established control, as desktops do
Without it the director's sticky lane is never reached, so a cell restart under load
cannot exercise the sticky re-booking.
* fix(relay-load): close a control after 75 s of silence, as desktops do
The load balancer keeps a dead backend's client websocket open, so a stopped cell's
simulated hosts never noticed and never re-assigned.
* fix(relay): staging sleep runs under the admission selector instead of refusing
The selector generation is monotonic and staging is past 0, so the nightly sleep could
never run again. Under a selector, admission belongs to the selector and the selector-era
wake restores it, so the sleep proves every cell empty and not reserve, never writes a
cell's admission, and refuses if the selector moved during the sleep.
* fix(relay): staging sleep re-reads reserve cells just before stopping the MIGs
The cell-flags workflow runs outside the staging lock, so a reserve flip can land
while Cloud Run scales down.
Staging auth mints relay load principals for one workflow file only, so the step-5
staging run's load lives as a mode of that workflow. Load runs take their own
concurrency group and no Cloud SQL lease, so deploys and switch writes can run
under load. Origins, region, principals and sharding are pinned by the workflow.
* fix(relay): the cell switch tool changes only the named switches and checks every key on read-back
A cell resets a key missing from its switch file to its default, so the tool now reads the
current object and changes only what --set names. The read-back compares every known key and
fails when the cell reports ignoredKeys.
* fix(relay): the cell switch tool refuses a switch the cell's image does not support
It reads supportedFlags from runtime-status (#27057). An image from before that field takes only
readinessLocal and ticketCheck off/shadow, so ticketCheck=enforce cannot void an old cell's object.
* fix(relay): the switch tool validates the whole object, refuses voided files, and records admitMode in Postgres
Every key and value of the object to be written is checked against what the image accepts (an
image without the list: only the keys it applies, ticketCheck off/shadow). A file the cell did
not apply is refused. admitMode=reserve needs a typed RESERVE <cell>; Postgres records reserve
before the file is written, and db only after the cell reports every control leased again.
* feat(relay): the switch tool knows reregisterInFlight (1-16), the flip-back pace
* fix(relay): the switch tool closes the cache gap before reserve, undoes a refused write, and has a break-glass db flip
After recording reserve in Postgres it waits past every director's 5 s cache and records it
again (re-running the open-flow check) before writing the file. A file write refused outright
(412 or 4xx) puts Postgres back to db; after a 5xx it stays reserve. An object the cell did not
apply is repaired when the new object passes the image check. Break glass (typed BREAK GLASS
<cell>) records db for a dead or unreachable reserve cell without its read-back.
* fix(relay): a director deploy refuses a traffic move across reserve-mode support while any cell is reserve
* fix(relay): the flip-back wait scales with the cell's controls and pace, and says how to finish
* fix(relay): the switch tool refuses to re-arm a tripped cell, breaks glass only on a silent one, and undoes reserve if the second post is refused
* feat(relay): director deploys set reserve placement, and refuse turning it off while any cell is reserve
* fix(relay): Terraform keeps the deploy-owned placement and shadow cells, and the deploy guard reads placement as built and refuses dropping a reserve cell from the shadow list
* fix(relay): director deploys take --admin-audience alone, name a drifted deploy identity, and carry the shadow seat-feed cells
Staging passes --admin-audience for the reserve guard without durable rehome
verification, which the parser refused. Only the generation and control
origin now travel together, and they still need the audience.
The guard decodes the caller from its token and refuses, by name, a revision
whose ORCA_RELAY_DEPLOY_SERVICE_ACCOUNT is another account, instead of
reading into a 401.
--shadow-seat-feed-cells (preserve, all, or a cell list) is a workflow input
on both director deploys; preserve keeps the serving value. A contract test
parses every workflow's real argv for both deploy scripts.
* fix(relay): the flag tool paces a flip back from the pool size the cell reports
An image from before the field falls back to the deployed sizes.
* feat(relay): log metrics and alerts for fenced and fatal rejections and the lease shadow
orca_relay_database_rejection_fenced and orca_relay_process_fatal become
incident counters on directors and cells, each with an alert at the first
event. The lease shadow line gets one distribution metric per class plus
the hello database read p99, and an alert when bad signatures persist on
a cell for 15 minutes.
* fix(relay): one fleet-wide fenced-rejection alert, director-only fatal page, console-only bad-signature alert
A database fault reaches every process at once, so fenced rejections sum
into one series. Cells' fatal rejections already page through the cell
exit alert, so the fatal page covers directors only. The bad-signature
alert can be held up by one client, so it stays in the console.
* fix(relay): fleet-wide fenced-rejection alert as one PromQL sum, like the fix-level alert
* feat(relay): metric for the lease shadow's database-refused count
* feat(relay): one-cell flag write workflow with dry run, compare-and-swap and read-back
Operate Relay Production Cell Flags writes cells/<cellId>.json for one
cell: it checks the origin answers as that cell with a flag channel,
refuses a stale expected generation, writes with ifGenerationMatch,
stamps the actor and run on the object, and waits up to 15 s for the
cell's runtime status to report the new generation applied. The deploy
Workload Identity condition admits the new workflow.
* fix(relay): flag workflow retries a transient 5xx, shows the plan before the generation check, and names the failure
Runtime-status reads go through the shared one-retry admin helper, and a
failed read-back poll no longer ends the wait early. The plan line is
logged before the generation check, a failure is logged as one JSON line
the job summary carries, and the object and summary name the triggering
actor.
* feat(relay): Cloud Storage bucket for the per-cell runtime switch files
<project_id>-relay-control, versioned, uniform access, public access
prevented. Cells and directors read; only the flag workflow's deploy
identity writes. Apply with a targeted apply, never a root plan.
* fix(relay): scope the switch-file writer to cells/ with a minimal custom role; soft-delete keeps history
The deploy identity is shared, so its grant now reaches only cells/<cellId>.json and carries
only get, create and delete. A 90-day soft-delete window keeps every deleted generation
recoverable, since overwrite needs delete and delete also reaches old versions.
* refactor(relay): drop the speculative directors.json read grant from the control bucket
* feat(relay): alert on cell disconnect bursts and database pool herds
Two alert policies that separate a load-balancer mass disconnect from a
database stall, after the 2026-10-06 18:34Z asia-east2 drop:
- Disconnect burst: over 120 code-1006 desktop control closes on one cell
in a minute, counted from the cell's own close line. The LB's
internal_error log entries carry the stream start as their timestamp
(median 2.4 h before the drop), so an LB-log rate never shows the pulse.
- Pool herd: a cell runtime sample with more than 50 requests queued for a
PostgreSQL connection (interval maximum, bar in the filter).
Also publishes orca_relay_host_hellos_shed for the refusal counter from
#26362, and pins both filters to the relay's text in a contract test.
* fix(relay): raise the cell pool-herd alert bar to 200 waiters
asia-east2 cells reach 184-196 waiters outside any herd, so >50 would have
fired in 84 episodes since 2026-10-03. >200 fires in 10, covering every known
herd. Drops the false 'healthy peak is 2 waiters' baseline from the policy text.
* docs(relay): describe pool-herd baseline as stalls, not routine load
A healthy cell peaks at 2 waiters (p99); the 50-196 readings are single-cell
stalls that already time out. The 200 bar is unchanged.
* fix(relay): let the same-cap roll's post-roll verify outlast one DB stall
The verify step's admin reads gave up after three 2 s retries; on 2026-10-07
c18 failed its roll on 503 'no healthy upstream' during a DB stall and was
healthy 16 s later. The step now retries 5xx for up to 60 s, which covers a
stall plus readiness grace and two LB health checks, and still fails a cell
that stays down. 4xx stays final.
* test(relay): surface curl stderr when the verify retry test fails
* fix(relay): retry the post-roll verify reads in bash, since curl 7.81 ignores --retry under --fail-with-body
* fix(relay): never report a stale body after a refused verify read; stop pinning inert curl retry flags
* feat(relay): slower same-cap drain paces (900000, 1200000) for every cell class
c28's 1,901-host drain at the 300 s pace (~6.3 hosts/s, 2026-10-07) held
database lock time over its bar for about 10 minutes. Add 15- and 20-minute
pace windows to the same-cap closed set, allowed on every cell class.
- The cell's /v1/admin/drain cap rises from 5 to 20 minutes.
- The drain script fails instead of falling back to an unpaced drain when a
cell rejects a window above 300000, since an older image answers both with
the same 400.
- The fast-pace US-only rule and the canary PASS rule apply only below the
default; a canary still authorizes its own pace or slower.
- The restart-safe timeout adds the window's excess over 300000 (quiet starts
after the last host leaves), unchanged at 300000/60000/30000; the job
timeout rises from 90 to 120 minutes.
* fix(relay): refuse a slow drain pace before isolating a cell that cannot accept it
The cell advertises its drain pace cap on /health; the same-cap job reads it
before the isolate, so a pace above an old image's 300000 cap stops with the
cell untouched instead of isolated. Also refreshes two stale timeout comments
and pins the drain step under its one-hour ID token and the job's 70-min
remainder.
* feat(relay): give Asia cell c34 a promotion wave so it can become a general cell
c34 launched on 2026-10-05 as a migration-only spare with no promotion path. This adds
it to the Asia admission promotion waves, the workflow's promote and canary cases, and
the canary evidence map, so the reviewed Asia workflow can promote it with the same
five-minute canary c30 and c31 ran.
The same-cap migration-only list is deliberately unchanged: a same-cap job reads a
cell's class from that list, and c34 must be rolled to the director's image as a
migration-only cell before promotion can run. The list moves after promotion, in its
own change.
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* chore(relay): move Asia cell c34 to the general same-cap lists after promotion
A same-cap job reads a cell's class from SAME_CAP_MIGRATION_ONLY_CELLS, not from the
selector. Once c34's Asia canary promotes it, leaving it there would make a same-cap
rollback demote it to migration-only. This moves c34 to the general same-cap list and
the shadow gate's fleet pool list (its pool is the Asia 16).
Merge only after c34's promotion succeeds. This file set is trusted evidence code, so
merging invalidates any sealed monitor or same-cap canary: merge outside a
monitor-to-enable window and before the next same-cap canary seals.
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* docs(relay): scope the c34 same-cap pause to the window after promotion
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* docs(relay): rewrap the c34 paragraph
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* fix(relay): commit the cell counter in one round trip; cells boot without the DB
Step 2 (option B) cell image:
- One-round-trip counter commit at acquireActivity, releaseActivity and
activateControl: the final counter UPDATE and COMMIT go as one simple-query
message. Server errors mean COMMIT never ran (retry as today; 22012 = no row,
rolled back and disambiguated outside the transaction); a lost connection is
never retried.
- Cells skip the schema apply and region backfill, so they listen while the
database is down and turn ready on their first successful query.
- G13: rehome target connection headroom folded into the existing NOWAIT
UPDATE, excluding the host's own reservation by key.
- fixLevel on every runtime metrics line, plus declared (not applied) cell
fix-level metrics and alert.
- Per-desktop drain disconnect-gap measurement from existing log lines.
- Census test that fails on floating database promises; fixes two shutdown
sites. Lock-wait sample keeps the combined role.
* fix(relay): make the outdated-image alert creatable: one PromQL condition, 1 h lookback, fixed floor
A PromQL condition must be the only condition in its policy, and alerts on
log-based metrics may look back at most 25 h. Replace the 6-day/7-day design
with relay_cell_min_fix_level (tfvars, raised by a targeted apply after each
wave) and one query: a serving cell below the floor or reporting no level,
sustained 6 h. Drops the separate without-level metric.
* fix(relay): review fixes: gap-script ordering, wider promise census, fused-path guard, row-busy as scheduled
- Drain gap script: sort closes by time (gcloud exports newest first) and
refuse an invalid drain start.
- Census: any floating promise in relay src, including callback-discarded
and never-read ones, with a reviewed never-rejects list.
- Test the fused counter commit through the store the server builds, so a
wrapper that stops forwarding commitWithFinal fails CI.
- Same-cap shadow gate: a row-busy refusal (the host's own release still
holds its row) is a scheduled 503, like an own early retry. No client
change.
* test(relay): judge drain redials by no host refused twice, not a refusal count
The row-busy count tracks how many releases are still in flight at the
dial (80 of 180 every run at 1 s, against a bar of 90). What matters is
that the release has finished by the next dial: assert no host is refused
twice, keep the time-to-placed p95 bound.
* fix(relay): cap row-busy as scheduled at the drain-return admissions; bound the gap script's window
Shadow gate: a row-busy refusal of a drained host follows its drain-return
lane admission, so per minute only that many (plus a rounding margin of 2)
are scheduled; the rest stay non-drain, so row contention the drain does
not explain still fails the budget. Gap script: --drain-ended-at excludes
the new container's closes after the roll; later grants still close a gap.
* feat(relay): give Asia cell c34 a promotion wave so it can become a general cell
c34 launched on 2026-10-05 as a migration-only spare with no promotion path. This adds
it to the Asia admission promotion waves, the workflow's promote and canary cases, and
the canary evidence map, so the reviewed Asia workflow can promote it with the same
five-minute canary c30 and c31 ran.
The same-cap migration-only list is deliberately unchanged: a same-cap job reads a
cell's class from that list, and c34 must be rolled to the director's image as a
migration-only cell before promotion can run. The list moves after promotion, in its
own change.
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* docs(relay): scope the c34 same-cap pause to the window after promotion
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* docs(relay): rewrap the c34 paragraph
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* fix(relay): publish the reviewed commit before main can move, and quiet the deploy driver
The publish workflow builds main's head at dispatch. The driver checked main
at preflight but dispatched the publish about four minutes later, after the
inspects and the typed phrase, so a busy main stopped the first real deploy.
It now dispatches the publish seconds after the check, before anything else,
and a build of a moved main stops with the --commit/--publish-run command that
deploys it once reviewed. Typed prompts end in a newline, and runs are
summarised (status changes plus every 5 min) instead of streaming gh run watch.
* fix(relay): a main-moved stop prints only the command that reuses the build
The generic re-run line named the reviewed commit without --publish-run, which
would only build the moved main again.
* feat(relay): add an operator-local driver for director deploys
One command runs the audited director deploy: preflight, pause rehome if
enabled, publish, deploy, optional cell configure, a digest-bound inspect,
the monitor dry-run, and re-enable. It only dispatches the existing
workflows, reads the published digest from the registry and the run log,
reads the monitor verdict from its sealed state, and enables with the
digests gcloud reports after the deploy. It stops at the first failure,
records state, and resumes from it.
* fix(relay): report and own the rehome pause; operator types every phrase
- Read the control back from pause and enable runs whatever their conclusion,
and report PAUSED or UNCONFIRMED loudly.
- Resume re-enables only the pause this driver recorded (generation and run).
- Ctrl-C and SIGTERM print the same state and resume report.
- The operator types every workflow confirmation. A 5-minute soak gated on
director 5xx runs before configure.
- A dry run keeps no state file. The quiet check pages through all runs.
Run IDs come only from the printed URL. One step table drives execute,
dry run and resume. The monitor verdict reuses verify-authority.
* fix(relay): read back only the driver's own rehome run; anchor the soak at the traffic switch
- The pause and enable read-back accepts only the control line its own step's mode prints, at the
generation its own dispatch expected. A run adopted after a crash is settled even when green.
- The soak window opens a minute before the deploy run completed and is read a minute after it
ends, for log ingestion lag.
* fix(relay): say what typing ENABLE_REGIONAL_REHOMING commits to
* fix(relay): the ENABLE prompt also names the 150 s evidence budget
* refactor(relay): derive every deploy decision from live state; no resume machinery
The driver keeps no state between runs. Each run reads the serving
director, its configured cells and the rehome control, and skips what is
already done.
- The only rehome fact it owns is the run that paused rehome. A re-run
names it (--pause-run), and the driver checks it against that run's log
and the live generation.
- A recover-enable line counts as the driver's own pause only with
recovered: true. A director safety pause is never adopted (F3).
- Publish runs before the pause. A fresh run that finds rehome paused
stops unless given --pause-run or --rehome-disabled (F2).
- Interrupts report a pause or enable still in flight as REHOME IS
CHANGING (F1). PAUSE UNCONFIRMED and ENABLE UNCONFIRMED are distinct.
- A tripped soak is judged again on fresh traffic (F5). SIGHUP is
handled, and a pending signal stops the driver before its next dispatch.
- Every stop prints the single command that finishes the deploy.
Removes the state file, --resume, step statuses, monitor adoption and
interrupted-dispatch adoption.
* fix(relay): never report done over an unexplained pause; prove pause ownership by actor
- G1: always read rehome; no early DONE.
- G2: --pause-run must be a rehome-control run by the same user.
- G3: a failed enable run is never an enable.
- C1: a pause or enable is reported as changing from the moment it is
dispatched.
- C2: --leave-rehome-paused (was --rehome-disabled) refuses an enabled
switch. Every printed command parses.
- G4: the enable-in-flight report prints both finishing commands.
- G5: the driver's own runs never block the quiet-lane check.
* test(relay): port the round-3 probes: hard kill mid-pause, unexplained disable after a failed enable
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* feat(relay): drain pace window as a reviewed same-cap input, with drain-aware 503 gates
The same-cap roll drained every cell over a fixed 300 s window, so a US roll
re-placed hosts at ~2/s and spent ~10 minutes draining and waiting for quiet.
The window is now a dispatch input from a closed set (300000, 60000, 30000),
defaulting to today's 300000.
- Below the default is refused for anything but US general cells; Asia drains
are bound by their targets' accept rate, and migration-only cells hold no hosts.
- A non-default window must be named in the confirmation, the canary authority
records it (v2), and a batch may run at its canary's window or slower only.
- Each cell job re-checks the window, scales the restart-safe timeout with it
(15-min lease + window, unchanged at the default), and records what the cell
applied and when it settled.
- The report-only shadow gate takes the director's drain-return deferrals out
of the 503 count (window and baselines), adds the rung's 5-min non-drain 503
budget and a Retry-After check, and reports the measured re-placement rate.
- relay-workflows.md documents the pace ladder and what each rung records.
* fix(relay): judge paced drains on counted 503s against the pre-drain minutes, and seal the canary's pace verdict
Review of #25639 replayed the shadow gate: it read 10-01 c29 as unverified (the
log read stopped at 20k entries), false-blocked 10-02 c22, and was blind on four
cells whose 24 h/48 h baseline held an incident.
- Director 503s now come from Cloud Run's request_count, aligned per minute by
Cloud Monitoring, so volume cannot truncate the count.
- The background is the median of the 10 same-day minutes before the drain;
the 24 h/48 h baselines are gone.
- Scheduled 503s come out: drain-return deferrals and sticky/placement answers
to a host's own early retry (host-rate-limited, host-in-flight), each split
across the minutes its 30 s sample covers.
- The rung budget counts only sustained breaches: two straight minutes over
max(1.5x, +20) warn, over max(2x, +40) would-block.
- The report carries a paceVerdict over the three pace checks. seal_canary
downloads the canary cell's report and seals that verdict; a batch below the
default pace needs PASS, from a report on the same cell that drained at that
pace.
- Docs: the step-down rule reads paceVerdict, 30 s waits for the lane service
time (#25645), and the staging step is dropped since staging drains unpaced.
Replayed read-only: 10-01 c29 would-block (9 minutes over 41.5/min); all nine
10-02 cells and 10-01 c25 paceVerdict PASS.
* fix(relay): a partial count already past a block line blocks, in the shadow gate's Cloud SQL and pool checks
A truncated FATAL count is a floor, and one runtime sample over the SQL-failure
line is a fact, so neither waits for a complete read. The waiter-run rule still
needs a complete run, since holes can join two runs into one.
* fix(relay): a canary pace PASS needs a real cohort and whole telemetry; one median-based 503 check
From the final review of #25639:
- canaryPaceVerdict seals PASS only from a report that drained at least 400
hosts (about half a 10-02 US cell), so a near-empty canary cannot authorize
a fast batch.
- A director-metrics sub-window with fewer samples than one instance emits
is unverified, so an empty or short Logging answer is never a calm drain.
- seal_canary names the shadow artifact, report path and cell from the gate's
normalized cell list, as cell_1 uploads it.
- director503 folds into nonDrain503Budget as a single-minute spike rule,
max(10x median, 200), dropping the pre-drain peak statistic. All 30 replayed
windows keep their verdicts.
Adds a sixth asia-east2 cell at the C31 shape (cap 3000, 6000 request
units, pool 16, e2-standard-4) in asia-east2-c, pinned to the f30b5cb1
cell image. It gets its own topology and registration wave but no
promotion wave, so the admission script and workflow refuse to promote
it; it stays a migration-only landing zone and out of the fleet pool list.
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
Both were promoted to general on 2026-10-01 (selector 344 shows them general),
but the same-cap job still classed them migration-only. Rollback mode on either
would isolate (general -> migration-only) before its no-op check failed, demoting
a live cell. They stay out of the fleet pool list (pool 10, not 16).
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* refactor(relay): sample fleet health inside the same-cap roll instead of a separate monitor run
A same-cap wave no longer consumes a 15-minute monitor dry-run and its sealed,
single-use, five-minute-fresh evidence. Each apply wave now samples fleet health
itself right before isolation, with the monitor's evaluator, thresholds, and
tolerances, for a window sized to the cell's host count (3/5/8 min), plus three
lookback rules: no cell container exit in 10 min, no minute over 500 director
503s in 10 min, and director concurrency p99 within the monitor bar over 4 min.
Removes the monitor-run inputs, the gate's consume/authorize steps, the
break-glass override, and the same-cap-only authorization shapes in
relay-monitor-evidence.mjs. The monitor workflow and the rehome enable path are
unchanged.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): bound the pre-drain sample overrun and keep the drain token fresh
Review follow-ups: alternating tolerated readings could hold the sample open
until its step timeout, so cap the overrun at three samples past the window;
record why a read failed; mint a fresh admin ID token for the drain after the
sample; raise the job timeout to 90 min so a long sample cannot cancel the
job past the failsafe.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* feat(relay): exempt the rolled cell and existing-only cells from the pre-drain crash rule
The exit rule counted every relay container exit fleet-wide, so a cell that
crashes every few hours (c25, 12 a week) blocked the very roll that fixes it,
and existing-only legacy cells (c5, 15 a week) blocked rolls they take no part
in. Exits are now grouped by instance, each instance is named by its own newest
runtime-metrics log line, and only exits on general or migration-only cells
other than the target count. An exit no configured cell can be named for trips
the rule; a failed lookup is a failed read. relay-observability.tf joins the
evidence-code set because the rule depends on its filter.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* test(relay): cover re-asking for an unnamed exiting instance; note the boot-exit risk
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* feat(relay): declare US cells c32 and c33 at the 3,000-host shape
Declares two us-central1 cells at the Asia shape (cap 3000, 6000 request
units, e2-standard-4) with the US default pool of 10, and generalises the
Asia topology and admission ladder to derive each wave's region from its
reviewed zone, leaving every Asia wave's behaviour unchanged.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* docs(relay): note the US canary tie-break and leave the fleet pool list to promotion
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): plan C32 and C33 as one topology wave
The live-image overlay refuses a declared non-target cell with no template,
so a lone C32 plan would fail on C33. Registration and promotion stay
one cell at a time.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): accept MIG version-name reconciliation and recreate stranded cells without rewriting the MIG
The stranded-rollback recovery ran a gcloud rolling action, which renames the
MIG version outside Terraform. Every later plan for that cell then reverted the
label, and the capacity-plan validator refused the revert as an unreviewed MIG
change, so the cell could be neither rolled nor rolled back.
The validator now accepts a MIG field moving back to what relay-gce-cells.tf
declares (version name and update policy), in every mode, and a test pins those
values to the Terraform file. The stranded branch recreates the cell's single
instance with recreate-instances, which leaves the MIG untouched.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): let a label-only MIG plan through and recreate on it in a stranded rollback
A stranded rollback whose template is already in place plans only the version
name revert. The validator still required the MIG template to move, so that
plan was refused, and the recreate gate (changes == 0) would have skipped a
plan of one change and left the drain flag set. Require the template move only
when no declared field reconciles, and recreate whenever the template was not
replaced (changes < 2).
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): anchor same-cap monitor evidence freshness to the run's authorisation, not job startup
The same-cap gate now verifies the dry-run on its own clock and records the
authorisation instant in the single-use consumed marker; each cell job checks
the evidence age at that instant and bounds its own start after it.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): refuse a re-run same-cap gate before it consumes evidence; tighten order tests
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): let a draining cell pass restart-safe through refused redials
A draining cell refuses every control and host proof, so once no session,
splice, or queued byte remains, an in-flight or reserved connection unit can
only belong to a dial the cell is about to refuse. The restart-safe wait no
longer resets its pace-window streak on those units, and its progress line
now prints every counter the gate reads.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* test(relay): pin fail-closed parsing of handshake counters on a draining cell
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
The parent workflow collapses canary-apply and batch-apply into the job mode
apply, so the headroom step's canary-apply/batch-apply condition never held and
the gate was skipped on every real roll. Run it wherever the drain runs (apply,
rollback before its restart) and in read-only verify; skip only a resumed
rollback, which drains nothing. A new workflow-shape test fails on any job
step comparing against a mode the parent cannot pass, and on a drain that can
run without the headroom check.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): gate the Asia canary on region fallbacks against a pre-canary baseline
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): gate the Asia canary on region fallbacks against a pre-canary baseline
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): let a same-cap drain finish when only unplaceable hosts remain
The c28 canary on 2026-10-01 drained the cell to zero live connections, but four
hosts with no free slot anywhere kept redialling and held director leases on it,
so the restart-safe wait timed out and left the cell isolated and empty.
The drain wait now also passes once the runtime has carried nothing for a
sustained quiet window while a small, capped number of leases remain, and logs
the escape. Apply modes also refuse a cell whose hosts exceed 80% of the free
slots on the other general cells, so a wave cannot strand hosts in the first place.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): define restart-safe by the cell runtime, not director leases
Replaces the opt-in stranded-host escape with a corrected definition. A
restart is safe when the cell runtime carries nothing live and no migration
is open, sustained for the drain pace window. Director activity leases lag
hosts that already left or cannot be placed, so they are reported in a
progress line and the verified result instead of blocking the restart.
The same-cap drain passes its existing pace window. The headroom script is
added to the trusted evidence code paths with the other production scripts.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): print stranded director counts on every restart-safe sample
Each restart-safe poll now prints its sample count and the director's
restart-blocking leases, request units, reserved remainder, and migrations
under `stranded`; the verified line carries the same object. Open migrations
still block because each is pinned to the cell incarnation a restart replaces.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): require the pace window for every live restart-safe wait
Pre-auth and total connections no longer reset the restart-safe window:
on drained c28 they flickered with unauthenticated redials in a third of
samples, which a restart does not lose. They stay in the progress output.
Every live restart-safe call must now pass --pace-window-ms. The capacity
job and staging proof drain unpaced, so they pass the production 300000 ms
window, and the calls that relied on the 180000 ms default get 480000 ms.
Headroom free slots now follow the director's placement rule: the admission
pause minus the larger of observed and enforced units, minus outstanding
control reservations.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
Second audit wave, targeting three more junk patterns:
- assertion-free cases that run code and assert nothing, so they pass no
matter what the code does;
- inventory literals re-typed from a production declaration, where the only
way the assertion can fail is someone editing one of the two copies;
- export key-set and export-shape loops (`typeof x === 'function'` over every
export) that restate what TypeScript already enforces.
Yield is much smaller than wave 1 on purpose: the assertion-free scanner has
a high false-positive rate, because many flagged blocks assert through a
shared helper or their oracle is "this must not throw". Those were kept.
`mobileWebCheckArgs` in `config/scripts/run-mobile-web-app-checks.mjs` is
de-exported — after the inventory comparison went away, nothing outside the
module read it.
Deletes 101 test files and trims 112 more, all matching documented junk
patterns: exact source/import/string greps, copied inventories and export
lists, duplicate invocations of a contract another test already owns,
typeof-shape checks TypeScript already enforces, and self-comparisons.
The largest group read a production `.ts` file and asserted on its text —
for example a TaskPage test that required the source to contain
`selectedRepos.find((r) => r.id === newIssueRepoId) ?? selectedRepos[0] ?? null`.
Any behavior-preserving rename broke it; no behavior change ever did.
Production-side follow-through: exports that only these tests imported are
de-exported or deleted, stale comments pointing at removed censuses are
dropped, and the reliability-gate registry, `cloud/package.json` test lists,
and orphaned source-reading helpers are updated so nothing references a
deleted file.
Two files kept their real coverage and lost only the census scaffolding:
`agent-status-producer-census.test.ts` now drives all five producers end to
end instead of grepping the source tree, and `config-toml-trust-stale-writes`
replaces an export-list parity check.
* fix(relay): drain a same-cap cell over 5 minutes, not 2
The 2026-09-23 c27 roll drained 2,145 hosts over the 2-minute window,
about 18 re-dials/s, while the director re-places roughly 8/s through
its single-slot sticky lane. The overflow queued behind slow
re-placements and timed out, so /v1/assign returned 503 fleet-wide for
about 5 minutes. 5 minutes is the cell's maximum pace window and keeps
the remaining 2,650-host cells near lane capacity. The transition wait
already outlasts a 5-minute window (17 min).
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* test(relay): pin the same-cap drain window contract at 5 minutes
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): keep the drain wait at lease plus the 5-minute window
The transition wait after a drain was set to the 15-minute migration
lease plus the pace window. Widening the window to 5 minutes without
moving the wait left 12 minutes for a migration that can hold for 15,
so a late-window migration would time the wave out into rollback.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
Jinwoo Hong
Neil