* chore(cloud): update relay runtimes and security scanning
* docs(cloud): record Go binary advisory reachability
* chore(cloud): rerun security checks after resolving main conflict
* fix(relay-ops): make each reserve-watch sample one Cloud Logging read
The project allows 60 ListLogEntries a minute shared by every watch and tool. Each watch sample
made four to six reads (two director runtime-metrics sub-windows, the ledger summaries, the
seat-behind-row lines, the cell's phone metrics, and since #31484 the 503 request log), so on
10-21 18:39Z twelve watches drew 429s until c21 and c22 tripped unverifiable.
A sample now sends its queries as one OR filter and splits the lines back client-side. The
per-instance 503 split counts each director's own logged /v1/assign 503s from the runtime
metrics it already reads, with the rest of the fleet's count as one unattributed instance,
instead of reading the request log.
* fix(relay-ops): judge unattributed 503s on their 3-minute sum against the whole fleet line
* docs(relay-ops): state that an instance that stops logging trips the unattributed rule
* fix(relay-ops): keep one stalled director instance from tripping the reserve watch's 503 rule
The reserve watch's director503 rule counted fleet-wide non-drain 503s. On 10-21 07:55Z one
director instance's database calls stalled for 3 minutes; its 503s alone carried the fleet over
the line and all four watched cells flipped back to db.
Over the fleet line the rule now also needs at least 2 director instances each above a fifth of
it for the same 3 minutes, counted from Cloud Run's request log by labels.instanceId less each
instance's scheduled 503s. Above max(10x background, 200) for 3 minutes it trips with no split.
* fix(relay-ops): size the instance share to the directors seen and count platform 503s
* fix(relay-ops): size the instance share per judged minute and refuse a missing count
The deploy passed `--max`, which is gcloud's service-wide cap. Cloud Run divides
that cap among the revisions taking traffic, so during the 20/80 stage a
candidate under a cap of 5 got one instance while the serving floor kept four,
and the spread check could never see the full floor (deploy M, 1 of 5 for six
minutes). Candidates now pin `--max-instances` per revision, and the deploy
resets a service cap below two full revisions to Cloud Run's default before it
creates any candidate. The rollout overlap of two full directors is already in
the checked Cloud SQL connection budget.
* fix(relay): move director traffic in two steps so a deploy does not land on one instance
Cloud Run routes a new director revision's floor instances their first request
roughly 60-110 s after they start, while a traffic move reaches clients in
15-30 s. The one-step 100% move therefore put every assignment on the single
instance started with the revision, which sits at its own admission limit at
peak; the 2026-10-20 deploy returned ~550 assign 503s in two minutes that way.
With a floor above one, the deploy now moves 20% first, waits until Cloud Run's
request log shows every floor instance serving (six-minute deadline), then
moves the rest. A timeout or failure at that stage returns all traffic to the
previous serving revision, which still holds its warm floor. The staging deploy
identity gains log read access for the same check.
* fix(relay): bound the director spread log read, log the instances seen at timeout, and document split recovery
A hung gcloud log read could hold the 20/80 split past the six-minute deadline;
each read now times out after 30 s and counts as an unread poll. A timeout logs
the distinct instances seen, so a partial share packed onto fewer instances than
the floor shows up. The runbook covers restoring a split left by a cancelled job
and the missing selector-rollback tag after a timed-out deploy.
* fix(relay): judge the reserve watch's 503 rule on non-drain 503s, not a host's own re-dials
The watch counted every director 503, and on 2026-10-18 ~98% were host-rate-limited re-dials that
grow with traffic, so it flipped production-gce-c7 back twice with no fault. It now uses the roll
ladder's non-drain measure (request_count 503s less drain-return deferrals and host-rate-limited /
host-in-flight re-dials from the director's runtime metrics) for both the background and the live
samples. A judged minute no runtime-metrics line covers is unverifiable, not zero.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(relay): state the reserve watch's metrics coverage honestly and share the metrics read limit
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
The folder is the cloud relay that reaches phones; under its old name it read as the legacy SSH
relay. git mv plus import and path-literal rewrites only.
* feat(relay): watch a reserve cell and flip it back on the first step 5 stop rule
A manual, inert-until-dispatched workflow samples one reserve cell every minute for up to
300 minutes against the step 5 §9 stop rules, logs one JSON line per sample, and on the
first trip (or N unreadable samples in a row) runs the flag tool's own admitMode=db write
under the flag workflow's concurrency group. dry-run logs the flag tool's plan instead.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): keep the reserve watch's flip back out of the flag group, and baseline 503s before the flip
- The flip back holds no concurrency group: GitHub keeps one pending run per group and a
newer one cancels it. The flag tool's CAS orders it instead; a generation conflict is
re-read and retried (5 tries, backoff) while the switch file still says reserve.
- The 503 background comes from the 10 minutes before a required flipped-at, or an explicit
value; neither is refused.
- A failed or cancelled write-mode watch flips back too. An unreachable cell fails with the
exact break-glass inputs.
- Missing 503 data (empty series, unpublished minute, unparseable time) is unverifiable.
- Seat-behind-row key adds the director instance and cell; six segments share one command.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): read the reserve watch's 503 publishing front from every response code
A quiet fleet has no 503 series, so the watch read it as unverifiable and flipped a healthy
cell back within minutes. The 503 read now groups request_count by response code in one
query; the newest minute any code has a point for is how far the series is published, so a
503 minute at or before it with no point is zero and minutes past it are left out.
Also: the flip-back job's steps run on always(), flipped-at ages against the watch's start
in later segments, a pending pre-flip background no longer adds to the unverifiable streak,
the flip back retries runtime-status before calling for break glass, and the concurrency
rationale lives once in the docs.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): state what a cancel during the flip-back lease wait can leave, and the 503 publish-skew assumption
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): fail a segment closed on a corrupt watch state file, and keep no checkout token on disk
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): alarm on a failed flip back, refuse a watch under a rollout lease, wait for reserve to apply, and control the duplicate read
- A flip back whose write or plan never finished (no ID token, a failed login, a silent cell)
raises an ::error:: annotation and a FLIP BACK FAILED summary, as the clock-skew monitor does.
- The first segment reads the Cloud SQL rollout lease through the lease action's own reader and
refuses to start while any run holds it, or when it cannot be read.
- A cell not yet reporting reserve is waited on for up to 10 minutes, judging nothing, before the
watch ends refused without a write.
- The duplicate-seat read takes the per-minute ledger summary in the same query and window; with
no fresh summary in it the sample is unverifiable, not zero duplicates.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): page on a refused reserve watch or a failed flip back
A refused watch can leave a cell in reserve and unwatched, the same as a flip back that did not
land. Both now raise an escaped ::error:: annotation through one `alarm` command, and in write
mode write orca_relay_reserve_watch_alarm with gcloud logging write. A new log-based metric and
alert policy page on any such line, per cell, through relay_alert_notification_channels, and the
deploy identity gets roles/logging.logWriter in production to write it.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): name the log-writer risk, check the alarm's project, and say a failed login cannot page
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): note a force-cancel can skip the reserve-watch flip-back write
* fix(relay): end the restart-safe quiet at the drain's pace window, not a window after it
A paced drain tells its last host when the window ends, so the restart-safe
wait has to see an empty runtime reaching past that point. It counted a full
window of quiet samples from the first empty one instead, and the runtime only
empties as the window ends, so every same-cap cell waited the window twice:
about 25 minutes per Asia cell at 1200 s on 2026-10-13, and 5 minutes per
300 s cell.
The window is now measured from the verifier's own start, which runs after the
cell accepted the drain and so ends no earlier than the cell's own window. A
pass still needs two consecutive restart-safe samples, the last taken at or
after that end, and any non-zero runtime counter or open migration still
restarts the count. Progress lines report paceWindowRemainingMs in place of the
old sample target.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* fix(relay): drop the same-cap drain timeout's double-wait allowance and stale quiet wording
The drain step's timeout added the window's excess over 300000 only because
restart-safe used to wait a second window after the cell emptied, so it is
now the 15-minute migration lease plus the window for every pace (35 minutes
at 1200000, unchanged at 300000 and below). Docs, workflow comments and the
shadow gate's settledAfterSeconds note now describe the window-end rule. The
always-quiet test starts its clock late, so a window measured from zero
fails it, and the overlapping late-control test is folded away.
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): state the restart-safe window-end rule exactly
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* docs(relay): the slowest restart-safe wait is now 35 minutes
Claude-Session: f2fe8cb5-4467-4a25-8aab-98598b70791f
* feat(relay): declare Asia spare cell c35 as migration-only
Adds a seventh asia-east2 cell at the C34 shape (cap 3000, 6000 request
units, pool 16, e2-standard-4) in asia-east2-a, pinned to image F. It gets
its own topology and registration wave but no promotion wave, so the
admission script and workflow refuse to promote it; it stays a
migration-only spare and out of the fleet pool list. Mirrors #25336.
* test(relay): accept the c35 spare under the same-cap capacity allowlist
* fix(relay): keep verifying host tokens from the last served keys while auth is down
jose refetches the relay JWKS once its cache is 10 min old and rejects the token when that
fetch fails, so an auth outage turned every still-valid host token into a 4401. The cell now
falls back to the last set auth served for up to an hour, skips refetching for 30 s after a
failure, and still rejects any key that set never had. A fresh set replaces it as soon as auth
answers, so a rotation takes effect at the first successful fetch.
* fix(relay): bound stale relay keys at 30 min and document the emergency purge
Thirty minutes still covers a full auth outage (no token the stale keys could accept outlives
~16 min of one) and halves how long a rotated-out key stays trusted on a cell kept from auth.
A restarted process holds no set and refuses host tokens until a fetch succeeds; the ops doc
names that as the purge after a signing-key leak, and a test pins the fail-closed start.
* test(relay): pin the 30 min stale relay key bound (29 min accepts, 31 min rejects)
* docs(relay): director Cloud SQL sidecar enablement and rollback, as run on staging
The 0% apply revision is not started by Cloud Run, so the first sidecar boot is the deploy's
rollback candidate. A new log-based metric can take minutes to appear, so its alert policy
can fail the first apply. Both observed on staging 2026-10-11.
* docs(relay): production sidecar enablement needs the tfvars PR, a passed drill and the rollout lease
* feat(relay): slower same-cap drain paces (900000, 1200000) for every cell class
c28's 1,901-host drain at the 300 s pace (~6.3 hosts/s, 2026-10-07) held
database lock time over its bar for about 10 minutes. Add 15- and 20-minute
pace windows to the same-cap closed set, allowed on every cell class.
- The cell's /v1/admin/drain cap rises from 5 to 20 minutes.
- The drain script fails instead of falling back to an unpaced drain when a
cell rejects a window above 300000, since an older image answers both with
the same 400.
- The fast-pace US-only rule and the canary PASS rule apply only below the
default; a canary still authorizes its own pace or slower.
- The restart-safe timeout adds the window's excess over 300000 (quiet starts
after the last host leaves), unchanged at 300000/60000/30000; the job
timeout rises from 90 to 120 minutes.
* fix(relay): refuse a slow drain pace before isolating a cell that cannot accept it
The cell advertises its drain pace cap on /health; the same-cap job reads it
before the isolate, so a pace above an old image's 300000 cap stops with the
cell untouched instead of isolated. Also refreshes two stale timeout comments
and pins the drain step under its one-hour ID token and the job's 70-min
remainder.
* feat(relay): give Asia cell c34 a promotion wave so it can become a general cell
c34 launched on 2026-10-05 as a migration-only spare with no promotion path. This adds
it to the Asia admission promotion waves, the workflow's promote and canary cases, and
the canary evidence map, so the reviewed Asia workflow can promote it with the same
five-minute canary c30 and c31 ran.
The same-cap migration-only list is deliberately unchanged: a same-cap job reads a
cell's class from that list, and c34 must be rolled to the director's image as a
migration-only cell before promotion can run. The list moves after promotion, in its
own change.
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* chore(relay): move Asia cell c34 to the general same-cap lists after promotion
A same-cap job reads a cell's class from SAME_CAP_MIGRATION_ONLY_CELLS, not from the
selector. Once c34's Asia canary promotes it, leaving it there would make a same-cap
rollback demote it to migration-only. This moves c34 to the general same-cap list and
the shadow gate's fleet pool list (its pool is the Asia 16).
Merge only after c34's promotion succeeds. This file set is trusted evidence code, so
merging invalidates any sealed monitor or same-cap canary: merge outside a
monitor-to-enable window and before the next same-cap canary seals.
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* docs(relay): scope the c34 same-cap pause to the window after promotion
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* docs(relay): rewrap the c34 paragraph
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* feat(relay): give Asia cell c34 a promotion wave so it can become a general cell
c34 launched on 2026-10-05 as a migration-only spare with no promotion path. This adds
it to the Asia admission promotion waves, the workflow's promote and canary cases, and
the canary evidence map, so the reviewed Asia workflow can promote it with the same
five-minute canary c30 and c31 ran.
The same-cap migration-only list is deliberately unchanged: a same-cap job reads a
cell's class from that list, and c34 must be rolled to the director's image as a
migration-only cell before promotion can run. The list moves after promotion, in its
own change.
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* docs(relay): scope the c34 same-cap pause to the window after promotion
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* docs(relay): rewrap the c34 paragraph
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* fix(relay): publish the reviewed commit before main can move, and quiet the deploy driver
The publish workflow builds main's head at dispatch. The driver checked main
at preflight but dispatched the publish about four minutes later, after the
inspects and the typed phrase, so a busy main stopped the first real deploy.
It now dispatches the publish seconds after the check, before anything else,
and a build of a moved main stops with the --commit/--publish-run command that
deploys it once reviewed. Typed prompts end in a newline, and runs are
summarised (status changes plus every 5 min) instead of streaming gh run watch.
* fix(relay): a main-moved stop prints only the command that reuses the build
The generic re-run line named the reviewed commit without --publish-run, which
would only build the moved main again.
* feat(relay): add an operator-local driver for director deploys
One command runs the audited director deploy: preflight, pause rehome if
enabled, publish, deploy, optional cell configure, a digest-bound inspect,
the monitor dry-run, and re-enable. It only dispatches the existing
workflows, reads the published digest from the registry and the run log,
reads the monitor verdict from its sealed state, and enables with the
digests gcloud reports after the deploy. It stops at the first failure,
records state, and resumes from it.
* fix(relay): report and own the rehome pause; operator types every phrase
- Read the control back from pause and enable runs whatever their conclusion,
and report PAUSED or UNCONFIRMED loudly.
- Resume re-enables only the pause this driver recorded (generation and run).
- Ctrl-C and SIGTERM print the same state and resume report.
- The operator types every workflow confirmation. A 5-minute soak gated on
director 5xx runs before configure.
- A dry run keeps no state file. The quiet check pages through all runs.
Run IDs come only from the printed URL. One step table drives execute,
dry run and resume. The monitor verdict reuses verify-authority.
* fix(relay): read back only the driver's own rehome run; anchor the soak at the traffic switch
- The pause and enable read-back accepts only the control line its own step's mode prints, at the
generation its own dispatch expected. A run adopted after a crash is settled even when green.
- The soak window opens a minute before the deploy run completed and is read a minute after it
ends, for log ingestion lag.
* fix(relay): say what typing ENABLE_REGIONAL_REHOMING commits to
* fix(relay): the ENABLE prompt also names the 150 s evidence budget
* refactor(relay): derive every deploy decision from live state; no resume machinery
The driver keeps no state between runs. Each run reads the serving
director, its configured cells and the rehome control, and skips what is
already done.
- The only rehome fact it owns is the run that paused rehome. A re-run
names it (--pause-run), and the driver checks it against that run's log
and the live generation.
- A recover-enable line counts as the driver's own pause only with
recovered: true. A director safety pause is never adopted (F3).
- Publish runs before the pause. A fresh run that finds rehome paused
stops unless given --pause-run or --rehome-disabled (F2).
- Interrupts report a pause or enable still in flight as REHOME IS
CHANGING (F1). PAUSE UNCONFIRMED and ENABLE UNCONFIRMED are distinct.
- A tripped soak is judged again on fresh traffic (F5). SIGHUP is
handled, and a pending signal stops the driver before its next dispatch.
- Every stop prints the single command that finishes the deploy.
Removes the state file, --resume, step statuses, monitor adoption and
interrupted-dispatch adoption.
* fix(relay): never report done over an unexplained pause; prove pause ownership by actor
- G1: always read rehome; no early DONE.
- G2: --pause-run must be a rehome-control run by the same user.
- G3: a failed enable run is never an enable.
- C1: a pause or enable is reported as changing from the moment it is
dispatched.
- C2: --leave-rehome-paused (was --rehome-disabled) refuses an enabled
switch. Every printed command parses.
- G4: the enable-in-flight report prints both finishing commands.
- G5: the driver's own runs never block the quiet-lane check.
* test(relay): port the round-3 probes: hard kill mid-pause, unexplained disable after a failed enable
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* feat(relay): report lane service times, 503 causes, per-site hold p99 and lock-wait split
Adds the step-1 observability: drain-return and sticky slot service times,
every /v1/assign 503 by cause with a non-drain total, a p99 per lock hold
site (drain-return regional rows get their own site), a director-side
pg_stat_activity sample that splits lock waiters by director vs cell, and
a log line for each reserved_requests drift reconciliation corrects.
Log-based metrics for the new fields are declared in Terraform, not applied.
* fix(relay): classify a lock waiter by the first relay table its statement names
* fix(relay): attribute lock waiters to the root holder; document the director hold alert change
Every waiter after the first in a row-lock convoy is blocked by the first
waiter, so the sample now walks pg_blocking_pids to the root, reading it once
per waiter. The sampler is single-flight. Drain-return regional holds feeding
the non-paging director hold policy is documented as expected.
* docs(relay): note the lock-wait sample undercounts director waiters when the pool is full
* feat(relay): drain pace window as a reviewed same-cap input, with drain-aware 503 gates
The same-cap roll drained every cell over a fixed 300 s window, so a US roll
re-placed hosts at ~2/s and spent ~10 minutes draining and waiting for quiet.
The window is now a dispatch input from a closed set (300000, 60000, 30000),
defaulting to today's 300000.
- Below the default is refused for anything but US general cells; Asia drains
are bound by their targets' accept rate, and migration-only cells hold no hosts.
- A non-default window must be named in the confirmation, the canary authority
records it (v2), and a batch may run at its canary's window or slower only.
- Each cell job re-checks the window, scales the restart-safe timeout with it
(15-min lease + window, unchanged at the default), and records what the cell
applied and when it settled.
- The report-only shadow gate takes the director's drain-return deferrals out
of the 503 count (window and baselines), adds the rung's 5-min non-drain 503
budget and a Retry-After check, and reports the measured re-placement rate.
- relay-workflows.md documents the pace ladder and what each rung records.
* fix(relay): judge paced drains on counted 503s against the pre-drain minutes, and seal the canary's pace verdict
Review of #25639 replayed the shadow gate: it read 10-01 c29 as unverified (the
log read stopped at 20k entries), false-blocked 10-02 c22, and was blind on four
cells whose 24 h/48 h baseline held an incident.
- Director 503s now come from Cloud Run's request_count, aligned per minute by
Cloud Monitoring, so volume cannot truncate the count.
- The background is the median of the 10 same-day minutes before the drain;
the 24 h/48 h baselines are gone.
- Scheduled 503s come out: drain-return deferrals and sticky/placement answers
to a host's own early retry (host-rate-limited, host-in-flight), each split
across the minutes its 30 s sample covers.
- The rung budget counts only sustained breaches: two straight minutes over
max(1.5x, +20) warn, over max(2x, +40) would-block.
- The report carries a paceVerdict over the three pace checks. seal_canary
downloads the canary cell's report and seals that verdict; a batch below the
default pace needs PASS, from a report on the same cell that drained at that
pace.
- Docs: the step-down rule reads paceVerdict, 30 s waits for the lane service
time (#25645), and the staging step is dropped since staging drains unpaced.
Replayed read-only: 10-01 c29 would-block (9 minutes over 41.5/min); all nine
10-02 cells and 10-01 c25 paceVerdict PASS.
* fix(relay): a partial count already past a block line blocks, in the shadow gate's Cloud SQL and pool checks
A truncated FATAL count is a floor, and one runtime sample over the SQL-failure
line is a fact, so neither waits for a complete read. The waiter-run rule still
needs a complete run, since holes can join two runs into one.
* fix(relay): a canary pace PASS needs a real cohort and whole telemetry; one median-based 503 check
From the final review of #25639:
- canaryPaceVerdict seals PASS only from a report that drained at least 400
hosts (about half a 10-02 US cell), so a near-empty canary cannot authorize
a fast batch.
- A director-metrics sub-window with fewer samples than one instance emits
is unverified, so an empty or short Logging answer is never a calm drain.
- seal_canary names the shadow artifact, report path and cell from the gate's
normalized cell list, as cell_1 uploads it.
- director503 folds into nonDrain503Budget as a single-minute spike rule,
max(10x median, 200), dropping the pre-drain peak statistic. All 30 replayed
windows keep their verdicts.
Adds a sixth asia-east2 cell at the C31 shape (cap 3000, 6000 request
units, pool 16, e2-standard-4) in asia-east2-c, pinned to the f30b5cb1
cell image. It gets its own topology and registration wave but no
promotion wave, so the admission script and workflow refuse to promote
it; it stays a migration-only landing zone and out of the fleet pool list.
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
Both were promoted to general on 2026-10-01 (selector 344 shows them general),
but the same-cap job still classed them migration-only. Rollback mode on either
would isolate (general -> migration-only) before its no-op check failed, demoting
a live cell. They stay out of the fleet pool list (pool 10, not 16).
Claude-Session: 1145a80d-dec4-4a9b-9373-bbbb876b9041
* refactor(relay): sample fleet health inside the same-cap roll instead of a separate monitor run
A same-cap wave no longer consumes a 15-minute monitor dry-run and its sealed,
single-use, five-minute-fresh evidence. Each apply wave now samples fleet health
itself right before isolation, with the monitor's evaluator, thresholds, and
tolerances, for a window sized to the cell's host count (3/5/8 min), plus three
lookback rules: no cell container exit in 10 min, no minute over 500 director
503s in 10 min, and director concurrency p99 within the monitor bar over 4 min.
Removes the monitor-run inputs, the gate's consume/authorize steps, the
break-glass override, and the same-cap-only authorization shapes in
relay-monitor-evidence.mjs. The monitor workflow and the rehome enable path are
unchanged.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): bound the pre-drain sample overrun and keep the drain token fresh
Review follow-ups: alternating tolerated readings could hold the sample open
until its step timeout, so cap the overrun at three samples past the window;
record why a read failed; mint a fresh admin ID token for the drain after the
sample; raise the job timeout to 90 min so a long sample cannot cancel the
job past the failsafe.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* feat(relay): exempt the rolled cell and existing-only cells from the pre-drain crash rule
The exit rule counted every relay container exit fleet-wide, so a cell that
crashes every few hours (c25, 12 a week) blocked the very roll that fixes it,
and existing-only legacy cells (c5, 15 a week) blocked rolls they take no part
in. Exits are now grouped by instance, each instance is named by its own newest
runtime-metrics log line, and only exits on general or migration-only cells
other than the target count. An exit no configured cell can be named for trips
the rule; a failed lookup is a failed read. relay-observability.tf joins the
evidence-code set because the rule depends on its filter.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* test(relay): cover re-asking for an unnamed exiting instance; note the boot-exit risk
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* feat(relay): declare US cells c32 and c33 at the 3,000-host shape
Declares two us-central1 cells at the Asia shape (cap 3000, 6000 request
units, e2-standard-4) with the US default pool of 10, and generalises the
Asia topology and admission ladder to derive each wave's region from its
reviewed zone, leaving every Asia wave's behaviour unchanged.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* docs(relay): note the US canary tie-break and leave the fleet pool list to promotion
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): plan C32 and C33 as one topology wave
The live-image overlay refuses a declared non-target cell with no template,
so a lone C32 plan would fail on C33. Registration and promotion stay
one cell at a time.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): accept MIG version-name reconciliation and recreate stranded cells without rewriting the MIG
The stranded-rollback recovery ran a gcloud rolling action, which renames the
MIG version outside Terraform. Every later plan for that cell then reverted the
label, and the capacity-plan validator refused the revert as an unreviewed MIG
change, so the cell could be neither rolled nor rolled back.
The validator now accepts a MIG field moving back to what relay-gce-cells.tf
declares (version name and update policy), in every mode, and a test pins those
values to the Terraform file. The stranded branch recreates the cell's single
instance with recreate-instances, which leaves the MIG untouched.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): let a label-only MIG plan through and recreate on it in a stranded rollback
A stranded rollback whose template is already in place plans only the version
name revert. The validator still required the MIG template to move, so that
plan was refused, and the recreate gate (changes == 0) would have skipped a
plan of one change and left the drain flag set. Require the template move only
when no declared field reconciles, and recreate whenever the template was not
replaced (changes < 2).
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): anchor same-cap monitor evidence freshness to the run's authorisation, not job startup
The same-cap gate now verifies the dry-run on its own clock and records the
authorisation instant in the single-use consumed marker; each cell job checks
the evidence age at that instant and bounds its own start after it.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): refuse a re-run same-cap gate before it consumes evidence; tighten order tests
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
The parent workflow collapses canary-apply and batch-apply into the job mode
apply, so the headroom step's canary-apply/batch-apply condition never held and
the gate was skipped on every real roll. Run it wherever the drain runs (apply,
rollback before its restart) and in read-only verify; skip only a resumed
rollback, which drains nothing. A new workflow-shape test fails on any job
step comparing against a mode the parent cannot pass, and on a drain that can
run without the headroom check.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): gate the Asia canary on region fallbacks against a pre-canary baseline
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): gate the Asia canary on region fallbacks against a pre-canary baseline
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): let a same-cap drain finish when only unplaceable hosts remain
The c28 canary on 2026-10-01 drained the cell to zero live connections, but four
hosts with no free slot anywhere kept redialling and held director leases on it,
so the restart-safe wait timed out and left the cell isolated and empty.
The drain wait now also passes once the runtime has carried nothing for a
sustained quiet window while a small, capped number of leases remain, and logs
the escape. Apply modes also refuse a cell whose hosts exceed 80% of the free
slots on the other general cells, so a wave cannot strand hosts in the first place.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): define restart-safe by the cell runtime, not director leases
Replaces the opt-in stranded-host escape with a corrected definition. A
restart is safe when the cell runtime carries nothing live and no migration
is open, sustained for the drain pace window. Director activity leases lag
hosts that already left or cannot be placed, so they are reported in a
progress line and the verified result instead of blocking the restart.
The same-cap drain passes its existing pace window. The headroom script is
added to the trusted evidence code paths with the other production scripts.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): print stranded director counts on every restart-safe sample
Each restart-safe poll now prints its sample count and the director's
restart-blocking leases, request units, reserved remainder, and migrations
under `stranded`; the verified line carries the same object. Open migrations
still block because each is pinned to the cell incarnation a restart replaces.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): require the pace window for every live restart-safe wait
Pre-auth and total connections no longer reset the restart-safe window:
on drained c28 they flickered with unauthenticated redials in a third of
samples, which a restart does not lose. They stay in the progress output.
Every live restart-safe call must now pass --pace-window-ms. The capacity
job and staging proof drain unpaced, so they pass the production 300000 ms
window, and the calls that relied on the 180000 ms default get 480000 ms.
Headroom free slots now follow the director's placement rule: the admission
pause minus the larger of observed and enforced units, minus outstanding
control reservations.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
Delivery collapsed on 2026-09-29 once send volume doubled: the worker, the
retention pruner and the request path share a two-connection pool, and the
database transaction rate pinned at ~140/s regardless of how many notifications
were delivered. Raise the pool to six so worker and pruner stop serialising on
one connection. The budget precondition stays satisfied (2 x 6 x 3 = 36 <= 64).
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): lock only the target cell row, last and NOWAIT, in the rehome commit
The idle-rehome commit runs on the source cell. From an Asia cell each
statement is a cross-region round trip, and the transaction locked every
relay_cells row plus every runtime, capability and safety row before about
twenty more statements, so each Asia-source rehome held the whole fleet's
cell rows for ~3.6 s and every reconnect, renewal and placement queued or
timed out behind it.
The commit now reads the cell inventory and the runtime, capability and
safety tables unlocked, keeps the control and worker rows locked (now
NOWAIT), and takes one cell lock: the target row, in a single statement
that locks it NOWAIT, re-checks enabled, general admission and capacity,
and reserves the units, issued as the last statement before COMMIT. A
target that changed admission, filled up, or is locked by another writer
rolls the whole commit back and answers deferred (candidate-ineligible).
The hold is sampled under a site label, so cellInventoryHoldMsMax still
sees rehome holds and rehomeTargetRowHoldMsMax reports them apart.
Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb
* fix(relay): fail closed on the rehome target-row lock clause
The target-row statement now carries FOR UPDATE ... NOWAIT unless the
dialect is explicitly SQLite, so a wrapper that omits the optional dialect
can no longer run the reservation unlocked. Test wrappers and the fault
injection entry forward the dialect they wrap.
The latency test also probes the admission and region tables at every
round trip; only the target's admission row may be locked, and only
before COMMIT. The runbook notes that an Asia-sourced commit holds the
rehome control row for about 6.5 s, so a pause that fails once is retried.
Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb
* feat(relay): alert on relay cell table lock convoys
Adds a log-based metric and alert for cell-inventory lock holds of at
least 1,000 ms, and a Cloud SQL log metric and alert for relay-only lock
timeout cancels at 20 or more per minute. NOWAIT refusals are excluded:
background sweeps produce about 160 per minute even with rehome paused.
Replayed over 2026-09-20 14:00 to 2026-09-22 15:00 UTC: the hold filter
matches all 93 asia-east2 rehome holds plus 9 director holds, and every
one of the 88 cancel burst minutes overlaps an asia-east2 hold.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
* fix(relay): page only on cell lock holds; director holds stay visible
Director holds of 1-2.5 s recur several times a day with rehoming paused,
and pausing rehome does not stop them. The paging hold policy now selects
role=cell samples only; a separate policy with no notification channel
keeps director holds visible. The burst documentation no longer claims no
burst happens while paused, and the runbook points a burst with no cell
hold at the director policy.
Replayed cell-only: 93 of 93 asia-east2 holds, 0 director holds over
2026-09-20 14:00 to 2026-09-22 15:00 UTC; 0 from then to 2026-09-23 07:30.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
The source cell runs the rehome commit. An Asia source pays a cross-ocean
round trip per statement while holding relay_cells row locks every cell
needs, which convoys the fleet database. Selection now drops source cells
outside the director's region before building the decision window, so the
incumbent_region filter shrinks while Asia cells stay valid targets. The
preview counts the same hosts as source-outside-director-region and the
poll summary reports skippedOffRegionSourceCells. Temporary stopgap.
Claude-Session: ced32ebb-7155-4413-adad-1eccd14c2010
C30 was promoted to general on 2026-09-23 (selector generation 286). The
same-cap wave now rolls it as a general cell instead of handing it back
isolated, and the shadow gate reads its pool beside C27-C29. Follow-up to
#22375.
Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb
* fix(cloud): gate the Asia canary on its own cell's SQL failures, not the directors'
The production canary summed sqlFailuresDelta over every director and the
canary cell and required zero. Directors log a steady baseline of
relay_cells NOWAIT and lock-timeout refusals unrelated to the canary cell,
so a C30 canary failed most attempts on that noise. The canary now requires
zero SQL failures from the canary cell's own metrics and records the
director sum as directorSqlFailures without gating on it. Directors keep
every other rule (unavailable regions, fallbacks, pool waiting, transient
waiter and wait-time bounds). Staging keeps the combined zero rule and its
evidence shape unchanged.
Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb
* fix(cloud): gate the Asia canary's pool bounds on its own cell too
Directors also show a steady pool-wait baseline (waiting above zero and
waits over 50 ms in about 6 of every 60 minutes), so a five-minute canary
still failed about half the time on director pool pressure unrelated to
the canary cell. With gateDirectorDatabase off, the production canary now
applies databasePoolWaitingMax, databasePoolWaitersMax and
databasePoolWaitMsMax to the canary cell's metrics only and records the
director values under director-prefixed names. Directors still gate Asia
selections, region fallbacks and unavailable regions. The staging path
keeps its combined values, key order and validation order.
Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb
The topology workflow's Cloud SQL gate carried a hard-coded 400 for the
instance's tier default while the consumer contract records the value
measured on the live instance (SHOW max_connections = 500, 2026-09-16,
#21163). The gate compares the two and the first production plan run
(35815654836) failed silently on that mismatch before Terraform ran.
The verified default now lives beside the tier and version it is verified
for, as VERIFIED_DEFAULT_MAX_CONNECTIONS, so the contract and the workflow
are two independent records of the same measurement and the gate keeps
its cross-check. The test pins the new source and forbids a bare literal.
Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb
* feat(relay): declare Asia cell c30 at the c27 shape
Adds production-gce-c30 in asia-east2-a at the reviewed Asia shape (6,000
request units, 3,000/60 connection limits, 16-connection pool, disabled) and
the rehome trust the other Asia cells carry.
Every Asia enumeration now knows C30. The topology, admission, and director
tools treat it as its own reviewed wave so its plan and registration never
touch the live launch cells. C30 promotion requires C27 general and fresh
staging evidence. The topology and director validators now pin the committed
production pool of 16 instead of the stale 10, which had made the topology
workflow reject the committed launch cells.
* fix(relay): plan C30 at live images and prove it with its own canary
The shared URL map pulls every cell into the C30 topology plan, so the workflow
now plans each non-target cell at the image its live template serves, and the
validator names any change to a cell outside the wave. C30 promotion runs the
same five-minute production canary and automatic rollback C27 used, with the
load report proving the canary control was placed on C30, instead of relying
on staging evidence. C30 leaves the shadow gate's fleet pool list until it
serves, rollback rejects mixed partial sets, and a budget test pins the
mixed-Asia-pool refusal.
* fix(relay): pin C30 to the production director's live image digest
C30 promotion requires the director and C30 to report one digest, so C30
takes the director's sha256:4158d8a2 (read 2026-09-22). C27-C29 keep their
committed lines; every Asia check compares only the cells named in a run.
* fix(relay): read the committed cell map from a plan, not console
terraform console evaluates every output against state, and the Relay
deployments output indexes each cell's MIG, so it fails with Invalid index
while C30 is declared but not created. Read the map from a no-refresh,
unlocked plan over the same targets instead, and refuse empty overlay input.
* fix(relay): keep console readers working and C30 migration-only until promotion
relay_gce_cell_deployments indexed each cell's MIG, backend, and template,
so once C30 is declared but not applied every production terraform console
reader printed a warning to stdout and broke its jq parse. Wrap those six
lookups in try(..., null).
Same-cap listed C30 as general, so a rollback dispatch on a migration-only
C30 would restore it with activate and skip its canary. List it with the
migration-only cells until the promotion follow-up moves it.
* fix(push): bound the delivery claim and stop keeping finished batches
* fix(push): bound claim scans to the notification TTL and document the queue
* test(push): boot waits out a table writer for the queue indexes; queue stays correct without them
* fix(push): share the claim lock so a previous-revision claim cannot re-lease a delivery
During a deploy overlap the previous revision's claim scans under an exclusive
push-worker-claim lock and re-reads the row without checking lease_until, so it
could overwrite a lease this revision had just committed and send twice. The
new claim now takes the same key shared: new claimers never block each other,
and the previous claim waits until their leases commit before it scans.
Droppable one release after every worker runs this revision.
* fix(push): install one queue index at boot, not three
push_batches_leased_device indexed lease_until, so every lease, renew and finish
UPDATE lost heap-only eligibility and rewrote every index. push_batches_pending_due
was unused: on a synthetic 902k-row table the candidate scan plans onto the
existing (state, due_at) and expiry indexes with or without it. The per-device
pending index stays; the head check and the busy anti-join use it. Fewer
boot-time builds also shorten the SHARE lock the first boot takes on the table.
* test(push): pin the claim's TTL scan bound and the server's worker connection cap
Removing either guard left the suite green. The claim test captures every row
the candidate scan returns and plants one row that only the TTL term excludes;
the server test drives the real worker through createPushServer and fails when
the request-connection reservation is unwired (peak 4 instead of 2).
* fix(push): renew delivery leases outside the background connection cap
Renew shared the single background slot with claim retries and prune batches,
so a heartbeat could wait long enough for a lease to lapse and the delivery to
be re-leased mid-send. It is a keyed one-row UPDATE, so request traffic cannot
starve it on the ungated pool.
* docs(push): describe the shared claim lock for mixed-revision deploys
* fix(relay-ops): roll a cell a wave stranded after its drain
A wave that stops any time after its drain leaves the cell migration-only and
draining on the rollback image, and nothing clears it: the drain flag is a
one-way latch on the running process, and the failsafe restarts nothing. Both
recovery modes then refuse the cell. Apply wants it general and not draining.
Rollback sees the rollback image, reads it as a resume, refuses the draining,
and would not have restarted it anyway.
The image alone cannot separate a rollback that failed after its template apply
from a wave that stopped before one. The restart can: the first left a fresh
process, the second did not. Classify on that, so the cell that never restarted
takes the rolling path instead of the resuming one.
Its template still carries the image it serves, so that is the predecessor its
plan is reviewed against, and a template already moved on to the target is
refused rather than rolled backwards under a stale review. When the reviewed
template is already in place the plan changes nothing, so the MIG is rolled
explicitly on the same replacement policy a template change uses; the existing
incarnation check is what proves the instance came back.
Every other combination of mode, live image, and drain flag keeps the value it
had, held by a census that runs the real block over all nine.
* fix(relay-ops): pin the replacement method on the explicit MIG roll
gcloud persists every rolling-action bound into the group's update policy, and
it defaults the replacement method to substitute on a group with no stateful
config. Passing surge and unavailable without the method would patch the policy
off the declared RECREATE, and the next targeted plan would then carry a MIG
change outside version.0.instance_template, which the plan validator refuses.
Pass all three so the patch is identical to the declared policy, and read the
declared values in the census instead of restating two of them. Dropping the
flag, or moving any of the three in Terraform, now fails the census.
c17's canary stopped at the pre-apply predecessor check with
`runtime predecessor mismatch fields=draining`. The flag is residue: the
previous canary (run 35290908836) drained c17 at 00:26:01, its terraform apply
then failed, and the failsafe re-isolates without restarting the VM, so nothing
cleared it. The same run had passed this very check a second earlier, which is
what proves a parked cell is not draining at rest.
Draining means connections are being shed, and a migration-only cell holds
none, so the flag is not a precondition there. Accept it on entry for that
class only. The replacement VM is still required not to be draining, on every
path, and the incarnation check still proves it was replaced.
Both predecessor checks now read one decision instead of computing the rule
twice, so the assertion and its diagnostic cannot disagree. Every general-cell
and rollback path keeps the value it had; a census test runs the real block
over all eight mode and class combinations to hold that.
c17's canary-apply failed closed at plan validation. Its instance template is
from 2026-08-07 and predates the ORCA_RELAY_CAPACITY_SERVICE_ACCOUNT line that
every cell rolled since already carries, so the plan legitimately added it. The
same-cap validator holds the whole startup script identical before and after
except the image, and that line is not one it excluded, so the wave stopped
with nothing applied.
Pin the line for same-cap-cell exactly as bootstrap-cell already does, and
exclude it from the before/after comparison. The cell may gain it; the pin is
what refuses a roll that drops it or rewrites it to another identity. Both plan
validations in the job now pass the capacity identity the job already requires.
The same-cap contract is otherwise unchanged: any other stale line still fails
closed, and needs a convergence apply before the cell can roll.
c17 and c18 hold no hosts and sit outside general admission, so rolling one
displaces nobody. They are the only zero-displacement canary for a new cell
image, but the same-cap wave refused them at the dispatch validator and would
have promoted them to general at the end if it had not.
Add them to the approved list and teach the wave a cell's entry admission
class: the precheck demands the class the cell is declared to serve in, the
restore hands it back that class, the isolate on an already-isolated cell is
asserted to change nothing, and the selector generation advances by 2 for a
general cell and by 0 for a migration-only one. One wave may not mix the two,
because every cell after the first offsets from a single per-wave delta.
Neither cell is a declared regional-rehome source, so its template carries no
rehome trust lines. The source-membership guard now fires exactly when a roll
expects those lines instead of for every US cell, which is the invariant it
was standing in for, and which limits c17 and c18 to rehome protocol 0.
* feat(relay): add a break-glass override for the same-cap monitor gate
Every mutating same-cap wave consumes a fresh 15-minute aggregate monitor
dry-run. When a chronic fault is what the gate freezes on, waiting for a green
window means waiting for the condition the wave removes: the gate froze 44
consecutive times on the recurring Cloud SQL stall the rolling image fixes.
Add `gate-override-reason` and `gate-override-confirmation`
(`SKIP_RELAY_MONITOR_GATE <target-image-digest>`) to the same-cap dispatch. A
valid pair skips only the aggregate evidence download, provenance verification,
and single-use marker. A partial or mismatched override fails closed before any
mutation, in both the caller and the reusable job. Record the actor, reason, and
confirmation in the gate run summary and, for a canary, in the sealed artifact.
The live per-wave preflight still runs. Give it a `--no-monitor-state` source
that takes the expected selector from the dispatch inputs and pins the migration
policy to `strict`, rather than synthesising a state file that would claim a
dry-run it never ran.
Also give `director.instances` the two-consecutive-sample tolerance the cell
probes have: Cloud Run replaces an instance in place, so the count leaves the
[5, 6] band for one sample roughly twice a day, and a deploy overlap raises it
the same way. Min and max share one streak so an alternating count still freezes.
* fix(relay): canonicalise the break-glass preflight membership
The override path parsed the operator's membership with a bare schema parse,
while the live selector read from the director is normalised and the comparison
is an ordered `JSON.stringify`. Unsorted dispatch input would therefore read as
selector drift on a healthy fleet, and the every-configured-cell-exactly-once
check was lost with it.
Normalise through the same `normalizeSelectorMembership` call the monitor CLI
uses when it seals evidence, against the same durable Terraform cell set.
Tests use a collect stub that returns the director's canonical selector rather
than echoing the expected one, so the ordering is actually exercised: unsorted
input must canonicalise, and a duplicated, missing, or unknown cell must be
rejected.
An unreadable sample (collector_failed) now gets the same two consecutive
sample budget per source as an unread signal, so one failed Cloud Monitoring
read no longer restarts the continuous window. monitor_gap keeps zero
tolerance because it means the run itself stopped sampling.
The pre-drain lineage cap moves from 25 to 35 minutes so a 15-minute window
plus one restart still reaches a verdict, and the collector error message is
now logged instead of being swallowed.
* fix(cloud): give same-cap waves ten minutes to consume gate evidence
The live preflight rejected monitor evidence older than five minutes, but
the same-cap job only reaches that step about five minutes after the
monitor completes: runner queue, the gate job, and a full-branch checkout.
On 2026-09-17 the first green gate in 44 attempts died at 302 s. The
preflight still takes live samples, so the older baseline is safe.
* docs(cloud): state the ten-minute preflight evidence bound
* fix(relay): treat pool connect failures as transient, not director faults
pg-pool raises connection-acquire failures as a plain Error with no SQLSTATE,
so the transient classifier matched only one of the three messages it can
produce. The other two reached the routes unclassified and became HTTP 500s,
which is what the rollout safety gate counts.
The acquire boundary now marks the errors it produces, so "Connection
terminated unexpectedly" counts as transient when the socket died during the
handshake and stays a hard failure mid-statement, where a retry could repeat a
commit whose outcome is unknown.
/v1/regions and /v1/admin/evacuation-status gain the transient handling
/v1/assign and /v1/resolve already had.
* fix(relay): mirror the pool-connect verdict in failure diagnostics
The query-failure event's connectionTimeout boolean matched one of the two
messages connectionTimeoutMillis can produce, so the 210 dialling timeouts in
the last day logged as false and were invisible to the field meant to find them.
The pool-connect vocabulary now lives beside the acquire boundary that owns it,
and both the router's classifier and the diagnostics read it from there, so the
two cannot drift. The event also carries the routing verdict the caller already
computed, making "how much of this burst reached users as a 500" one field.
* fix(relay): null-safe transient classification and honest transient docs
The classifier now runs inside the query catch, where a thrown null or
undefined would have turned a database failure into a TypeError that buried it.
The diagnostics doc claimed transient maps to a 503 or a 500. Sweeps, startup
reconciliation, and admin routes that answer 409 all emit the same event, so
counting the false ones over-states user-facing hard failures.
Neil
Jinwoo Hong
