* fix(relay): bound the same-cap shadow gate and apply a resumed backend update
Two findings both adversarial reviews of tonight's merged set agree on.
The report-only shadow health gate (#21849) had `continue-on-error: true` but
no step timeout. That bounds the step's contribution to the job outcome, not
its clock. Its reads are serialised, and a failure that answers nothing slowly
— an expired credential, a project-wide Logging 429 storm — makes every read
cost its full 3 x 60 s retry budget, so the cost scales with the roll window:
roughly 8S + 2 reads for S ten-minute sub-windows. A 40-minute window is about
34 reads, or 108 minutes, against the job's `timeout-minutes: 75`. A cancelled
job cannot be absorbed by continue-on-error, fires the failure-gated cleanup
isolation on an already-restored cell, and stops the strict next-cell chain.
Give the step `timeout-minutes: 5` and the artifact upload `timeout-minutes: 2`.
A timed-out step is a failed step, which continue-on-error covers, so the job
stays green. Inside the script, stop reading after an overall four-minute
deadline and report the remaining checks unverified, so the normal outcome is a
written verdict rather than a killed process; the step timeout is then only for
a hung process. The census test pins both timeouts and that the deadline leaves
the step time to write its verdict.
The resume branch (#21860) accepted `changes == 0` with a non-empty
`backendUpdate` as complete and applied nothing, so a resumed cell silently
kept the 300-second drain and no request logging behind a green resume. That
shape means the template and MIG are converged and only this cell's reviewed
backend update is left, so apply the saved resume plan — the validator has
already bounded it to this cell's backend and neither attribute restarts an
instance — then continue as converged. Template-and-MIG drift still applies
nothing, which is what a resume means, and a stranded cell's explicit MIG
replace is unchanged.
Claude-Session: relay-same-cap-gate-timeout-and-resume
* fix(relay): raise the shadow gate bounds clear of a healthy gate's read time
A healthy gate is already minutes of serial reads on the 2-vcpu runner, so a
four-minute deadline would report unverified tails on ordinary days and stop
the shadow roll measuring the comparison it exists for. Raise both together:
the step to eight minutes and the script's own deadline to seven, keeping the
census pin that the deadline leaves the step room to write its verdict. The
job budget is unaffected: a ~14-minute cell plus eight is well inside 75.
Claude-Session: relay-same-cap-gate-timeout-and-resume
* chore(relay): target each cell's backend service from the same-cap job
The 60 s connection drain timeout merged in #21848 has no safe apply path.
A root plan scoped to the backend services alone still pulls every
`google_compute_instance_template.relay_gce_cell` in as a dependency, and
standing image drift turns all 29 into replacements, so applying it would roll
the fleet at once.
Add `google_compute_backend_service.relay_gce_cell["${TARGET_CELL_ID}"]` to
both plan invocations in the per-cell same-cap job, next to the template and
MIG it already targets, and teach the reviewed plan validator to allow exactly
one extra change: an in-place update of that one cell's backend whose only
changed attribute is `connection_draining_timeout_sec`, landing on the
constant `validate-relay-asia-topology-plan.mjs` exports. Any other attribute,
any other resource, or a backend for another cell still fails the validator.
The accepted update is reported as `connectionDrainUpdate` and kept out of
`changes`, so the apply step's stranded branch and the resume step's drift
branch keep reading the template-and-MIG count they were written against; the
resume branch additionally accepts a plan whose only pending change is that
drain update, which restarts nothing.
Claude-Session: relay-same-cap-targets-cell-backend
* fix(relay): also let the same-cap wave apply this cell's LB request logging
A read-only production plan for production-gce-c7 showed the live US cell
backends carry no `log_config` at all, while relay-gce-cells.tf has declared
`log_config { enable = true, sample_rate = var.relay_gce_cell_log_sample_rate }`
on every cell backend since the Terraform root landed in 3eec77c11a (#18413).
Nothing has applied it because every production apply since is a per-cell
targeted plan that names only the template and the MIG.
So the real canary plan's backend moves two paths, not one:
`["connection_draining_timeout_sec", "log_config.0"]`. The drain-only validator
rejected exactly that plan, which would have stranded the cell mid-wave after
the drain had already started.
Accept both, each optional, for this cell's backend only: the drain landing on
RELAY_CELL_CONNECTION_DRAIN_SECONDS, and a log_config of exactly one block with
`enable = true` and `sample_rate` equal to RELAY_CELL_LOG_SAMPLE_RATE, the
declared default of a variable no environment file overrides. Any third path,
a different sample rate, disabled logging, another cell, or a replacement still
fails. The accepted paths are reported as `backendUpdate`, which the resume
branch now reads instead of the drain-only flag.
Verified against the real production plan: the masked, three-target plan for
production-gce-c7 contains exactly that cell's template, MIG, and backend
service and nothing else, and this validator returns
`{"changes":2,"backendUpdate":["connection_draining_timeout_sec","log_config.0"]}`.
Claude-Session: relay-same-cap-targets-cell-backend
* perf(relay): cut the cell LB drain to 60s and widen the same-cap batch to ten cells
Two independent sources of relay roll wall clock, neither of which protects a
host:
1. `connection_draining_timeout_sec` on the per-cell backend services was 300s.
The same-cap job drains every host off the cell to a restart-safe condition
before Terraform runs, so the LB drain only ever covers a host still
mid-handshake. Measured 2026-09-16 over ten same-cap cell jobs, it sat as
~5m55s of dead time between `Apply complete` and the old VM powering off,
inside an 8.5-minute `wait-until --stable` step. Now 60s, and pinned in the
topology `check` block beside the other fixed-one invariants.
2. The same-cap wave capped a batch at four cells, so a 22-cell roll needed six
batches, six single-use monitor gates, and a human handoff per batch. The
wave workflow now declares cell_1..cell_10 with the identical serial shape
and chaining, and the validator accepts two to ten.
The shared wave-index rule (`relay-monitor-evidence.mjs` and the relay-ops
preflight CLI) widens from 0-3 to 0-9 so the later cells can present the same
evidence; each job workflow keeps its own narrower range, so the capacity wave
stays at four. Cells remain strictly serial, one at a time behind the rollout
lease, each with its own live preflight.
Claude-Session: https://claude.ai/session/relay-roll-drain-timeout-and-batch-cap
* fix(relay): align the Asia topology plan validator with the 60s cell drain
`validate-relay-asia-topology-plan.mjs` rejected any Asia backend whose
`connection_draining_timeout_sec` was not 300, and
`cloud-deploy-relay-asia-topology.yml` targets
`google_compute_backend_service.relay_gce_cell["<cell>"]` per cell. With the
Terraform local at 60 that workflow would have failed its own plan review.
The validator's two restated topology values are now named exports, and a new
census test reads `relay-gce-cells.tf` and equates three statements of each:
the `relay_gce_topology` local, the topology `check` assert that pins it, and
the validator constant. Terraform cannot export a local to JS, so reading the
source is the only way to stop them drifting; the test was confirmed to fail
when the local alone is moved back to 300.
Repo-wide grep finds no other pin of the drain value.
Claude-Session: https://claude.ai/session/relay-roll-drain-timeout-and-batch-cap
c17's canary-apply failed closed at plan validation. Its instance template is
from 2026-08-07 and predates the ORCA_RELAY_CAPACITY_SERVICE_ACCOUNT line that
every cell rolled since already carries, so the plan legitimately added it. The
same-cap validator holds the whole startup script identical before and after
except the image, and that line is not one it excluded, so the wave stopped
with nothing applied.
Pin the line for same-cap-cell exactly as bootstrap-cell already does, and
exclude it from the before/after comparison. The cell may gain it; the pin is
what refuses a roll that drops it or rewrites it to another identity. Both plan
validations in the job now pass the capacity identity the job already requires.
The same-cap contract is otherwise unchanged: any other stale line still fails
closed, and needs a convergence apply before the cell can roll.
c17 and c18 hold no hosts and sit outside general admission, so rolling one
displaces nobody. They are the only zero-displacement canary for a new cell
image, but the same-cap wave refused them at the dispatch validator and would
have promoted them to general at the end if it had not.
Add them to the approved list and teach the wave a cell's entry admission
class: the precheck demands the class the cell is declared to serve in, the
restore hands it back that class, the isolate on an already-isolated cell is
asserted to change nothing, and the selector generation advances by 2 for a
general cell and by 0 for a migration-only one. One wave may not mix the two,
because every cell after the first offsets from a single per-wave delta.
Neither cell is a declared regional-rehome source, so its template carries no
rehome trust lines. The source-membership guard now fires exactly when a roll
expects those lines instead of for every US cell, which is the invariant it
was standing in for, and which limits c17 and c18 to rehome protocol 0.
* feat(relay): add a break-glass override for the same-cap monitor gate
Every mutating same-cap wave consumes a fresh 15-minute aggregate monitor
dry-run. When a chronic fault is what the gate freezes on, waiting for a green
window means waiting for the condition the wave removes: the gate froze 44
consecutive times on the recurring Cloud SQL stall the rolling image fixes.
Add `gate-override-reason` and `gate-override-confirmation`
(`SKIP_RELAY_MONITOR_GATE <target-image-digest>`) to the same-cap dispatch. A
valid pair skips only the aggregate evidence download, provenance verification,
and single-use marker. A partial or mismatched override fails closed before any
mutation, in both the caller and the reusable job. Record the actor, reason, and
confirmation in the gate run summary and, for a canary, in the sealed artifact.
The live per-wave preflight still runs. Give it a `--no-monitor-state` source
that takes the expected selector from the dispatch inputs and pins the migration
policy to `strict`, rather than synthesising a state file that would claim a
dry-run it never ran.
Also give `director.instances` the two-consecutive-sample tolerance the cell
probes have: Cloud Run replaces an instance in place, so the count leaves the
[5, 6] band for one sample roughly twice a day, and a deploy overlap raises it
the same way. Min and max share one streak so an alternating count still freezes.
* fix(relay): canonicalise the break-glass preflight membership
The override path parsed the operator's membership with a bare schema parse,
while the live selector read from the director is normalised and the comparison
is an ordered `JSON.stringify`. Unsorted dispatch input would therefore read as
selector drift on a healthy fleet, and the every-configured-cell-exactly-once
check was lost with it.
Normalise through the same `normalizeSelectorMembership` call the monitor CLI
uses when it seals evidence, against the same durable Terraform cell set.
Tests use a collect stub that returns the director's canonical selector rather
than echoing the expected one, so the ordering is actually exercised: unsorted
input must canonicalise, and a duplicated, missing, or unknown cell must be
rejected.
* fix(cloud): pin the asia cell database pool in the same-cap plan validator
Raising `database_pool_max` from 10 to 16 for production-gce-c27, c28 and c29
made every same-cap roll of those three cells fail closed at plan validation.
The cell startup template emits `ORCA_RELAY_DATABASE_POOL_MAX` only for a cell
whose region differs from the root region or whose pool is off the default, so
the asia cells carry that line while the us-central1 cells do not. The plan
validator requires the before and after startup scripts to normalize to the
same text, masking only the lines it independently pins to a reviewed value.
The pool line was neither masked nor pinned, so the live template's `'10'` and
the plan's `'16'` were read as unreviewed drift.
The validator gains an optional `--database-pool-max`, accepted in
`same-cap-cell` mode alone. When it is supplied the after-script must contain
exactly that pool line and the line is masked from the equality check; when it
is not supplied the after-script must contain no pool line at all. Masking
without the pin would have removed the guard rather than moved it.
The same-cap job resolves the expected pool next to the hard cap, cross-checks
it against the committed `relay_gce_cells` map (asserting the default 10 for
the us-central1 cells), and passes the flag to both validator invocations only
for the cells that emit the line.
* test(cloud): require the pool pin for a line the live template already carries
The five `node ... | tee` steps in cloud-operate-relay-production-rehome-job.yml
reported tee's exit code, so a thrown inspect or apply passed green. The Aug 28
21:25Z and Aug 29 inspects and today's first inspect all printed
"director returned an invalid regional rehome control" (the durable control had
moved to generation 12 when the Aug 28 rehome aborted) and still succeeded.
`shell: bash` adds `-o pipefail`. A test pins the default and the tee count.
Phase 6 of the relay split: the relay's deploy/operate surface moves under cloud/ with 24 cloud-* workflows gated on ORCA_CLOUD_OPERATIONS_ENABLED, the Cloud SQL rollout lease action, the relay Terraform root (dual-accept identities for both repositories), scripts, docs, CODEOWNERS, and a terraform validate job in Cloud Verify.
Jinwoo Hong