* fix(relay-ops): roll a cell a wave stranded after its drain
A wave that stops any time after its drain leaves the cell migration-only and
draining on the rollback image, and nothing clears it: the drain flag is a
one-way latch on the running process, and the failsafe restarts nothing. Both
recovery modes then refuse the cell. Apply wants it general and not draining.
Rollback sees the rollback image, reads it as a resume, refuses the draining,
and would not have restarted it anyway.
The image alone cannot separate a rollback that failed after its template apply
from a wave that stopped before one. The restart can: the first left a fresh
process, the second did not. Classify on that, so the cell that never restarted
takes the rolling path instead of the resuming one.
Its template still carries the image it serves, so that is the predecessor its
plan is reviewed against, and a template already moved on to the target is
refused rather than rolled backwards under a stale review. When the reviewed
template is already in place the plan changes nothing, so the MIG is rolled
explicitly on the same replacement policy a template change uses; the existing
incarnation check is what proves the instance came back.
Every other combination of mode, live image, and drain flag keeps the value it
had, held by a census that runs the real block over all nine.
* fix(relay-ops): pin the replacement method on the explicit MIG roll
gcloud persists every rolling-action bound into the group's update policy, and
it defaults the replacement method to substitute on a group with no stateful
config. Passing surge and unavailable without the method would patch the policy
off the declared RECREATE, and the next targeted plan would then carry a MIG
change outside version.0.instance_template, which the plan validator refuses.
Pass all three so the patch is identical to the declared policy, and read the
declared values in the census instead of restating two of them. Dropping the
flag, or moving any of the three in Terraform, now fails the census.
c17's canary stopped at the pre-apply predecessor check with
`runtime predecessor mismatch fields=draining`. The flag is residue: the
previous canary (run 35290908836) drained c17 at 00:26:01, its terraform apply
then failed, and the failsafe re-isolates without restarting the VM, so nothing
cleared it. The same run had passed this very check a second earlier, which is
what proves a parked cell is not draining at rest.
Draining means connections are being shed, and a migration-only cell holds
none, so the flag is not a precondition there. Accept it on entry for that
class only. The replacement VM is still required not to be draining, on every
path, and the incarnation check still proves it was replaced.
Both predecessor checks now read one decision instead of computing the rule
twice, so the assertion and its diagnostic cannot disagree. Every general-cell
and rollback path keeps the value it had; a census test runs the real block
over all eight mode and class combinations to hold that.
c17's canary-apply failed closed at plan validation. Its instance template is
from 2026-08-07 and predates the ORCA_RELAY_CAPACITY_SERVICE_ACCOUNT line that
every cell rolled since already carries, so the plan legitimately added it. The
same-cap validator holds the whole startup script identical before and after
except the image, and that line is not one it excluded, so the wave stopped
with nothing applied.
Pin the line for same-cap-cell exactly as bootstrap-cell already does, and
exclude it from the before/after comparison. The cell may gain it; the pin is
what refuses a roll that drops it or rewrites it to another identity. Both plan
validations in the job now pass the capacity identity the job already requires.
The same-cap contract is otherwise unchanged: any other stale line still fails
closed, and needs a convergence apply before the cell can roll.
c17 and c18 hold no hosts and sit outside general admission, so rolling one
displaces nobody. They are the only zero-displacement canary for a new cell
image, but the same-cap wave refused them at the dispatch validator and would
have promoted them to general at the end if it had not.
Add them to the approved list and teach the wave a cell's entry admission
class: the precheck demands the class the cell is declared to serve in, the
restore hands it back that class, the isolate on an already-isolated cell is
asserted to change nothing, and the selector generation advances by 2 for a
general cell and by 0 for a migration-only one. One wave may not mix the two,
because every cell after the first offsets from a single per-wave delta.
Neither cell is a declared regional-rehome source, so its template carries no
rehome trust lines. The source-membership guard now fires exactly when a roll
expects those lines instead of for every US cell, which is the invariant it
was standing in for, and which limits c17 and c18 to rehome protocol 0.
* feat(relay): pace the drain send during a same-cap cell roll
A same-cap roll drains a cell with graceMs 0, which sends `drain` to all
~800 controls in one pass. Every desktop re-dials on receipt regardless of
graceMs, so the whole cell reconnects inside a second. On 2026-09-16 that
stampede hit a Cloud SQL stall: attaches timed out, each leaving 10 minutes
of late-arrival debt on connection headroom, and placement answered
relay_capacity_exhausted fleet-wide for ~13 minutes.
Spreading the sends spreads the re-dials. `HostSessionRegistry.drain` takes
an optional pacing window and schedules each session's send evenly across
it; admission is fenced for every session up front, and each host keeps its
own full grace after its own send. /v1/admin/drain accepts `paceWindowMs`
(<= 5 min) and echoes what it applied. The same-cap job asks for 120 s, and
the drain-completion wait grew by the same amount. A cell still on an older
image rejects the field, so the deploy script falls back to an unpaced
drain rather than failing the roll.
* fix(relay): scope the drain fence to the hosts already told
Review of the paced drain found two problems, both from treating "this cell
is draining" as one instant when pacing makes it a window.
Timers: the sends queued by a paced drain were neither cleared when a later
drain superseded them nor unref'd. A SIGTERM mid-window left up to 800 no-op
timers holding the event loop open until systemd escalated to SIGKILL. Drain
timers are now tracked, cleared on the next drain, and unref'd, so a retry
re-arms a session's teardown instead of stacking a second one.
Phones: the client fence read the global draining flag, so every phone was
refused for the whole window even though its own host had not been told yet
and was still serving. The director keeps pointing phones at this cell until
their host moves, so they would have looped for up to two minutes. A session
is now fenced when its drain is sent, not when the drain starts, and the
client paths key off that. New control connections and re-attaches stay
fenced globally: nothing new should land on a cell that is going away.
* fix(cloud): pin the asia cell database pool in the same-cap plan validator
Raising `database_pool_max` from 10 to 16 for production-gce-c27, c28 and c29
made every same-cap roll of those three cells fail closed at plan validation.
The cell startup template emits `ORCA_RELAY_DATABASE_POOL_MAX` only for a cell
whose region differs from the root region or whose pool is off the default, so
the asia cells carry that line while the us-central1 cells do not. The plan
validator requires the before and after startup scripts to normalize to the
same text, masking only the lines it independently pins to a reviewed value.
The pool line was neither masked nor pinned, so the live template's `'10'` and
the plan's `'16'` were read as unreviewed drift.
The validator gains an optional `--database-pool-max`, accepted in
`same-cap-cell` mode alone. When it is supplied the after-script must contain
exactly that pool line and the line is masked from the equality check; when it
is not supplied the after-script must contain no pool line at all. Masking
without the pin would have removed the guard rather than moved it.
The same-cap job resolves the expected pool next to the hard cap, cross-checks
it against the committed `relay_gce_cells` map (asserting the default 10 for
the us-central1 cells), and passes the flag to both validator invocations only
for the cells that emit the line.
* test(cloud): require the pool pin for a line the live template already carries
`relay_region_rehome_source_cell_ids` listed only the 16 US cells, and that
list is the sole thing that stamps ORCA_RELAY_REHOME_DIRECTOR_SERVICE_ACCOUNT
and ORCA_RELAY_REHOME_AUDIENCE into a cell's startup script. A cell reports
regionalRehomeProtocol 1 only when both are present, so c27-c29 have always
reported 0. That leaves them ineligible as rehome sources and, once the worker
is bidirectional, as targets too, which strands the US desktops homed there.
This is a prerequisite only. Merge and roll it ONLY AFTER the bidirectional
rehome director change is deployed. Two live gates still hard-code the primary
region and would reject an Asia source no matter what the template stamps:
`cloud/apps/relay/src/app.ts` line 610 fails the trust probe with 409 when the
source cell's region is not RELAY_DEFAULT_REGION, and
`cloud/apps/relay/src/assignment-store.ts` line 5476 skips such a cell as
source_ineligible during rehome source selection. The bidirectional lane
removes both.
The topology check asserted every source sits in the primary region. That
mirrored those two gates rather than protecting anything Terraform owns, so it
is now advisory: it requires only a configured, unfenced cell with an explicit
connection limit, and the comment records that region eligibility belongs to
the director's own source and target predicates. Every cell's region is
already constrained by the assert above it.
The same-cap census test cross-checked membership against us-central1. Every
reviewed serving cell now carries the trust, so it asserts protocol 1 for all,
plus one non-source cell to keep the validator's protocol-0 branch covered.
Roll sequencing, because this apply is not self-contained:
- After the apply the Asia templates carry the two rehome lines, and the
`unexpectedRehome` rule at `cloud/dev/scripts/validate-relay-capacity-plan.mjs`
lines 243-247 rejects a protocol-0 plan that contains them. So c27-c29 have
no dispatchable protocol-0 same-cap roll until the director gate is gone or
this is reverted.
- The same-cap job runs the per-host trust probe after isolate, drain, and the
targeted apply. A 409 there leaves the cell serving but isolated and
migration-only, which is what happened to c13 on 2026-09-06.
- The only safe path: deploy the bidirectional rehome director, then dispatch
`Deploy Relay Production Same-Cap` canary-apply for one Asia cell with
target-rehome-protocol 1 and rollback-rehome-protocol 0, then batch-apply the
remaining two. That job runs its own targeted template and MIG apply.
- Never reach these cells with an untargeted root apply. The current plan
carries 60 changes and 50 destroys of unrelated standing drift.
The same-cap wave validator approves 19 cells (c7-c26 plus the Asia cells
c27-c29), but the canary script it drives hard-rejected anything outside the
16 US capacity cells, so the first Asia same-cap canary failed closed at
isolate. Give the canary an explicit --approved-cells switch that selects the
same-cap allowlist, and pass it from the four same-cap job invocations. With no
switch the behaviour is unchanged, so the US-only capacity workflow keeps its
scope.
Jinwoo Hong