* feat(relay): pace the drain send during a same-cap cell roll
A same-cap roll drains a cell with graceMs 0, which sends `drain` to all
~800 controls in one pass. Every desktop re-dials on receipt regardless of
graceMs, so the whole cell reconnects inside a second. On 2026-09-16 that
stampede hit a Cloud SQL stall: attaches timed out, each leaving 10 minutes
of late-arrival debt on connection headroom, and placement answered
relay_capacity_exhausted fleet-wide for ~13 minutes.
Spreading the sends spreads the re-dials. `HostSessionRegistry.drain` takes
an optional pacing window and schedules each session's send evenly across
it; admission is fenced for every session up front, and each host keeps its
own full grace after its own send. /v1/admin/drain accepts `paceWindowMs`
(<= 5 min) and echoes what it applied. The same-cap job asks for 120 s, and
the drain-completion wait grew by the same amount. A cell still on an older
image rejects the field, so the deploy script falls back to an unpaced
drain rather than failing the roll.
* fix(relay): scope the drain fence to the hosts already told
Review of the paced drain found two problems, both from treating "this cell
is draining" as one instant when pacing makes it a window.
Timers: the sends queued by a paced drain were neither cleared when a later
drain superseded them nor unref'd. A SIGTERM mid-window left up to 800 no-op
timers holding the event loop open until systemd escalated to SIGKILL. Drain
timers are now tracked, cleared on the next drain, and unref'd, so a retry
re-arms a session's teardown instead of stacking a second one.
Phones: the client fence read the global draining flag, so every phone was
refused for the whole window even though its own host had not been told yet
and was still serving. The director keeps pointing phones at this cell until
their host moves, so they would have looped for up to two minutes. A session
is now fenced when its drain is sent, not when the drain starts, and the
client paths key off that. New control connections and re-attaches stay
fenced globally: nothing new should land on a cell that is going away.
* fix(cloud): pin the asia cell database pool in the same-cap plan validator
Raising `database_pool_max` from 10 to 16 for production-gce-c27, c28 and c29
made every same-cap roll of those three cells fail closed at plan validation.
The cell startup template emits `ORCA_RELAY_DATABASE_POOL_MAX` only for a cell
whose region differs from the root region or whose pool is off the default, so
the asia cells carry that line while the us-central1 cells do not. The plan
validator requires the before and after startup scripts to normalize to the
same text, masking only the lines it independently pins to a reviewed value.
The pool line was neither masked nor pinned, so the live template's `'10'` and
the plan's `'16'` were read as unreviewed drift.
The validator gains an optional `--database-pool-max`, accepted in
`same-cap-cell` mode alone. When it is supplied the after-script must contain
exactly that pool line and the line is masked from the equality check; when it
is not supplied the after-script must contain no pool line at all. Masking
without the pin would have removed the guard rather than moved it.
The same-cap job resolves the expected pool next to the hard cap, cross-checks
it against the committed `relay_gce_cells` map (asserting the default 10 for
the us-central1 cells), and passes the flag to both validator invocations only
for the cells that emit the line.
* test(cloud): require the pool pin for a line the live template already carries
`relay_region_rehome_source_cell_ids` listed only the 16 US cells, and that
list is the sole thing that stamps ORCA_RELAY_REHOME_DIRECTOR_SERVICE_ACCOUNT
and ORCA_RELAY_REHOME_AUDIENCE into a cell's startup script. A cell reports
regionalRehomeProtocol 1 only when both are present, so c27-c29 have always
reported 0. That leaves them ineligible as rehome sources and, once the worker
is bidirectional, as targets too, which strands the US desktops homed there.
This is a prerequisite only. Merge and roll it ONLY AFTER the bidirectional
rehome director change is deployed. Two live gates still hard-code the primary
region and would reject an Asia source no matter what the template stamps:
`cloud/apps/relay/src/app.ts` line 610 fails the trust probe with 409 when the
source cell's region is not RELAY_DEFAULT_REGION, and
`cloud/apps/relay/src/assignment-store.ts` line 5476 skips such a cell as
source_ineligible during rehome source selection. The bidirectional lane
removes both.
The topology check asserted every source sits in the primary region. That
mirrored those two gates rather than protecting anything Terraform owns, so it
is now advisory: it requires only a configured, unfenced cell with an explicit
connection limit, and the comment records that region eligibility belongs to
the director's own source and target predicates. Every cell's region is
already constrained by the assert above it.
The same-cap census test cross-checked membership against us-central1. Every
reviewed serving cell now carries the trust, so it asserts protocol 1 for all,
plus one non-source cell to keep the validator's protocol-0 branch covered.
Roll sequencing, because this apply is not self-contained:
- After the apply the Asia templates carry the two rehome lines, and the
`unexpectedRehome` rule at `cloud/dev/scripts/validate-relay-capacity-plan.mjs`
lines 243-247 rejects a protocol-0 plan that contains them. So c27-c29 have
no dispatchable protocol-0 same-cap roll until the director gate is gone or
this is reverted.
- The same-cap job runs the per-host trust probe after isolate, drain, and the
targeted apply. A 409 there leaves the cell serving but isolated and
migration-only, which is what happened to c13 on 2026-09-06.
- The only safe path: deploy the bidirectional rehome director, then dispatch
`Deploy Relay Production Same-Cap` canary-apply for one Asia cell with
target-rehome-protocol 1 and rollback-rehome-protocol 0, then batch-apply the
remaining two. That job runs its own targeted template and MIG apply.
- Never reach these cells with an untargeted root apply. The current plan
carries 60 changes and 50 destroys of unrelated standing drift.
The same-cap wave validator approves 19 cells (c7-c26 plus the Asia cells
c27-c29), but the canary script it drives hard-rejected anything outside the
16 US capacity cells, so the first Asia same-cap canary failed closed at
isolate. Give the canary an explicit --approved-cells switch that selects the
same-cap allowlist, and pass it from the four same-cap job invocations. With no
switch the behaviour is unchanged, so the US-only capacity workflow keeps its
scope.
Jinwoo Hong