Jinwoo Hong 5947d6b269 infra(relay): raise asia-east2 cell pools to 16 and record the measured connection ceiling (#21163) ...

* infra(relay): raise asia-east2 cell pools to 16 and retire four idle cells

The three asia-east2 cells sit 176 ms from the Cloud SQL instance in
us-central1. Server-side statement time there is 0.2 ms, so a pool slot is
held by the round trip, not by the query. At a pool of 10 they measured
94-156 waiters and 2 s waits, and client accepts ran a ~4 s p95 against
222-646 ms in us-central1. Raising those three pools to 16 is the agreed
first step; every other cell stays at 10.

c4 and c5 join the committed fence set. Both are existing-only capacity the
admission selector can never place on again, they carried ~1 connection each
on 40-day-old images, and each still holds 10 Postgres connections. The fence
set is the prerequisite the fence-source workflow confirms before it drains
and attests a cell; it is not itself the resize.

c17 and c18 are not fenced here. They are migration-only, and the runbook
requires retire-migration-cell to move a migration-only cell to existing-only
through a generation-bound selector CAS before it can be fenced. Terraform
cannot express that step.

The Cloud SQL consumer contract carried two stale numbers: auth at 2 instances
when production has run a cap of 20 since 2026-09-04, and a 400-connection
ceiling when the live instance reports 500. Both are corrected, and the budget
now asserts its headroom in two named gates instead of one aggregate boolean.
Those gates fail: auth alone accounts for 200 configured connections and a
215-connection rollout overlap, so the operating maximum is 713 against a
usable ceiling of 490. Nothing here caused that, and no pool was lowered to
hide it.

* infra(relay): move the Cloud SQL contract correction out of this branch

The contract correction (auth at its real 20-instance cap, the measured
500-connection ceiling) makes the budget gate fail for reasons that have
nothing to do with asia pools or fenced cells, and it held this branch red.
It moves to its own branch where the failure is the subject.

production-cloud-sql-app-consumers.json returns to main unchanged. The budget
test keeps main's single gate and only repins the cell figure that this branch
genuinely moves: 230 -> 228, being +18 for three asia pools at 16 and -20 for
fencing c4 and c5. Against main's 400-connection model that leaves an operating
maximum of 383 under a usable ceiling of 390.

* infra(relay): move the c4/c5 fence entries out of this branch

Terraform now sets a cell's MIG target size directly from relay_gce_fenced_cells
(relay-gce-cells.tf); the lifecycle ignore that used to protect operational
target_size drift is gone. So a fence entry sitting on main ahead of its
fence-source run is a standing instruction that any apply reaching that cell may
execute without the documented drain and attestation. Keeping the entry in the
same merge as an unrelated pool change widens that blast radius for no reason.

The two entries move to their own branch, to be merged immediately before
fence-source runs for c4 and then c5. This branch keeps the multi-line reflow of
the list, which makes that later diff two added lines instead of a rewritten one.

The cell figure in the budget test follows: 230 + 18 for the three asia-east2
pools at 16, with no fenced-cell subtraction. That is 403 operating against a
usable ceiling of 390, so the headroom gate now fails by 13. It fails against a
ceiling of 400 that is itself wrong; the instance reports 500. See the PR body.

* infra(cloud-sql): record the measured 500-connection ceiling

The budget's usable ceiling came from maxConnections: 400, described as the
tier default. It is a tier default, since no max_connections flag is set, but
the instance does not report 400. SHOW max_connections on it returns 500,
measured 2026-09-16.

On main the model sat at 385 against a usable ceiling of 390, five connections
of margin, so raising the three asia-east2 pools by 18 failed the gate by 13
against a ceiling that was never checked. Against the measured one it is 403
against 490, clearing by 87.

Only the ceiling and its source note change here. auth stays recorded at 2
instances, which is also wrong; PR #21165 corrects it, and with the true auth
figure the budget is over by 225 for reasons that have nothing to do with these
pools.

* test(cloud): state the cell pool arithmetic literally in the budget pin comment

2026-09-17 02:00:17 -04:00

..

production-cloud-sql-app-consumers.json

infra(relay): raise asia-east2 cell pools to 16 and record the measured connection ceiling (#21163)

2026-09-17 02:00:17 -04:00