infra(relay): raise asia-east2 cell pools to 16 and retire four idle cells

The three asia-east2 cells sit 176 ms from the Cloud SQL instance in
us-central1. Server-side statement time there is 0.2 ms, so a pool slot is
held by the round trip, not by the query. At a pool of 10 they measured
94-156 waiters and 2 s waits, and client accepts ran a ~4 s p95 against
222-646 ms in us-central1. Raising those three pools to 16 is the agreed
first step; every other cell stays at 10.

c4 and c5 join the committed fence set. Both are existing-only capacity the
admission selector can never place on again, they carried ~1 connection each
on 40-day-old images, and each still holds 10 Postgres connections. The fence
set is the prerequisite the fence-source workflow confirms before it drains
and attests a cell; it is not itself the resize.

c17 and c18 are not fenced here. They are migration-only, and the runbook
requires retire-migration-cell to move a migration-only cell to existing-only
through a generation-bound selector CAS before it can be fenced. Terraform
cannot express that step.

The Cloud SQL consumer contract carried two stale numbers: auth at 2 instances
when production has run a cap of 20 since 2026-09-04, and a 400-connection
ceiling when the live instance reports 500. Both are corrected, and the budget
now asserts its headroom in two named gates instead of one aggregate boolean.
Those gates fail: auth alone accounts for 200 configured connections and a
215-connection rollout overlap, so the operating maximum is 713 against a
usable ceiling of 490. Nothing here caused that, and no pool was lowered to
hide it.
This commit is contained in:
Jinwoo-H
2026-09-17 01:05:54 -04:00
parent 28a2b628bc
commit 380a5800fb
3 changed files with 61 additions and 23 deletions
@@ -6,26 +6,51 @@ import {
readRelayCloudSqlConnectionBudget
} from './relay-cloud-sql-connection-budget.mjs'
test('production shared consumers keep allowance and reserve below the ceiling', () => {
const shortfall = (report) =>
[
`cells ${report.consumers.cells}`,
`directors ${report.consumers.directors}`,
`auth ${report.consumers.auth}`,
`api ${report.consumers.api}`,
`= ${report.configuredMaximum} configured`,
`+ ${report.rolloutOverlap.maximum} rollout overlap`,
`+ ${report.maintenanceAdminAllowance} admin allowance`,
`= ${report.operatingMaximum} operating`,
`against ${report.maxConnections} max_connections less a ${report.explicitReserve} reserve`
].join(', ')
test('the production budget reads the committed pools and the measured ceiling', () => {
const report = readRelayCloudSqlConnectionBudget()
assert.deepEqual(report.consumers, { cells: 230, directors: 15, auth: 20, api: 50 })
assert.deepEqual(report.asia, { cells: 3, poolMax: 10 })
assert.equal(report.configuredMaximum, 315)
assert.equal(report.maxConnections, 500)
assert.deepEqual(report.consumers, { cells: 228, directors: 15, auth: 200, api: 50 })
assert.deepEqual(report.asia, { cells: 3, poolMax: 16 })
assert.equal(report.configuredMaximum, 493)
assert.equal(report.rolloutOverlap.relayDirectorCandidate, 30)
assert.equal(report.rolloutOverlap.apiCandidate, 65)
assert.equal(report.rolloutOverlap.authCandidate, 35)
assert.equal(report.rolloutOverlap.authCandidate, 215)
assert.equal(report.rolloutOverlap.relayCells, 15)
assert.equal(report.rolloutOverlap.retainedDirectorRollback, 15)
assert.equal(report.rolloutOverlap.maximum, 65)
assert.equal(report.rolloutOverlap.maximum, 215)
assert.equal(report.maintenanceAdminAllowance, 5)
assert.equal(report.explicitReserve, 10)
assert.equal(report.usableCeiling, 390)
assert.equal(report.operatingMaximum, 385)
assert.equal(report.remainingWithinUsableCeiling, 5)
assert.equal(report.budgetedTotal, 395)
assert.equal(report.unallocated, 5)
assert.equal(report.withinBudget, true)
assert.equal(report.usableCeiling, 490)
assert.equal(report.operatingMaximum, 713)
})
test('configured pools fit under the ceiling less the stated reserve', () => {
const report = readRelayCloudSqlConnectionBudget()
assert.ok(
report.configuredMaximum <= report.usableCeiling,
`configured pools exceed the usable ceiling: ${shortfall(report)}`
)
})
test('a serialized rollout still fits under the ceiling less the stated reserve', () => {
const report = readRelayCloudSqlConnectionBudget()
assert.ok(report.withinBudget, `the operating maximum exceeds the usable ceiling: ${shortfall(report)}`)
})
test('fails closed when pool growth consumes the explicit reserve', () => {