mirror of
https://github.com/stablyai/orca.git
synced 2026-10-04 08:02:09 +00:00
infra(relay): raise asia-east2 cell pools to 16 and retire four idle cells
The three asia-east2 cells sit 176 ms from the Cloud SQL instance in us-central1. Server-side statement time there is 0.2 ms, so a pool slot is held by the round trip, not by the query. At a pool of 10 they measured 94-156 waiters and 2 s waits, and client accepts ran a ~4 s p95 against 222-646 ms in us-central1. Raising those three pools to 16 is the agreed first step; every other cell stays at 10. c4 and c5 join the committed fence set. Both are existing-only capacity the admission selector can never place on again, they carried ~1 connection each on 40-day-old images, and each still holds 10 Postgres connections. The fence set is the prerequisite the fence-source workflow confirms before it drains and attests a cell; it is not itself the resize. c17 and c18 are not fenced here. They are migration-only, and the runbook requires retire-migration-cell to move a migration-only cell to existing-only through a generation-bound selector CAS before it can be fenced. Terraform cannot express that step. The Cloud SQL consumer contract carried two stale numbers: auth at 2 instances when production has run a cap of 20 since 2026-09-04, and a 400-connection ceiling when the live instance reports 500. Both are corrected, and the budget now asserts its headroom in two named gates instead of one aggregate boolean. Those gates fail: auth alone accounts for 200 configured connections and a 215-connection rollout overlap, so the operating maximum is 713 against a usable ceiling of 490. Nothing here caused that, and no pool was lowered to hide it.
This commit is contained in:
@@ -6,26 +6,51 @@ import {
|
||||
readRelayCloudSqlConnectionBudget
|
||||
} from './relay-cloud-sql-connection-budget.mjs'
|
||||
|
||||
test('production shared consumers keep allowance and reserve below the ceiling', () => {
|
||||
const shortfall = (report) =>
|
||||
[
|
||||
`cells ${report.consumers.cells}`,
|
||||
`directors ${report.consumers.directors}`,
|
||||
`auth ${report.consumers.auth}`,
|
||||
`api ${report.consumers.api}`,
|
||||
`= ${report.configuredMaximum} configured`,
|
||||
`+ ${report.rolloutOverlap.maximum} rollout overlap`,
|
||||
`+ ${report.maintenanceAdminAllowance} admin allowance`,
|
||||
`= ${report.operatingMaximum} operating`,
|
||||
`against ${report.maxConnections} max_connections less a ${report.explicitReserve} reserve`
|
||||
].join(', ')
|
||||
|
||||
test('the production budget reads the committed pools and the measured ceiling', () => {
|
||||
const report = readRelayCloudSqlConnectionBudget()
|
||||
|
||||
assert.deepEqual(report.consumers, { cells: 230, directors: 15, auth: 20, api: 50 })
|
||||
assert.deepEqual(report.asia, { cells: 3, poolMax: 10 })
|
||||
assert.equal(report.configuredMaximum, 315)
|
||||
assert.equal(report.maxConnections, 500)
|
||||
assert.deepEqual(report.consumers, { cells: 228, directors: 15, auth: 200, api: 50 })
|
||||
assert.deepEqual(report.asia, { cells: 3, poolMax: 16 })
|
||||
assert.equal(report.configuredMaximum, 493)
|
||||
assert.equal(report.rolloutOverlap.relayDirectorCandidate, 30)
|
||||
assert.equal(report.rolloutOverlap.apiCandidate, 65)
|
||||
assert.equal(report.rolloutOverlap.authCandidate, 35)
|
||||
assert.equal(report.rolloutOverlap.authCandidate, 215)
|
||||
assert.equal(report.rolloutOverlap.relayCells, 15)
|
||||
assert.equal(report.rolloutOverlap.retainedDirectorRollback, 15)
|
||||
assert.equal(report.rolloutOverlap.maximum, 65)
|
||||
assert.equal(report.rolloutOverlap.maximum, 215)
|
||||
assert.equal(report.maintenanceAdminAllowance, 5)
|
||||
assert.equal(report.explicitReserve, 10)
|
||||
assert.equal(report.usableCeiling, 390)
|
||||
assert.equal(report.operatingMaximum, 385)
|
||||
assert.equal(report.remainingWithinUsableCeiling, 5)
|
||||
assert.equal(report.budgetedTotal, 395)
|
||||
assert.equal(report.unallocated, 5)
|
||||
assert.equal(report.withinBudget, true)
|
||||
assert.equal(report.usableCeiling, 490)
|
||||
assert.equal(report.operatingMaximum, 713)
|
||||
})
|
||||
|
||||
test('configured pools fit under the ceiling less the stated reserve', () => {
|
||||
const report = readRelayCloudSqlConnectionBudget()
|
||||
|
||||
assert.ok(
|
||||
report.configuredMaximum <= report.usableCeiling,
|
||||
`configured pools exceed the usable ceiling: ${shortfall(report)}`
|
||||
)
|
||||
})
|
||||
|
||||
test('a serialized rollout still fits under the ceiling less the stated reserve', () => {
|
||||
const report = readRelayCloudSqlConnectionBudget()
|
||||
|
||||
assert.ok(report.withinBudget, `the operating maximum exceeds the usable ceiling: ${shortfall(report)}`)
|
||||
})
|
||||
|
||||
test('fails closed when pool growth consumes the explicit reserve', () => {
|
||||
|
||||
Reference in New Issue
Block a user