mirror of
https://github.com/stablyai/orca.git
synced 2026-09-21 16:02:20 +00:00
4feca6baa1b2a1d2eec77b2c5aa0e230e88e9be2
99
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jinwoo Hong
|
e476193bf5 |
chore(relay): bound the shadow health gate and apply a pending backend update on resume (#21865)
* fix(relay): bound the same-cap shadow gate and apply a resumed backend update Two findings both adversarial reviews of tonight's merged set agree on. The report-only shadow health gate (#21849) had `continue-on-error: true` but no step timeout. That bounds the step's contribution to the job outcome, not its clock. Its reads are serialised, and a failure that answers nothing slowly — an expired credential, a project-wide Logging 429 storm — makes every read cost its full 3 x 60 s retry budget, so the cost scales with the roll window: roughly 8S + 2 reads for S ten-minute sub-windows. A 40-minute window is about 34 reads, or 108 minutes, against the job's `timeout-minutes: 75`. A cancelled job cannot be absorbed by continue-on-error, fires the failure-gated cleanup isolation on an already-restored cell, and stops the strict next-cell chain. Give the step `timeout-minutes: 5` and the artifact upload `timeout-minutes: 2`. A timed-out step is a failed step, which continue-on-error covers, so the job stays green. Inside the script, stop reading after an overall four-minute deadline and report the remaining checks unverified, so the normal outcome is a written verdict rather than a killed process; the step timeout is then only for a hung process. The census test pins both timeouts and that the deadline leaves the step time to write its verdict. The resume branch (#21860) accepted `changes == 0` with a non-empty `backendUpdate` as complete and applied nothing, so a resumed cell silently kept the 300-second drain and no request logging behind a green resume. That shape means the template and MIG are converged and only this cell's reviewed backend update is left, so apply the saved resume plan — the validator has already bounded it to this cell's backend and neither attribute restarts an instance — then continue as converged. Template-and-MIG drift still applies nothing, which is what a resume means, and a stranded cell's explicit MIG replace is unchanged. Claude-Session: relay-same-cap-gate-timeout-and-resume * fix(relay): raise the shadow gate bounds clear of a healthy gate's read time A healthy gate is already minutes of serial reads on the 2-vcpu runner, so a four-minute deadline would report unverified tails on ordinary days and stop the shadow roll measuring the comparison it exists for. Raise both together: the step to eight minutes and the script's own deadline to seven, keeping the census pin that the deadline leaves the step room to write its verdict. The job budget is unaffected: a ~14-minute cell plus eight is well inside 75. Claude-Session: relay-same-cap-gate-timeout-and-resume |
||
|
Jinwoo Hong
|
2524737ef0 |
chore(relay): apply the cell backend drain and request-logging settings inside each same-cap wave (#21860)
* chore(relay): target each cell's backend service from the same-cap job
The 60 s connection drain timeout merged in #21848 has no safe apply path.
A root plan scoped to the backend services alone still pulls every
`google_compute_instance_template.relay_gce_cell` in as a dependency, and
standing image drift turns all 29 into replacements, so applying it would roll
the fleet at once.
Add `google_compute_backend_service.relay_gce_cell["${TARGET_CELL_ID}"]` to
both plan invocations in the per-cell same-cap job, next to the template and
MIG it already targets, and teach the reviewed plan validator to allow exactly
one extra change: an in-place update of that one cell's backend whose only
changed attribute is `connection_draining_timeout_sec`, landing on the
constant `validate-relay-asia-topology-plan.mjs` exports. Any other attribute,
any other resource, or a backend for another cell still fails the validator.
The accepted update is reported as `connectionDrainUpdate` and kept out of
`changes`, so the apply step's stranded branch and the resume step's drift
branch keep reading the template-and-MIG count they were written against; the
resume branch additionally accepts a plan whose only pending change is that
drain update, which restarts nothing.
Claude-Session: relay-same-cap-targets-cell-backend
* fix(relay): also let the same-cap wave apply this cell's LB request logging
A read-only production plan for production-gce-c7 showed the live US cell
backends carry no `log_config` at all, while relay-gce-cells.tf has declared
`log_config { enable = true, sample_rate = var.relay_gce_cell_log_sample_rate }`
on every cell backend since the Terraform root landed in
|
||
|
Jinwoo Hong
|
5b8ac36f41 |
chore(relay): add a report-only post-wave health gate to the same-cap cell job (#21849)
* feat(relay): report a post-wave health verdict on each same-cap cell, without gating on it After a same-cap cell finishes rolling, an operator reads five things by hand before dispatching the next cell: director 503s against the same clock hour a day and two days earlier, whether the cell's new container announced its listener and has stayed up, the cell's own pool pressure, the asia-east2 pool trio, and Cloud SQL FATALs. This runs those same reads automatically and records PASS / WARN / WOULD_BLOCK with its numbers, so its calls can be compared with the operator's over a full roll before it is ever allowed to stop one. It cannot fail a cell in this change. The script exits 0 on every verdict, and the step is continue-on-error, so even a crash stays off the job's outcome and the failure failsafe cannot fire on anything it observes. It also runs after the restore, so no cell waits on it to go back into admission. Cloud Logging returns only --limit entries and says nothing when it truncates, so every count is split into sub-windows of ten minutes and a sub-window that comes back at the limit is reported unverified rather than as a count. Windows are always explicitly bounded: --freshness does not bind on these logs. Claude-Session: https://claude.ai/session/ced32ebb-7155-4413-adad-1eccd14c2010 * fix(relay): bound the shadow gate's cell reads at the apply start and cap every read Four fixes from review, all in the report-only shadow health gate. The boot search opened at apply-completed-at, which is stamped after `terraform apply` and `wait-until --stable`. The new container announces its listener while the MIG is still converging, so that bound is already past the announcement it looks for and a healthy roll read as would-block. The job now stamps apply-started-at immediately before the apply, and the boot search opens there; apply-completed-at is kept, recorded rather than judged, so an operator comparing verdicts can see apply time next to boot time. The crash query started at the newest listener timestamp, which erased any crash before it. A crash-restart loop ends with an announcement that looks like a clean boot, so that is exactly the case it hid: against production, the 2026-09-20 c28 crash at 20:18:10 was dropped because the listener landed at 20:18:27. It now runs from the apply start, still scoped to the instance id the listener identified, and that crash is counted. A runtime-metrics read that came back at its 500-entry limit fed judgePool as though it were a complete sample run. A truncated run has holes and the consecutive-sample rule reads a hole as a recovery, so it now reports unverified. gcloud reads had no timeout. continue-on-error bounds the job's outcome but not its clock, so a stalled read could have spent the rollout's remaining minutes. Each read now gets 60 s and a timed-out read is just a failed read. Claude-Session: https://claude.ai/session/ced32ebb-7155-4413-adad-1eccd14c2010 * test(relay): require each shadow-gate stamp's presence before asserting its order The ordering assertion used indexOf, which answers -1 for an absent stamp, and -1 precedes every real offset. Deleting the apply-started-at line left the test green, so the census could not see the fix it was written to pin. Each stamp's presence is now asserted first, with a message naming the stamp and the step, and presence is judged inside the step that owns the stamp rather than anywhere in the file: a stamp written into a neighbouring step records the wrong instant but would satisfy a whole-file match. Control-run against a scratch copy of the job. Deleting drain-started-at, apply-started-at, or apply-completed-at each reds with its own message, and moving apply-started-at after terraform apply reds on the ordering assertion, so presence and order both fail independently. Claude-Session: https://claude.ai/session/ced32ebb-7155-4413-adad-1eccd14c2010 |
||
|
Jinwoo Hong
|
4f839cc8c9 |
chore(relay): cut the cell LB connection drain to 60 s and allow ten-cell same-cap batches (#21848)
* perf(relay): cut the cell LB drain to 60s and widen the same-cap batch to ten cells Two independent sources of relay roll wall clock, neither of which protects a host: 1. `connection_draining_timeout_sec` on the per-cell backend services was 300s. The same-cap job drains every host off the cell to a restart-safe condition before Terraform runs, so the LB drain only ever covers a host still mid-handshake. Measured 2026-09-16 over ten same-cap cell jobs, it sat as ~5m55s of dead time between `Apply complete` and the old VM powering off, inside an 8.5-minute `wait-until --stable` step. Now 60s, and pinned in the topology `check` block beside the other fixed-one invariants. 2. The same-cap wave capped a batch at four cells, so a 22-cell roll needed six batches, six single-use monitor gates, and a human handoff per batch. The wave workflow now declares cell_1..cell_10 with the identical serial shape and chaining, and the validator accepts two to ten. The shared wave-index rule (`relay-monitor-evidence.mjs` and the relay-ops preflight CLI) widens from 0-3 to 0-9 so the later cells can present the same evidence; each job workflow keeps its own narrower range, so the capacity wave stays at four. Cells remain strictly serial, one at a time behind the rollout lease, each with its own live preflight. Claude-Session: https://claude.ai/session/relay-roll-drain-timeout-and-batch-cap * fix(relay): align the Asia topology plan validator with the 60s cell drain `validate-relay-asia-topology-plan.mjs` rejected any Asia backend whose `connection_draining_timeout_sec` was not 300, and `cloud-deploy-relay-asia-topology.yml` targets `google_compute_backend_service.relay_gce_cell["<cell>"]` per cell. With the Terraform local at 60 that workflow would have failed its own plan review. The validator's two restated topology values are now named exports, and a new census test reads `relay-gce-cells.tf` and equates three statements of each: the `relay_gce_topology` local, the topology `check` assert that pins it, and the validator constant. Terraform cannot export a local to JS, so reading the source is the only way to stop them drifting; the test was confirmed to fail when the local alone is moved back to 300. Repo-wide grep finds no other pin of the drain value. Claude-Session: https://claude.ai/session/relay-roll-drain-timeout-and-batch-cap |
||
|
Jinwoo Hong
|
eb6068a434 |
fix(relay): stop a terminated checked-out PostgreSQL client from killing the cell (#21840)
pg-pool removes its own `error` listener when it hands a client out (pg-pool@3.14.0 index.js:344) and only reattaches it in `_release` (index.js:385). Between acquire and release the client therefore has no `error` listener, so when Cloud SQL terminates that session mid-statement the emit becomes an unhandled 'error' event and the process exits. `absorbPostgresIdleClientErrors` cannot see it: pg-pool routes to `pool.on('error')` only from the idle listener. Attach a per-checkout `error` listener in the one seam every relay checkout passes through, log a single warn line, and release the client with the error so pg-pool destroys it instead of pooling a dead connection. The listener is removed on release so it cannot accumulate. The in-flight query still rejects, so existing failure reporting and the transaction retry ladder are unchanged. Claude-Session: https://claude.ai/session/ced32ebb-7155-4413-adad-1eccd14c2010 |
||
|
Jinwoo Hong
|
68b11282a5 |
fix(relay): let the rehome evidence parser read a line the director grew (#21823)
The enable workflow reads the director's `[orca-relay] regional rehome inventory` line out of Cloud Logging and pins the whole line with one regex. Adding `hostNotArrivedLast24Hours` in #21813 made every healthy line stop matching, so "Read fresh aggregate completion and abort evidence" threw "no aggregate regional rehome inventory evidence" and the fail-closed step disabled the durable switch at control generation 26. The parser now requires the six original fields and tolerates further ones in any order. Extra fields stay fenced by value shape rather than by pinning the whole line: a field must be a bare name and a non-negative integer or `none`, so `hostId=someone` is still not a counter and cannot ride along. An absent count reads as null, not zero, because an older director not reporting leaks is not the same as reporting none. `hostNotArrivedLast24Hours` and `oldestActiveAgeMs` now reach the evidence JSON and the operator step summary. Two guards close the chain, each verified to fail on the regression it exists for: a census in the relay package feeds the real formatter's output to the real parser, and a script-side test pins the parser's output to the fields the workflow summary renders. Claude-Session: https://claude.ai/session/ced32ebb-7155-4413-adad-1eccd14c2010 |
||
|
Jinwoo Hong
|
fa0010e8d6 |
fix(relay): abort rehomes whose host never arrived, without disabling the switch (#21813)
A regional rehome whose host went offline right after accepting the move left its migration row open forever: the target had registered it, the host held nothing on the source, and the completion sweep could never finish it. Eight such rows filled REGIONAL_REHOME_CONCURRENT_LIMIT and every later candidate came back deferred, silently, for 21 hours. The only sweep that touched them fires at 24 hours and also sets enabled = 0 on the durable control, so the first leak to age out would have turned rehoming off, repeatedly. Adds a director sweep that rolls such an attempt back to its source after one migration lease, with abort_reason = 'host_not_arrived', reusing the existing rollback (assignment epoch bump back to the source, lease removal, superseded target reservation release) and leaving the switch untouched. The 24-hour sweep keeps its disable as a last-resort latch. The source cell now names why it deferred, on a new optional response field, and the director stops walking its candidate page on a deferral no later candidate can pass. Each poll that dispatched logs one summary line. Claude-Session: https://claude.ai/session/ced32ebb-7155-4413-adad-1eccd14c2010 |
||
|
Jinwoo Hong
|
030a1e0c77 |
fix(relay): stop holding a cell row across the whole control accept (#21563)
* fix(relay): stop holding a cell row across the whole control accept The cell accept path took the host's cell row FOR UPDATE at its first supersession statement and held it to COMMIT across a dozen round trips, which capped a cell far from Postgres at a couple of accepts a second. Fold every cell-row change on the path into one conditional delta write issued last, so the contended row is held only across the commit. * fix(relay): give relay_cells one global row lock order, taken last Moving the accept's cell-row write to the end of its transaction put it after the host's relay_control_connection_reservations rows, while every director path that reads the inventory took those rows the other way round. Pin one order for both roles -- host rows, then the shared cell row -- by locking the host's reservation rows before the inventory in the nine director paths that take both, document the tiers next to CellInventoryLockMode, and add a census that fails on a new path taking relay_cells first. |
||
 Jinwoo HongandClaude
|
7080eb0604 |
fix(relay): bound the idle-rehome candidate poll to a window of decisions (#21557)
* fix(relay): bound the idle-rehome candidate poll to a window of decisions The director's idle-regional-rehome poll built every (eligible host x target cell in its preferred region) pair, applied the cohort predicate downstream of that fan-out, sorted the lot, and took LIMIT 100 OFFSET n. Its cost was set by the size of the fleet and the width of the cohort, so raising the cohort from 10% to 100% pushed it past the serving pool's 5 s statement_timeout and the rollout stalled at 0.37 hosts/min. The poll now resolves the cell inventory once (tens of rows), takes a bounded window of decision rows in primary-key order from a keyset cursor with the cohort, freshness and cross-region predicates applied first, verifies only that window against the host-side gates, and ranks targets in the process. Same candidates in the same priority order; the work per poll no longer depends on the cohort or the fleet. Adds a once-a-minute aggregated poll summary so an operator can tell a poll gated by the dispatch budget from one that found nobody to move. Co-Authored-By: Claude <noreply@anthropic.com> * fix(relay): pin the rehome verification to the window's exact keys The window read and the verification read take separate snapshots. The verification repeated the window's predicate with its own LIMIT, so a decision that turned eligible between the two reads shifted that LIMIT and pushed the window's last host out of it -- while the cursor still advanced past that host, skipping it for a whole sweep. The verification now names the keys the window returned. Its LIMIT stays as the optimisation fence that stops Postgres flattening the subquery, but can no longer truncate a key set that is at most one window long. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Jinwoo Hong
|
164c7140fd |
fix(relay): stop reporting an unavailable home cell as exhausted capacity (#21518)
A host whose home cell is not live — readiness false, drained, or inside a boot window — is refused by the committed-fence branch in assignOnce() without any capacity being consulted. It answered relay_capacity_exhausted, so every cell boot and every readiness dip printed capacity rejections at 17% fleet utilisation and sent an investigation after headroom that was never short. The branch now raises RelayHomeCellUnavailableError, which carries the cell id and which of cellIsLive()'s conditions failed (draining / booting / unheard / not_ready). The director logs reason, cause and cell, and returns the new reason in the same retryable 503. Nothing on the wire reads the body: the desktop client discards it unread and branches on status only, and no log-based metric or alert parses the reason. The load harness, the only body-reading consumer, gets its own bucket so a home-cell rejection no longer inflates the capacity count. Hinted grants are now logged on whichever lane served them, so a host that failed sticky verification and was rehomed by placement leaves a record of where it landed. Unhinted placement grants stay silent. |
||
|
Jinwoo Hong
|
3467e5f6b5 |
fix(relay): check the rehome dispatch budget before planning the candidate join (#21517)
* fix(relay): check the rehome dispatch budget before planning the candidate join `selectIdleRegionalRehomeCandidates` read the enable control and the fleet safety snapshot, then ran the twenty-table candidate join, then handed every row to the worker, which POSTed each one to its source cell. Only there — in `commitIdleRegionalRehome`, three statements into a write transaction that takes `FOR UPDATE` on two global single-row tables — was the durable dispatch budget consulted. The budget is ten moves a minute (`next_dispatch_at = now + 6s`), and five directors poll every six seconds, so most of that work was spent to be told the budget was closed. A five-minute `paused_until` made every poll in the window do it. The gate is a single-row primary-key read, so it goes in front. An absent row means the budget has never been spent and opens the gate, matching the INSERT ... ON CONFLICT DO NOTHING the commit path already relies on. * test(relay): assign the closed budget field once so the case runs on Postgres The two gate cases zeroed both `next_dispatch_at` and `paused_until` and then set the one under test, which names that column twice in a single `SET`. SQLite accepts it; Postgres raises "multiple assignments to same column", so both cases failed whenever `ORCA_IDLE_REHOME_POSTGRES_URL` pointed the suite at a real server -- exactly the backend the gate has to hold on. Setup already leaves both fields at 0, so naming the other one bought nothing. |
||
|
Jinwoo Hong
|
ce5d8c02d4 |
fix(relay): wait out a cold proxy at boot instead of exiting the cell (#21516)
* fix(relay): wait out a cold proxy at boot instead of exiting the cell A cell container starts its relay process beside a cloud-sql-proxy that is itself still dialling. The first pool acquire therefore competes with a proxy cold start, and the 2s connect timeout that protects the request path fires before the proxy is listening. `openRelayDatabase` rejects out of the region backfill, the top-level await rejects, and the process exits; COS restarts the container and the next boot succeeds 1-3s later. The 2026-09-18 fleet roll saw 0-7 of these per cell, including on cells with zero hosts, so it is a property of the boot sequence rather than of database load. The boot open now retries on transient errors only, inside a 45s wall-clock window with exponential backoff from 250ms to 4s. The classifier is the one the request path already uses, so a rejected credential or a bad URL still exits on the first attempt. Each wait logs `orca_relay_boot_database_retry` and a give-up logs `orca_relay_boot_database_failed`, both with the bounded error category, so a rollout can tell a slow boot from a stuck one without reading container exit codes. The bounded startup retry is lifted out of `reconcileCellAdmissionAtStartup`, which had the same loop; its attempt budget, flat delay, and both log events are unchanged (a flat delay is a cap equal to the base). * fix(relay): retry the boot open only when Postgres is unreachable The boot open re-runs the schema apply, and applyPostgresSchema refuses to repeat a DDL lock timeout on purpose: relation locks are granted in queue order, so a repeat parks every writer behind the same statement again. Gating the boot retry on the full request-path classifier would have re-queued it up to 16 times in 45s on sustained 55P03 - the mechanism behind the 2026-09-16 outage. The boot call site now has its own predicate: pool connect failures (both connect-timeout messages and an acquire-marked early-ended socket) plus 08001 and 08006. Lock and overload SQLSTATEs - 55P03, 57014, 53300 - exit on the first attempt. The retry predicate moves onto the policy because what a step re-runs, not the request path, decides what it may repeat; the startup reconcile keeps the full classifier, which is what lets it wait out 55P03. |
||
|
Jinwoo Hong
|
6c913a917f |
fix(relay-ops): roll a cell a wave stranded after its drain (#21321)
* fix(relay-ops): roll a cell a wave stranded after its drain A wave that stops any time after its drain leaves the cell migration-only and draining on the rollback image, and nothing clears it: the drain flag is a one-way latch on the running process, and the failsafe restarts nothing. Both recovery modes then refuse the cell. Apply wants it general and not draining. Rollback sees the rollback image, reads it as a resume, refuses the draining, and would not have restarted it anyway. The image alone cannot separate a rollback that failed after its template apply from a wave that stopped before one. The restart can: the first left a fresh process, the second did not. Classify on that, so the cell that never restarted takes the rolling path instead of the resuming one. Its template still carries the image it serves, so that is the predecessor its plan is reviewed against, and a template already moved on to the target is refused rather than rolled backwards under a stale review. When the reviewed template is already in place the plan changes nothing, so the MIG is rolled explicitly on the same replacement policy a template change uses; the existing incarnation check is what proves the instance came back. Every other combination of mode, live image, and drain flag keeps the value it had, held by a census that runs the real block over all nine. * fix(relay-ops): pin the replacement method on the explicit MIG roll gcloud persists every rolling-action bound into the group's update policy, and it defaults the replacement method to substitute on a group with no stateful config. Passing surge and unavailable without the method would patch the policy off the declared RECREATE, and the next targeted plan would then carry a MIG change outside version.0.instance_template, which the plan validator refuses. Pass all three so the patch is identical to the declared policy, and read the declared values in the census instead of restating two of them. Dropping the flag, or moving any of the three in Terraform, now fails the census. |
||
|
Jinwoo Hong
|
27bddc6198 |
fix(relay-ops): accept a drained predecessor on a cell that holds no hosts (#21315)
c17's canary stopped at the pre-apply predecessor check with `runtime predecessor mismatch fields=draining`. The flag is residue: the previous canary (run 35290908836) drained c17 at 00:26:01, its terraform apply then failed, and the failsafe re-isolates without restarting the VM, so nothing cleared it. The same run had passed this very check a second earlier, which is what proves a parked cell is not draining at rest. Draining means connections are being shed, and a migration-only cell holds none, so the flag is not a precondition there. Accept it on entry for that class only. The replacement VM is still required not to be draining, on every path, and the incarnation check still proves it was replaced. Both predecessor checks now read one decision instead of computing the rule twice, so the assertion and its diagnostic cannot disagree. Every general-cell and rollback path keeps the value it had; a census test runs the real block over all eight mode and class combinations to hold that. |
||
|
Jinwoo Hong
|
acedcf2a97 |
fix(relay-ops): pin the capacity identity so a stale same-cap template can roll (#21314)
c17's canary-apply failed closed at plan validation. Its instance template is from 2026-08-07 and predates the ORCA_RELAY_CAPACITY_SERVICE_ACCOUNT line that every cell rolled since already carries, so the plan legitimately added it. The same-cap validator holds the whole startup script identical before and after except the image, and that line is not one it excluded, so the wave stopped with nothing applied. Pin the line for same-cap-cell exactly as bootstrap-cell already does, and exclude it from the before/after comparison. The cell may gain it; the pin is what refuses a roll that drops it or rewrites it to another identity. Both plan validations in the job now pass the capacity identity the job already requires. The same-cap contract is otherwise unchanged: any other stale line still fails closed, and needs a convergence apply before the cell can roll. |
||
|
Jinwoo Hong
|
ff8f7085cc |
fix(relay-ops): bind the canary cell's admission class into same-cap batch authority (#21313)
A batch-apply wave verified only that the sealed canary named some approved same-cap cell, so a canary rolled on the migration-only, zero-host, 600-cap c17 or c18 was accepted as authority for a general 1000/3000-cap batch. The verify step now hands the batch's own cells to the check, which requires the sealed cell's entry admission to equal the batch's class. |
||
|
Jinwoo Hong
|
399306c171 |
feat(relay-ops): allow the migration-only cells c17 and c18 in same-cap waves (#21307)
c17 and c18 hold no hosts and sit outside general admission, so rolling one displaces nobody. They are the only zero-displacement canary for a new cell image, but the same-cap wave refused them at the dispatch validator and would have promoted them to general at the end if it had not. Add them to the approved list and teach the wave a cell's entry admission class: the precheck demands the class the cell is declared to serve in, the restore hands it back that class, the isolate on an already-isolated cell is asserted to change nothing, and the selector generation advances by 2 for a general cell and by 0 for a migration-only one. One wave may not mix the two, because every cell after the first offsets from a single per-wave delta. Neither cell is a declared regional-rehome source, so its template carries no rehome trust lines. The source-membership guard now fires exactly when a roll expects those lines instead of for every US cell, which is the invariant it was standing in for, and which limits c17 and c18 to rehome protocol 0. |
||
|
Jinwoo Hong
|
91ade4b82a |
perf(relay): batch control lease renewals per cell instead of one write transaction per host (#21303)
* perf(relay): batch control lease renewals per cell instead of one write transaction per host Every connected desktop renewed its own control lease with its own single-row write transaction every 30s. At ~14,000 hosts that is ~470 write transactions per second fleet-wide, each with its own transaction id, all updating the same few heap pages of relay_assignments and relay_assignment_activity_leases. Sampling three onsets at 250ms showed no lock queue and no slow statement: 60-144 backends piled into LWLock:BufferContent and Timeout/SpinDelay inside that one statement, and Query Insights attributed 152 of 157 seconds of lightweight-lock wait in the onset minute to it. The heartbeat now enqueues a due renewal instead of issuing it. A cell flushes its queue once per second, or as soon as 500 rows are waiting, through one statement that unnests the parameter arrays and applies the same CTE row-wise. Concurrent writers drop from the host count to the cell count, and transaction ids with them. Measured against a 20,000-row table: 1 row 4.1ms, 12 rows 3.3ms, 100 rows 6.6ms, 500 rows 22.7ms. Per-session semantics are unchanged. Each enqueue still resolves on a renewal and rejects with the outcome as its message, so the completed-attempt counter, the staleness guard, and every close path route exactly as before, and one outcome per row is recorded against the flush latency. Lock order is (user_id, relay_host_id), the primary key of relay_assignments, applied in JavaScript and repeated as the statement's ORDER BY. EXPLAIN confirms LockRows sits above that Sort, so a batch acquires its assignment rows in one global order. Every writer in the store locks a host's assignment row before its migration or lease rows and only ever touches one host, so a batch can only wait on a row a single-host writer holds, never the reverse. One statement also means one contended row could fail the whole batch, so a failed batch degrades to the per-host statements it replaced rather than costing every other host on the cell its renewal. * perf(relay): batch control lease renewals per cell instead of one write transaction per host Every connected desktop renewed its own control lease with its own single-row write transaction every 30s. At ~14,000 hosts that is ~470 write transactions per second fleet-wide, each with its own transaction id, all updating the same few heap pages of relay_assignments and relay_assignment_activity_leases. Sampling three onsets at 250ms showed no lock queue and no slow statement: 60-144 backends piled into LWLock:BufferContent and Timeout/SpinDelay inside that one statement, and Query Insights attributed 152 of 157 seconds of lightweight-lock wait in the onset minute to it. The heartbeat now enqueues a due renewal instead of issuing it. A cell flushes its queue every second, or as soon as 200 rows are waiting, through one statement that unnests the parameter arrays and applies the same CTE row-wise. Concurrent writers drop from the host count to the cell count, and transaction ids with them. Per-host buffer traffic is unchanged: 30 hits for one row, 25.5 per host at 12 rows, 30.1 per host at 200, against the 28.7 the single-row statement reports in production. Per-session semantics are unchanged. Each enqueue still resolves on a renewal and rejects with the outcome as its message, so the completed-attempt counter, the staleness guard, and every close path route as before, and one outcome per row is recorded against the flush latency. Row locks live until the statement commits, so a batch that waited on a contended row would hold every other row's lock for that whole wait. The assignment pass therefore takes its locks with SKIP LOCKED and reports a contended host as assignment_lock_unavailable, which the registry retries on the next tick instead of closing the control. That keeps the hold to the statement's own execution: 11.5ms for 200 rows against a 20,000-row table, and 9.4ms with a host wedged in a per-host transaction, where a blocking FOR UPDATE spends the pool's whole 1s lock_timeout and then fails every row in the flush. An unlocked present_assignment probe separates a host with no assignment row from one the skip passed over, so a skipped row can never be mistaken for a missing assignment and close a live desktop. With no wait on the assignment pass the lock order is only needed for the two later passes, and it holds: every writer takes a host's assignment row before that host's lease rows, and a host whose assignment row is held was skipped, so the batch never reaches its lease. markMigrationTargetRegistered is the one writer that locks a migration row first, and it takes no further locks. * fix(relay): answer every row of a control-renewal batch from its own lease update Review findings on the batched renewal. Two control leases on one host in one batch made the second report control_activity_not_found although both were renewed: the assignment UPDATE is offered the same target row twice, applies one source row and returns one, so the other row_index never came back. The verdict now reads renewed_lease, which has a row per input row, and the assignment UPDATE groups per host so it also stops taking an arbitrary one of the two expiries instead of the later one. The queue now partitions per (userId, relayHostId) rather than per activity, so a second control activity for one host opens the next flush instead of sharing this one. Belt to the statement fix, not a substitute: the store API has to be right for the rows it accepts. renewControlActivities recorded no outcome for a one-row flush that threw, and none at all when every row failed validation, where a mixed batch recorded its invalid_* rows. Both now record in a finally, the way the single-row path's finally always did, and the error-to-outcome mapping both paths share is one function. The four flush fields the runtime metrics event emits had no log-based metric, so add them next to the existing controlRenewalLatencyMs* entries. Applying the Terraform is a separate manual step. controlRenewalLatencyMsP50/P95/Max now measure a batched row's flush duration rather than its own statement latency. Left named as they are for history, with a line at the emit site recording that the meaning changed here. |
||
|
Jinwoo Hong
|
ddbad2218b |
chore(relay): drop the live-basis partial index that the planner never picks (#21305)
Added in #21301 on the reasoning that the composite (active, deadline) index spans all 6.65M rows to find a few hundred live ones. Measured post-merge against production-shaped history, that reasoning does not hold: a basis is inserted active and flipped to 0, so the partial index accumulates one dead entry per deactivation exactly as the composite one does. Scan buffers are identical to composite-only in every state, 11,099 cold, 2,522 warm, 9 after VACUUM, and the planner picks the composite index throughout. The extra index costs ~65 bytes of WAL per basis insert, about 22% more. The relief is the reaper plus vacuum, which #21301 already ships. Removes the statement from SCHEMA, restores the plan test to pinning the composite index by name, and records why a narrower index is not the cure so the next reader does not re-derive it. |
||
|
Jinwoo Hong
|
cbd04704d6 |
perf(relay): index and gate the credential cleanup sweeps that seq-scan relay_invites (#21301)
* perf(relay): index and gate the credential cleanup sweeps that seq-scan relay_invites The credential cleanup ran every 30s in all 23 cells as well as the director. Both of its relay_invites passes matched columns no index covered, so each one seq-scanned the whole table inside the maintenance transaction: 56 calls/min fleet-wide, 129ms and 63ms typical and 57s at the tail, to return about one row every nine minutes. Adds partial indexes matching each sweep predicate, gives the cleanup the same owner as the assignment sweep, and reaps terminal invites after seven days so the table stops growing for the life of the database. Every index carries the schema-deferrable marker: an operator builds them with CREATE INDEX CONCURRENTLY, and the catalog pre-check skips them from then on. * perf(relay): index live bases and reap settled connection authorizations relay_connection_bases is the dominant cost in the cleanup transaction: 195 ms of the 268 ms average, with ~5,800 shared buffer hits per call even though it already uses relay_connection_bases_active_deadline. That index spans all 6.65M rows, and only a few hundred are ever active. Adds a partial index on the live rows alone, and reaps settled rows from relay_connection_bases and relay_direct_authorizations once their deadline is more than a day past. Both readers of either table require the row active/unconsumed and inside its deadline, and every deadline is set at most 30s past insert, so a settled row can never authorize anything again. The composite index stays: it is the only one covering active = 0, and it is what lets the drained reaper learn there is nothing to do from the index rather than the 1.5 GB heap. Measured at 200k rows, 5 buffers with it and 1,274 without. * test(relay): accept either bases index in the sweep plan assertion The negative assertion pinned a planner choice rather than the invariant: either index keeps the sweep off the 1.5 GB heap, and which one wins on cost is not something the test should fix. Matches how the same file already handles the two invite sweep indexes. Also names the column the authorization reaper actually measures, which is consumed_at rather than deadline. |
||
|
Jinwoo Hong
|
8e8a9b38ea |
perf(relay): stop indexing the column every control renewal writes (#21286)
* perf(relay): stop indexing the column every control renewal writes relay_assignment_activity_expiry indexes expires_at on relay_assignment_activity_leases, and expires_at is what every control renewal updates: ~471 calls/s, all of them non-HOT because a changed indexed column forbids HOT. The index has one reader, the 30s expiry sweep, which seq-scans the whole 14.8k-row table in under a millisecond. Drop it, and set fillfactor to 70 so a renewal has room for a second row version on its own page. Measured on postgres:16-alpine over 14.8k rows, WAL bytes per renewal and HOT ratio: index, fillfactor 100 (today) 0% HOT 371 B index, fillfactor 70 0% HOT 246 B no index, fillfactor 100 0.5% HOT 298 B no index, fillfactor 70 100% HOT 80 B Both are needed: the index makes HOT illegal, and the default fillfactor leaves no page space to make it possible. Neither statement can use the catalog pre-check as it stood. DROP INDEX IF EXISTS resolves the name before it locks, so once the index is gone it costs a catalog miss and takes no lock on the table - pinned in the lock-target census as the one exempt statement. ALTER TABLE SET does take a lock, so it gets a new 'reloption' target kind that asks pg_class.reloptions for the name=value pair, keeping the invariant that no lock-taking statement reaches a warm boot unchecked. * fix(relay): pre-check the activity-expiry drop and let it defer on a lock timeout The drop had no catalog pre-check, so it was sent on every boot, and a 55P03 from it was fatal: apply-postgres-schema throws on a lock timeout with no retry. On the migration boot that combination is a crash loop. All 28 directors reach the same DROP INDEX at once, it needs ACCESS EXCLUSIVE on a table written ~475/s with lock_timeout at 1s, and a boot that fails restarts the instance to re-queue the same DDL behind the same writers. Two changes: - A new 'index-by-name' lock target. A DROP INDEX names no table, so the existing index check could not serve it; this one resolves by name through the search_path with relkind = 'i', which is how the DROP itself resolves, and skips when absent. DROP INDEX now counts as lock-taking in the census, so it is covered rather than exempt, and IF EXISTS is required the way it is on DROP CONSTRAINT. - A 'schema-deferrable' marker, read from a statement's leading comment. A 55P03 on a marked statement logs orca_relay_postgres_schema_object_deferred and leaves the statement unapplied instead of failing the boot; the next boot re-sends it. Both activity-lease migrations carry it. Everything else keeps the old contract and still fails loudly. SchemaApplySummary gains a deferred count so a boot that skipped work is distinguishable from one with nothing to do. Verified against a real server: with the index present and the table held in ACCESS EXCLUSIVE by another session, both statements defer, the boot completes, nothing is half-applied, and the next boot finishes the job. A warm boot now sends neither statement at all. |
||
|
Jinwoo Hong
|
0e7948fa6d |
feat(relay): pace the drain send during a same-cap cell roll (#21284)
* feat(relay): pace the drain send during a same-cap cell roll A same-cap roll drains a cell with graceMs 0, which sends `drain` to all ~800 controls in one pass. Every desktop re-dials on receipt regardless of graceMs, so the whole cell reconnects inside a second. On 2026-09-16 that stampede hit a Cloud SQL stall: attaches timed out, each leaving 10 minutes of late-arrival debt on connection headroom, and placement answered relay_capacity_exhausted fleet-wide for ~13 minutes. Spreading the sends spreads the re-dials. `HostSessionRegistry.drain` takes an optional pacing window and schedules each session's send evenly across it; admission is fenced for every session up front, and each host keeps its own full grace after its own send. /v1/admin/drain accepts `paceWindowMs` (<= 5 min) and echoes what it applied. The same-cap job asks for 120 s, and the drain-completion wait grew by the same amount. A cell still on an older image rejects the field, so the deploy script falls back to an unpaced drain rather than failing the roll. * fix(relay): scope the drain fence to the hosts already told Review of the paced drain found two problems, both from treating "this cell is draining" as one instant when pacing makes it a window. Timers: the sends queued by a paced drain were neither cleared when a later drain superseded them nor unref'd. A SIGTERM mid-window left up to 800 no-op timers holding the event loop open until systemd escalated to SIGKILL. Drain timers are now tracked, cleared on the next drain, and unref'd, so a retry re-arms a session's teardown instead of stacking a second one. Phones: the client fence read the global draining flag, so every phone was refused for the whole window even though its own host had not been told yet and was still serving. The director keeps pointing phones at this cell until their host moves, so they would have looped for up to two minutes. A session is now fenced when its drain is sent, not when the drain starts, and the client paths key off that. New control connections and re-attaches stay fenced globally: nothing new should land on a cell that is going away. |
||
|
Jinwoo Hong
|
09622f0c28 |
feat(relay): add a break-glass override for the same-cap monitor gate (#21270)
* feat(relay): add a break-glass override for the same-cap monitor gate Every mutating same-cap wave consumes a fresh 15-minute aggregate monitor dry-run. When a chronic fault is what the gate freezes on, waiting for a green window means waiting for the condition the wave removes: the gate froze 44 consecutive times on the recurring Cloud SQL stall the rolling image fixes. Add `gate-override-reason` and `gate-override-confirmation` (`SKIP_RELAY_MONITOR_GATE <target-image-digest>`) to the same-cap dispatch. A valid pair skips only the aggregate evidence download, provenance verification, and single-use marker. A partial or mismatched override fails closed before any mutation, in both the caller and the reusable job. Record the actor, reason, and confirmation in the gate run summary and, for a canary, in the sealed artifact. The live per-wave preflight still runs. Give it a `--no-monitor-state` source that takes the expected selector from the dispatch inputs and pins the migration policy to `strict`, rather than synthesising a state file that would claim a dry-run it never ran. Also give `director.instances` the two-consecutive-sample tolerance the cell probes have: Cloud Run replaces an instance in place, so the count leaves the [5, 6] band for one sample roughly twice a day, and a deploy overlap raises it the same way. Min and max share one streak so an alternating count still freezes. * fix(relay): canonicalise the break-glass preflight membership The override path parsed the operator's membership with a bare schema parse, while the live selector read from the director is normalised and the comparison is an ordered `JSON.stringify`. Unsorted dispatch input would therefore read as selector drift on a healthy fleet, and the every-configured-cell-exactly-once check was lost with it. Normalise through the same `normalizeSelectorMembership` call the monitor CLI uses when it seals evidence, against the same durable Terraform cell set. Tests use a collect stub that returns the director's canonical selector rather than echoing the expected one, so the ordering is actually exercised: unsorted input must canonicalise, and a duplicated, missing, or unknown cell must be rejected. |
||
|
Jinwoo Hong
|
0ed2771fa5 |
fix(relay-ops): tolerate a single unreadable monitor sample (#21272)
An unreadable sample (collector_failed) now gets the same two consecutive sample budget per source as an unread signal, so one failed Cloud Monitoring read no longer restarts the continuous window. monitor_gap keeps zero tolerance because it means the run itself stopped sampling. The pre-drain lineage cap moves from 25 to 35 minutes so a 15-minute window plus one restart still reaches a verdict, and the collector error message is now logged instead of being swallowed. |
||
|
Jinwoo Hong
|
c4917d6e74 |
fix(cloud): retry transient director admin failures in the relay monitor and preflight (#21263)
The director's /v1/admin/cell-status maps any thrown operation error onto HTTP 404, so a Cloud SQL pool connect timeout arrived at the ops tooling as "Relay admin telemetry returned 404" and killed the whole sample. Retry the admin reads that carry a transient database error, and let the live preflight spend one of its existing attempts on a thrown collector instead of failing the wave. |
||
|
Jinwoo Hong
|
1957437005 |
fix(relay): stop admin routes reporting a stalled database as 404 or 409 (#21264)
Every admin handler collapsed a thrown error into one status, so a two-second pool connect timeout answered POST /v1/admin/cell-status with 404. The rollout tooling never retries a 4xx, by design, so the wave failed on a database that was briefly out of reach and recovered on its own. Transient database failures now answer 503 with Retry-After, the shape the public routes and the region catalog already use. Every other error keeps the route's existing 404 or 409 mapping. |
||
|
Jinwoo Hong
|
8b2502fc92 |
fix(cloud): give same-cap waves ten minutes to consume gate evidence (#21259)
* fix(cloud): give same-cap waves ten minutes to consume gate evidence The live preflight rejected monitor evidence older than five minutes, but the same-cap job only reaches that step about five minutes after the monitor completes: runner queue, the gate job, and a full-branch checkout. On 2026-09-17 the first green gate in 44 attempts died at 302 s. The preflight still takes live samples, so the older baseline is safe. * docs(cloud): state the ten-minute preflight evidence bound |
||
|
Jinwoo Hong
|
a3046cd27b |
fix(relay): treat database pool connect failures as transient, not director faults (#21243)
* fix(relay): treat pool connect failures as transient, not director faults pg-pool raises connection-acquire failures as a plain Error with no SQLSTATE, so the transient classifier matched only one of the three messages it can produce. The other two reached the routes unclassified and became HTTP 500s, which is what the rollout safety gate counts. The acquire boundary now marks the errors it produces, so "Connection terminated unexpectedly" counts as transient when the socket died during the handshake and stays a hard failure mid-statement, where a retry could repeat a commit whose outcome is unknown. /v1/regions and /v1/admin/evacuation-status gain the transient handling /v1/assign and /v1/resolve already had. * fix(relay): mirror the pool-connect verdict in failure diagnostics The query-failure event's connectionTimeout boolean matched one of the two messages connectionTimeoutMillis can produce, so the 210 dialling timeouts in the last day logged as false and were invisible to the field meant to find them. The pool-connect vocabulary now lives beside the acquire boundary that owns it, and both the router's classifier and the diagnostics read it from there, so the two cannot drift. The event also carries the routing verdict the caller already computed, making "how much of this burst reached users as a 500" one field. * fix(relay): null-safe transient classification and honest transient docs The classifier now runs inside the query catch, where a thrown null or undefined would have turned a database failure into a TypeError that buried it. The diagnostics doc claimed transient maps to a 503 or a 500. Sweeps, startup reconciliation, and admin routes that answer 409 all emit the same event, so counting the false ones over-states user-facing hard failures. |
||
|
Jinwoo Hong
|
7184b1dc5b |
fix(relay-ops): recalibrate the pre-roll monitor gate to chronic production baselines (#21241)
* fix(relay-ops): let the pre-roll gate ride out chronic production noise The 15-minute pre-drain dry-run froze 39 times out of 39 on conditions that have nothing to do with the roll it gates: - A cell probe is one HTTP round trip from one runner. When the Asia cells' readiness SQL probe times out behind a saturated pool, the load balancer answers "no healthy upstream" for ~30 s and the gate froze on a single sample. Cell probe signals now need more than cellProbeToleranceSamples consecutive failing samples to freeze; absorbed blips are recorded in the state artifact. Director and auth probes keep zero tolerance. - directorErrors 3 -> 15. Measured non-503 5xx per rolling five minutes over the 24 h to 2026-09-17: p90 3 / p95 5 / p99 9 / max 52. The old bar sat on the p90 and froze 29% of gates. - cloudSqlBackends 250 -> 320. Measured latest-sum over the same 24 h: p95 212 / p99 262 / max 282. The old bar sat under the observed peak and froze 22% of gates. Failure codes are unchanged so downstream matchers keep working, and the trusted evidence scripts are untouched. * fix(relay-ops): key probe tolerance by cell and extend it to live preflight Three review findings on the cell-probe tolerance: - The streak was keyed per signal, so a cell alternating between slow (latency over bar) and down (health/ready 0) held every individual streak at one and never reached the tolerance. A continuously unhealthy cell passed the gate. The streak is now keyed by cell id, so one cell's health, ready and latency readings share it. - The live preflight runs one sample before every mutating wave and retried only on freshness codes, so the same Asia blip could still fail a wave there. It now re-samples per-cell probe breaches on the same tolerance, spaced the existing interval. Director and auth probes still fail the wave on the first bad sample, as does any non-probe threshold. - docs/relay-incident-monitor.md still stated the old bars. Updated the threshold table, the 400-connection ceiling text, and the superseded 2026-08-26 and 2026-09-12 entries, and added a dated 2026-09-17 recalibration entry. Also pins the resumed-state case: a state file carrying a full streak now has a test proving it freezes on the next bad sample. Trusted evidence scripts remain untouched. |
||
|
Jinwoo Hong
|
560c42e1d1 |
fix(cloud): pin the asia cell database pool in the same-cap plan validator (#21171)
* fix(cloud): pin the asia cell database pool in the same-cap plan validator Raising `database_pool_max` from 10 to 16 for production-gce-c27, c28 and c29 made every same-cap roll of those three cells fail closed at plan validation. The cell startup template emits `ORCA_RELAY_DATABASE_POOL_MAX` only for a cell whose region differs from the root region or whose pool is off the default, so the asia cells carry that line while the us-central1 cells do not. The plan validator requires the before and after startup scripts to normalize to the same text, masking only the lines it independently pins to a reviewed value. The pool line was neither masked nor pinned, so the live template's `'10'` and the plan's `'16'` were read as unreviewed drift. The validator gains an optional `--database-pool-max`, accepted in `same-cap-cell` mode alone. When it is supplied the after-script must contain exactly that pool line and the line is masked from the equality check; when it is not supplied the after-script must contain no pool line at all. Masking without the pin would have removed the guard rather than moved it. The same-cap job resolves the expected pool next to the hard cap, cross-checks it against the committed `relay_gce_cells` map (asserting the default 10 for the us-central1 cells), and passes the flag to both validator invocations only for the cells that emit the line. * test(cloud): require the pool pin for a line the live template already carries |
||
|
Jinwoo Hong
|
5947d6b269 |
infra(relay): raise asia-east2 cell pools to 16 and record the measured connection ceiling (#21163)
* infra(relay): raise asia-east2 cell pools to 16 and retire four idle cells The three asia-east2 cells sit 176 ms from the Cloud SQL instance in us-central1. Server-side statement time there is 0.2 ms, so a pool slot is held by the round trip, not by the query. At a pool of 10 they measured 94-156 waiters and 2 s waits, and client accepts ran a ~4 s p95 against 222-646 ms in us-central1. Raising those three pools to 16 is the agreed first step; every other cell stays at 10. c4 and c5 join the committed fence set. Both are existing-only capacity the admission selector can never place on again, they carried ~1 connection each on 40-day-old images, and each still holds 10 Postgres connections. The fence set is the prerequisite the fence-source workflow confirms before it drains and attests a cell; it is not itself the resize. c17 and c18 are not fenced here. They are migration-only, and the runbook requires retire-migration-cell to move a migration-only cell to existing-only through a generation-bound selector CAS before it can be fenced. Terraform cannot express that step. The Cloud SQL consumer contract carried two stale numbers: auth at 2 instances when production has run a cap of 20 since 2026-09-04, and a 400-connection ceiling when the live instance reports 500. Both are corrected, and the budget now asserts its headroom in two named gates instead of one aggregate boolean. Those gates fail: auth alone accounts for 200 configured connections and a 215-connection rollout overlap, so the operating maximum is 713 against a usable ceiling of 490. Nothing here caused that, and no pool was lowered to hide it. * infra(relay): move the Cloud SQL contract correction out of this branch The contract correction (auth at its real 20-instance cap, the measured 500-connection ceiling) makes the budget gate fail for reasons that have nothing to do with asia pools or fenced cells, and it held this branch red. It moves to its own branch where the failure is the subject. production-cloud-sql-app-consumers.json returns to main unchanged. The budget test keeps main's single gate and only repins the cell figure that this branch genuinely moves: 230 -> 228, being +18 for three asia pools at 16 and -20 for fencing c4 and c5. Against main's 400-connection model that leaves an operating maximum of 383 under a usable ceiling of 390. * infra(relay): move the c4/c5 fence entries out of this branch Terraform now sets a cell's MIG target size directly from relay_gce_fenced_cells (relay-gce-cells.tf); the lifecycle ignore that used to protect operational target_size drift is gone. So a fence entry sitting on main ahead of its fence-source run is a standing instruction that any apply reaching that cell may execute without the documented drain and attestation. Keeping the entry in the same merge as an unrelated pool change widens that blast radius for no reason. The two entries move to their own branch, to be merged immediately before fence-source runs for c4 and then c5. This branch keeps the multi-line reflow of the list, which makes that later diff two added lines instead of a rewritten one. The cell figure in the budget test follows: 230 + 18 for the three asia-east2 pools at 16, with no fenced-cell subtraction. That is 403 operating against a usable ceiling of 390, so the headroom gate now fails by 13. It fails against a ceiling of 400 that is itself wrong; the instance reports 500. See the PR body. * infra(cloud-sql): record the measured 500-connection ceiling The budget's usable ceiling came from maxConnections: 400, described as the tier default. It is a tier default, since no max_connections flag is set, but the instance does not report 400. SHOW max_connections on it returns 500, measured 2026-09-16. On main the model sat at 385 against a usable ceiling of 390, five connections of margin, so raising the three asia-east2 pools by 18 failed the gate by 13 against a ceiling that was never checked. Against the measured one it is 403 against 490, clearing by 87. Only the ceiling and its source note change here. auth stays recorded at 2 instances, which is also wrong; PR #21165 corrects it, and with the true auth figure the budget is over by 225 for reasons that have nothing to do with these pools. * test(cloud): state the cell pool arithmetic literally in the budget pin comment |
||
|
Jinwoo Hong
|
69787e763a |
fix(relay): serve readiness from last-known-good during auth or SQL blips (#21161)
* fix(relay): serve readiness from last-known-good during auth or SQL blips
The load balancer health check hits /ready, which re-probed the auth JWKS
endpoint and Postgres on every poll and reported not-ready on the first
failure. On 2026-09-16 an auth outage therefore took every cell out of the
load balancer within ~30s and dropped every connected host, even though the
token verifier caches keys in process and kept verifying tokens.
/ready now remembers when each dependency last answered and keeps reporting
ready while the failed one stays inside a grace window
(ORCA_RELAY_READINESS_GRACE_MS, default 15 minutes, 0 disables). A process
that has never succeeded still gates on the real dependencies, so cold boot
is unchanged. Grace answers carry degraded plus the failure reason on the
existing readiness observation, and entering or leaving grace logs once.
MIG autohealing still uses the dependency-free /health endpoint.
* fix(relay): split readiness grace per dependency and probe both every poll
Review follow-ups on the last-known-good readiness window.
An unset environment variable arrives as an empty string, which z.coerce
reads as 0, so the single ORCA_RELAY_READINESS_GRACE_MS would have switched
the window off instead of falling back to its default. The two replacement
variables preprocess '' to undefined.
JWKS and SQL now get separate windows and separate clocks:
ORCA_RELAY_READINESS_JWKS_GRACE_MS defaults to 15 minutes, and
ORCA_RELAY_READINESS_SQL_GRACE_MS to 3 minutes. Each cell is its own load
balancer backend, so failing readiness never re-routes a host, it only makes
that hostname unreachable, and a host that lands on a SQL-dead cell gets
WRONG_CELL and is re-placed by the director. Three minutes rides a Cloud SQL
failover without hiding a per-cell fault for a quarter of an hour.
Both dependencies are probed on every poll. A JWKS failure used to
short-circuit the SQL probe, which let the SQL clock age with no evidence
behind it. Grace transitions are emitted per dependency, so JWKS recovering
while SQL fails logs both sides instead of nothing.
/ready keeps its 200 and its {ok:true} body when healthy, and adds
degraded plus the dependency list when the answer comes from a window.
|
||
|
Jinwoo Hong
|
0699d73fd6 |
fix(relay): skip boot-time DDL when the catalog already has the object (#21147)
* fix(relay): skip boot-time DDL when the catalog already has the object CREATE INDEX IF NOT EXISTS and ALTER TABLE ADD COLUMN IF NOT EXISTS take their relation lock before the server evaluates the existence test, so a boot on an already-migrated database still joins the lock queue. Relation locks are granted in queue order, so every writer queues behind it. The shared runner now asks pg_catalog whether the index or column is already there and skips the statement when a row comes back, and 55P03 is no longer retried by default: with the pre-check ahead of it, a lock timeout means the object is genuinely missing and each retry re-enters the queue. Push keeps the old retry behind an explicit option. * fix(relay): tie the index pre-check to its table and fail on an unreadable target Three defects found in review of the auth reference implementation: - The catalog query matched an index by name inside the table's namespace without checking it belonged to that table. Index names are unique per schema, not per table, so a same-named index on a sibling table answered yes and the real index was skipped forever. Added i.indrelid = t.oid. - Lock-target derivation read a keyword sitting in an identifier position as the object name: CREATE UNIQUE INDEX CONCURRENTLY ON t(c) yielded the name CONCURRENTLY, and ADD COLUMN IF NOT EXISTS with no column yielded IF. A wrong target is worse than none, so keywords are now excluded and an index or column statement whose target cannot be read throws at boot with the statement text instead of falling through to the lock path. - A concurrent-create collision retried the CREATE INDEX, taking SHARE on the table again for an object another director had just finished creating. The catalog is re-asked instead and a present object counts as skipped. * fix(relay): pre-check constraint swaps so a warm boot sends no DDL at all The two ALTER TABLE constraint statements were the last lock-taking statements without a pre-check, so every boot still took ACCESS EXCLUSIVE on relay_region_rehome_attempts twice. A lock target now carries the catalog answer that means there is nothing left to do. ADD CONSTRAINT skips when pg_constraint already names it; DROP CONSTRAINT IF EXISTS is the inverse and skips when it does not, because nothing to drop is nothing to do. The match is by name only: the CHECK body is generated from RELAY_REGIONS, so comparing it would re-run the swap on every region change. Changing a definition under the same name is an operator migration, and the rule comment beside SCHEMA says so. A bare DROP CONSTRAINT gets no target and throws at boot, because skipping it would swallow the undefined_object the server is supposed to raise. The census invariant is now that every lock-taking statement has a pre-check, with no exceptions, and the warm-boot Postgres test asserts zero statements sent rather than two. * fix(relay): refuse a multi-action ALTER TABLE instead of pre-checking its first action `ALTER TABLE t ADD COLUMN IF NOT EXISTS a TEXT, ADD COLUMN IF NOT EXISTS b TEXT` derived the target for `a` alone, so once `a` existed the whole statement was skipped and `b` was never added. The first subcommand parses, so neither the parse throw nor the census caught it. A lock-taking ALTER TABLE with a comma outside parentheses, quotes and comments now throws at boot. One action per statement, or no pre-check is possible. Commas inside a parenthesised type, a CHECK body, a quoted default or a comment are unaffected, and push's 18 statements still parse. * fix(relay): strip every comment before classifying, fold catalog names, count brackets Four findings from the bot reviews on #21147: - A comment between two keywords (ALTER TABLE t ADD /* note */ COLUMN c TEXT) was invisible to both the classification regexes and the must-parse shapes, so the statement got no target AND no throw and ran with no pre-check. Every comment is now stripped quote-aware before classification, nested block comments included. The server is still sent the original text. - hasTopLevelComma counted parentheses but not square brackets, so ADD COLUMN c bigint[] DEFAULT ARRAY[1, 2] read as two subcommands and failed the boot. - bareIdentifier split a qualified name on '.' regardless of quoting, so "a.b" became b", and it kept the written case while Postgres folds an unquoted identifier to lower case before storing it in relname, attname and conname. The name is now tokenised quote-aware and folded, with the qualified table text still passed to to_regclass as written. - sqlWithoutLeadingComments is renamed sqlWithoutComments to match. Relay's 74 statements and push's 18 all still parse, and no relay target name changed: every identifier there was already lower case. * fix(relay): treat a dollar-quoted body as opaque in both scanners A comment marker, comma, parenthesis or bracket inside `$$...$$` or `$tag$...$tag$` is text. The closing delimiter has to match the opening tag exactly, so an inner `$$` inside a `$tag$` body is more text rather than the end, and a tag cannot start with a digit, which keeps a `$1` placeholder from reading as an opener. Relay's pg_stat_statements DO block is the only dollar-quoted statement in the schema, and it now survives the stripper byte-identical. A test asserts that against the real statement. |
||
 Neil
|
2569a71ce8 | fix(deps): update vulnerable dependencies without new overrides | ||
|
Jinwoo Hong
|
77cd61df39 |
fix(relay): keep pool pressure a per-cell rehome exclusion, not a fleet stop (#21126)
The fleet safety gate returned database_pool_pressure whenever the Math.max of database_pool_waiters_max or database_pool_wait_ms_max across every general cell crossed 16 waiters or 250ms. Measured 2026-09-16, the asia-east2 cells breach continuously at 94-156 waiters and ~2000ms while their server-side execution is 0.2ms, which is a client pool too narrow for a 176ms round trip rather than database distress, and the us-central1 cells breach in bursts on about a third of polls. Worse, the bar flaps: the pre-check passes, the commit re-check reads fresh rows seconds later and trips, and that path durably disables the control instead of merely deferring. Drop the pool check from the fleet gate. Pool pressure stays a per-cell exclusion in regionalRehomeCellSafetyIsClean, which already drops a breaching cell as both source and target on selection and again on the commit path. The fleet bars that remain (stale monitoring, sql failure storms, control-recovery failures, reconnect storms) all signal database-wide distress. Nothing cells publish, no stored row and no exported constant changes. |
||
|
Jinwoo Hong
|
71e308e574 |
feat(relay): count failed cell-inventory lock acquisitions (#21067)
* feat(relay): count failed cell-inventory lock acquisitions The cell inventory lock is taken NOWAIT, so contention errors with 55P03 and retries instead of waiting. CellInventoryHoldSamples.record only runs after a successful acquisition, so the hold metrics were structurally blind to the dominant failure mode: production showed ~65 failed fleet-wide acquisitions per minute while cellInventoryHoldMsMax read a benign 53ms mean. Count failures next to the holds and publish them as cellInventoryLockUnavailable in orca_relay_runtime_metrics. Drained on both the commit and the rollback path, since a 55P03 rolls its transaction back. * fix(relay): separate request-path lock timeouts from sweep deferrals Review caught that the first counter only incremented under failIfUnavailable, which is the sweep mode. Background sweeps take the inventory NOWAIT and re-derive a skipped candidate next tick, so those deferrals are by design and already reported as orca_relay_sweep_cell_inventory_busy. The request path uses a bounded lock_timeout instead, whose expiry raises the same 55P03 without NOWAIT and was not counted at all -- so the metric measured only the benign population and missed the user-visible one. Split them: cellInventoryLockUnavailable for NOWAIT deferrals, cellInventoryLockTimeouts for expired bounded waits. Production over 30 minutes shows why the distinction matters -- roughly 1,200 fleet-wide sweep deferrals against roughly 10/min request-path timeouts. Adds transaction-path coverage for both drains, which were previously unpinned. Timeouts count per attempt, not per request, since 55P03 is retryable. * fix(relay): publish the cell-inventory lock metrics to Cloud Monitoring google_logging_metric.relay_snapshot only creates metrics for fields listed in relay_runtime_metrics, and the cellInventoryHold* fields were never added when the hold telemetry landed. They have been log-only since, so nothing could alert on the lock and the contention stayed invisible in exactly the way the telemetry was meant to prevent. Maps the three hold fields and both new failure counters. Also corrects the field comment: the split is by wait policy, not by caller. assignOnce takes the inventory fail-fast on its first placement attempt, so request-reachable sites land in cellInventoryLockUnavailable too; that lane reads as contention pressure, and the expired bounded wait is the stall lane. |
||
|
Jinwoo Hong
|
b07c4032ea | Log bounded PostgreSQL acquisition and execution failure diagnostics (#20749) | ||
|
Jinwoo Hong
|
d51747e4c4 | feat(relay): expose preloaded PostgreSQL statement statistics (#20712) | ||
 Neil
|
2162e31f80 |
test(relay): differential coverage for the single-pass host-data owner lookup (#20426)
The production change (single insertion-order scan of the session inventory, reused by the unfenced leg) landed in #20219. This carries the regression coverage for it: a 1,000-session differential suite that counts iterator visits and pendingConns.has probes against the pre-change two-find oracle, ordering under duplicate connection IDs, and attach-ownership tests on the client-accept path. Folds host-session-owner-scan.test.ts into that suite. |
||
|
Neil
|
101cdc45f8 | perf(relay): sort latency samples once per percentile pair (#20427) | ||
|
Neil
|
0537c6eb3b | perf: index drain migration inventories by assignment identity (#20344) | ||
|
Neil
|
ce82438828 | perf(relay): reuse host data attachment owner without inventory copies (#20219) | ||
|
Jinwoo Hong
|
e1bac25041 | fix(cloud): diagnose wrapped relay trust probe failures (#20403) | ||
|
Jinwoo Hong
|
1a9a5f9bc7 | fix(cloud): tolerate sparse director errors in rollout monitor (#20238) | ||
|
Jinwoo Hong
|
9f7fd9a270 | fix(relay): reuse canary across completed rollout batches (#20214) | ||
|
Jinwoo Hong
|
f7238ce469 |
fix(relay): bound idle rehome polling and skip disabled scans (#20203)
* fix(relay): bound idle rehome polling and skip disabled scans * test(relay): align sweep jitter expectation with polling budget |
||
|
Jinwoo Hong
|
113e58f34e |
feat(relay): support protocol 3 in cell rollout gates (#20174)
* feat(relay): support protocol 3 in cell rollout gates * fix(relay): validate and prove protocol-3 cell rollouts * docs(relay): clarify regional capability deployment prerequisite * test(relay): cover protocol-3 plans across rollout cells |
||
|
Jinwoo Hong
|
1d7bb47a11 | test(relay): harden regional rehome race coverage (#20136) | ||
|
Jinwoo Hong
|
cd9aa43a2c |
feat(relay): correct regional placement only when the source is idle (#20105)
* feat(relay): correct regional placement only at an idle source * test(relay): lock source activity capacity semantics |
||
|
Jinwoo Hong
|
08a24efaba | fix(push): preserve distinct Android alerts while offline (#20066) |