Commit Graph
4 Commits
Author SHA1 Message Date
Jinwoo Hong b378101901 docs(cloud): reconcile the 2026-08-23 retry figure with the gate metric (#18581) 2026-09-04 01:40:55 -04:00
Jinwoo Hong 79d5fb469a fix(cloud): recalibrate the relay monitor's postgres-retry freeze to a measured bar (#18580)
The global relay_cells FOR UPDATE lock made successful retries a
steady-state rate: fleet-wide p50 430 / p90 924 / p99 1320 / max 1504
per five minutes over the last 24 h, 55% of windows over the 300 bar,
only 22% of 15-minute gates clean. Three read-only dry-runs on
2026-09-04 froze on it, blocking the same-cap roll that carries #18521
and the beginProof crash guard to the 23 cells. 2000 clears every
measured healthy gate; the exhausted-retry, director concurrency, and
pool bars keep the incident discriminator role.
2026-09-04 01:23:54 -04:00
Jinwoo Hong 4101505b6b fix(cloud): recalibrate the relay monitor's exhausted-retry freeze to a measured bar (#18569)
* fix(cloud): recalibrate the relay monitor's exhausted-retry freeze to a measured bar

The pre-drain dry-run froze at minute one on relayPostgresRetryExhausted: 0
in every run since #18521 reached the director, blocking the cell roll that
carries the same fix. #18521 made contended request-path waiters fail fast
(500 ms) instead of succeeding slowly, so exhaustion is now a steady
contention rate: 236/236 five-minute windows non-zero over 23 h; post-#18521
p50 42 / p90 147 / max 220 fleet-wide; the 2026-08-23 incident peaked at 467.
300 clears every measured healthy window and stays under the incident shape.
/v1/assign 503 share was unchanged by #18521 (13.9% vs 12.3%).

* test(cloud): pin the exhausted-retry freeze boundary at exactly 300

* docs(cloud): reword relay comments that still described the zero exhausted-retry bar
2026-09-04 00:20:28 -04:00
Jinwoo Hong 3eec77c11a chore(cloud): add the relay fence broker, ops console, Terraform root, scripts, and 24 cloud-* workflows (#18413)
Phase 6 of the relay split: the relay's deploy/operate surface moves under cloud/ with 24 cloud-* workflows gated on ORCA_CLOUD_OPERATIONS_ENABLED, the Cloud SQL rollout lease action, the relay Terraform root (dual-accept identities for both repositories), scripts, docs, CODEOWNERS, and a terraform validate job in Cloud Verify.
2026-09-03 06:55:14 -04:00