Files
orca/cloud/apps/relay-ops/src
Jinwoo Hong a3c1d32995 fix(relay-ops): per-region cell latency bar and attributable preflight failures (#18877)
The incident monitor froze three healthy 15-minute production gates on
2026-09-05 because asia-east2 cells are judged against a bar calibrated
for us-central1. A cell's /ready fetches the auth JWKS and runs SELECT 1
against Cloud SQL, both in us-central1, so from the US GitHub runner the
asia-east2 round trip measures p50 0.88 s / max 2.7 s against 0.08-0.5 s
for us-central1 cells.

Give cell.<id>.latency_ms a per-region threshold (us-central1 2000,
asia-east2 4000) carried on IncidentCellExpectation from the tfvars
region. Director and auth latency rules keep the flat 2000 bar, and hard
faults are still caught by the .health/.ready equal-1 checks and the
probe's 8 s fetch timeout.

Also name the signal and its observed/threshold in the live preflight
failure message, keeping the source/code tokens other tooling matches on.
2026-09-05 16:21:50 -04:00
..