mirror of
https://github.com/stablyai/orca.git
synced 2026-09-29 16:02:50 +00:00
Jinwoo-H
6785019551
US desktops were homed on asia-east2 cells for weeks in 2026-08 with every existing relay alert green. Roughly 226 of 332 hosts on those cells were non-APAC, and a phone connect took ~10 s there against ~0.6 s in region, but nothing in Cloud Monitoring could see distance: the connection, queue, heap, and SQL bars all measure a cell's own health, which was fine. Three policies close that gap. Two read distance per cell, from the accept and control-RTT timing added in the parent commit: phone-accept p95 above 2 s, and control ping p50 above 150 ms. The third reads the cause fleet-wide, as the asia-east2 share of the region hints desktops send the director, so a mis-picking client probe is visible before it lands anyone on a far cell. All three are MQL rather than the metric filters the other relay policies use. Every runtime metric is a DELTA DISTRIBUTION, and a filter condition can only align one with a percentile; each alert needs the sum of the extracted values as a volume floor so a sparse window cannot page. None of these metrics exists in the project yet, so what was checked against production is the query shape: the same MQL run over existing metrics of the same kind. The skew denominator needs one log-based metric per hint key, so `requestedRegionsDelta` now has one per relay region plus the unhinted bucket. Those ride the existing snapshot metric family, which adds map entries without touching the live metrics. A ratchet test pins the key list to relay-contract's RELAY_REGIONS: a region added there without a metric would shrink the denominator, so the test fails rather than letting the share quietly inflate.