mirror of
https://github.com/stablyai/orca.git
synced 2026-09-22 00:02:31 +00:00
Jinwoo Hong
7184b1dc5b
* fix(relay-ops): let the pre-roll gate ride out chronic production noise The 15-minute pre-drain dry-run froze 39 times out of 39 on conditions that have nothing to do with the roll it gates: - A cell probe is one HTTP round trip from one runner. When the Asia cells' readiness SQL probe times out behind a saturated pool, the load balancer answers "no healthy upstream" for ~30 s and the gate froze on a single sample. Cell probe signals now need more than cellProbeToleranceSamples consecutive failing samples to freeze; absorbed blips are recorded in the state artifact. Director and auth probes keep zero tolerance. - directorErrors 3 -> 15. Measured non-503 5xx per rolling five minutes over the 24 h to 2026-09-17: p90 3 / p95 5 / p99 9 / max 52. The old bar sat on the p90 and froze 29% of gates. - cloudSqlBackends 250 -> 320. Measured latest-sum over the same 24 h: p95 212 / p99 262 / max 282. The old bar sat under the observed peak and froze 22% of gates. Failure codes are unchanged so downstream matchers keep working, and the trusted evidence scripts are untouched. * fix(relay-ops): key probe tolerance by cell and extend it to live preflight Three review findings on the cell-probe tolerance: - The streak was keyed per signal, so a cell alternating between slow (latency over bar) and down (health/ready 0) held every individual streak at one and never reached the tolerance. A continuously unhealthy cell passed the gate. The streak is now keyed by cell id, so one cell's health, ready and latency readings share it. - The live preflight runs one sample before every mutating wave and retried only on freshness codes, so the same Asia blip could still fail a wave there. It now re-samples per-cell probe breaches on the same tolerance, spaced the existing interval. Director and auth probes still fail the wave on the first bad sample, as does any non-probe threshold. - docs/relay-incident-monitor.md still stated the old bars. Updated the threshold table, the 400-connection ceiling text, and the superseded 2026-08-26 and 2026-09-12 entries, and added a dated 2026-09-17 recalibration entry. Also pins the resumed-state case: a state file carrying a full streak now has a test proving it freezes on the next bad sample. Trusted evidence scripts remain untouched.