Jinwoo Hong
8cf6228655
feat(relay): alert on cell disconnect bursts and database pool herds ( #26363 )
...
* feat(relay): alert on cell disconnect bursts and database pool herds
Two alert policies that separate a load-balancer mass disconnect from a
database stall, after the 2026-10-06 18:34Z asia-east2 drop:
- Disconnect burst: over 120 code-1006 desktop control closes on one cell
in a minute, counted from the cell's own close line. The LB's
internal_error log entries carry the stream start as their timestamp
(median 2.4 h before the drop), so an LB-log rate never shows the pulse.
- Pool herd: a cell runtime sample with more than 50 requests queued for a
PostgreSQL connection (interval maximum, bar in the filter).
Also publishes orca_relay_host_hellos_shed for the refusal counter from
#26362 , and pins both filters to the relay's text in a contract test.
* fix(relay): raise the cell pool-herd alert bar to 200 waiters
asia-east2 cells reach 184-196 waiters outside any herd, so >50 would have
fired in 84 episodes since 2026-10-03. >200 fires in 10, covering every known
herd. Drops the false 'healthy peak is 2 waiters' baseline from the policy text.
* docs(relay): describe pool-herd baseline as stalls, not routine load
A healthy cell peaks at 2 waiters (p99); the 50-196 readings are single-cell
stalls that already time out. The 200 bar is unchanged.
2026-10-07 21:03:49 -04:00
..
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-30 22:25:36 -04:00
2026-09-30 22:25:36 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-09 16:22:17 -04:00
2026-09-11 14:25:10 -04:00
2026-09-11 14:25:10 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-10-05 23:15:25 -04:00
2026-10-05 23:15:25 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-22 23:41:09 -04:00
2026-10-06 00:53:51 -04:00
2026-10-06 00:53:51 -04:00
2026-10-05 23:15:27 -04:00
2026-10-05 23:15:27 -04:00
2026-09-23 14:37:17 -04:00
2026-09-23 14:37:17 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-10-01 15:34:57 -04:00
2026-10-01 15:34:57 -04:00
2026-10-05 00:26:33 -04:00
2026-10-05 00:26:33 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-10-07 19:07:02 -04:00
2026-10-07 19:07:02 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-10-05 00:26:33 -04:00
2026-10-05 00:26:33 -04:00
2026-09-03 06:55:14 -04:00
2026-09-29 01:21:53 -07:00
2026-09-10 17:59:46 -04:00
2026-09-10 17:59:46 -04:00
2026-09-29 01:21:53 -07:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-11 14:25:10 -04:00
2026-09-11 14:25:10 -04:00
2026-10-07 21:00:44 -04:00
2026-09-04 22:04:21 -04:00
2026-09-04 22:04:21 -04:00
2026-09-21 04:18:57 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-10-05 23:15:27 -04:00
2026-10-05 23:15:27 -04:00
2026-10-05 00:26:33 -04:00
2026-09-07 00:30:21 -04:00
2026-10-05 00:26:33 -04:00
2026-10-05 18:51:18 -04:00
2026-10-01 15:36:17 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-22 23:41:09 -04:00
2026-09-22 23:41:09 -04:00
2026-09-18 17:55:35 -04:00
2026-09-03 06:55:14 -04:00
2026-09-26 12:59:47 -07:00
2026-09-26 12:59:47 -07:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-10-01 15:34:57 -04:00
2026-10-01 15:34:57 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-23 03:24:09 -04:00
2026-10-01 15:36:17 -04:00
2026-10-01 15:36:17 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 07:18:29 -04:00
2026-09-09 16:22:17 -04:00
2026-10-07 19:07:02 -04:00
2026-10-07 19:07:02 -04:00
2026-09-07 02:58:32 -04:00
2026-10-07 21:03:49 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-07 04:51:48 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-20 15:50:43 -04:00
2026-09-20 15:50:43 -04:00
2026-09-04 20:54:40 -04:00
2026-09-03 15:57:00 -04:00
2026-10-01 15:36:17 -04:00
2026-10-07 00:10:52 -04:00
2026-10-07 00:10:52 -04:00
2026-10-06 00:53:51 -04:00
2026-10-07 00:10:52 -04:00
2026-10-01 03:11:32 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 15:57:00 -04:00
2026-09-10 17:59:46 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-10-05 00:26:33 -04:00
2026-10-05 00:26:33 -04:00
2026-10-01 08:12:49 -04:00
2026-10-01 08:12:49 -04:00
2026-10-01 05:58:43 -04:00
2026-10-01 05:58:43 -04:00
2026-09-03 06:55:14 -04:00
2026-09-03 06:55:14 -04:00
2026-09-10 17:59:46 -04:00