From ca7e1c290b1718da703e7ac9d4e1c8dc1584d4f2 Mon Sep 17 00:00:00 2001 From: Jinwoo-H Date: Fri, 4 Sep 2026 07:43:00 -0400 Subject: [PATCH] docs(cloud): c27 crash loop; per-hour die counts --- cloud/docs/relay-reconnect-2026-09-findings.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/cloud/docs/relay-reconnect-2026-09-findings.md b/cloud/docs/relay-reconnect-2026-09-findings.md index ac500414318..a089c7454e7 100644 --- a/cloud/docs/relay-reconnect-2026-09-findings.md +++ b/cloud/docs/relay-reconnect-2026-09-findings.md @@ -390,7 +390,9 @@ SQL CPU 0.84 -> 0.78 in the preceding minutes, director concurrency 18–22 (idl 09:39 (c21, c24), 09:55 (c16, c8), 09:59 (c20), 10:05 (c8, c20), 10:19 (c16 stalled, no crash), then a 58-min lull, 11:04 (c9; c28 died 13x in 4 min, autoheal recreate 11:09Z, its 3rd recreate today), 11:17 (c10, c28 again, c22, c23, c14 x9 looping; 23 dies in ~90 s; fleet 13.3k -> 10.8k), 11:31 (c14, c23, c25, c15, c24, c19), 11:34 (c20, c26, c29 x4, c14, c27 x3, c25; fleet 13.1k -> 10.3k). -Three cascades in 17 min. Every cell that has died today is on 5aedbca5; c7 (85bf6799, 5 h) has not. +Three cascades in 17 min. 11:38–11:42 c27 crash-looped 11x (autoheal recreate pending), c29 recreating. +Every cell that has died today is on 5aedbca5; c7 (85bf6799, 5 h) has not. Cell dies per hour today: +01Z 5, 02Z 7, 03Z 4, 04Z 7, 05Z 4, 06Z 9, 07Z 11, 08Z 30, 09Z 26, 10Z 2, 11Z 68+ (to 11:42). Director concurrency pinned at 85 for 09:32–09:33; 503s 4,141 and 4,396 per minute. 09:39: c21, c24 (2,870 503s). Crashes per instance 08:10–09:40Z: c28 x14, c27 x5, c23 x5, c22 x4, c14 x4, c13/c20 x3, then c26/c9/c24/c16/c8 x2. Mean gap between cascades since 08:40: ~12 min. Every 15-min gate attempt