starred/orca
1
0
Fork 0
mirror of https://github.com/stablyai/orca.git synced 2026-09-29 08:03:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(relay): record c27 recovery #2 miss (laptop sleep) and gate #43

Browse Source
This commit is contained in:
Jinwoo-H
2026-09-05 15:07:46 -04:00
parent dab47f4eb0
commit c78dfaedb0
1 changed files with 3 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
cloud/docs/relay-reconnect-2026-09-findings.md
+3
View File
@@ -965,3 +965,6 @@ Owner: "sure, feel free to drive these." Sequence chosen: Roll 1 first (highest
| Monitor dry-run #41 result | **Green** 09:58:51Z at gen 143, `9f2a9a248e`. |
| c27 recovery #1 (run 33959789773, `mode=rollback`) | **Failed closed, no mutation** 10:09:21Z at `Verify monitor evidence provenance`: `relay monitor dry-run authority is incomplete or stale`. The dry-run authority is valid for 5 min after `completedAt` at wave 0 (`EVIDENCE_MAX_AGE_MS`); the gate completed 09:58:51Z but the operator poller (20 s `gh run view` loop) only observed completion at 10:07:09Z during a local network outage, so the dispatch landed at 10:07:11Z, 8 m 20 s after completion. Failed before the rollout lease, isolate, or any Terraform step; c27 unchanged (migration-only, drained, `5aedbca5`, gen 143). Every prior canary dispatched ≤15 s after gate green, so this is a dispatch-latency miss, not a job defect; the freshness bound behaved as designed. |
| Monitor dry-run #42 | Dispatched 18:39Z at gen 143 on main `af82126058` (trusted paths byte-identical to `9f2a9a248e`), run 33984753269. Recovery script re-armed behind it (same `mode=rollback` onto `519f4914`, protocol 0). |
| Monitor dry-run #42 result | **Green** 18:55:47Z at gen 143, `af82126058`. |
| c27 recovery #2 (run 33985902062, `mode=rollback`) | **Failed closed, no mutation** 19:05:02Z, same `authority is incomplete or stale`. Dispatch landed 19:02:39Z, 6 m 52 s after the gate completed. Root cause of both misses is the operator laptop sleeping during the 15 min gate wait (`pmset -g log`: asleep 18:52:28Z → 19:02:17Z; the morning miss coincided with a sleep/dark-wake cycle too), so the 20 s poller never ran inside the 5 min window. Not a job or evidence defect: the freshness bound did its job. Operator fix: poller now runs under `caffeinate -i`. |
| Monitor dry-run #43 | Dispatched 19:06Z at gen 143 on main `af82126058`, run 33986121849. Recovery armed behind it under `caffeinate`. |