mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-04 16:09:05 +00:00
Shrinath Rao
801f96fc8e
## Problem
The Stuck-Run-Reaper conflated **two different situations** and
described both as "waiting for an EKS runner / freed a runner":
1. **runner-wait** — a job queued with no EKS runner assigned (the real
target).
2. **zombie run** — a run left in a non-completed state with **0 jobs
ever dispatched**. It never held a runner.
A real example: a **2026-05-26** run stuck ~**1842h** with 0 jobs was
cancelled and announced as *"freed a runner stuck for 1842 hours"* —
which is nonsense; nothing was ever running.
## Fix — label the two distinctly
| Case | Threshold | Wording |
|---|---|---|
| runner-wait (job queued, no runner) | **60 min** (unchanged) |
"waiting for an EKS runner that was never assigned" |
| zombie run (0 jobs dispatched) | **24h** (new
`DEFAULT_ZOMBIE_RUN_THRESHOLD_HOURS` / `zombie_run_threshold_hours`
input) | "🧟 abandoned in a non-completed state with no jobs ever
dispatched — **held no runner**, stale-run cleanup" |
- **Cancellation behaviour is unchanged** — zombies are still swept;
they're just described accurately.
- Separate longer threshold so a brief GitHub scheduling blip isn't
labelled a zombie.
- Wait time for zombies uses `run.created_at`; `humanize()` now renders
days (`76d 18h`, not `1842h 0m`).
- Job-summary table gains a **Reason** column; PR comment wording
branches per kind.
## Validation
- YAML parses; embedded `github-script` body passes `node --check`
(async-wrapped as github-script runs it).
- OSS/ENT files differ only in the `WORKFLOWS` list + label comment
(verified by diff).
- Companion PR on the same branch `test/reaper-zombie-run-labeling` in
the sister repo.