fix: trigger flow error handler on unrecoverable (OOM/zombie) step failures (#9662)

* fix: trigger flow error handler on unrecoverable (OOM/zombie) step failures

When a worker is OOM-killed mid-step, the zombie job handler fails the step
via handle_job_error with unrecoverable=true. update_flow_status_after_job_completion
had `false if unrecoverable => false`, which silently completed the flow with the
error and skipped the flow's failure module (error handler). It would also have
pinned the failure module to the dead worker via same_worker.

Unrecoverable failures now route to the failure module instead of being retried or
silently dropped, and the error-handler step is pushed as a regular queued job that
any live worker can pick up.

Fixes WIN-2070

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: skip retry on unrecoverable flow failures, add retry-skip regression test

Address review: the failure-module-on-unrecoverable change must also bypass the per-step retry policy in push_next_flow_job, otherwise an OOM/zombie-killed step with a retry config would be retried instead of routing to the error handler. Gate the retry evaluation on !unrecoverable and add a regression test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: add sqlx offline cache for new flow-step zombie test query

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: route unrecoverable continue_on_error step failures to the error handler

Addresses Codex/Pi review (P1): with continue_on_error on the failed step, the
step counter was advanced before the unrecoverable decision branch, so
push_next_flow_job pushed the next normal step instead of the failure module —
hiding the worker death and letting the flow complete successfully.

- Do not advance the step counter (inc) for an unrecoverable continue_on_error failure.
- Let the Failure arm in push_next_flow_job route to the failure step even on a
  continue_on_error module when unrecoverable.
- Add a regression test (a[continue_on_error] -> b + failure_module): asserts the
  failure module runs and step b does not.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Ruben Fiszel
2026-06-19 09:57:05 +00:00
committed by GitHub
co-authored by Claude Opus 4.8
parent a425431e90
commit 7e4df02bd6
4 changed files with 324 additions and 7 deletions
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "SELECT q.id FROM v2_job_queue q JOIN v2_job j USING (id)\n WHERE j.parent_job = $1 AND q.running = true",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": "Uuid"
}
],
"parameters": {
"Left": [
"Uuid"
]
},
"nullable": [
false
]
},
"hash": "fac563138c316998d4f523edd633db97dba77d649b637c2529e4f232dce16c88"
}