Ruben Fiszel 7e4df02bd6 fix: trigger flow error handler on unrecoverable (OOM/zombie) step failures (#9662) ...

* fix: trigger flow error handler on unrecoverable (OOM/zombie) step failures

When a worker is OOM-killed mid-step, the zombie job handler fails the step
via handle_job_error with unrecoverable=true. update_flow_status_after_job_completion
had `false if unrecoverable => false`, which silently completed the flow with the
error and skipped the flow's failure module (error handler). It would also have
pinned the failure module to the dead worker via same_worker.

Unrecoverable failures now route to the failure module instead of being retried or
silently dropped, and the error-handler step is pushed as a regular queued job that
any live worker can pick up.

Fixes WIN-2070

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: skip retry on unrecoverable flow failures, add retry-skip regression test

Address review: the failure-module-on-unrecoverable change must also bypass the per-step retry policy in push_next_flow_job, otherwise an OOM/zombie-killed step with a retry config would be retried instead of routing to the error handler. Gate the retry evaluation on !unrecoverable and add a regression test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: add sqlx offline cache for new flow-step zombie test query

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix: route unrecoverable continue_on_error step failures to the error handler

Addresses Codex/Pi review (P1): with continue_on_error on the failed step, the
step counter was advanced before the unrecoverable decision branch, so
push_next_flow_job pushed the next normal step instead of the failure module —
hiding the worker death and letting the flow complete successfully.

- Do not advance the step counter (inc) for an unrecoverable continue_on_error failure.
- Let the Failure arm in push_next_flow_job route to the failure step even on a
  continue_on_error module when unrecoverable.
- Add a regression test (a[continue_on_error] -> b + failure_module): asserts the
  failure module runs and step b does not.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-19 09:57:05 +00:00

..

fixtures

block operators from inline preview job execution (#9572)

2026-06-15 00:07:46 +02:00

scripts

feat: add sandbox annotations, volume mounts, for AI sandbox starting with claude (#8058)

2026-03-05 06:19:51 +00:00

agent_workers.rs

feat: support assigning a worker tag to app inline scripts (#9002)

2026-05-01 17:21:17 +00:00

app_anonymous_execution_mode.rs

feat: workspace protection rule to restrict anonymous app deployment (#9509)

2026-06-10 13:47:19 +00:00

app_custom_path_cross_workspace.rs

fix: actionable error when a custom_path is taken by an app in another workspace (#9190)

2026-05-16 08:29:53 +00:00

app_preview_auth.rs

require pinned sha for inline raw_code in viewer app run mode (#9570)

2026-06-14 23:48:03 +02:00

asset_trigger_dispatch.rs

feat: Data Pipelines alpha (#9193)

2026-06-18 18:09:02 +02:00

batch_rerun.rs

fix: handle singlestepflow zombies and stop filtering them from runs page (#9055)

2026-05-07 08:00:38 +00:00

bun_jobs.rs

fix(bun): pass --preserve-symlinks on unbundled execution (#9147)

2026-05-13 13:04:31 +00:00

ci_tests.rs

feat: Db-backed user drafts (#9351)

2026-06-15 10:23:16 +02:00

dedicated_workers.rs

refactor: convert read-hot globals to AtomicBool/I64 and ArcSwap (#8815)

2026-04-14 00:04:10 +00:00

delete_after_secs.rs

feat: add scheduled job deletion with configurable retention period (#8753)

2026-04-08 04:15:28 +00:00

dependency_map.rs

feat: Db-backed user drafts (#9351)

2026-06-15 10:23:16 +02:00

end_user_email.rs

fix: pair PG arg type with actual Rust binding to keep query_typed_raw safe (#8999)

2026-05-01 17:08:59 +00:00

error_handler.rs

fix: pair PG arg type with actual Rust binding to keep query_typed_raw safe (#8999)

2026-05-01 17:08:59 +00:00

flow_engine_parity.rs

feat(flows): opt-in to include the stopping step's result in early-stop errors (#9446)

2026-06-04 18:57:05 +00:00

folder_default_permissioned_as.rs

feat: folder default_permissioned_as rules for ownership defaults on deploy (#8801)

2026-04-12 20:14:07 +00:00

fork_deployment_requests.rs

feat: add deploy restriction rule and fork review requests (#8804)

2026-04-13 17:10:37 +00:00

inline_preview_auth.rs

block operators from inline preview job execution (#9572)

2026-06-15 00:07:46 +02:00

instance_config.rs

fix: treat empty global setting strings as unset (#8793)

2026-04-10 14:23:37 +00:00

job_payload.rs

feat: support restart from steps inside BranchOne, ForLoop, Subflow (#8955)

2026-04-28 20:00:03 +00:00

jobs_read_auth.rs

fix(backend): authorize single-job read endpoints by job/flow visibility (#9416)

2026-06-02 22:10:16 +00:00

list_jobs.rs

feat: support assigning a worker tag to app inline scripts (#9002)

2026-05-01 17:21:17 +00:00

mcp_resource_authz.rs

fix(ai): enforce resource authz when loading MCP tools in agent worker (#9571)

2026-06-14 23:50:18 +02:00

mcp_token_exfil.rs

fix(mcp): resolve MCP resource token via caller RLS + SSRF-guard url (#9428)

2026-06-03 09:00:43 +00:00

nativets_dedicated.rs

feat(cli): better stale scripts detection #3 (#8480)

2026-03-23 18:20:19 +00:00

nativets_jobs.rs

feat: support assigning a worker tag to app inline scripts (#9002)

2026-05-01 17:21:17 +00:00

nativets_stress.rs

feat: support assigning a worker tag to app inline scripts (#9002)

2026-05-01 17:21:17 +00:00

operator_db_sync.rs

fix: add secretKeyRef support for jwt_secret and rsa_keys (#8698)

2026-04-03 18:17:27 +00:00

otel.rs

feat(otel): connect jobs to the inbound distributed trace (#9456)

2026-06-05 16:50:07 +00:00

preserve_on_behalf_of.rs

fix: replace email with permissioned_as for triggers/schedules (#8439)

2026-03-20 16:28:38 +00:00

preview_native_tag.rs

fix(backend): route //native TypeScript previews to native workers (WIN-2007) (#9407)

2026-06-02 10:23:39 +00:00

protection_rules.rs

feat: add deploy restriction rule and fork review requests (#8804)

2026-04-13 17:10:37 +00:00

python_jobs.rs

feat: Data Pipelines alpha (#9193)

2026-06-18 18:09:02 +02:00

relock_skip.rs

feat: Db-backed user drafts (#9351)

2026-06-15 10:23:16 +02:00

restart_at_step_api.rs

feat: support restart from steps inside BranchOne, ForLoop, Subflow (#8955)

2026-04-28 20:00:03 +00:00

retry.rs

fix(flows): skip_if evaluates wrong previous_result during retry (#9547)

2026-06-12 17:31:31 +02:00

script_auto_kind_failure.rs

feat: Db-backed user drafts (#9351)

2026-06-15 10:23:16 +02:00

script_auto_parent_archived.rs

fix: reset parent_hash in auto_parent when all versions at path are archived (#9172)

2026-05-16 07:43:35 +00:00

script_modules.rs

feat: Data Pipelines alpha (#9193)

2026-06-18 18:09:02 +02:00

success_handler.rs

fix: pair PG arg type with actual Rust binding to keep query_typed_raw safe (#8999)

2026-05-01 17:08:59 +00:00

suspend_resume.rs

fix: pair PG arg type with actual Rust binding to keep query_typed_raw safe (#8999)

2026-05-01 17:08:59 +00:00

trigger_listener_queries.rs

feat: Db-backed user drafts (#9351)

2026-06-15 10:23:16 +02:00

volume_tests.rs

feat: support assigning a worker tag to app inline scripts (#9002)

2026-05-01 17:21:17 +00:00

worker.rs

fix: trigger flow error handler on unrecoverable (OOM/zombie) step failures (#9662)

2026-06-19 09:57:05 +00:00

workspace_dependencies.rs

fix: pair PG arg type with actual Rust binding to keep query_typed_raw safe (#8999)

2026-05-01 17:08:59 +00:00

workspace_export.rs

feat: Db-backed user drafts (#9351)

2026-06-15 10:23:16 +02:00

workspace_fairness.rs

fix(queue): duration-weighted workspace fairness signal (#9329)

2026-05-26 11:47:45 +00:00

ws_specific.rs

feat: add workspace-specific flag for resources and variables (#8836)

2026-05-07 13:35:54 +00:00