* feat(backend): native script retry without one-step-flow wrapping
Schedules and data pipelines that retry a single script previously wrapped
it in a one-step flow (JobKind::SingleStepFlow), creating extra job rows, a
v2_job_status row, and UI projection complexity. This adds native retry on a
plain JobKind::Script job.
- RetrySettings: flatten Retry into a deduped retry_settings table, carried
via the existing runnable_settings_handle (lazy, off the hot path).
- push() materializes a bare-script-with-retry SingleStepFlow into a native
Script job (gated on min-version + no handlers/retry_if).
- add_completed_job re-pushes the next attempt on failure with backoff,
tracking the attempt counter in v2_job_queue.extras and the chain via
parent_job; schedule completion handlers fire only on the terminal attempt.
- frontend: ScriptRetryChain shows the attempt chain on the run page.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(backend): native retry_if eval + per-occurrence schedule handlers
Extends native script retry to the two cases that previously stayed on the
one-step-flow path:
- retry_if: evaluated natively on the failure path via a feature-gated
windmill-jseval dep (quickjs) over the failure result + flow_input; push
materializes such policies natively only when quickjs is available.
- on_failure_times / on_recovery: apply_schedule_handlers now resolves each
past scheduled occurrence's terminal status across its native-retry chain
(root OR any parent_job=root child succeeded) and excludes the current
occurrence, so the counting is per-occurrence rather than per-attempt.
All scheduled-script retries now go native (schedule.rs gate removed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(backend): always materialize retry_if natively; unsupported without quickjs
retry_if is evaluated by the worker (which always has quickjs), not the
pusher, so gating materialization on the pusher's feature was wrong. The
flow path was never a real fallback either — the flow runtime needs quickjs
to evaluate retry_if too. retry_if now always goes native; on a worker
without quickjs it is unsupported and fails closed (no retry).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(backend): un-park asset-cascade (pipeline) retry
Native retry resolves the blocker that parked pipeline retry: a retried
subscriber is now a Script job (not a one-step flow / flow step), so it
stays eligible for asset dispatch and can trigger its own downstream on
recovery.
- scripts.rs: persist // retry <count> [<delay>] to script_trigger on asset
edges (was dropped with a TODO warning).
- asset_dispatch.rs: is_eligible_kind keys off flow_step_id, not parent_job,
so native-retry attempts dispatch on success while flow steps stay excluded.
- tests: retry-bearing subscriber now dispatches as a native Script carrying
the policy in runnable_settings_handle; native-retry attempt is eligible.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(backend): cap native retry interval, lazy result serialization, idempotent retry push
Hardening from a self-review of the native retry path:
- Cap the backoff at MAX_RETRY_INTERVAL to match the flow-runtime path
(evaluate_retry); the exponential formula could otherwise schedule up to
~18h vs the flow path's 6h.
- Serialize the failure result lazily (only when a retry_if policy needs it),
so the common failure no longer pays the serialization on the failure path.
- Push each retry with a deterministic id per (root, attempt). If a worker
dies between enqueueing the retry and finalizing the current attempt, the
reaper re-handles the attempt and lands here again — push rejects the
duplicate id, so the retry is enqueued exactly once (no double-retry).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(backend): defer schedule handlers idempotently on retry-push replay (review P1)
Address local-review findings:
- P1: retry_pending was derived from the retry push *result*, so on a worker
crash + reaper replay the duplicate-id push returned Err → retry_pending
flipped to false → apply_schedule_handlers fired for the non-terminal
attempt (and the terminal attempt later fired them again). Pre-check whether
the deterministic retry id already exists and report it as pending without
re-pushing, so the handler-deferral invariant is crash-idempotent too.
- P2: refresh the stale 'wrap the script in a one-step flow' comment in the
asset-cascade retry push — it now materializes a native Script.
- Add RetrySettings <-> Retry round-trip unit tests (clamping edges).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(backend): native retry chain + per-occurrence status sqlx tests
Close the two integration-test gaps flagged in local review:
- chains_attempts_and_is_idempotent: drives maybe_enqueue_native_script_retry
through attempt0 -> retry1 -> retry2 -> exhausted (counter, backoff, max-attempts)
and asserts crash-replay idempotency (the P1 fix: a replayed completion reports
pending without double-enqueueing).
- per_occurrence_status_counts_recovered_as_success: pins the exact per-occurrence
terminal-status query from jobs_ee::apply_schedule_handlers — a retried-but-
recovered occurrence counts as success, retries (parent_job set) are excluded
from occurrence counting, and the current occurrence is excluded.
- canceled_job_does_not_retry: cancellation wins over a pending retry.
Runtime sqlx API (no .sqlx cache entry needed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): exclude schedule handlers from the retry-attempt chain
The retry chain listed all script children of the root by parent_job, but
schedule completion handlers (on_failure/on_recovery/on_success) are also
script children — when the occurrence has no retries, the handler's parent is
the root itself, so a successful, never-retried job rendered a bogus
'Retries (1)' badge pointing at the handler. Filter children to re-runs of the
same script (matching script_hash); real retries keep the root's hash, handlers
run a different script.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(frontend): surface schedule handlers on the run page
Extend the run-page chain component with schedule completion handlers:
- A 'Handlers' row on a scheduled job links to the on_failure/on_recovery/
on_success runs that fired for that occurrence (found as children of the
terminal attempt, identified by their synthetic created_by).
- A handler's own run page now shows a 'Failure/Recovery/Success handler'
label with a link back to the run it handled and its schedule. on_recovery
and on_success share created_by, disambiguated by the recovery-only
error_started_at arg.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(backend): restore folder_default_permissioned_as sqlx caches dropped by prepare
An earlier `cargo sqlx prepare` on this branch ran before #8801's
folder_default_permissioned_as test merged in, so it pruned the 3 query caches
that test needs; cargo_test then failed under SQLX_OFFLINE. Restore them from main.
* fix(backend): only cascade assets from native retry attempts, not handlers (review P1)
is_eligible_kind keyed dispatch on flow_step_id alone, so every parented Script
child became asset-eligible — including schedule/error/recovery handlers (Script
jobs with parent_job set and no flow_step_id). A handler that declares assets
would then trigger a cascade the old parent_job IS NULL guard prevented. Gate
parented jobs on being a genuine retry attempt: a re-run of the SAME runnable as
its chain parent (handlers run a different script). Runtime query, no sqlx cache.
* fix(backend): cache the private-gated retry_setting asset-dispatch test query
The same prepare-without-private that dropped the folder_default caches also
pruned the cache for the retry_setting_dispatches_subscriber_as_native_script
test query (asset_trigger_dispatch.rs:721). Regenerated with --features private.
* fix(backend): exclude handler children from per-occurrence recovery (review)
A scheduled occurrence's on_failure/on_success handler runs as a successful
child (parent_job = occurrence), and the per-occurrence success EXISTS counted
ANY successful child — so a failed occurrence whose error handler succeeded was
marked 'recovered', breaking on_recovery (test_script/flow_schedule_handlers in
the merge) and on_failure_times counting. EE query now scopes the EXISTS to
same-runnable children (only native retry attempts); regenerate sqlx cache + bump
ee-repo-ref. native_retry_test gains a handler-child regression case.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(backend): scheduled-script retry is a native Script, not SingleStepFlow
test_push_script_with_retry / test_try_schedule_with_retry (from main) asserted
the old SingleStepFlow wrapping for scheduled-script retry; this PR makes it a
native Script. Update both to assert kind='script' and that the retry policy is
carried via runnable_settings_handle.
* fix(backend): preserve dedicated_worker on native retry + saturate count casts (cubic)
Address cubic CI review:
- P1: the SingleStepFlow->native Script materialization dropped dedicated_worker,
so a dedicated-worker scheduled script lost its dedicated pool on retry. Resolve
it from the script row in push so the materialized Script keeps the dedicated tag.
- P2: saturate the u32->i32 retry-attempt narrowings (RetrySettings::from) and the
u32->i16 // retry count narrowing (scripts.rs) instead of wrapping.
* fix(backend): use a retry-specific signal, not runnable equality (codex review)
Address Codex CI review:
- P1: is_native_retry_attempt treated any same-runnable parented Script child as
a retry. WAC v2 inline children have that exact shape, so an inline child of an
asset producer would cascade. Use a retry-specific signal instead: the job
carries a retry_settings policy (always re-inserted by maybe_enqueue) and has no
flow_innermost_root_job. Apply the same flow_innermost guard to the EE
per-occurrence EXISTS (WAC inline children must not count as a recovery).
- P1: the deterministic retry-id pre-check raced with push; a concurrent duplicate
now resolves as 'retry pending' (re-check on the duplicate-id error) instead of
flipping retry_pending to false and firing handlers early.
- Tests: native_retry + asset_trigger_dispatch gain WAC-inline-child cases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(backend): explicit native_retry_attempt marker, drop heuristics
Replace the per-site "is this a retry?" inference (parent_job + runnable match +
flow_innermost / retry_settings) with one explicit marker: a sparse
native_retry_attempt(job_id, attempt) table, written in maybe_enqueue. The marker
also carries the attempt counter (previously in v2_job_queue.extras), so it's the
single source of truth.
- asset_dispatch: is_native_retry_attempt is now one indexed EXISTS on the marker.
- EE per-occurrence query: joins the marker instead of guessing by runnable/flow_innermost.
- maybe_enqueue: reads/writes the marker (persistent) instead of queue extras.
- Lifecycle: swept with the job in retention (log_cleanup), no FK to keep bulk delete cheap.
- Eliminates handler / WAC-inline-child misclassification by construction.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(backend): sweep native_retry_attempt markers in the periodic retention path too (codex)
The marker has no FK and relies on retention cleanup; log_cleanup.rs swept it but
the periodic monitor.rs path deleted v2_job rows without it, orphaning markers.
Add the same WHERE job_id = ANY(...) sweep there.
* fix(backend): widen native_retry_attempt.attempt to integer (cubic)
The smallint column was cast to/from u32 and could wrap a retry chain longer than
i16::MAX into premature exhaustion. Use integer, matching the retry policy's i32
attempt count, so no narrowing occurs on the maybe_enqueue read/write path.
* feat(frontend): mark retries via is_retry on listJobs; drop SAVEPOINT
- Expose an is_retry flag on jobs (UnifiedJob/CompletedJob/QueuedJob + openapi),
computed from the native_retry_attempt marker. The run-page chain now filters
retry attempts by is_retry instead of the script_hash heuristic, so WAC v2
inline children (same script, parent_job) no longer render as retries (codex).
- Revert the marker-cleanup SAVEPOINT (an unused pattern in this codebase): keep
the plain catch-and-continue matching the other side-table deletes; the table is
created by a startup migration so it always exists when cleanup runs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(backend): mark is_retry sqlx(default) so non-list job queries can omit it
The single-job GET query maps directly to CompletedJob/QueuedJob via FromRow but
does not select is_retry, which errored with "no column found". Only the list
endpoint populates the marker; #[sqlx(default)] lets every other query omit the
column and default to None.
* feat(backend): select is_retry in single-job GET too for consistency
The list endpoint already exposes the marker; populate it on the single-job GET
(both completed and queued variants) as well so a run loaded directly reflects
its retry status. #[sqlx(default)] stays as a safety net for any other query.
* feat(backend): reap orphaned native_retry_attempt markers via periodic sweep
The marker has no FK to v2_job (to keep the hot bulk retention delete cheap), so
direct job deletions (workspace/job delete, schedule clearing) would leave marker
rows orphaned. Rather than add explicit cleanup to every v2_job delete site (which
must then be remembered for every future path), reap orphans in the periodic
delete_expired_items pass: DELETE FROM native_retry_attempt WHERE NOT EXISTS (the
job). The table is sparse so the anti-join drives off it and probes v2_job by PK —
cheap. Retention still sweeps markers inline (keeps the table small so this stays
cheap); a transient orphan is harmless (nothing reads is_retry for a gone job).
* fix(frontend): include flow handlers in retry chain handler row (codex)
Schedule on_failure/on_recovery/on_success handlers can be flow paths (flow/...),
whose handler job is a flow, not a script. The chain fetched children with
jobKinds:'script', hiding flow handlers. Drop the kind filter — retry attempts
are still selected by is_retry and handlers by created_by, so both kinds surface.
* fix(backend): carry concurrency/debouncing settings into native retries
maybe_enqueue re-pushed the next attempt with ConcurrencySettings/DebouncingSettings
::default(), dropping the script/pipeline concurrency settings the failed job carried
in its runnable_settings_handle. A retry of a concurrency-limited script then inserted
no concurrency_key and ran unbounded. Resolve both from the same handle (cached) and
pass them in the payload, which push forwards to the materialized retry. Adds a
regression test asserting the retry's handle resolves to the concurrency settings.
* fix(backend): carry concurrency/debounce into scheduled-retry root + document retry-helper auth (codex)
P1a (schedule.rs): the scheduled-retry materialization fetched the script's
concurrency/debounce settings but passed ConcurrencySettings/DebouncingSettings
::default() into the SingleStepFlow payload, so the root attempt's handle held only
the retry policy and the whole chain ran unbounded. Pass the fetched settings.
Regression test asserts the root handle resolves to retry + concurrency.
P1b (jobs.rs): document maybe_enqueue_native_script_retry's authorization contract
— it is pub only for the integration test; the sole production caller is the worker
completion path passing a DB-derived, already-authorized MiniCompletedJob.
* docs(backend): attach native-retry auth contract to the function itself (codex)
The doc block was merged with eval_retry_if's doc and bound to that function,
leaving maybe_enqueue_native_script_retry undocumented. Split them: eval_retry_if
keeps its own doc; the native-retry + authorization contract now sits directly
above maybe_enqueue_native_script_retry.
* docs(backend): regenerate served openapi-deref with is_retry + fix stale comments (codex)
- Regenerate openapi-deref.{yaml,json} (served from lib.rs): they were stale since
1.734.0 and lacked is_retry on QueuedJob/CompletedJob, so clients reading the
served spec couldn't see the field. Now current at 1.739.0.
- schedule.rs: a retry_if gate is evaluated at failure time and fails closed without
quickjs (no retry); it does not fall back to a flow path.
- windmill-types jobs.rs: is_retry is selected by both the list and single-job GET
endpoints (not list-only).
* docs(backend): fix remaining stale retry_if/quickjs comments (codex)
The retry_if block and the push materialization comments claimed push keeps
retry_if on a flow path / the worker always has quickjs. The code always
materializes native retry and the no-quickjs eval_retry_if path fails closed —
correct the comments to that constraint.
* docs(backend): fix stale quickjs-fallback + schedule-handler-restriction comments (codex)
- Cargo.toml quickjs feature: without quickjs a retry_if gate cannot be evaluated
and the job does not retry (no one-step-flow fallback).
- jobs.rs handler-defer comment: apply_schedule_handlers resolves per-occurrence
failure/recovery status across the retry chain, so the old 'restricted to
schedules whose handlers don't need per-occurrence counting' claim is dropped.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: reuse outer tx for schedule push in commit_completed_job
Instead of calling handle_maybe_scheduled_job(db) which opens its own
connections (peak=3), inline the schedule push using a savepoint on the
outer transaction. Auth is fetched via the tx connection using
fetch_authed_from_permissioned_as_conn, and push_scheduled_job runs
on a savepoint so failures roll back only the push, not the completion.
On push failure: savepoint rolls back, schedule is disabled on the outer
tx, and the zombie return path is preserved if disabling also fails.
Peak connections drop from 3 to 1 (or 2 on cold RunnableSettings cache).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* all
* fix: extract shared try_schedule_next_job to unify schedule push paths
Replace the two diverging schedule-push implementations (inlined in
commit_completed_job and standalone handle_maybe_scheduled_job) with a
single try_schedule_next_job that reuses the caller's transaction via
savepoints. This eliminates extra pool connection usage in the
worker_flow.rs path and ensures consistent retry/error semantics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test: add failpoint markers to try_schedule_next_job
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: remove plan.md
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: remove inner retry loop from try_schedule_next_job, add caller-level retries
The 10-retry x 5s-sleep loop inside try_schedule_next_job held locks on
v2_job_completed/v2_job_queue for up to ~45s when running inside the
outer commit_completed_job transaction.
Now try_schedule_next_job makes a single attempt and returns errors to
the caller. Non-retryable errors (QuotaExceeded, NotFound) disable the
schedule immediately inside the function. Transient errors are returned
for the caller to retry:
- commit_completed_job path: outer backon retry (10x3s) retries the
entire transaction including the schedule push, so no locks are held
during sleep.
- handle_flow path: new backon retry (10x3s) wraps begin/push/commit
with a fresh transaction per attempt.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: clear push_err after successful schedule disable to prevent stuck schedules
When try_schedule_next_job disables the schedule for non-retryable errors
(NotFound, QuotaExceeded), clear the error so the caller commits the tx
(persisting the disable). Previously, the error propagated up, causing the
tx to be dropped and rolling back the disable — leaving the schedule
permanently enabled but broken.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: add 5s timeout on push_scheduled_job, clean up handle_flow error handling
- Add tokio::time::timeout(5s) around push_scheduled_job inside
try_schedule_next_job to bound worst-case lock holding per attempt
- Remove unreachable QuotaExceeded/NotFound match arms in handle_flow
(these errors are handled internally by try_schedule_next_job)
- Add report_error_to_workspace_handler_or_critical_side_channel in
handle_flow when post-exhaustion schedule disable fails
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: return SchedulePushZombieError when both schedule push and disable fail
When handle_flow cannot push the next scheduled job AND cannot disable the
schedule, return a SchedulePushZombieError so the worker leaves the flow job
in the queue for zombie detection to restart. This prevents stuck schedules
where neither the next tick was pushed nor the schedule was disabled.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* app compiles with every ee substituted
* Replace all oss files content
* Revert "Replace all oss files content"
This reverts commit ea4017d59f.
* delete all ee
* hide all _ee files under private flag
* hide every oss stuff when private flag set
* pub use *
* gitignore and substitute script
* pub mod for ee needed for ee repo
* small mistakes
* remove oidc_oss impl
* ee ref (temp)
* ee ref
* fix --all-features selecting private in OSS CI
* ee repo ref
* allow unused
* basic cron schedule editing ui
* schedules run in a user-specified timezone
* fix other uses of CronInput component
* use now() from database to schedule next job
* offset -> IANA timezone conversion on db migration
* sqlx ci
---------
Co-authored-by: Ruben Fiszel <ruben@rubenfiszel.com>
Ruben Fiszel
Pyra
dieriba
Diego Imbert
Ruben Fiszel
Lucas Abel
HugoCasa
Guillaume Bouvignies
Kai Jellinghaus
Oliver Veal