Ruben FiszelandClaude Opus 4.8 e90b2be8fa perf(monitor): skip protected prefix in retention delete via cross-batch watermark (WIN-2088) (#9744) ...

The expired-job retention loop re-scanned the same oldest rows on every batch. When
the oldest completed jobs are undeletable (children of a still-active root flow), the
ORDER BY completed_at ASC scan walked that protected prefix on each of the up-to-20
batches, doing a v2_job PK lookup per row — quadratic in prefix size (measured ~9s/batch,
~180s/cleanup-cycle on a 1.5M-row prefix).

Carry a completed_at watermark (max deleted) across batches and re-apply it as
completed_at >= floor so each batch resumes past the already-processed prefix. Also skip
the v2_job join entirely when no old root flow is active (the common case), since nothing
is protected then. Measured: subsequent batches 9000ms -> 159ms; empty-set path 154 -> 36ms.

The watermark only ever skips rows the current run already deleted, was protecting, or
skip-locked — all deferred to the next run, identical to the unbounded scan's row set
(verified: union of batched deletes == single delete, 0 diff). Mirrored in
windmill-api-settings log_cleanup.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-23 20:49:55 +00:00

..

src

perf(monitor): skip protected prefix in retention delete via cross-batch watermark (WIN-2088) (#9744)

2026-06-23 20:49:55 +00:00

Cargo.toml

refactor: create windmill-ai crate (part 1 — types, traits, base modules) (#8530)

2026-04-22 11:20:27 +00:00