* fix: ignore NotFound errors when deleting log files from object store
Periodic and manual log cleanup delete log files from instance object
storage. S3's DeleteObjects silently ignores missing keys, but GCS
returns a 404 for each individual delete, which the object_store crate's
default delete_stream surfaces as Error::NotFound. This produced noisy
error/warning logs on every cleanup cycle even though the cleanup
succeeded (DB records are removed regardless).
Treat a NotFound delete as a successful no-op in both delete handlers:
- monitor.rs: skip logging NotFound errors
- log_cleanup.rs: count NotFound as deleted instead of an error
Fixes WIN-2081
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat: report 404 (already-absent) count in object store log cleanup
Track delete calls that returned 404 (object already absent) separately
from real deletes so operators can see how many of the attempted deletes
were no-ops, instead of those numbers silently folding into s3_deleted.
- monitor.rs: emit a final info summary per cleanup cycle:
"N deleted, M already absent (404), K failed" (only when work occurred)
- log_cleanup.rs: add s3_not_found to LogCleanupProgress (serde default for
backward-compatible deserialization of in-flight rows), thread it through
s3_bulk_delete and all call sites, and log a final summary on release
- openapi.yaml + generated client + ObjectStoreConfigSettings.svelte:
surface the 404 count in the manual cleanup status UI
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: import ObjectStoreError directly from object_store_reexports
The object_store_reexports module already re-exports object_store::Error
under the name ObjectStoreError, so `Error as ObjectStoreError` failed to
resolve (no `Error` in that module). This compiles only behind the
parquet feature, which the local dev `cargo watch` doesn't enable, so it
was caught by CI's full-feature check rather than locally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: log cleanup scans S3 orphans and works cross-server
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: don't skip service log orphan scan when job retention is disabled
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: time-based heartbeat + flag partial folder sizes on list errors
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: move background_task module from common to api-settings
Only log_cleanup and storage_usage use it today, both in windmill-api-settings.
Keeping it in the consumer crate narrows the blast radius; if workers or
indexer later need cross-server lease+progress coordination they can move it
back to common then.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* backend
* iterate
* all
* all
* all
* iterate
* revert
* all
* add tracing to get of authed client
* all
* all
* lal
* all
* update
* fix
* push
* all
* all
* revert
* frontend
* fix checks
* avoid deadlock
* safer
* fix
* fix
Ruben FiszelandClaude Opus 4.8
Guilhem
hugocasa
Henri Courdent
Ruben Fiszel