feat: day-partition the service log index and expire whole chunks (#10893)

* feat: day-partition the service log index and expire whole chunks

The service log index becomes one tantivy index per UTC day. The substance is
in windmill-ee-private#753; this side carries the EE ref and moves the log
indexer writer instead of cloning it, because sealing a chunk takes sole
ownership of its tantivy writer.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EeUmYWCJeaaHutiHLfZBQX

* fix: do not adopt the superseded watermark after an explicit index clear

A clear asks for the retention window to be read again, and a watermark says
it already has been — and the v3 copy in object storage is kept for rollback,
so it outlives the local one the clear removes. Both copies of that watermark
are now read and the newer wins, for the same reason the v4 one is taken from
the store when it is ahead: a replica that lost the lock keeps a local file
frozen where it stopped while the store went on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EeUmYWCJeaaHutiHLfZBQX

* fix: delete a day's raw files at its checkpoint, and rebuild whole days

Bumps the EE ref for windmill-ee-private#753.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EeUmYWCJeaaHutiHLfZBQX

* fix: make an interrupted rebuild detectable, and pin the rebuild floor

Bumps the EE ref for windmill-ee-private#753.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EeUmYWCJeaaHutiHLfZBQX

* fix: keep the rebuild marker in the object store, not on local disk

Bumps the EE ref for windmill-ee-private#753.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EeUmYWCJeaaHutiHLfZBQX

* fix: two more routes to a partial index being accepted as complete

Bumps the EE ref for windmill-ee-private#753.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EeUmYWCJeaaHutiHLfZBQX

* fix: trust a local chunk only when the tracker vouches for it

Bumps the EE ref for windmill-ee-private#753.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EeUmYWCJeaaHutiHLfZBQX

* chore: condense the stale-chunk guard's doc to the four-line limit

Bumps the EE ref for windmill-ee-private#753.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EeUmYWCJeaaHutiHLfZBQX

* chore: update ee-repo-ref to 17ef439b087b400889ff19109be9d2c810142278

This commit updates the EE repository reference after PR #753 was merged in windmill-ee-private.

Previous ee-repo-ref: 3e79901b4742906d2285dd943e24fac0f735f199

New ee-repo-ref: 17ef439b087b400889ff19109be9d2c810142278

Automated by sync-ee-ref workflow.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
This commit is contained in:
Ruben Fiszel
2026-08-30 07:16:09 +02:00
committed by GitHub
co-authored by Claude Opus 5 windmill-internal-app[bot]
parent 7639d83a42
commit d91ee4614a
2 changed files with 5 additions and 3 deletions
+1 -1
View File
@@ -1 +1 @@
f3c923975012e5e499fb07d8b390b61808bb8374
17ef439b087b400889ff19109be9d2c810142278
+4 -2
View File
@@ -1264,10 +1264,12 @@ Windmill Community Edition {GIT_VERSION}
#[cfg(all(feature = "tantivy", feature = "parquet"))]
let log_indexer_f = {
let log_indexer_rx = killpill_rx.resubscribe();
let log_index_writer2 = log_index_writer.clone();
// Moved, not cloned: sealing a chunk takes sole ownership of its
// tantivy writer, which a second live handle would silently prevent.
let moved_log_index_writer = log_index_writer;
async {
if let Some(db) = conn.as_sql() {
if let Some(log_index_writer) = log_index_writer2 {
if let Some(log_index_writer) = moved_log_index_writer {
windmill_indexer::service_logs_oss::run_indexer(
db.clone(),
log_index_writer,