mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-09-07 00:01:49 +00:00
feat: day-partition the service log index and expire whole chunks (#10893)
* feat: day-partition the service log index and expire whole chunks The service log index becomes one tantivy index per UTC day. The substance is in windmill-ee-private#753; this side carries the EE ref and moves the log indexer writer instead of cloning it, because sealing a chunk takes sole ownership of its tantivy writer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EeUmYWCJeaaHutiHLfZBQX * fix: do not adopt the superseded watermark after an explicit index clear A clear asks for the retention window to be read again, and a watermark says it already has been — and the v3 copy in object storage is kept for rollback, so it outlives the local one the clear removes. Both copies of that watermark are now read and the newer wins, for the same reason the v4 one is taken from the store when it is ahead: a replica that lost the lock keeps a local file frozen where it stopped while the store went on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EeUmYWCJeaaHutiHLfZBQX * fix: delete a day's raw files at its checkpoint, and rebuild whole days Bumps the EE ref for windmill-ee-private#753. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EeUmYWCJeaaHutiHLfZBQX * fix: make an interrupted rebuild detectable, and pin the rebuild floor Bumps the EE ref for windmill-ee-private#753. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EeUmYWCJeaaHutiHLfZBQX * fix: keep the rebuild marker in the object store, not on local disk Bumps the EE ref for windmill-ee-private#753. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EeUmYWCJeaaHutiHLfZBQX * fix: two more routes to a partial index being accepted as complete Bumps the EE ref for windmill-ee-private#753. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EeUmYWCJeaaHutiHLfZBQX * fix: trust a local chunk only when the tracker vouches for it Bumps the EE ref for windmill-ee-private#753. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EeUmYWCJeaaHutiHLfZBQX * chore: condense the stale-chunk guard's doc to the four-line limit Bumps the EE ref for windmill-ee-private#753. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EeUmYWCJeaaHutiHLfZBQX * chore: update ee-repo-ref to 17ef439b087b400889ff19109be9d2c810142278 This commit updates the EE repository reference after PR #753 was merged in windmill-ee-private. Previous ee-repo-ref: 3e79901b4742906d2285dd943e24fac0f735f199 New ee-repo-ref: 17ef439b087b400889ff19109be9d2c810142278 Automated by sync-ee-ref workflow. --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
This commit is contained in:
co-authored by
Claude Opus 5
windmill-internal-app[bot]
parent
7639d83a42
commit
d91ee4614a
@@ -1 +1 @@
|
||||
f3c923975012e5e499fb07d8b390b61808bb8374
|
||||
17ef439b087b400889ff19109be9d2c810142278
|
||||
|
||||
+4
-2
@@ -1264,10 +1264,12 @@ Windmill Community Edition {GIT_VERSION}
|
||||
#[cfg(all(feature = "tantivy", feature = "parquet"))]
|
||||
let log_indexer_f = {
|
||||
let log_indexer_rx = killpill_rx.resubscribe();
|
||||
let log_index_writer2 = log_index_writer.clone();
|
||||
// Moved, not cloned: sealing a chunk takes sole ownership of its
|
||||
// tantivy writer, which a second live handle would silently prevent.
|
||||
let moved_log_index_writer = log_index_writer;
|
||||
async {
|
||||
if let Some(db) = conn.as_sql() {
|
||||
if let Some(log_index_writer) = log_index_writer2 {
|
||||
if let Some(log_index_writer) = moved_log_index_writer {
|
||||
windmill_indexer::service_logs_oss::run_indexer(
|
||||
db.clone(),
|
||||
log_index_writer,
|
||||
|
||||
Reference in New Issue
Block a user