Design at:
[https://github.com/openobserve/designs/blob/main/infrastructure/file-list/trace-time-index.md](https://github.com/openobserve/designs/blob/main/infrastructure/file-list/trace-time-index.md)
## Summary
- add an hourly Metadata time index per trace stream to resolve a trace
ID to its full time range
- add external time-range and trace-details APIs, including bounded
locate/expand behavior and standard search responses
- integrate index generation into trace ingestion and preserve query
visibility through the synchronous Metadata write path
- synchronize retention and deletion for associated Metadata streams
- reuse the shared trace stream permission check across trace APIs
## Why
Trace detail lookup currently depends on a caller-provided time range,
which can omit spans outside that window. The time index provides a
bounded, best-effort range lookup for `stream + trace_id` without
scanning the full trace stream.
Closes#13823
## Environment variables
| Variable | Default | Effect |
| --- | --- | --- |
| `ZO_TRACE_TIME_INDEX_ENABLED` | `true` | Enables per-trace-stream
Metadata indexes, index writes during trace ingestion, and indexed
time-range lookup. Only traces ingested after the feature is enabled are
indexed; there is no historical backfill. |
| `ZO_QUERY_TIMEOUT` | 600 seconds | Existing setting reused as the
cumulative Trace Timer locate/expand deadline and as the default timeout
for the trace-details search. |
When enabled, each traces stream can create a related Metadata stream
named `trace_time_index_{stream}`. Its retention settings follow the
source stream, and it is removed with the source stream.
## API changes
All timestamp parameters and returned time ranges use microseconds.
### Get a trace time range
```http
GET /api/{org_id}/{stream_name}/traces/{trace_id}/time_range
```
Query parameters:
- `hint_ts` (optional): timestamp hint used to start the index lookup
Returns the source stream, trace ID, and the resolved `start_time` /
`end_time`. The `range` field is `null` when the index cannot resolve
the trace.
### Get trace details
```http
GET /api/{org_id}/{stream_name}/traces/{trace_id}/details
```
Query parameters:
- `start_time` and `end_time` (optional, but must be supplied together):
caller-provided range
- `hint_ts` (optional): timestamp hint for the time-index lookup
- `timeout` (optional): trace-details search timeout in seconds
The endpoint unions the caller range with the indexed range, so it never
narrows a caller-provided range. If neither the caller range nor the
index can provide a range, it returns `400`. The response uses the
standard search response and includes the effective range in
`new_start_time` and `new_end_time`.
## Validation
- `cargo test -p openobserve-api-search --lib` — 145 passed
- `cargo test -p openobserve-core traces::time_index::tests --lib` — 7
passed
- `cargo test -p infra
db::trace_time_index::tests::test_coverage_start_cache -- --exact
--nocapture` — passed
- `cargo check -p openobserve-api-search`
- `cd web && npm run type-check:app`
- end-to-end ingestion and query validation with the data generator
## Validation notes
- targeted ESLint could not start because the local web dependencies are
missing `@eslint/css`
- enterprise-only compilation could not be completed from this OSS
checkout because the local enterprise overlay is unavailable
## Summary
- run Clippy against every workspace member and target in the unit-test
CI
- fix the existing test-target Clippy findings, including the
non-snake-case test name
- keep warnings as hard failures so `unit_tests_summary` can be used as
the required check
## Validation
- `cargo fmt --all -- --check`
- `cargo clippy --workspace --all-targets -- -D warnings`
- `cargo test -p search --lib
catalog_functions_documents_what_only_the_server_can_supply`
- `cargo test -p config --lib` — 2967 passed, 2 ignored
- `cargo test -p infra --lib` — 1243 passed, 35 ignored; the full
parallel run also exposed two pre-existing local-only failures: an
environment-sensitive default-index-field assertion and a shared mock
database initialization race. The ratelimit test passes in isolation.
Design at: #13538
## Summary
With `ZO_FEATURE_WAL_PACK_ENABLED` (#13497), metrics streams persist
into pack files under `{data_wal_dir}/pack/`, which the
`zo_ingest_parquet_files` collector (a scan of
`{data_wal_dir}/files/**/*.parquet`) never sees — so the pending-upload
backlog of packed streams was invisible.
This PR adds two **unlabeled** gauges (the pack path is metrics-only, so
per-org/per-stream-type labels add no information):
- `zo_ingest_pack_files` — number of wal pack files on disk (pack
registry size).
- `zo_ingest_pack_segments` — number of pack segments pending upload to
object store (segment index totals, via the existing
`get_segment_index_stats()`).
Both are cheap O(streams)/O(packs) reads of in-memory state — no disk
scans, no per-segment walking under locks — and are set unconditionally
every cycle of the existing ingester metrics job, so there is no
per-label bookkeeping and no stale-series problem by construction.
## Changes
- `src/config/src/metrics.rs`: define + register the two gauges.
- `src/ingester/src/pack.rs`: `collect_pack_metrics()` + unit test.
- `src/jobs/src/job/metrics.rs`: call it from
`update_parquet_metrics()`.
## Testing
- `cargo test -p ingester` — 60 passed.
- `cargo test -p config metrics::` — 20 passed.
- `cargo check` clean on config/ingester/jobs; `cargo fmt --check`
clean.
Design at: https://github.com/openobserve/openobserve/issues/13509Closes#13509
## Problem
On clusters with 100k+ metrics streams, every memtable rotation persists
**one parquet file per stream × hour**, producing thousands of tiny
files per second on the ingester. The legacy file mover cannot keep up:
it re-scans the whole `wal/files` tree every `file_push_interval`
(capped at `file_push_limit` = 10000 files per cycle ≈ a hard ceiling of
~1000 files/s), repeatedly stats not-yet-expired files, and pays a full
DataFusion plan + S3 PUT per stream×hour for a few KB of data.
Generation outpaces consumption and wal files pile up unboundedly.
## Design
Behind `ZO_FEATURE_WAL_PACK_ENABLED` (bool, default **off**), applying
to **metrics streams only** — the stream-cardinality problem is
metrics-specific; other stream types keep the legacy per-stream persist
path. With the flag on, the "split by stream" step moves from persist
time (disk) to upload time (memory):
- **Persist**: all streams of a memtable are appended into a few large
**pack** files (`wal/pack/{idx}/{memtable_id}.{seq}.pack`, rolled at
`ZO_MAX_FILE_SIZE_ON_DISK`, default 512MB). Each segment inside a pack
is a **complete, self-contained parquet file**; the pack footer (JSON +
gxhash trailer, versioned) indexes segments with stream, partition key,
offset/length and FileMeta. One rotation now writes O(1) sequential
files instead of O(streams) tiny ones.
- **Query**: an in-memory segment index (rebuilt from footers on
startup) resolves stream → byte ranges; segments are materialized into
record batches and reuse the memtable search table-building path. Dedup
against in-memory data uses the existing memtable_ids mechanism.
- **Mover**: pending streams are snapshotted from the segment index (no
readdir/stat storm, no `file_push_limit` cap). A stream is flushed when
its segments are big enough or the oldest exceeds
`max_file_retention_time`. A **single segment is uploaded as-is (zero
decode/re-encode)**; multiple segments are merged in memory via
DataFusion MemTable + the existing `merge_parquet_files`. Object keys,
file_list, incremental compaction trigger, tantivy index,
cache_latest_files and enterprise service-streams all follow the legacy
mover. **S3 layout, file_list schema, compactor and querier are
completely unchanged.**
- **Crash recovery**: packs follow the same `.lock` five-step flow as
legacy files (`.pack.tmp` → `.lock` → delete wal → rename → unlock),
keeping exactly-once semantics. Lock recovery stays synchronous on
startup (it must run before the pack index rebuild), while the orphan
`.par` sweep over the potentially millions-of-files `wal/files` tree
moved to a background task so startup no longer blocks.
- **Pack recycling**: a registry tracks per-pack `{total, consumed,
readers}`; a pack file is deleted once all segments are consumed and no
in-flight query read holds it (read guards replace the legacy
`lock_files`/pending-delete mechanism for packs).
Also included: shared-memtable hash changed from `{thread_id}_{org_id}`
to `{org_id}_{stream_name}` so one stream maps to exactly one bucket,
removing the per-request round-robin fan-out that multiplied
file/segment counts.
## Testing
- Unit tests: pack format roundtrip, size-based rollover,
corruption/truncation rejection, real-parquet segment read through the
index with time-range/memtable-id filtering, consume lifecycle (pack
deleted when fully consumed), reader-guard deferred deletion, flush
conditions.
- End-to-end with the real binary (4 phases):
1. wal binary format is byte-identical to main (`git diff main... --
src/wal src/ingester/src/entry.rs` is empty);
2. old-format wal written with the flag off **replays into packs** with
the flag on — no migration needed;
3. mixed-source search returns exact counts with legacy wal parquet +
pack segments + memtable coexisting (300/400/500 across three batches,
filters and row fetches correct);
4. pack mover merges multiple segments into one storage file, generates
the tantivy index, deletes consumed packs; final storage-side queries
and `match_all` are exact.
## Unrelated fixes included
Two startup paths shared the same "treat any db error as absence" bug:
- `init_db` used to treat ANY error from `get_db_schema_version` as
version 0 ("assuming default of 0 and trying upgrade") and kicked off
the full db upgrade — so a merely unreachable or overloaded database got
migration DDL piled on top of its existing load. Now
`get_db_schema_version` returns 0 only when the version is genuinely
absent (missing key, or the meta table itself not created yet on a fresh
install); transient errors are retried 5 times with backoff and then
abort the startup so the orchestrator restarts the node once the
database recovers.
- bootstrap did the same with the **instance id**: `instance::get()`
reported a missing key as an error, so "not stored yet" and "db
unreachable" landed in one arm that generated a fresh instance id — and
could overwrite the stored one once the db recovered. The instance id
must stay stable for the lifetime of a deployment, so a new one is now
generated only when it is genuinely absent; read errors are retried with
backoff and then abort the startup, and a failed write of a newly
generated id is no longer ignored.
## Notes for reviewers
- Flag is default **off**; with the flag off, behavior is unchanged
except the startup fixes and the shared-memtable hash change. The flag
only gates the persist path — pack search always runs (no-op on an empty
index), so packs already on disk stay searchable after the flag is
turned off while the mover drains them.
- Verified with the real binary that with the flag on, a metrics stream
is packed while a logs stream keeps the legacy per-stream parquet, both
queryable (SQL + PromQL).
- Pairs with o2-enterprise branch `feat/ingester-wal-pack-drain` (drain
completion check also scans `wal/pack`); that change must land
before/with enabling the flag in production, otherwise a draining node
could shut down with pack segments not yet uploaded.
- Known follow-ups (not in this PR): pack-specific metrics, extending
pack to other stream types, large-scale load test for segment-index
memory sizing, `wal-reader`-style debugging tool for packs.
Fixes#13408
## Problem
`ZO_MAX_FILE_SIZE_IN_MEMORY` has no effect. `rotate()` checks the
thresholds twice — once before taking the wal write lock, once after, to
guard against a concurrent rotation. The pre-check considered both the
wal and the memtable, but the post-check only looked at the wal:
```rust
if !self.check_wal_threshold(wal.size(), entry_bytes_size) {
return Ok(()); // check again to avoid race condition
}
```
So every rotation triggered *solely* by the memtable threshold was
silently discarded. The memtable could only be rotated by
`max_file_size_on_disk` — which measures pre-conversion JSON bytes — or
by the retention timer.
The reporter's log shows exactly this:
```
finish persist file: /data/wal/logs/0/default/logs/1784801072944124.wal,
json_size: 299395197, arrow_size: 4000015100, file_num: 3
batch_num: 260618, records: 260618, took: 1457 ms
```
With `ZO_MAX_FILE_SIZE_IN_MEMORY=512`, `json_size` (285MB) sits under
the 512MB disk threshold while `arrow_size` (3.7GB) is 7x over the 512MB
memory threshold. `batch_num == records` means one row per
`RecordBatch`, so arrow overhead is ~15KB/row and memory grows far ahead
of JSON size — precisely the case this limit exists to catch.
The only remaining backstop is `check_memtable_size()`, which *rejects
writes* (`MemoryTableOverflowError`) rather than rotating. So the
practical consequence is memory climbing to `mem_table_max_size` (25–50%
of RAM) and ingestion then failing, not merely a setting being ignored.
Introduced in fbeae17f1f (#5924), which split the single locked check
into a pre-check and a post-check; the prior version in 99a7485f6a held
the lock throughout and had no second check.
## Fix
Both call sites now go through a single `should_rotate()` helper. The
root cause was two copies of one condition drifting apart, not a wrong
condition, so sharing the helper is what actually prevents a recurrence.
The race guard is preserved: after a concurrent rotation the fresh wal
is exactly `FILE_TYPE_IDENTIFIER_LEN` (13) bytes and
`check_wal_threshold` requires `> 13`, while the fresh memtable has
`json_size == 0` and `check_mem_threshold` requires `> 0`. Both return
false, so the redundant rotation is still skipped.
**Lock ordering:** `should_rotate()` acquires the memtable read lock
while the second caller holds the wal write lock. All ten lock sites in
`writer.rs` order as `wal -> memtable`, and `rotate()` drops the wal
guard before `memtable.write()`, so there is no reverse path and no
deadlock.
## Testing
- `cargo test -p ingester` — 46 passed, 0 failed
- `cargo clippy -p ingester --all-targets` — no new warnings
- `cargo fmt -p ingester -- --check` — clean
No regression test added: reproducing "memory over limit, disk under
limit" requires shrinking `max_file_size_in_memory`, and `CONFIG` is a
process-global `ArcSwap` shared by all 46 tests in the binary, so
mutating it would be flaky. Making this properly testable would mean
resolving the thresholds once into `Writer` fields instead of calling
`get_config()` per check — happy to do that as a follow-up if coverage
here is wanted.
## Out of scope: the retention timer fires at 2x the configured interval
Noting this separately since the issue also reports "set 5mins dump
once, but the logs still shows every 10mins" — that is a **second,
independent bug**, not a consequence of the one fixed here, and it is
deliberately left out of this PR.
The ttl job in `lib.rs` sleeps for exactly `max_file_retention_time`,
while `check_wal_threshold` requires `created_at + retention <= now`.
`created_at` is stored after the rotation completes, so it always lands
slightly after the tick that caused it (write-queue wait, wal file
creation, `wal.sync()`). Every other tick therefore misses the deadline
by that delta:
```
tick t+300: created_at(t+delta) + 300 > t+300 -> skip
tick t+600: deadline passed -> rotate
```
Giving a stable rotate/skip/rotate/skip pattern at 2x the configured
interval — which also explains why it reads as "the setting isn't being
read at all", since 2 x 300s coincides with the 600s default.
After this PR the practical impact is much smaller: with the memory
threshold working, a stream at the reporter's volume rotates on size
roughly every 75s, so `created_at` is refreshed long before the timer
matters. The doubling only shows up on low-volume streams where the
timer is the sole trigger.
Fixing it properly means either sleeping until the earliest writer
deadline or applying an explicit tolerance to the comparison — a design
choice worth its own discussion rather than a magic polling constant
bolted onto this PR. Happy to open a separate issue or PR for it.
## Summary
Resolves#13172
The ingest write path already converts JSON data to an Arrow RecordBatch
for the memtable. This PR reuses that RecordBatch and serializes it as
an Arrow IPC stream for the WAL entry, instead of serializing the parsed
data back to JSON. WAL replay decodes the RecordBatch directly (the IPC
stream carries the schema), skipping JSON parsing and schema inference
entirely.
Write path: any protocol -> JSON -> format -> RecordBatch -> memtable +
WAL (Arrow IPC).
## Benchmarks
Micro-benchmark, per 3846-record batch (7.3MB pretty-printed JSON, k8s
logs), Apple M-series:
| | JSON (before) | Arrow IPC (after) | change |
|---|---|---|---|
| WAL entry serialize | 5.25 ms | 0.42 ms | 12x faster |
| WAL entry size | 7.35 MB | 2.87 MB | 61% smaller |
| WAL entry decode (replay) | 17.1 ms | 0.46 ms | 37x faster |
End-to-end, 10GB HTTP `_json` ingest (1200 x 8.4MB requests, 8
concurrent, 4 alternating rounds, single node):
- throughput: ~734 MB/s -> ~788 MB/s (**+8-11%**, one round reversed
under background-load noise)
- server CPU for the same work: **-11~14%** (consistent in every round)
- record count verified exact after every run (4,619,046)
End-to-end crash recovery (ingest 1GB / 461k records, `kill -9`,
restart):
- WAL replay: 2.95s -> 0.61s (**~5x faster**), recovered count exact
## Compatibility
- New entries start with a `0xFFFF` marker. The legacy format starts
with the stream-name length (u16), which can never be 0xFFFF (names are
bounded far below 64KB by the HTTP request-head limit), so
`Entry::from_bytes` detects the format per entry.
- **Upgrade:** legacy JSON WAL files replay through the unchanged legacy
path (covered by tests).
- **Downgrade caveat — draining the WAL before rollback is mandatory:**
an old binary cannot decode Arrow-format WAL entries. Entries smaller
than 64KiB fail with `ReadDataError` and are skipped per entry; entries
**larger than 64KiB** fail with `FromUtf8Error` (the 0xFFFF marker is
read as a 65,535-byte stream name containing non-UTF8 IPC bytes), which
the old replay loop does not skip — it aborts `replay_wal_files` and
leaves all remaining WAL files unreplayed. Always shut down gracefully
(drain WAL) before downgrading.
- Rotation checks use the actual serialized WAL bytes (Arrow IPC size),
matching the units of `wal::Writer::size()`; JSON size estimates remain
only for stats accounting.
- `data_size` left as 0 by a caller is normalized from the data at
preprocess time, and memtable emptiness is now determined by the batch
row count (the legacy `into_bytes()` used to backfill `data_size`, and 0
means "skip" downstream).
- `data_size` (original JSON size, used for stats/billing) is now the
size estimated at ingest time rather than re-serialized JSON length; the
two differ only marginally.
## Tests
- Arrow entry roundtrip (fields + batch equality), empty-batch and
unicode-field roundtrips
- Legacy JSON roundtrips still pass, with an added assertion that legacy
bytes are not mis-detected as Arrow
- `wal-reader` debug tool handles both formats
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Design at: #12596closes#12596
## Problem
`index_updated_at` is a single stream-level timestamp, bumped whenever
any field is added to `full_text_search_keys` or `index_fields`. The
tantivy optimize path (`SimpleCount` / `SimpleHistogram` /
`SimpleMultiHistogram` / `SimpleTopN` / `SimpleTopNMulti` /
`SimpleDistinct`) excludes every file with `min_ts < index_updated_at`,
because optimize results are used directly as final answers with no
DataFusion re-check.
Consequence: adding one index field to a long-retention stream knocks
every histogram/count/top-N over historical data off the fast path until
the cutoff ages out of retention — even when the queried fields have
been indexed since day one.
## Design
### Storage
`StreamSettings` gains `index_fields_updated_at: HashMap<String, i64>`,
recording when each field was added to `full_text_search_keys` or
`index_fields`:
- A field add stamps only genuinely-new fields. A no-op re-add of an
existing field no longer bumps anything (previously it pointlessly
bumped the stream-level value and disabled the index for all history).
- The stream-level `index_updated_at` is still bumped on every real add,
so older queriers in a mixed-version cluster keep the old conservative
behavior during rolling upgrades.
- A field removal prunes its entry (in `normalize_stream_settings`), so
a later re-add stamps a fresh time — files written in the gap lack that
field's index, and a single timestamp can't represent a gap.
- Bloom-only fields auto-folded into `index_fields` by normalization are
stamped like a normal add.
- The `reset-index-updated-at` CLI clears the map, since per-field
entries override the scalar.
- The empty map is skipped during serialization; legacy settings without
the key deserialize to an empty map. Fully backward/forward compatible,
no migration.
A field present in both `full_text_search_keys` and `index_fields`
shares one entry (last add wins). That is over-conservative for at most
one of the two uses and only in the rare case where the same field is
added to both lists at different times.
### Query
The cutoff in `flight.rs` is now computed from the fields the query
actually reads from the index — the optimize-rule target fields
(`SimpleTopN` / `SimpleTopNMulti` / `SimpleMultiHistogram` /
`SimpleDistinct`):
```
cutoff = max over referenced fields of:
BASE_TIME if the field is a default-enabled index field
else the per-field entry, else stream-level index_updated_at, else stream created_at
```
- Default-enabled index fields (`SQL_SECONDARY_INDEX_SEARCH_FIELDS`:
`trace_id`, `service_name`, `operation_name`, plus
`feature_secondary_index_extra_fields`) are indexed since the stream's
first file, so they always resolve to `BASE_TIME` and never constrain
the cutoff — even if they also carry a per-field entry.
- Index-condition fields (equality and FTS/`match_all`) are deliberately
**not** part of the cutoff: each file's tantivy schema is the per-file
source of truth there, so the timestamps add nothing.
- A query that references no per-stream fields — e.g. a count or
histogram without filters — gets cutoff `0`: its tantivy result only
depends on data every `.ttv` file has. The existing global
`ttv_timestamp_updated_at` / `ttv_secondary_index_updated_at` checks are
unchanged and still apply on top. This also covers the comment on #12596
that pure histograms don't need the `index_updated_at` check — it falls
out of the rule instead of being special-cased.
The regular filter-pushdown path is untouched: it never used
`index_updated_at` (it checks the actual per-file tantivy schema and
adds the filter back to DataFusion when a field is missing).
### Example
Stream created 2024-01 (so `trace_id` etc. indexed since then),
`user_id` added to `index_fields` on 2026-06-01:
| Query | Before | After |
|---|---|---|
| top-N on `service_name` (default field) | index only for files after
2026-06-01 | all indexed files (`BASE_TIME`) |
| top-N / distinct on `user_id` | files after 2026-06-01 | files after
2026-06-01 (correct: older files lack the `user_id` index) |
| `histogram(_timestamp)` / `count(*)` | files after 2026-06-01 | all
indexed files |
## Intentionally out of scope
The cutoff still conflates data time with index build time: a file the
compactor merges *after* a settings change carries a fresh, complete
index but keeps its old `min_ts`, so it stays off the optimize path.
Fixing that requires fetching `created_at` per file from file_list — one
more column over millions of rows on every query — to rescue roughly one
compaction cycle of data around the settings change, which self-heals
within the next hour. Not worth it.
This PR add bloom filter for parquet again under a env
`ZO_BLOOM_FILTER_PARQUET_ENABLED=false`, if you want to generate bloom
filter for parquet set it to `true`.
> We no longer use it even you generate it.
## Summary
Design at:
https://github.com/openobserve/designs/blob/main/infrastructure/bloomfilter/DESIGN.md
A bloom-filter pruning layer that sits **between `file_list` and
tantivy** in the search path. It drops candidate files before any `.ttv`
is opened, for **positive equality / IN** lookups on high-cardinality
fields (`trace_id`, `request_id`, …).
It is built **on top of** tantivy: the `.bf` is constructed by iterating
the tantivy term dictionary, so a field must be a tantivy secondary
index (`index_fields`) **and** opted into `bloom_filter_fields`. Only
the intersection is built — a bloom-only field produces nothing. To make
that intersection non-empty by construction, enabling a field for bloom
now folds it into `index_fields` automatically (see Config). Values are
not inspected; the operator chooses which fields are worth it (see
guidance below).
Design notes live in the module-level docs under `src/infra/src/bloom/`
(`mod.rs`, `writer.rs`, `reader.rs`, `sbbf.rs`).
## Design
- **Transposed (block-major) `.bf` layout.** All files in a `.bf` share
`num_blocks = B`, so `block_index = fastmap(hash(value), B)` is
identical across files and the blocks a query needs form **one
contiguous row**. The search side reads **one `M × 32`-byte row per
`.bf`** to answer membership for all `M` files — prune cost is
`O(groups)`, not `O(files)`. (`infra::bloom`, own ~120-LOC SBBF over
gxhash64.)
- **Sub-grouping.** A hour's new files are sorted by record count and
split into chunks of ≤ `ZO_BLOOM_FILTER_MAX_FILES_PER_BF` (default 256),
one `.bf` each. Bounds compactor build memory (`M × B × 32`) and `.bf`
object size; query reads become `ceil(files / 256)` rows.
- **Footer-only reader + batched fetch.** Footer (≈ `M × 12` B) comes
from `BLOOM_FOOTER_CACHE` or one 16 KB suffix GET (with a precise
re-read when a large footer overflows the probe); rows are pulled with
one `cache_storage::get_ranges` call. `cache_storage` was reworked to do
real batched cache reads (memory: 1 lookup + N slices; disk: 1
`File::open` + N `pread`).
- **B sizing.** `num_blocks_for(max_records_in_chunk,
ZO_BLOOM_FILTER_FPP)` (default fpp `0.01`) — record count is a safe NDV
upper bound, so the SBBF never saturates for high-cardinality fields.
- **Versioned, with orphan cleanup.** A file's `bloom_ver` is write-once
— a row is never re-stamped, so dumped rows keep pointing at the `.bf`
they were built into. But `bloom_ver` is **not** append-only at the
bucket level: a merge produces new output files with a fresh `bloom_ver`
and deletes the small inputs, so the old inputs' `.bf` can become
orphaned. After each merge round the compactor retires any `.bf` whose
`bloom_ver` is no longer referenced by a live `file_list` row in the
bucket (enqueued through the existing `file_list_deleted` path).
`bloom_ver` is preserved through every file_list INSERT/SELECT path
(incl. the SQLite read-cache) and the dump parquet.
- **Observability.** Prometheus metrics for prune
(`bloom_prune_keep_ratio`, `bloom_prune_duration_seconds`);
`bloom-inspect` CLI dumps a `.bf` footer (fields, B, row bytes, file
names); footer-cache hit/miss via the shared `BytesCache` tagged
metrics; `Local::get_ranges` now reports `STORAGE_READ_*` (previously
bypassed).
## When it helps (measured, S3, no disk cache)
Same stream `benchtest`, 170 files / 14.64 GB index, single matching
row. The only variable is whether the field's values are
time-correlated:
| Field | Type | tantivy only | bloom + tantivy | Outcome |
|---|---|---|---|---|
| `trace_id` (16-byte random) | random | **2584 ms** (opens all 170
`.ttv`) | 24 ms prune + 65 ms = **89 ms** | **~29× faster** |
| `request_id` (`…-7e74-…`, UUIDv7) | time-ordered | **154 ms** (sparse
index range-prunes) | 42 ms + 154 ms = **196 ms** | ~1.3× **slower** |
The whole 170-file group was answered by **one row read of 5440 bytes**
(`170 × 32`) + a cached footer.
**Guidance — enable only when all three hold (advisory; not enforced):**
1. **High cardinality** field.
2. **Random values** (not time-ordered) — so tantivy can't range-prune
and must scan every file. Time-ordered ids (UUIDv7, snowflake) are
already pruned by tantivy's sparse index in memory → bloom is pure
overhead there.
3. **Many files per hour** — hourly volume ≳ 10 ×
`ZO_COMPACT_MAX_FILE_SIZE` (≳10 `.ttv` files), so there's something to
prune. Bloom turns N file opens into one row read; at 1–2 files/hour
it's net overhead.
The 29× result above had 170 files/hour all answered by one `.bf`. ❌
Low-cardinality / non-indexed / range / `LIKE` / regex / `!=` / OR: no
effect.
## Config
- `ZO_BLOOM_FILTER_ENABLED` (default true) — global gate.
- Per stream: a field must be in **both** `index_fields` and
`bloom_filter_fields`. Adding a field to `bloom_filter_fields` auto-adds
it to `index_fields` (bloom is built on the secondary index), and the
default index fields include the default bloom fields.
- `ZO_BLOOM_FILTER_FPP` (default 0.01) — target false-positive
probability; sizes the SBBF block count (smaller fpp = fewer false
survivors, larger `.bf`). Out-of-range falls back to 0.01.
- `ZO_BLOOM_FILTER_MAX_FILES_PER_BF` (default 256) — files per `.bf`;
trades query reads vs build memory.
- `ZO_BLOOM_FILTER_DEFAULT_FIELDS` (default empty) — fields bloomed for
every stream.
- `ZO_BLOOM_FOOTER_CACHE_MAX_SIZE` (default 1% mem, clamped 32–256 MB).
Removed in this PR: `ZO_BLOOM_FILTER_DISABLED_ON_SEARCH`,
`ZO_BLOOM_FILTER_NDV_RATIO` (B is now sized from record count + fpp, not
an ndv ratio).
## Test plan
- [x] `cargo build`, `cargo clippy -p infra -p openobserve --lib -- -D
warnings`, `cargo fmt --check` clean
- [x] Unit tests: `cargo test -p infra --lib bloom`, `cargo test -p
openobserve --lib bloom`
- [x] E2E on S3 (results above): random field prunes 170→1 and is ~29×
faster; transposed read = 1 request / 5440 B per group
- [x] Reviewer: confirm gating expectations (operator opts fields in; no
auto value detection)
- [x] Test: orphan bloom: 2 * 256MB -> merge to 512MB -> merge to
1024MB, the first .bf should be deleted
- [x] Test: orphan bloom: 6 * 256MB -> merge to 1GB + 512MB -> 512 MB
merge to 1024MB, the first .bf include 2 files(1GB + 512MB), after merge
again, the first still have 2 files but one of them already deleted. we
should keep the first .bf, because still have files in use.
## Notes
- `.bf` format is `VERSION 0x01` (transposed). Not compatible with any
pre-merge experimental `.bf`; rebuild by wiping the `bloom/` dir and
resetting `bloom_ver=0`.
- Hash is gxhash64 (project-wide `config::utils::hash`), not the Parquet
spec's xxhash64 — `.bf` is internal, never read by external tools.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
https://github.com/openobserve/openobserve/pull/8876 had made
this variable private, resulting in failed enterprise builds.
It also removed one of the imports causing build failure.
This commit reverts those specific changes.
Implements cloud-agnostic graceful shutdown mechanism to support
Kubernetes preStop hooks and other orchestration platforms.
Changes:
- Add IS_INGESTER_DRAINING global flag with atomic operations
- Enhance /enable?value=false to trigger flush and set draining mode
- Add /drain_status endpoint to monitor shutdown readiness
- Modify parquet upload job to prioritize draining mode
- Exit upload loop when all files are uploaded to S3
API Usage:
- PUT /node/enable?value=false - Initiates drain (flushes memory, sets flag)
- GET /node/drain_status - Returns readyForShutdown status
Benefits:
- Eliminates fixed 900s wait in Helm charts
- Cloud-agnostic (works with K8s, ECS, Docker Swarm, systemd)
- Data-safe: ensures all WAL files uploaded before shutdown
### **User description**
- During ingestion there are instances where the writer channel is
closed. And this will need an app restart.
- So instead when we do get_writer check if channel is closed and if it
is creates a new writer
___
### **PR Type**
Bug fix
___
### **Description**
- Detect closed writer channels and refresh
- Remove stale writers from cache on close
- Add channel-closed helper on Writer
- Improve rotate error log with writer index
___
### Diagram Walkthrough
```mermaid
flowchart LR
GW["get_writer()"] -- "cache hit + channel open" --> RET["return cached Writer"]
GW -- "cache hit but channel closed" --> RM["remove from cache"]
RM -- "proceed to slow path" --> CREATE["create new Writer"]
GW -- "cache miss" --> CREATE
W["Writer"] -- "is_channel_closed()" --> GW
TTL["check_ttl()"] -- "log with idx on error" --> LOG["enhanced error log"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Bug
fix</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>writer.rs</strong><dd><code>Handle closed writer
channels and improve logging</code>
</dd></summary>
<hr>
src/ingester/src/writer.rs
<ul><li>Check cached writer channel state before returning.<br> <li>
Remove closed-channel writers from cache and recreate.<br> <li> Add
Writer::is_channel_closed helper method.<br> <li> Enhance rotate error
log to include writer index.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8155/files#diff-d1a9b1542cc2758afe4fa67abd454dd818718f4e1c7570303519af5a58d6301f">+22/-2</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
Hengfei Yang
Huaijin
Loakesh IndiranandHengfei Yang
ByteBakerandHengfei Yang
漂流
Sai Nikhil
Yashodhan Joshi
Simha G
Harsh Mahajan
Zhao.wei
udev
Yuta YamaguchiandHengfei Yang