mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-04 00:08:55 +00:00
Hengfei Yang
42e0d0448e
## Summary - add `ZO_FILE_DOWNLOAD_MIN_RECORDS` with a default of 100 records - skip undersized files before latest-file batch downloads and background queue insertion - apply the threshold consistently to Parquet and index files - preserve caching when `records == 0`, which can represent missing metadata from older gRPC senders ## Why Downloading and caching very small files consumes queue capacity and cache space without providing enough reuse benefit. Filtering them before enqueueing avoids unnecessary download work. ## Impact Files with a known positive record count below the configured threshold are read from object storage without being added to the download cache. A zero count is treated as unknown and remains cache-eligible; files at or above the threshold keep the existing behavior. ## Validation - `cargo fmt --all -- --check` - focused minimum-record boundary test - `cargo check -p search -p openobserve-core -p openobserve-api` - `git diff --check` Fixes #13288