related to
- https://github.com/openobserve/openobserve/pull/8003
- https://github.com/openobserve/openobserve/issues/7986
# Why We Can Delete `enable_align_histogram`
## What It Was
`enable_align_histogram` was a query parameter introduced in [PR
#8003](https://github.com/openobserve/openobserve/pull/8003) to fix
[Issue #7986](https://github.com/openobserve/openobserve/issues/7986).
The bug was that in the old two-step search flow (`_search_partition` →
`_search`), the two APIs independently generated histogram intervals,
leading to mismatched time buckets and inaccurate chart rendering in
dashboards.
The flag told `_search_partition` to:
1. Align partition boundaries to histogram interval boundaries (e.g.,
round to the start of a minute/hour/day).
2. Return the computed `histogram_interval` in the response so the
frontend could pass it to `_search`.
## Why It Is Now Dead Code
### 1. The frontend no longer calls `_search_partition`
All frontend search now uses `_search_stream` (HTTP/2 streaming), which
internally calls `search_partition()` with `enable_align_histogram =
false`. The old `_search_partition` endpoint is no longer consumed by
the UI. The only frontend caller (`getValuesPartition` in
`IndexList.vue`) is defined but never invoked.
### 2. `search_stream` handles alignment inherently
In the streaming search flow, partitions themselves define the search
time ranges. Each partition search runs on the exact aligned time
boundaries returned by the partition step. There is no separate
histogram query that needs an externally provided interval — alignment
is "baked into" the partition boundaries.
### 3. Histogram queries are already covered by `is_complex_query`
For histogram queries (which have `GROUP BY`), `is_complex_query`
returns `true`. Combined with `has_ts_column = true`, this triggers
`calculate_partition_settings()` to align `min_step` to the histogram
interval — the same alignment behavior that `enable_align_histogram`
previously provided, but without the extra flag.
### 4. All backend callers pass `false`
Of the 5 callers of `search_partition()`:
| Caller | Value |
|--------|-------|
| gRPC handler | hardcoded `false` |
| streaming execution | hardcoded `false` |
| traces handler | hardcoded `false` |
| HTTP `_search_partition` | from query param (frontend always passed
`true`, but frontend no longer calls this) |
| HTTP `_search_partition_multi` | from query param (frontend always
passed `true`, but frontend no longer calls this) |
No active code path ever sets `enable_align_histogram = true`.
## What Was Removed
- `get_enable_align_histogram_from_request()` — the HTTP query param
parser
- The `enable_align_histogram` parameter from:
- `search_partition()` function signature
- `search_partition_multi()` function signature
- `calculate_partition_settings()` function signature
- The alignment logic in `search_partition()`:
- `add_mini_partition` (always `false` without the flag)
- `PartitionGenerator` align parameter (now driven by `is_histogram`
alone)
- `resp.histogram_interval` override block
- The `|| enable_align_histogram` branch in
`calculate_partition_settings`
- The query parameter from both HTTP handler `utoipa` specs
- The `false` argument from all 5 call sites
## Summary
- **Dedup**: adds `queued_files` module so the same file can only sit in
the queue once — prevents repeated alert runs re-enqueuing the same
uncached files under high blob latency
- **Priority queue OOM fix**: removes the unbounded `VecDeque`
drain-task architecture; `queue_download` now pushes directly to a
capped `VecDeque` (`FILE_DOWNLOAD_QUEUE_SIZE`), eliminating the mpsc
indirection that bypassed all back-pressure
- **LIFO fix**: corrects `pop_front` → `pop_back` so workers process the
most recently enqueued (freshest) files first, as originally intended in
#6786
- **Notify**: replaces 5s polling sleep with `tokio::sync::Notify`;
fixes missed-wakeup race and burst-drain chaining
- **Tests**: queue cap honored (normal + priority), dedup correctness
Fixes https://github.com/openobserve/openobserve/issues/11371
- close https://github.com/openobserve/o2-enterprise/issues/1656
# Performance
## Datasets
3TB original data, 12 cpus
## Querys
### query 1
```sql
SELECT
histogram(_timestamp) AS zo_sql_key,
kubernetes_namespace_name AS zo_sql_breakdown,
count(*) AS zo_sql_num
FROM
default
GROUP BY
zo_sql_key, zo_sql_breakdown ORDER BY zo_sql_key DESC
```
### query 2
```sql
SELECT
histogram(_timestamp) AS zo_sql_key,
kubernetes_namespace_name AS zo_sql_breakdown,
count(*) AS zo_sql_num
FROM
default
WHERE
kubernetes_pod_name <> 'ebs-csi-controller-76499b5578-trbwq'
GROUP BY
zo_sql_key, zo_sql_breakdown ORDER BY zo_sql_key DESC
```
after filter it will keep 99.79%'s data
### query 3
```sql
SELECT
histogram(_timestamp) AS zo_sql_key,
kubernetes_namespace_name AS zo_sql_breakdown,
count(*) AS zo_sql_num
FROM
default
WHERE
match_all('zinc')
GROUP BY
zo_sql_key, zo_sql_breakdown ORDER BY zo_sql_key DESC
```
after filter it will keep 41%'s data
### query 4
```sql
SELECT
histogram(_timestamp) AS zo_sql_key,
kubernetes_namespace_name AS zo_sql_breakdown,
count(*) AS zo_sql_num
FROM
default
WHERE
match_all('datafusion')
GROUP BY
zo_sql_key, zo_sql_breakdown ORDER BY zo_sql_key DESC
```
after filter it will keep 15%'s data
### query 5
```sql
SELECT
histogram(_timestamp) AS zo_sql_key,
kubernetes_namespace_name AS zo_sql_breakdown,
count(*) AS zo_sql_num
FROM
default
WHERE
match_all('error')
GROUP BY
zo_sql_key, zo_sql_breakdown ORDER BY zo_sql_key DESC
```
after filter it will keep 0.44%'s data
## Result
### main branch
query 1: 6.3s
query 2: 9.1s
query 3: 21s
query 4: 25s
query 5: 21s
### this pr
query 1: 4.2s
query 2: 5.1s
query 3: 1.8s
query 4: 1s
query 5: 300ms
# Summary
This PR adds a **multi-histogram** tantivy index optimization that
accelerates queries of the form:
```sql
SELECT histogram(_timestamp), <secondary_field>, count(*)
FROM table WHERE <filters>
GROUP BY histogram(_timestamp), <secondary_field>
```
When a secondary index exists on the breakdown field, the aggregation
(histogram + terms) is pushed down to the tantivy index level, avoiding
a full scan of the parquet data.
## Motivation
The existing `SimpleHistogram` optimizer only handles `GROUP BY
histogram(_timestamp)` with no secondary breakdown. Queries that
additionally group by an indexed field (e.g., `level`, `status_code`)
fall back to the full scan path and miss the tantivy optimization.
## Changes
- **New `SimpleMultiHistogram` optimization mode** — extended the
`IndexOptimizeMode` enum with a `SimpleMultiHistogram(min_value,
bucket_width, num_buckets, field)` variant that carries the breakdown
field name alongside histogram parameters.
- **Plan optimizer rule** (`is_simple_multi_histogram`) — a new tree
visitor in the physical optimizer that detects `AggregateExec` with
exactly 2 group-by expressions (one `date_bin`/histogram on
`_timestamp`, one column on an indexed field) and a single `count(*)`
aggregate.
- **Tantivy-level aggregation** (`handle_simple_multi_histogram`) —
executes a nested tantivy aggregation with an outer
`HistogramAggregation` on `_timestamp` and a sub-`TermsAggregation` on
the breakdown field, returning `Vec<(timestamp, breakdown_value,
count)>`.
- **Plumbing through the stack** — wired the new mode through the
follower index optimizer, tantivy search dispatch, multi-file result
merging, result cache (`CacheEntry::MultiHistogram`), and Arrow
`RecordBatch` conversion.
- **`index_fields` propagation** — the `FollowerIndexOptimizerRule` now
receives the set of indexed fields so the multi-histogram rule can
validate that the breakdown column actually has a secondary index.
In alert manager pipeline handler function, we should not clear the
trigger data (containing the period_end_time representing the timestamp
used for the last scheduled pipeline query), if the pipeline is
disabled. Because, when enabling, this should have the data to know
where to start the pipeline from.
## ENVs
Replace `ZO_DATAFUSION_FILE_STAT_CACHE_MAX_ENTRIES` to
`ZO_DATAFUSION_FILE_STAT_CACHE_MAX_SIZE`
fixed https://github.com/openobserve/openobserve/issues/11835
- Changed `datafusion_file_stat_cache_max_entries` to
`datafusion_file_stat_cache_max_size` in configuration, allowing for
memory size limits instead of entry counts.
- Implemented logic to set default cache size to 5% of total memory if
not specified.
- Added new metrics for cache hits, misses, eviction counts, and
eviction time to monitor cache performance.
- Updated cache eviction logic to track current memory usage and log
warnings when limits are exceeded.
## Summary
- Drop `PARQUET_MAX_ROW_GROUP_SIZE` from 1M to 128k. The constant was
previously load-bearing for row_id → row_group mapping, which is why the
old comment said it couldn't change.
- Write the row-group-size value in effect at index-build time into each
`.ttv` puffin file's footer (`row_group_size` property) so readers can
recover it later.
- Generate parquet access plans from tantivy hits using the **per-file**
value (carried through `TantivyResult::RowIdsBitVec` → `CacheEntry` →
`FileKey.row_group_size` → `SegmentInfo` in the SEGMENTS cache).
- Falls back to `1024 * 1024` for legacy `.ttv` files written before the
property existed, so old data keeps working unchanged.
## Why this matters
With the constant in the path, the assumed row-group size at read time
had to match the one at write time. Now each tantivy index is
self-describing, so the writer's row-group-size limit can be tuned
freely going forward without breaking row_id lookups against historical
files.
## Notes for reviewers
- No DB schema change. No parquet footer reads at query time. No
tantivy/puffin binary format break — the property is additive via
`PuffinMeta.properties` (already `#[serde(default)]`).
- `FileKey.row_group_size` is set locally on the querier after tantivy
search; it's not serialized over gRPC (parallels `segment_ids`).
- `helpers.rs::generate_parquet_access_plan` now uses the per-file size;
legacy `.ttv` falls back to `LEGACY_ROW_GROUP_SIZE = 1024 * 1024`.
## Test plan
- [x] `cargo check --all-targets --workspace` clean
- [x] `cargo test --package tantivy_utils` — all 153 tests pass
- [x] `cargo test --package config --lib test_file_key` — pass
- [ ] Ingest with the new 128k size, then search — confirm row_ids
resolve correctly
- [ ] Search against a pre-existing `.ttv` (no `row_group_size`
property) — confirm fallback path still finds rows
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Bug
Queries with `ORDER BY` on a non-timestamp column and `size: -1`
(unlimited) returned empty results when the TopKHeap path was active.
## Root Cause
In `execution.rs`, when `size = -1`, `original_size` was computed as
`0`:
```rust
let original_size = if req.query.size > 0 { req.query.size as usize } else { 0 };
// Per-partition: size = (from + 0) = 0 → partition fetches 0 rows
```
Every partition received `size=0`, so `TopKHeap` got no rows and
returned `[]`.
## Fixes
**`streaming/execution.rs`**
- When `size=-1`: use `ZO_QUERY_DEFAULT_LIMIT` as `heap_k` instead of 0
- Keep `size=-1` for per-partition requests so each partition returns
all its data
**`streaming/sorting.rs` — `TopKHeap::new`**
- Guard: `k=0` falls back to `ZO_QUERY_DEFAULT_LIMIT` (safety net;
`execution.rs` fix prevents this in practice)
## Fixed Behavior
Queries with non-ts `ORDER BY` and `size=-1` now correctly:
- Use `ZO_QUERY_DEFAULT_LIMIT` rows as the heap bound
- Send `size=-1` to each partition (not `size=0`)
- Return results instead of `[]`
Note: CTE and subquery queries that have no `_timestamp` in the outer
result take a **single partition** path (via `ts_column=None`) and never
reach `TopKHeap` — fixed in the enterprise repo (see companion PR).
## Tests
7 new `TopKHeap` unit tests in `sorting.rs`:
- Bounded `k` evicts correctly
- `k=1000` (unlimited) keeps all rows
- `from` offset applied after drain
- ASC order
- Multi-partition push
- `size=-1` non-ts ORDER BY (all rows survive)
- `k=0` fallback guard
## Test Queries (against a stream with `level`, `_timestamp` fields)
```sql
-- Should return data, correct DESC order
SELECT level, COUNT(*) as cnt FROM "stream" GROUP BY level ORDER BY cnt DESC
-- size: -1
-- Bounded
SELECT level, COUNT(*) as cnt FROM "stream" GROUP BY level ORDER BY cnt DESC
-- size: 10, from: 0
-- Paginated
SELECT level, COUNT(*) as cnt FROM "stream" GROUP BY level ORDER BY cnt DESC
-- size: 5, from: 5
```
## Related
Enterprise companion PR: openobserve/o2-enterprise#1727 (fix: suppress
_timestamp wildcard default for CTE and inline subquery sources)
Design at: feat: impl tantivy footer cache
- close https://github.com/openobserve/o2-enterprise/issues/1684
### 1. Generic Bytes Cache (`infra::cache::bytes_cache::BytesCache`)
A new reusable cache for byte buffers, tagged so multiple caches can
coexist with independent metrics.
```
┌──────────────────────────────────────────┐
│ BytesCache │
│ │
│ DashMap<String, Bytes> (readers) │
│ Mutex<VecDeque<String>> (cacher/FIFO) │
│ AtomicI64 (current_memory)│
│ usize (max_bytes) │
│ String (tag) │
│ │
│ get(key) → Option<Bytes> │
│ put(key, value) → evict if over limit │
│ evict() → batch-evict 5% oldest entries │
└──────────────────────────────────────────┘
```
**Key characteristics:**
- **Concurrency:** `DashMap` for reads, `parking_lot::Mutex` around the
FIFO queue for writes.
- **Eviction:** FIFO (not LRU), evicting 5% of entries in a batch when
memory exceeds `max_bytes`. Batch eviction avoids locking on every
insertion while staying bounded.
- **Metrics:** Per-tag gauges for memory size, entry count; counters for
hits, misses, GC runs; histogram for GC time.
- **Use case flexibility:** The `tag` parameter lets different
subsystems reuse the same cache code (e.g., `"tantivy_footer_cache"`,
`"puffin_footer_cache"`).
### 2. FooterCache L2 (`FOOTER_DATA_CACHE`)
A global `BytesCache` instance wraps the `FooterCache` as an L2 cache:
```
Query
│
▼
FooterCache::from_directory(dir, cache_key)
│
├─ FOOTER_DATA_CACHE.get(cache_key) ──► hit → deserialize & return
│
└─ miss
│
├─ read .footer_cache file from directory
├─ FOOTER_DATA_CACHE.put(cache_key, bytes) ← store for next time
└─ deserialize & return
```
**Serialization format** (binary, no external dependencies):
```
┌──────────────────────┐
│ raw footer data │ ← concatenated OwnedBytes for each (file, range)
├──────────────────────┤
│ JSON metadata │ ← FooterCacheMeta { files: Map<path, Vec<Item>> }
├──────────────────────┤
│ u64 footer_offset │ ← LE byte offset to metadata
├──────────────────────┤
│ u32 version (1) │ ← LE version for forward compatibility
└──────────────────────┘
```
**Configuration:**
- `ZO_INVERTED_INDEX_FOOTER_CACHE_MAX_SIZE` (env): max MB for the L2
`BytesCache`, default 5% total memory.
### 3. Puffin Footer Caching
The Puffin footer reader (`PuffinFooterBytesReader`) now also uses
`FOOTER_DATA_CACHE`. On first read, the footer tail + payload are
combined and cached under the key `"puffin_footer_{location}"`.
Subsequent reads for the same location skip the storage round-trip
entirely.
### 4. Parallel Warm-Up
`warm_up_terms` now dispatches all four warm-up categories concurrently
via `tokio::try_join!` instead of awaiting them sequentially:
```
Before (sequential):
warm_up_fields → warm_up_fields_term → warm_up_terms → warm_up_fast_fields
After (parallel):
┌─ warm_up_terms ──────────┐
├─ warm_up_fields ─────────┤
├─ warm_up_fields_term ────┤
└─ warm_up_fast_fields ────┘
all joined
```
## Configuration Reference
| Env Variable | Default | Description |
|---|---|---|
| `ZO_INVERTED_INDEX_FOOTER_CACHE_MAX_SIZE` | 5% total memory | Max
memory for the footer L2 bytes cache |
## Metrics Reference
| Metric | Labels | Type | Description |
|---|---|---|---|
| `zo_bytes_cache_memory_size` | `tag` | Gauge | Current memory used by
a bytes cache |
| `zo_bytes_cache_entry_count` | `tag` | Gauge | Number of entries in a
bytes cache |
| `zo_bytes_cache_gc_time` | `tag` | Histogram | Time spent per GC
eviction run |
| `zo_bytes_cache_gc_count` | `tag` | Counter | Total number of GC
eviction runs |
| `zo_bytes_cache_hits_total` | `tag` | Counter | Total cache hits |
| `zo_bytes_cache_miss_total` | `tag` | Counter | Total cache misses |
On my local machine, merging a 256GB file into 10GB files with 6 merge
threads requires too much memory. In the current main branch, each merge
thread uses about 8GB of memory, so 6 threads require 48GB in total.
Since my machine has only 32GB of memory, it starts using swap, which
makes the merge process slow.
this pr reduce the memory usage for merge files, that speed up the merge
speed.
Note: if the merge do not trigger the swap the merge speed almost same,
because most of time spend for merge parquet is write parquet file,
collect all data by datafusion is only 10%-20%.
## main
2026-05-15T10:02:45.357306+00:00 INFO
openobserve::service::search::datafusion::peak_memory_pool: [trace_id ]
DataFusion peak memory usage: 9093509452 bytes (8672.25 MB)
2026-05-15T10:02:45.369673+00:00 INFO
openobserve::service::compact::merge: [COMPACTOR:WORKER:0] merged 20
files into a new file:
files/default/logs/default/2026/04/29/08/74609930655246581765c7b.parquet,
original_size: 10250729590, compressed_size: 26601268, took: 29443 ms
## this pr
2026-05-15T10:05:27.015179+00:00 INFO
openobserve::service::search::datafusion::peak_memory_pool: [trace_id ]
DataFusion peak memory usage: 65057480 bytes (62.04 MB)
2026-05-15T10:05:27.015429+00:00 INFO
openobserve::service::compact::merge: [COMPACTOR:WORKER:3] merged 20
files into a new file:
files/default/logs/default/2026/04/29/08/746099374351712256066ac.parquet,
original_size: 10453255400, compressed_size: 25995715, took: 4811 ms
## Problem
`ORDER BY <non-timestamp col>` (e.g. `ORDER BY duration DESC`) gave
wrong results and risked OOM because per-partition \`hits_to_skip\`
assumed rows are time-ordered — but the globally k-th row can be in any
partition.
## Solution
Each partition fetches its local top-`k = from+size` rows. A
**TopKHeap** on the leader merges them globally, skips `from`, returns
`size`. Heap is bounded to `k` elements at all times — O(k) memory
regardless of partition count.
**Multi-column ORDER BY** (`ORDER BY duration DESC, _timestamp DESC`) is
fully honored: heap comparison and final sort respect all columns in
order, using secondary columns as tiebreakers.
## Scenario behavior
| Query | Behavior |
|---|---|
| `ORDER BY _timestamp DESC` | Unchanged — per-partition path |
| `ORDER BY duration DESC` | Heap merge path — correct pagination |
| `ORDER BY duration DESC, _timestamp DESC` | Heap merge; ties in
`duration` broken by `_timestamp` |
| `ORDER BY duration DESC` page 2 | Heap collects global top-k, skips
`from`, returns `size` |
| JOIN / subquery / GROUP BY | `is_aggregate=true` → single partition
(unchanged) |
| `ZO_DISABLE_PARTITIONS_FOR_NON_TS_ORDER_BY=true` | Forces 1 partition
(circuit breaker preserved) |
| Partial cache hit + non-ts ORDER BY | TODO — pre-existing bug, no
regression |
## What is NOT changed
- `ORDER BY _timestamp` execution path
- `_search` (non-streaming) — DataFusion handles it correctly
- Aggregate / histogram queries
## Problem
When executing a histogram query with `use_cache=true`, the same result
row was emitted **3 times** instead of once.
**Root cause:** `invalidate_cached_response_by_stream_min_ts` was
designed for raw log retention but didn't account for histogram bucket
boundaries. It trimmed `response_start_time` to `stream_min_ts` (the
first live document timestamp), which can fall **mid-bucket**.
**Example:**
- Cache covers `Jan 15 → May 14`, contains 1 bucket: `Apr 16` (26,000
docs)
- `stream_min_ts = Apr 16 11:47` (first real document, mid-bucket)
- Old code pushed `response_start_time` to `Apr 16 11:47`, past the
bucket key `Apr 16 00:00`
- This created a large delta `Jan 14 → Apr 16 11:47` that **contained
the bucket key**
- Delta sub-partitions re-queried the cache and found the same bucket →
3× duplicates
## Fix
For histogram queries, instead of trimming `response_start_time`, check
each bucket individually. A bucket is only stale when its **entire
interval** is before `stream_min_ts`:
```
bucket "Apr 16": covers Apr 16 00:00 → Apr 17 00:00
stream_min_ts : Apr 16 11:47
Apr 17 00:00 > Apr 16 11:47 → bucket still has live data → KEEP ✓
bucket "Jan 16": covers Jan 16 00:00 → Jan 17 00:00
Jan 17 00:00 > Apr 16 11:47 → NO → fully expired → DROP ✓
```
`response_start_time` is left untouched (stays at `Jan 15`). The delta
becomes `Jan 14 → Jan 15` — tiny, nowhere near the bucket key. No
sub-partition can re-find the bucket. No duplicates.
## Changes
- `src/service/search/cache/cacher.rs` —
`invalidate_cached_response_by_stream_min_ts`
- Added `histogram_interval` parameter
- For histogram queries: retain/drop individual buckets based on whether
the full bucket interval has expired
- For non-histogram queries: original behaviour unchanged
This adds support for current cycle for stripe users as period for usage
page
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
- [x] add timestamp filter to tantivy when we can
- [x] fix secondary index do not able to use tantivy optimzier
- [x] refactor tantivy file group logical to fit datafusion file group
logical
## Summary
Cherry-picks two correlation fixes from `branch-v0.90.0` to `main`:
- `a077b760c` fix: corelation custom groups (#11662) — adds `group`
field to `FoundGroup` and threads `semantic_groups` through
`calculate_dimension_analytics`
- `211876a97` fix: corelation settings fixes (#11666) — validates
tracked alias IDs against bundled defaults + custom DB groups; sanitizes
payload (drops `service` and orphan IDs) before save
Conflicts resolved by taking v0.90.0 versions (the fixes are the point
of the cherry-pick). Companion PR on o2-enterprise:
openobserve/o2-enterprise#1648 — that PR is currently red because it
depends on the new `FoundGroup.group` field landing here first.
## Test plan
- [ ] CI green (build, unit, playwright)
- [ ] Verify save Identity Config rejects invalid alias IDs but accepts
default + custom IDs
- [ ] Verify `_analytics` endpoint returns `group` category on
FoundGroup
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
- **fix: llm insights tab**
- **remove custom attributes and use gen ai spec**
---------
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
# Case 1: 5 files, 2 target partition
## before
one partition will have 3 files, another partition will have 2 files
## this pr
each partition have 2.5 files
# Case 2: 2 files, 4 target partition
## before
only have 2 partition, one partition one file
## this pr
have 4 partition, each partition have 0.5 files
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
## Summary
Two compactor optimizations focused on the historical (old-data) merge
path:
1. **Replace `ZO_COMPACT_OLD_DATA_MIN_RECORDS` with a file-size check.**
The hour-selection SQL in `query_old_data_hours` now uses `original_size
< ZO_COMPACT_MAX_FILE_SIZE / 2` instead of `records < 100`. Record count
is a weak proxy for "mergeable" — a file's size depends on schema width
and compression, while the compactor's actual goal is to produce files
near `max_file_size`. The size check directly answers the right
question.
2. **Push the 95%-of-max-size filter into `query_for_merge` SQL.**
Previously, every file in the target hour was fetched from `file_list`,
then files at ≥95% of `max_file_size` were discarded in-process by the
worker. The filter is now applied in the SQL (`AND original_size <=
max_file_size * 95 / 100`), reducing the rowset shipped from the
database — meaningful for healthy streams where most files are already
compacted. The redundant in-process check in `merge.rs` was removed.
### Env var change (breaking)
- **Removed:** `ZO_COMPACT_OLD_DATA_MIN_RECORDS` (no replacement; the
size threshold is derived from `ZO_COMPACT_MAX_FILE_SIZE`).
- Unchanged: `ZO_COMPACT_OLD_DATA_INTERVAL`,
`ZO_COMPACT_OLD_DATA_STREAMS`, `ZO_COMPACT_OLD_DATA_MAX_DAYS`,
`ZO_COMPACT_OLD_DATA_MIN_HOURS`, `ZO_COMPACT_OLD_DATA_MIN_FILES`.
Docs PR: openobserve/openobserve-docs (separate).
## Test plan
- [x] `cargo fmt && cargo clippy --all-targets` clean for the touched
files
- [x] `cargo check --bin openobserve` succeeds
- [x] `cargo test -p config` passes (sanitizer + defaults test updated)
- [x] Manually verify on a stream with many small old files that
`query_old_data_hours` selects them and the merge proceeds
- [x] Manually verify on a stream where most files are already at full
size that `query_for_merge` returns a smaller rowset than before
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- Normalize `ZO_FILE_FORMAT` to `parquet` for non-enterprise builds.
- Preserve configured file format values for enterprise builds.
- Add targeted tests for both non-enterprise and enterprise behavior.
## Why
Non-enterprise builds should not fail startup because `ZO_FILE_FORMAT`
is configured to an enterprise-only format. Instead, they now continue
with parquet as the supported storage format.
## Validation
- `cargo test -p config test_non_enterprise_file_format_forces_parquet`
- `cargo test -p config --features enterprise
test_enterprise_file_format_preserves_configured_value`
- `git diff --check`
Note: the test run reports an existing unrelated unused import warning
in `src/config/src/meta/user.rs`.
design at: designs/anomaly-detection
## Summary
Adds super cluster sync support for anomaly detection configs and fixes
a missing FK constraint on `anomaly_detection_config.folder_id`.
**Super cluster sync:**
- Add `infra::table::anomaly_detection` module with config CRUD ops
(`create`, `create_if_not_exists`, `update`, `put`, `delete`,
`get_by_id`, `list`)
- Add `super_cluster_queue::anomaly_detection` handler for
`ConfigCreate`, `ConfigUpdate`, and `ConfigDelete` events
- Refactor `service::anomaly_detection` and `scheduler::handlers` to use
the new table module instead of raw SeaORM entity queries
- Refactor `super_cluster_queue::scheduler` to use new table module
- Reschedule trigger instead of deleting when config not found locally —
handles SC race where trigger row arrives before `ConfigCreate` NATS
message is processed
**Migration / referential integrity fix:**
- Add migration `m20260504` to retrofit the missing `FOREIGN KEY
(folder_id) REFERENCES folders(id)` constraint on
`anomaly_detection_config` (alerts and dashboards have had this since
creation; anomaly detection was missing it)
- Pre-migration repair: any rows whose `folder_id` no longer exists in
`folders` (orphaned by folder deletion) are re-pointed to the org's
default Alerts folder before the constraint is added — prevents
migration failure on existing deployments
- Add folder existence check inside transaction in `create()` —
eliminates TOCTOU race between folder lookup and INSERT
- Add folder existence check in `update()` when folder is changing —
matches `alerts::update` behaviour
- Add `PutAnomalyConfigError::FolderDoesNotExist` error variant matching
the alert/dashboard error pattern
## Files changed (11)
- `src/infra/src/table/anomaly_detection/` — new module (config.rs,
mod.rs, models.rs)
- `src/super_cluster_queue/anomaly_detection.rs` — new SC queue handler
- `src/infra/src/table/mod.rs` — register new module
- `src/super_cluster_queue/mod.rs` — wire anomaly_detection handler
- `src/service/anomaly_detection.rs` — refactored to use new table
module
- `src/service/alerts/scheduler/handlers.rs` — refactored; reschedule on
missing config
- `src/super_cluster_queue/scheduler.rs` — refactored to use new table
module
- `src/infra/src/errors/mod.rs` — add
`PutAnomalyConfigError::FolderDoesNotExist`
-
`src/infra/src/table/migration/m20260504_000001_add_anomaly_detection_config_folder_fk.rs`
— new migration
- `src/infra/src/table/migration/mod.rs` — register migration
## Test plan
- [x] Anomaly config create/update/delete broadcasts appear in SC queue
logs on primary region
- [x] Follower region receives and applies `ConfigCreate`,
`ConfigUpdate`, `ConfigDelete`
- [x] Trigger reschedule (not delete) when config not yet synced to
follower
- [x] Create with non-existent folder returns clean `FolderDoesNotExist`
error
- [x] Update moving config to non-existent folder returns clean
`FolderDoesNotExist` error
- [x] Migration runs clean on existing DB with no orphaned rows
- [x] Migration repairs orphaned `folder_id` rows before adding FK
constraint
- [x] `cargo check --workspace` passes
Huaijin
Hengfei Yang
Loakesh Indiran
Subhra264
udev
Ashish Kolhe
若骏
Harsh Mahajan