mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-05 08:09:22 +00:00
Hengfei Yang
cf8d122394
Design at: #12596 closes #12596 ## Problem `index_updated_at` is a single stream-level timestamp, bumped whenever any field is added to `full_text_search_keys` or `index_fields`. The tantivy optimize path (`SimpleCount` / `SimpleHistogram` / `SimpleMultiHistogram` / `SimpleTopN` / `SimpleTopNMulti` / `SimpleDistinct`) excludes every file with `min_ts < index_updated_at`, because optimize results are used directly as final answers with no DataFusion re-check. Consequence: adding one index field to a long-retention stream knocks every histogram/count/top-N over historical data off the fast path until the cutoff ages out of retention — even when the queried fields have been indexed since day one. ## Design ### Storage `StreamSettings` gains `index_fields_updated_at: HashMap<String, i64>`, recording when each field was added to `full_text_search_keys` or `index_fields`: - A field add stamps only genuinely-new fields. A no-op re-add of an existing field no longer bumps anything (previously it pointlessly bumped the stream-level value and disabled the index for all history). - The stream-level `index_updated_at` is still bumped on every real add, so older queriers in a mixed-version cluster keep the old conservative behavior during rolling upgrades. - A field removal prunes its entry (in `normalize_stream_settings`), so a later re-add stamps a fresh time — files written in the gap lack that field's index, and a single timestamp can't represent a gap. - Bloom-only fields auto-folded into `index_fields` by normalization are stamped like a normal add. - The `reset-index-updated-at` CLI clears the map, since per-field entries override the scalar. - The empty map is skipped during serialization; legacy settings without the key deserialize to an empty map. Fully backward/forward compatible, no migration. A field present in both `full_text_search_keys` and `index_fields` shares one entry (last add wins). That is over-conservative for at most one of the two uses and only in the rare case where the same field is added to both lists at different times. ### Query The cutoff in `flight.rs` is now computed from the fields the query actually reads from the index — the optimize-rule target fields (`SimpleTopN` / `SimpleTopNMulti` / `SimpleMultiHistogram` / `SimpleDistinct`): ``` cutoff = max over referenced fields of: BASE_TIME if the field is a default-enabled index field else the per-field entry, else stream-level index_updated_at, else stream created_at ``` - Default-enabled index fields (`SQL_SECONDARY_INDEX_SEARCH_FIELDS`: `trace_id`, `service_name`, `operation_name`, plus `feature_secondary_index_extra_fields`) are indexed since the stream's first file, so they always resolve to `BASE_TIME` and never constrain the cutoff — even if they also carry a per-field entry. - Index-condition fields (equality and FTS/`match_all`) are deliberately **not** part of the cutoff: each file's tantivy schema is the per-file source of truth there, so the timestamps add nothing. - A query that references no per-stream fields — e.g. a count or histogram without filters — gets cutoff `0`: its tantivy result only depends on data every `.ttv` file has. The existing global `ttv_timestamp_updated_at` / `ttv_secondary_index_updated_at` checks are unchanged and still apply on top. This also covers the comment on #12596 that pure histograms don't need the `index_updated_at` check — it falls out of the rule instead of being special-cased. The regular filter-pushdown path is untouched: it never used `index_updated_at` (it checks the actual per-file tantivy schema and adds the filter back to DataFusion when a field is missing). ### Example Stream created 2024-01 (so `trace_id` etc. indexed since then), `user_id` added to `index_fields` on 2026-06-01: | Query | Before | After | |---|---|---| | top-N on `service_name` (default field) | index only for files after 2026-06-01 | all indexed files (`BASE_TIME`) | | top-N / distinct on `user_id` | files after 2026-06-01 | files after 2026-06-01 (correct: older files lack the `user_id` index) | | `histogram(_timestamp)` / `count(*)` | files after 2026-06-01 | all indexed files | ## Intentionally out of scope The cutoff still conflates data time with index build time: a file the compactor merges *after* a settings change carries a fresh, complete index but keeps its old `min_ts`, so it stays off the optimize path. Fixing that requires fetching `created_at` per file from file_list — one more column over millions of rows on every query — to rescue roughly one compaction cycle of data around the settings change, which self-heals within the next hour. Not worth it.