# feat: Dashboard Table Chart — per-column formatting
## 1. Entry point — config panel
Column formatting lives in the panel **Config** panel, under a **Column
Formatting**
section. A **"Configure column formatting"** button opens the **Column
Formatting**
dialog.
<img width="1920" height="907" alt="image"
src="https://github.com/user-attachments/assets/21d05294-3d3e-4f84-893f-cf3b235adf0d"
/>
## 2. Column Formatting dialog — empty state
A 3-pane dialog. When nothing is configured yet it shows an empty state
(`OEmptyState`)
with a single **"+ Add field"** action. **Add field** is a dropdown
listing the panel's
columns (each with a **NUM / TEXT** type badge); the menu matches the
trigger width.
<img width="1920" height="907" alt="image"
src="https://github.com/user-attachments/assets/866fb119-76f3-48b2-98b6-50df24660a8c"
/>
## 3. Column Formatting dialog — editing a field
- **Left** — "+ Add field" + the list of added fields (type badge, name,
hover-to-remove ✕).
- **Middle** — the shared controls for the selected field:
- **Field Type** — `Auto` / `Num` / `Text` (Auto detects from the data;
Num/Text force it).
- **Value Formatting** — unit dropdown (numbers, bytes, durations,
percentage, currency, custom) — numeric columns only.
- **Alignment** — `Auto` / `Left` / `Center` / `Right` (Auto =
type-based default).
- **Styling** — Text color, Background color (curated swatches +
custom), and a Unique-value-color toggle.
- **Conditional Styling** — threshold rules (numeric columns only).
- **Right** — a **live preview** of the column rendered with the current
settings.
<img width="1920" height="907" alt="image"
src="https://github.com/user-attachments/assets/276289d8-f92d-46f2-a413-b58e840e2f6b"
/>
## 4. Conditional styling
Per-column rules read as a sentence: **If value** `[operator]` `[value]`
**then text
color** `[…]` **and background** `[…]`. Operators: `<`, `>`, `<=`, `>=`,
`=`, `!=`.
Add multiple rules — **the last matching rule wins**, so later rules
override earlier
ones (e.g. `> 1000` takes precedence over `> 400` for `2301`).
<img width="1920" height="907" alt="image"
src="https://github.com/user-attachments/assets/b47bde1c-11e4-4ec9-8d6b-03f702ae8902"
/>
**Example** — flag `count` when it's low:
> Conditional Styling → **Add rule** → operator `<`, value `50` → Text
`#b91c1c`, Background `#fef2f2`.
Persists in `override_config` as:
```json
{
"field": { "matchBy": "name", "value": "count" },
"config": [
{
"type": "conditional_styles",
"rules": [
{ "operator": "<", "threshold": 50, "textColor": "#b91c1c", "bgColor": "#fef2f2" }
]
}
]
}
```
## 5. Rendered result in the live table
Cells matching a rule render with the configured colors; non-matching
cells stay
normal. Thresholds may be decimals (e.g. `0.5`).
<img width="1920" height="907" alt="image"
src="https://github.com/user-attachments/assets/4c4146f9-b018-465e-81ae-fc04149cdc83"
/>
---
## Data model
Overrides persist as an array at `config.override_config`; each entry
targets one column
and carries typed config items. The discriminants (`tableConfigUtils.ts`
`OVERRIDE_CONFIG_TYPES`) are kept in sync with the Rust v8 schema
(`src/config/src/meta/dashboards/v8/mod.rs`):
```
field_type · unit · alignment · text_color · background_color · unique_value_color · conditional_styles
```
Unset values use `null` (inherit the panel default); `field_type`
defaults to `auto`.
Backward compatible: dashboards saved before this change load and render
unchanged (the
v8 `Config` enum has no `deny_unknown_fields` and no newly-required
field).
## Architecture
| Piece | File |
| --- | --- |
| Dialog (3-pane) |
`web/src/components/dashboards/OverrideConfigPopup.vue` |
| Shared sections |
`web/src/components/dashboards/ColumnFormatControls.vue` |
| Color picker | `web/src/components/dashboards/ColorSwatchPicker.vue` |
| Model + (de)serialize |
`web/src/composables/dashboard/useColumnFormatting.ts` |
| Parse / apply / format | `web/src/utils/dashboard/tableConfigUtils.ts`
|
| Render (cell styling) |
`web/src/components/dashboards/panels/TableRenderer.vue` |
| Config-panel entry |
`web/src/components/dashboards/addPanel/OverrideConfig.vue` |
| Rust v8 schema | `src/config/src/meta/dashboards/v8/mod.rs` |
## Scope changes (vs the original proposal)
- **Cell types removed** — the `Text / Bar / Spark` toggle, in-cell
**progress bars**,
and **sparklines** were prototyped and then dropped (frontend + the v8
`cell_type`
variant). Formatting is type/unit/alignment/color/conditional only.
- **Inline header popover removed** — formatting is reached via the
config-panel dialog
(no per-header tune-icon popover / "Edit all").
- **Conditional precedence** — last matching rule wins (was
first-match).
## Notable correctness fixes
- **Value-mapping ranges** — numeric values arriving as strings (e.g.
`"5"`) again match
range mappings (restored origin behavior; empty/null cells don't
false-match a
`0`-bounded range).
- **Conditional threshold** — decimal thresholds (e.g. `0.5`)
deserialize correctly on the
Rust side (flexible `OrdF64` deserializer).
- **Reset-on-type-change** — forcing a numeric column to `Text` clears
its now-hidden
unit/conditions (keyed by field, so removing a column never wipes a
different one).
---
# Column Filtering
> Detailed design: [`table-filtering.md`](./table-filtering.md)
In-table, multi-select, spreadsheet-style filtering for table panels.
Each column header
gets a filter button; its dropdown lists that column's **distinct
values** as a checkbox
list — tick/untick to narrow the rows. Filtering is **purely
client-side** (no re-query).
## 1. Enable it
A single **Column Filtering** toggle in the panel **Config** panel
(`config.table_filtering`, an `Option<bool>` in the v8 schema). Off by
default; when on,
the table renderers receive `:enable-filtering` and the header filter
buttons appear.
(Filtering is suppressed in print/export mode.)
<img width="1680" height="907" alt="image"
src="https://github.com/user-attachments/assets/d834a916-c7eb-4dce-9984-95276e996e87"
/>
## 2. Filter a column
Click a header's filter button to open the dropdown:
- **Distinct values** of the column as a checkbox list — pick any
subset.
- **Searchable** — a search box narrows the value list.
- **Clear** a column's filter in one click.
- Values display formatted (units/mappings) but match on the raw value.
- Multiple column filters combine with **AND**; sorting preserves active
filters.
<img width="1680" height="907" alt="image"
src="https://github.com/user-attachments/assets/0671b75e-e049-442f-bc56-53e861b9ff41"
/>
## Architecture
| Piece | File |
| --- | --- |
| Config toggle |
`web/src/components/dashboards/addPanel/ConfigPanel.vue`
(`table_filtering`) |
| Renderer wiring |
`web/src/components/dashboards/PanelSchemaRenderer.vue`
(`:enable-filtering`) |
| Filter UI + logic | `web/src/components/TenstackTable.vue` (gated by
`enableColumnFilter`) |
| v8 schema flag | `src/config/src/meta/dashboards/v8/mod.rs`
(`table_filtering: Option<bool>`) |
Implemented on the same branch as the formatting work;
`enableColumnFilter` is a no-op
when false, so non-table charts and filtering-off panels are unaffected.
</content>
Design At: dashboards/table-enhancement
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
## Problem
The enterprise build is broken on `main` (`error[E0061]`):
```
error[E0061]: this function takes 3 arguments but 2 arguments were supplied
--> src/super_cluster_queue/short_urls.rs:33:13
33 | infra::table::short_urls::add(&short_id, &original_url).await?;
| argument #3 of type `&str` is missing
```
#12829 ("fix: org id as mandatory check") made `org_id` a mandatory
third argument of `infra::table::short_urls::add`, but the
**enterprise-gated** super-cluster queue handler
`super_cluster_queue::short_urls::process` was not updated. This module
is `#[cfg(feature = "enterprise")]`, so OSS-only CI compiled fine while
the enterprise build (and every downstream Playwright shard) failed.
## Fix
The super-cluster queue message (`emit_put_event` → `short_url_put`)
only carries the `original_url` over the wire — `org_id` is never
published to the queue. So the receiver passes an empty `org_id`.
This matches the legacy semantics introduced in #12829, where
`short_urls::get` accepts empty-org rows from any org:
```rust
infra::table::short_urls::add(&short_id, &original_url, "").await?;
```
## Note / possible follow-up
With an empty `org_id`, short URLs replicated **across a super cluster**
are stored org-agnostic (accepted from any org on lookup), the same as
pre-existing legacy rows. If org-scoped short URLs need to survive
super-cluster replication, a follow-up should thread `org_id` through
the queue message value (producer `emit_put_event` and the
`short_url_put` payload) — a larger change touching the wire format.
This PR only unblocks the build with behavior consistent with #12829's
design.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This attempts to refactor the process_node fn to make it more readable
and change-able.
The commits should be reviewed individually, first extracts individual
match arms into their own fns, second does cleanup of fn params by
extracting them into structs.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Currently in pipeline execution, if any null record is encountered, it
results in errors. Instead it should just ignore the null records and
continue. It is very common in pipeline vrls to use -`. = null` when
certain condition matches.
Currently if you try to self consume the histogram metrics emitted by
o2, it fails with the below message -
```
ERROR openobserve::job::promql_self_consume: error in ingesting self-metrics : missing value
```
This pr fixes this .
- Add logs to analyze the time taken by pipelines (can be turned on by
`ZO_PRINT_KEY_EVENT=true`)
- Add pipeline processing time to the usage reporting
- Introduce new metrics (`zo_pipeline_exec_batch_size` and
`zo_pipeline_exec_time_ms`) for time taken by pipeline.
## Summary
Eliminates 1 of 3 redundant \SearchService::cache::search\ calls per
trace request by merging the Q2a (per-trace aggregate fields) SQL
directly into Q1.
### Changes
- **\uild_llm_trace_query\**: Added \extra_selects: &str\ param — Q2a
fields injected before \FROM\ for both regular and LLM trace queries
- **\get_latest_traces\**: Moved schema-dependent expression computation
before Q1 SQL building; extended Q1 with Q2a columns (span_count,
error_count, max_duration, service_count, root_service_name,
first_service_name, first_operation_name); removed entire Q2a search
block (~200 lines)
- **\process_latest_traces_stream\**: Same Q1 extension; added
early-break after 3 consecutive empty partitions for absolute time
ranges
- **\ un_q2_for_traces\**: Removed Q2a search call; populates all fields
directly from Q1 hits; keeps Q2b (service breakdown) for multi-service
traces
### Before (3 search calls per request)
1. Q1 — basic trace list
2. Q2a — \ race_id IN (...)\ for span aggregates (removed)
3. Q2b — \ race_id IN (...)\ for service breakdown
### After (2 search calls, or 1 for single-service traces)
1. Q1 — trace list with all aggregates
2. Q2b — service breakdown (multi-service only)
### Risk
Low. The removed Q2a code was entirely internal — column aliases like
\min_start_time\/\max_end_time\ had zero external references.
\TraceResponseItem\ struct unchanged. All fields now populated from Q1
with equivalent values.
---------
Co-authored-by: Giriraj Bidwai <girirajbidwai@users.noreply.github.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
## Summary
Opening a `.ttv` (puffin) inverted-index file on a footer-cache **miss**
previously issued **two** sequential ranged reads against object
storage:
1. read the last 12 bytes (footer tail) to learn `payload_size`, then
2. read the payload region (`HeadMagic + JSON`).
This mirrors the one-shot tail optimization already used for
bloom-filter (`.bf`) files: pull a single fixed-size suffix that almost
always covers the whole footer region, eliminating the first round-trip.
## Why 4 KB
A `.ttv` is always a **single-segment** tantivy index, so its footer is
a small uncompressed JSON over a **fixed ~6-blob** set (`meta.json`, the
`.term/.idx/.pos/.fast` segment files, and `footer_cache`; at most ~9
even with `.store/.fieldnorm/.del`). The footer size does **not** grow
with the number of indexed fields.
Measured across 195 real `.ttv` files (4 KB – 176 MB):
| metric | min | p50 | p99 | max |
|---|---|---|---|---|
| footer payload JSON | 949 B | 1013 B | 1124 B | 1139 B |
| one-shot tail needed | 965 B | 1029 B | 1140 B | 1155 B |
| blobs / file | 6 | 6 | 7 | 7 |
So **4 KB** covers the entire footer region in one read with ~2×
headroom over the structural ceiling (~2 KB), while being far smaller
than the bloom filter's 16 KB (whose footer scales with group
cardinality, unlike here).
## Behavior
- **Common case:** 1 ranged read instead of 2 → saves ~one RTT per cold
footer open. Queries scanning many partitions save `N × RTT`.
- **Rare larger footer:** re-reads exactly the footer region, so the
worst case is no worse than before (still 2 reads).
- **Cache unchanged:** only the exact footer region (not the full 4 KB
probe) is stored in `FOOTER_DATA_CACHE`, so cache memory is identical to
today.
- **Footer-cache hit path:** untouched.
## Tests
- `cargo test -p tantivy_utils puffin` — 153 passed, incl. new
`test_slice_payload_from_suffix_roundtrip` covering both the single-read
fast path and the short-probe fallback against a real in-memory puffin
file.
- `cargo clippy -p tantivy_utils --tests` — clean.
- Verified working locally on real `.ttv` data.
- [x] ~add sort when to many groups, we can do this, but for now do not
do this~
- [x] remove the unsafe for memory pool
- [x] add approx_distinct improve, already fork, will add in next pr
- [x] fork vortex to add cast fix, and export
`.exclude_schemes([IntDictScheme.id()])`
- [x] ~enable_physical_uncorrelated_scalar_subquery test~
## Problem
When **"Correlate without service attribute"** (`service_optional =
true`) is enabled, the same physical service can appear as two (or more)
separate entries in **Discovered Services**.
**Root cause:** `ServiceKey` is always `(service_name, set_id,
disambiguation)`. When a log has no `service.name` attribute, the
processor falls back to the stream name (e.g. `repro_logs`) as
`service_name`. A trace for the same workload carries the real
`service.name` (e.g. `ecs-api-service`). Both entries share identical
`(set_id, disambiguation)` but are stored as separate rows — they never
merge.
**What the DB looks like (bug):**
| service_name | set_id | disambiguation | logs_streams | traces_streams
|
|---|---|---|---|---|
| `ecs-api-service` | `kubernetes` | `{"k8s-namespace":"ecs-prod"}` |
`[]` | `["repro_traces"]` |
| `repro_logs` | `kubernetes` | `{"k8s-namespace":"ecs-prod"}` |
`["repro_logs"]` | `[]` |
Two rows. `disambiguation` is byte-for-byte identical. Only
`service_name` differs.
In production this gets worse — ECS metric streams (which also have no
`service.name`) each become their own phantom row under the same
disambiguation. We observed 85 rows for a single ECS service, with only
2 rows carrying actual streams.
## Fix
`GET /api/:org_id/service_streams` now performs a read-time merge when
`service_optional = true`:
1. Fetches raw rows from the DB (unchanged)
2. Loads the org's service identity config (already cached)
3. If `service_optional = true` — groups rows by `(set_id,
disambiguation)`, unions their `logs_streams` / `metrics_streams` /
`traces_streams` arrays, and picks the real service name (the one that
does NOT appear in its own stream arrays) as the display name
4. Returns the merged list
**After fix — what the API returns:**
```json
[{
"service_name": "ecs-api-service",
"set_id": "kubernetes",
"disambiguation": {"k8s-namespace": "ecs-prod"},
"logs_streams": ["repro_logs"],
"traces_streams": ["repro_traces"]
}]
```
One entry. Real service name. All streams unified.
The DB rows are not changed — merge is purely at read time in the
handler. When `service_optional = false` the handler returns raw rows as
before (no change in behaviour).
Fixes#1848
This adds a UI and API based trigger for re triggering license check
limits, so we do not have to restart nodes
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Currently, on the roles edit page, the frontend makes role permission
api for each and every resources. So, for current 41 resources, there
are 41 concurrent api calls, each going through the openfga rbac check
middleware. That means, there are 41 concurrent redundant openfga
checks, and then 41 openfga query to get the role permissions.
This pr adds another api `org_id/roles/role_name/permissions` that
combines all the resource permissions of the given role name and return
the same eliminating the need for making api calls for each rbac
resource.
## Summary
- Preserve canonical/resource-derived trace fields when resource or span
attributes collide with flattened span fields like `service_name`.
- Keep resource `service.name` as the canonical `service_name` while
moving colliding resource fields such as resource `name` to
`service_attr_name`.
- Prefix colliding span attributes with `attr_` before serde flattening.
- Add regression coverage for span/resource `service_name` collisions
and normalized resource-field collisions.
Fixes#12448
## Testing
- `git diff --check`
Not run: cargo test, because this local environment does not have
rustc/cargo installed.
---------
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
Design at: #12620Closes#12620Fixes#12564
## What
Rewrites both tantivy histogram optimize-rule collectors following the
same approach as the recent `TopNCollector` /
`SingleSegmentDocIdCollector` rewrites, and adds timezone support to
both rules.
- `MultiHistogramCollector` (replaces the aggregation framework for
`SimpleMultiHistogram`)
- `SimpleHistogramCollector` (replaces
`tantivy::collector::HistogramCollector` for `SimpleHistogram`)
### Timezone support (#12564)
`histogram(ts, interval, tz)` rewrites to `date_bin(interval,
to_timestamp_micros(_timestamp + offset), origin)`; the optimizer's
`get_column_name` hit the `BinaryExpr` and silently skipped both rules.
Now:
- the visitors extract the fixed offset from that shape
(`get_timestamp_offset`), so the rules fire with a timezone
- `IndexOptimizeMode::SimpleHistogram` / `SimpleMultiHistogram` carry a
`ts_offset` field (no proto change — histogram modes never cross gRPC,
they're re-derived per node)
- bucket edges are computed in shifted wall-clock space and the
collectors bucket on `_timestamp + ts_offset`, so emitted keys match
`date_bin` exactly when merging with non-tantivy partials
## Testing
- New collector unit tests: dense vs sparse equality, ts_offset
shifting, bounds filtering, missing column / missing breakdown docs,
per-bucket truncation with tie-breaks
- Visitor tests now include real-DataFusion-plan timezone cases
(`'+08:00'` simple, `'-05:30'` multi) asserting the extracted shifted
bounds and offset
- `cargo test -p config` (1936 passed), plus `index_optimizer` /
`search::tantivy` / `tantivy_optimize_exec` suites in the main crate;
clippy clean
- Low cardinality: fast `20%` than main
```
histogram(_timestamp, '1h') AS ts, count(*) FROM default GROUP BY ts ORDER BY ts ASC
```
- Low cardinality: fast `30%` than main
```
histgoram(_timestamp, '1h') AS ts, breakdown, count(*) FROM default GROUP BY ts, breakdown ORDER BY ts ASC
```
- High cardinality: faster `5x` than main
```
histgoram(_timestamp, '1h') AS ts, breakdown, count(*) FROM default GROUP BY ts, breakdown ORDER BY ts ASC
```
Close#12345
## Problem
OTLP histogram metrics sent over HTTP/JSON (e.g. otel-collector with
`encoding: json`) created streams but stored 0 events, with no
`_count`/`_sum`/`_bucket` streams. Int-valued counters/gauges were
silently recorded as 0. The protobuf path was unaffected, which is why
this only surfaced for collectors explicitly configured with JSON
encoding.
## Root cause
`otlp_json` deserializes the body directly into opentelemetry-proto's
serde-derived types, which only partially implement the proto3 JSON
mapping (64-bit integers encoded as strings, enums by name, zero/empty
fields omitted). `Metric.data` is a flattened `Option` oneof, and serde
converts any inner deserialization error of a flattened `Option` into
`None` instead of failing — so the whole metric silently became `data:
None`, while the stream itself had already been created by the schema
setup. Same mechanism on `NumberDataPoint.value` turned string `asInt`
values into 0.
## Changes
- Upgrade opentelemetry stack 0.31 → 0.32, which fixes string-encoded
u64 handling for plain histograms and summaries upstream.
- Add `otlp_json_compat` normalizer for the gaps still present in 0.32
(each verified empirically, with unit tests):
- string `asInt` values on data points and exemplars
- `aggregationTemporality` as enum name
- `Exemplar`'s value oneof is not flattened in 0.32
- missing `serde(default)` on
`ExponentialHistogramDataPoint`/`Buckets`/`ValueAtQuantile`/`Exemplar`,
which rejected omitted zero/empty fields
- Count and log metrics whose data oneof failed to decode
(`partial_success.rejected_data_points`) instead of silently skipping
them, and stop overwriting specific partial-success error messages with
the generic retention message.
- Adapt to OTLP 0.32 API additions (`KeyValue.key_strindex`,
`AnyValue::StringValueStrindex`).
- Upgrade workspace reqwest 0.12 → 0.13, required because
opentelemetry-http 0.32 only implements `HttpClient` for reqwest 0.13
(`rustls` feature replaces `rustls-tls-native-roots` via the platform
verifier; `json`/`query`/`form` are now explicit features). The
`segment` crate is pinned to reqwest 0.12, so `TELEMETRY_CLIENT` moved
into `telemetry.rs` as a plain `reqwest::Client` that posts the Track
message itself, replicating `segment::HttpClient::send`.
- Remove the unused pyroscope profiling integration (optional feature,
dependencies, agent setup, `ZO_PROF_PYROSCOPE_*` config).
## Verification
- 10 unit tests in `otlp_json_compat` covering the full payload matrix
(collector-style string integers, enum names, omitted zero fields,
exemplars with and without trace context, already-normalized payloads,
malformed input).
- End-to-end: posted the issue's collector-style JSON payload to
`/api/default/v1/metrics` — `_bucket` (3 events, correct cumulative
counts with exemplars attached), `_count`, `_sum` populated for
histogram and exponential histogram, summary quantiles stored, `asInt`
counter stored as 42 instead of 0.
- `cargo clippy -- -D warnings` clean; lib test failures are
pre-existing local-environment issues that reproduce identically on
main.
## Problem
`get_sre_agent_credentials` early-returned on the `Ok(record)` branch
when the SRE-agent SA DB row already existed, so `ensure_sys_rca_agent`
only ran for orgs whose SA was *missing*. If the startup backfill ever
failed for an org (FGA transient, OpenFGA not yet ready, etc.), there
was no runtime path to retry → every subsequent RCA trigger
short-circuited past the self-heal, the SA's OpenFGA tuples stayed
broken, MCP returned 0 tools, and the agent failed with `Tool
'GetIncident' not found`.
Also bumps the `sre_readonly_afolder_incidents` migration flag (`v1` →
`v2`) so the OpenFGA top-up patch re-runs once on clusters seeded by
older code — required to add the new `mcp:_all_{org}` grants from the
companion PR. Also bumps the reqwest timeout to o2-ai (300s → 600s) —
multi-turn RCA legitimately needs the headroom.
## Fix
Always run `ensure_sys_rca_agent` before the DB lookup. Idempotent, safe
to call every request, self-heals the SA + role state on the next
trigger after a transient backfill failure.
## Pairs with
openobserve/o2-enterprise#1885 — drops the equivalent early-return in
`create_and_assign_sre_readonly_role` and adds the missing `mcp:`
resource grants.
Resolves https://github.com/openobserve/o2-enterprise/issues/1901
Design at: #12596closes#12596
## Problem
`index_updated_at` is a single stream-level timestamp, bumped whenever
any field is added to `full_text_search_keys` or `index_fields`. The
tantivy optimize path (`SimpleCount` / `SimpleHistogram` /
`SimpleMultiHistogram` / `SimpleTopN` / `SimpleTopNMulti` /
`SimpleDistinct`) excludes every file with `min_ts < index_updated_at`,
because optimize results are used directly as final answers with no
DataFusion re-check.
Consequence: adding one index field to a long-retention stream knocks
every histogram/count/top-N over historical data off the fast path until
the cutoff ages out of retention — even when the queried fields have
been indexed since day one.
## Design
### Storage
`StreamSettings` gains `index_fields_updated_at: HashMap<String, i64>`,
recording when each field was added to `full_text_search_keys` or
`index_fields`:
- A field add stamps only genuinely-new fields. A no-op re-add of an
existing field no longer bumps anything (previously it pointlessly
bumped the stream-level value and disabled the index for all history).
- The stream-level `index_updated_at` is still bumped on every real add,
so older queriers in a mixed-version cluster keep the old conservative
behavior during rolling upgrades.
- A field removal prunes its entry (in `normalize_stream_settings`), so
a later re-add stamps a fresh time — files written in the gap lack that
field's index, and a single timestamp can't represent a gap.
- Bloom-only fields auto-folded into `index_fields` by normalization are
stamped like a normal add.
- The `reset-index-updated-at` CLI clears the map, since per-field
entries override the scalar.
- The empty map is skipped during serialization; legacy settings without
the key deserialize to an empty map. Fully backward/forward compatible,
no migration.
A field present in both `full_text_search_keys` and `index_fields`
shares one entry (last add wins). That is over-conservative for at most
one of the two uses and only in the rare case where the same field is
added to both lists at different times.
### Query
The cutoff in `flight.rs` is now computed from the fields the query
actually reads from the index — the optimize-rule target fields
(`SimpleTopN` / `SimpleTopNMulti` / `SimpleMultiHistogram` /
`SimpleDistinct`):
```
cutoff = max over referenced fields of:
BASE_TIME if the field is a default-enabled index field
else the per-field entry, else stream-level index_updated_at, else stream created_at
```
- Default-enabled index fields (`SQL_SECONDARY_INDEX_SEARCH_FIELDS`:
`trace_id`, `service_name`, `operation_name`, plus
`feature_secondary_index_extra_fields`) are indexed since the stream's
first file, so they always resolve to `BASE_TIME` and never constrain
the cutoff — even if they also carry a per-field entry.
- Index-condition fields (equality and FTS/`match_all`) are deliberately
**not** part of the cutoff: each file's tantivy schema is the per-file
source of truth there, so the timestamps add nothing.
- A query that references no per-stream fields — e.g. a count or
histogram without filters — gets cutoff `0`: its tantivy result only
depends on data every `.ttv` file has. The existing global
`ttv_timestamp_updated_at` / `ttv_secondary_index_updated_at` checks are
unchanged and still apply on top. This also covers the comment on #12596
that pure histograms don't need the `index_updated_at` check — it falls
out of the rule instead of being special-cased.
The regular filter-pushdown path is untouched: it never used
`index_updated_at` (it checks the actual per-file tantivy schema and
adds the filter back to DataFusion when a field is missing).
### Example
Stream created 2024-01 (so `trace_id` etc. indexed since then),
`user_id` added to `index_fields` on 2026-06-01:
| Query | Before | After |
|---|---|---|
| top-N on `service_name` (default field) | index only for files after
2026-06-01 | all indexed files (`BASE_TIME`) |
| top-N / distinct on `user_id` | files after 2026-06-01 | files after
2026-06-01 (correct: older files lack the `user_id` index) |
| `histogram(_timestamp)` / `count(*)` | files after 2026-06-01 | all
indexed files |
## Intentionally out of scope
The cutoff still conflates data time with index build time: a file the
compactor merges *after* a settings change carries a fresh, complete
index but keeps its old `min_ts`, so it stays off the optimize path.
Fixing that requires fetching `created_at` per file from file_list — one
more column over millions of rows on every query — to rescue roughly one
compaction cycle of data around the settings change, which self-heals
within the next hour. Not worth it.
## Summary
- Super cluster leader was hardcoding `0` for `took_wait_in_queue` in
all search responses
- Added `wait_in_queue` field to `ScanStats` proto and struct
- Follower now propagates its queue wait time via `ScanStats`
- Leader reads aggregated value (`max` across followers) instead of `0`
Single-cluster deployments were unaffected.
## Summary
Rewrites the tantivy `GROUP BY ... count(*) ... ORDER BY cnt LIMIT n`
optimization around a custom flat collector over columnar term ordinals,
replacing the nested `TermsAggregation` approach from #12348 that was
disabled in #12528 for performance. One unified `SimpleTopN` path now
serves **1..=4 group by fields** at every layer (SQL matcher, internal
enum, wire protocol, collector, result types).
Closes the performance issue documented in
https://github.com/openobserve/openobserve/pull/12348#issuecomment-4659231020
and re-enables multi-field group by.
## Design
**`TopNCollector`** (`config::tantivy::query::topn_collector`), per
file:
1. read each group field's term ordinal per doc, fetched in blocks via
`Column::first_vals` (vectorized for full columns)
2. count into a **dense per-ordinal array** indexed by the mixed-radix
ordinal when the fields' dictionary-size product fits 4M cells (no
hashing — ordinals are dense), falling back to a hash map keyed by
bit-packed ordinals (`u64` for 1-2 fields, `u128` for 3-4) beyond that
3. keep groups per the exactness rule below, selecting top-k with a
bounded heap directly off the counting array (no full materialization)
4. resolve only the surviving ordinals to strings, in one sorted forward
pass per field via `Dictionary::sorted_ords_to_term_cb` (point lookups
with `ord_to_term` re-scan a dictionary block per call and were the
dominant cost)
DataFusion re-aggregates the per-file partials and applies the final
ORDER BY + LIMIT.
## Exactness contract
Per-file top-K partials are only exact when nothing is truncated; once
truncation kicks in, a group's total can be spread thin across files and
the merged top-N is approximate. Measured on near-uniform data, no
practical keep size restores accuracy (keeping 100k of 1.4M groups per
file still matched 0/5 of the true top-5), so the rule is bimodal,
decided per file at harvest where the distinct count is known for free:
- **distinct groups <= max(limit-derived top-k,
`ZO_INVERTED_INDEX_TOPN_MAX_GROUP_NUM`)** (env default 1000): the file
returns all groups — its contribution is **exact**
- **beyond**: keep only the limit-derived top-k — **approximate but
fast**; correct for the skewed distributions top-n queries target (true
heavy hitters survive), and the env lets deployments trade speed for
accuracy (raising it above the per-file cardinality reproduces exact
counts; verified)
## Wire format
`SimpleTopN` now carries `repeated string fields` (wire-compatible with
the former singular `field`: same tag, length-delimited) and
`SimpleTopNMulti` is removed with its oneof tag reserved — it only
existed unreleased since #12348.
## Benchmarks
112.7M rows, 72 files, single node, mimalloc build,
`ZO_QUERY_THREAD_NUM=12`:
Group By | Cardinality | Limit | Parquet | this PR | Winner
-- | -- | -- | -- | -- | --
path | low (16) | 1000 | 259ms | 48ms | 🟢 Tantivy 5.4×
trace_id | high (112M) | 1000 | 2459ms | 245ms | 🟢 Tantivy 10.0×
trace_id | high (112M) | 10000 | 2450ms | 466ms | 🟢 Tantivy 5.3×
trace_id, path | high × low | 1000 | 2385ms | 610ms | 🟢 Tantivy 3.9×
trace_id, path | high × low | 10000 | 2400ms | 899ms | 🟢 Tantivy 2.7×
trace_id, server_name | high × high | 1000 | 2380ms | 723ms | 🟢 Tantivy
3.3×
trace_id, server_name | high × high | 10000 | 2403ms | 1624ms | 🟢
Tantivy 1.5×
Correctness verified against DataFusion (`sum(1)` baseline): exact count
match for every query within the exact regime, including cross-segment
merging and the legacy-file (missing column) guard.
## Notes
- files whose tantivy index predates a field being added to
`index_fields` log a warning and contribute nothing, same convention as
the histogram handlers
- a debug log marks per-file truncation so operators can see when a
result enters the approximate regime
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Adds support to show tentative cost in usage page in cloud
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
This PR add bloom filter for parquet again under a env
`ZO_BLOOM_FILTER_PARQUET_ENABLED=false`, if you want to generate bloom
filter for parquet set it to `true`.
> We no longer use it even you generate it.
## Summary
Makes the compactor's bloom-filter **build side** enterprise-only by
moving it into the `o2-enterprise` crate (see
openobserve/o2-enterprise#1889). The search-side pruner and shared
`infra::bloom` SBBF support are unchanged.
- Delete `src/service/compact/bloom_build.rs` and
`src/service/tantivy/bloom_builder.rs` — they now live in
`o2_enterprise::enterprise::bloom`.
- Gate the `merge.rs` build hook behind `#[cfg(feature =
"enterprise")]`, calling
`o2_enterprise::enterprise::bloom::compact::build_for_stream`. In
non-enterprise builds the hook is a no-op.
- Drop the now-unused `get_tantivy_directory` re-export from
`grpc::storage`.
## Dependency
Requires openobserve/o2-enterprise#1889 (the enterprise build is broken
until that lands).
## Test plan
- [x] `cargo check --bin openobserve` (OSS, default features) —
warning-clean
- [x] `cargo check --bin openobserve --features enterprise` — gated call
site resolves
- [x] `cargo check -p o2_enterprise`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- close https://github.com/openobserve/openobserve/issues/12533
it will improve all case when we use tantivy to get row_id, then give
parquet to execute.
# Performance
400M records, original size 750GB
## Q1
```sql
SELECT
kubernetes_namespace_name,
avg(code) AS cnt
FROM
default
WHERE
match_all('zinc')
GROUP BY
kubernetes_namespace_name ORDER BY cnt DESC LIMIT 10
```
the `zinc` apprear in 140M records, so after match_all('zinc'), tantivy
will return 140M row_ids.
### main
tantivy time: 3s
totol time: 4.5s
### this pr
tantivy time: 95ms
total time: 1.5s
## Summary
- **Cloud visibility**: `list()` in `src/service/db/alerts/templates.rs`
was gating `DEFAULT_ORG` templates behind `!cfg!(feature = \"cloud\")`,
so any non-default org on cloud received an empty template list. Removed
the flag entirely — all `DEFAULT_ORG` templates (prebuilt and custom)
are now visible to every org, matching the existing ENT behaviour. This
is safe because cloud org identifiers are UUID-assigned; no regular user
can create an org named `default`.
- **Prebuilt templates fully read-only for non-root**: previously only
delete was blocked (403); updates were silently allowed. Now both
`save()` and `delete()` return `PrebuiltReadOnly` (403) for non-root
callers. Root users retain full access via `is_root_user()` bypass
(consistent with existing patterns in the codebase). Create with a
reserved name returns `ReservedName` (400) for everyone including root.
- **Frontend hardcoding removed**: `AddDestination.vue` was using a
`DEFAULT_TEMPLATE_SENTINEL` (`"__default__"`) and constructing prebuilt
template names client-side. Dropdown is now populated from the
`isPrebuilt: true` entries returned by `GET /alerts/templates`.
`TemplateList.vue` edit button now matches delete: shown-but-disabled
with `systemReadOnly` tooltip for prebuilt rows.
## Root cause
The `!cfg!(feature = "cloud")` flag in `list()` was added to prevent
`DEFAULT_ORG` data bleeding across cloud tenants. This correctly scopes
custom templates but also hides system prebuilt templates that every org
needs. Since prebuilt templates are now enforced as read-only
(non-root), exposing all DEFAULT_ORG templates to all orgs carries no
cross-tenant mutation risk.
## Test plan
- [ ] Non-default org on cloud: `GET /api/{org}/alerts/templates`
returns `prebuilt_slack`, `prebuilt_email`, etc. with `isPrebuilt: true`
- [ ] Non-root user: `PUT /api/{org}/alerts/templates/prebuilt_slack`
returns 403
- [ ] Non-root user: `DELETE /api/{org}/alerts/templates/prebuilt_slack`
returns 403
- [ ] Root user: `PUT /api/{org}/alerts/templates/prebuilt_slack`
succeeds
- [ ] Root user: `DELETE /api/{org}/alerts/templates/prebuilt_slack`
succeeds
- [ ] Any user: `POST /api/{org}/alerts/templates` with name
`prebuilt_slack` returns 400
- [ ] Destinations page: prebuilt destination template dropdown shows
`prebuilt_<type>` sourced from API
- [ ] `TemplateList`: edit button disabled (not hidden) for prebuilt
rows, with "System Read Only" tooltip
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Subhra264
Abhay Padamaniandgithub-actions[bot]
Shrinath RaoandClaude Opus 4.8
Yashodhan JoshiandYashodhan Joshi
Huaijin
Harsh Mahajan
Loakesh Indiran
udev
Rohith MatamandHengfei Yang
Manan Shukla