- [x] ~add sort when to many groups, we can do this, but for now do not
do this~
- [x] remove the unsafe for memory pool
- [x] add approx_distinct improve, already fork, will add in next pr
- [x] fork vortex to add cast fix, and export
`.exclude_schemes([IntDictScheme.id()])`
- [x] ~enable_physical_uncorrelated_scalar_subquery test~
## Problem
When **"Correlate without service attribute"** (`service_optional =
true`) is enabled, the same physical service can appear as two (or more)
separate entries in **Discovered Services**.
**Root cause:** `ServiceKey` is always `(service_name, set_id,
disambiguation)`. When a log has no `service.name` attribute, the
processor falls back to the stream name (e.g. `repro_logs`) as
`service_name`. A trace for the same workload carries the real
`service.name` (e.g. `ecs-api-service`). Both entries share identical
`(set_id, disambiguation)` but are stored as separate rows — they never
merge.
**What the DB looks like (bug):**
| service_name | set_id | disambiguation | logs_streams | traces_streams
|
|---|---|---|---|---|
| `ecs-api-service` | `kubernetes` | `{"k8s-namespace":"ecs-prod"}` |
`[]` | `["repro_traces"]` |
| `repro_logs` | `kubernetes` | `{"k8s-namespace":"ecs-prod"}` |
`["repro_logs"]` | `[]` |
Two rows. `disambiguation` is byte-for-byte identical. Only
`service_name` differs.
In production this gets worse — ECS metric streams (which also have no
`service.name`) each become their own phantom row under the same
disambiguation. We observed 85 rows for a single ECS service, with only
2 rows carrying actual streams.
## Fix
`GET /api/:org_id/service_streams` now performs a read-time merge when
`service_optional = true`:
1. Fetches raw rows from the DB (unchanged)
2. Loads the org's service identity config (already cached)
3. If `service_optional = true` — groups rows by `(set_id,
disambiguation)`, unions their `logs_streams` / `metrics_streams` /
`traces_streams` arrays, and picks the real service name (the one that
does NOT appear in its own stream arrays) as the display name
4. Returns the merged list
**After fix — what the API returns:**
```json
[{
"service_name": "ecs-api-service",
"set_id": "kubernetes",
"disambiguation": {"k8s-namespace": "ecs-prod"},
"logs_streams": ["repro_logs"],
"traces_streams": ["repro_traces"]
}]
```
One entry. Real service name. All streams unified.
The DB rows are not changed — merge is purely at read time in the
handler. When `service_optional = false` the handler returns raw rows as
before (no change in behaviour).
Fixes#1848
This adds a UI and API based trigger for re triggering license check
limits, so we do not have to restart nodes
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Currently, on the roles edit page, the frontend makes role permission
api for each and every resources. So, for current 41 resources, there
are 41 concurrent api calls, each going through the openfga rbac check
middleware. That means, there are 41 concurrent redundant openfga
checks, and then 41 openfga query to get the role permissions.
This pr adds another api `org_id/roles/role_name/permissions` that
combines all the resource permissions of the given role name and return
the same eliminating the need for making api calls for each rbac
resource.
## Summary
- Preserve canonical/resource-derived trace fields when resource or span
attributes collide with flattened span fields like `service_name`.
- Keep resource `service.name` as the canonical `service_name` while
moving colliding resource fields such as resource `name` to
`service_attr_name`.
- Prefix colliding span attributes with `attr_` before serde flattening.
- Add regression coverage for span/resource `service_name` collisions
and normalized resource-field collisions.
Fixes#12448
## Testing
- `git diff --check`
Not run: cargo test, because this local environment does not have
rustc/cargo installed.
---------
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
Design at: #12620Closes#12620Fixes#12564
## What
Rewrites both tantivy histogram optimize-rule collectors following the
same approach as the recent `TopNCollector` /
`SingleSegmentDocIdCollector` rewrites, and adds timezone support to
both rules.
- `MultiHistogramCollector` (replaces the aggregation framework for
`SimpleMultiHistogram`)
- `SimpleHistogramCollector` (replaces
`tantivy::collector::HistogramCollector` for `SimpleHistogram`)
### Timezone support (#12564)
`histogram(ts, interval, tz)` rewrites to `date_bin(interval,
to_timestamp_micros(_timestamp + offset), origin)`; the optimizer's
`get_column_name` hit the `BinaryExpr` and silently skipped both rules.
Now:
- the visitors extract the fixed offset from that shape
(`get_timestamp_offset`), so the rules fire with a timezone
- `IndexOptimizeMode::SimpleHistogram` / `SimpleMultiHistogram` carry a
`ts_offset` field (no proto change — histogram modes never cross gRPC,
they're re-derived per node)
- bucket edges are computed in shifted wall-clock space and the
collectors bucket on `_timestamp + ts_offset`, so emitted keys match
`date_bin` exactly when merging with non-tantivy partials
## Testing
- New collector unit tests: dense vs sparse equality, ts_offset
shifting, bounds filtering, missing column / missing breakdown docs,
per-bucket truncation with tie-breaks
- Visitor tests now include real-DataFusion-plan timezone cases
(`'+08:00'` simple, `'-05:30'` multi) asserting the extracted shifted
bounds and offset
- `cargo test -p config` (1936 passed), plus `index_optimizer` /
`search::tantivy` / `tantivy_optimize_exec` suites in the main crate;
clippy clean
- Low cardinality: fast `20%` than main
```
histogram(_timestamp, '1h') AS ts, count(*) FROM default GROUP BY ts ORDER BY ts ASC
```
- Low cardinality: fast `30%` than main
```
histgoram(_timestamp, '1h') AS ts, breakdown, count(*) FROM default GROUP BY ts, breakdown ORDER BY ts ASC
```
- High cardinality: faster `5x` than main
```
histgoram(_timestamp, '1h') AS ts, breakdown, count(*) FROM default GROUP BY ts, breakdown ORDER BY ts ASC
```
Close#12345
## Problem
OTLP histogram metrics sent over HTTP/JSON (e.g. otel-collector with
`encoding: json`) created streams but stored 0 events, with no
`_count`/`_sum`/`_bucket` streams. Int-valued counters/gauges were
silently recorded as 0. The protobuf path was unaffected, which is why
this only surfaced for collectors explicitly configured with JSON
encoding.
## Root cause
`otlp_json` deserializes the body directly into opentelemetry-proto's
serde-derived types, which only partially implement the proto3 JSON
mapping (64-bit integers encoded as strings, enums by name, zero/empty
fields omitted). `Metric.data` is a flattened `Option` oneof, and serde
converts any inner deserialization error of a flattened `Option` into
`None` instead of failing — so the whole metric silently became `data:
None`, while the stream itself had already been created by the schema
setup. Same mechanism on `NumberDataPoint.value` turned string `asInt`
values into 0.
## Changes
- Upgrade opentelemetry stack 0.31 → 0.32, which fixes string-encoded
u64 handling for plain histograms and summaries upstream.
- Add `otlp_json_compat` normalizer for the gaps still present in 0.32
(each verified empirically, with unit tests):
- string `asInt` values on data points and exemplars
- `aggregationTemporality` as enum name
- `Exemplar`'s value oneof is not flattened in 0.32
- missing `serde(default)` on
`ExponentialHistogramDataPoint`/`Buckets`/`ValueAtQuantile`/`Exemplar`,
which rejected omitted zero/empty fields
- Count and log metrics whose data oneof failed to decode
(`partial_success.rejected_data_points`) instead of silently skipping
them, and stop overwriting specific partial-success error messages with
the generic retention message.
- Adapt to OTLP 0.32 API additions (`KeyValue.key_strindex`,
`AnyValue::StringValueStrindex`).
- Upgrade workspace reqwest 0.12 → 0.13, required because
opentelemetry-http 0.32 only implements `HttpClient` for reqwest 0.13
(`rustls` feature replaces `rustls-tls-native-roots` via the platform
verifier; `json`/`query`/`form` are now explicit features). The
`segment` crate is pinned to reqwest 0.12, so `TELEMETRY_CLIENT` moved
into `telemetry.rs` as a plain `reqwest::Client` that posts the Track
message itself, replicating `segment::HttpClient::send`.
- Remove the unused pyroscope profiling integration (optional feature,
dependencies, agent setup, `ZO_PROF_PYROSCOPE_*` config).
## Verification
- 10 unit tests in `otlp_json_compat` covering the full payload matrix
(collector-style string integers, enum names, omitted zero fields,
exemplars with and without trace context, already-normalized payloads,
malformed input).
- End-to-end: posted the issue's collector-style JSON payload to
`/api/default/v1/metrics` — `_bucket` (3 events, correct cumulative
counts with exemplars attached), `_count`, `_sum` populated for
histogram and exponential histogram, summary quantiles stored, `asInt`
counter stored as 42 instead of 0.
- `cargo clippy -- -D warnings` clean; lib test failures are
pre-existing local-environment issues that reproduce identically on
main.
## Problem
`get_sre_agent_credentials` early-returned on the `Ok(record)` branch
when the SRE-agent SA DB row already existed, so `ensure_sys_rca_agent`
only ran for orgs whose SA was *missing*. If the startup backfill ever
failed for an org (FGA transient, OpenFGA not yet ready, etc.), there
was no runtime path to retry → every subsequent RCA trigger
short-circuited past the self-heal, the SA's OpenFGA tuples stayed
broken, MCP returned 0 tools, and the agent failed with `Tool
'GetIncident' not found`.
Also bumps the `sre_readonly_afolder_incidents` migration flag (`v1` →
`v2`) so the OpenFGA top-up patch re-runs once on clusters seeded by
older code — required to add the new `mcp:_all_{org}` grants from the
companion PR. Also bumps the reqwest timeout to o2-ai (300s → 600s) —
multi-turn RCA legitimately needs the headroom.
## Fix
Always run `ensure_sys_rca_agent` before the DB lookup. Idempotent, safe
to call every request, self-heals the SA + role state on the next
trigger after a transient backfill failure.
## Pairs with
openobserve/o2-enterprise#1885 — drops the equivalent early-return in
`create_and_assign_sre_readonly_role` and adds the missing `mcp:`
resource grants.
Resolves https://github.com/openobserve/o2-enterprise/issues/1901
Design at: #12596closes#12596
## Problem
`index_updated_at` is a single stream-level timestamp, bumped whenever
any field is added to `full_text_search_keys` or `index_fields`. The
tantivy optimize path (`SimpleCount` / `SimpleHistogram` /
`SimpleMultiHistogram` / `SimpleTopN` / `SimpleTopNMulti` /
`SimpleDistinct`) excludes every file with `min_ts < index_updated_at`,
because optimize results are used directly as final answers with no
DataFusion re-check.
Consequence: adding one index field to a long-retention stream knocks
every histogram/count/top-N over historical data off the fast path until
the cutoff ages out of retention — even when the queried fields have
been indexed since day one.
## Design
### Storage
`StreamSettings` gains `index_fields_updated_at: HashMap<String, i64>`,
recording when each field was added to `full_text_search_keys` or
`index_fields`:
- A field add stamps only genuinely-new fields. A no-op re-add of an
existing field no longer bumps anything (previously it pointlessly
bumped the stream-level value and disabled the index for all history).
- The stream-level `index_updated_at` is still bumped on every real add,
so older queriers in a mixed-version cluster keep the old conservative
behavior during rolling upgrades.
- A field removal prunes its entry (in `normalize_stream_settings`), so
a later re-add stamps a fresh time — files written in the gap lack that
field's index, and a single timestamp can't represent a gap.
- Bloom-only fields auto-folded into `index_fields` by normalization are
stamped like a normal add.
- The `reset-index-updated-at` CLI clears the map, since per-field
entries override the scalar.
- The empty map is skipped during serialization; legacy settings without
the key deserialize to an empty map. Fully backward/forward compatible,
no migration.
A field present in both `full_text_search_keys` and `index_fields`
shares one entry (last add wins). That is over-conservative for at most
one of the two uses and only in the rare case where the same field is
added to both lists at different times.
### Query
The cutoff in `flight.rs` is now computed from the fields the query
actually reads from the index — the optimize-rule target fields
(`SimpleTopN` / `SimpleTopNMulti` / `SimpleMultiHistogram` /
`SimpleDistinct`):
```
cutoff = max over referenced fields of:
BASE_TIME if the field is a default-enabled index field
else the per-field entry, else stream-level index_updated_at, else stream created_at
```
- Default-enabled index fields (`SQL_SECONDARY_INDEX_SEARCH_FIELDS`:
`trace_id`, `service_name`, `operation_name`, plus
`feature_secondary_index_extra_fields`) are indexed since the stream's
first file, so they always resolve to `BASE_TIME` and never constrain
the cutoff — even if they also carry a per-field entry.
- Index-condition fields (equality and FTS/`match_all`) are deliberately
**not** part of the cutoff: each file's tantivy schema is the per-file
source of truth there, so the timestamps add nothing.
- A query that references no per-stream fields — e.g. a count or
histogram without filters — gets cutoff `0`: its tantivy result only
depends on data every `.ttv` file has. The existing global
`ttv_timestamp_updated_at` / `ttv_secondary_index_updated_at` checks are
unchanged and still apply on top. This also covers the comment on #12596
that pure histograms don't need the `index_updated_at` check — it falls
out of the rule instead of being special-cased.
The regular filter-pushdown path is untouched: it never used
`index_updated_at` (it checks the actual per-file tantivy schema and
adds the filter back to DataFusion when a field is missing).
### Example
Stream created 2024-01 (so `trace_id` etc. indexed since then),
`user_id` added to `index_fields` on 2026-06-01:
| Query | Before | After |
|---|---|---|
| top-N on `service_name` (default field) | index only for files after
2026-06-01 | all indexed files (`BASE_TIME`) |
| top-N / distinct on `user_id` | files after 2026-06-01 | files after
2026-06-01 (correct: older files lack the `user_id` index) |
| `histogram(_timestamp)` / `count(*)` | files after 2026-06-01 | all
indexed files |
## Intentionally out of scope
The cutoff still conflates data time with index build time: a file the
compactor merges *after* a settings change carries a fresh, complete
index but keeps its old `min_ts`, so it stays off the optimize path.
Fixing that requires fetching `created_at` per file from file_list — one
more column over millions of rows on every query — to rescue roughly one
compaction cycle of data around the settings change, which self-heals
within the next hour. Not worth it.
## Summary
- Super cluster leader was hardcoding `0` for `took_wait_in_queue` in
all search responses
- Added `wait_in_queue` field to `ScanStats` proto and struct
- Follower now propagates its queue wait time via `ScanStats`
- Leader reads aggregated value (`max` across followers) instead of `0`
Single-cluster deployments were unaffected.
## Summary
Rewrites the tantivy `GROUP BY ... count(*) ... ORDER BY cnt LIMIT n`
optimization around a custom flat collector over columnar term ordinals,
replacing the nested `TermsAggregation` approach from #12348 that was
disabled in #12528 for performance. One unified `SimpleTopN` path now
serves **1..=4 group by fields** at every layer (SQL matcher, internal
enum, wire protocol, collector, result types).
Closes the performance issue documented in
https://github.com/openobserve/openobserve/pull/12348#issuecomment-4659231020
and re-enables multi-field group by.
## Design
**`TopNCollector`** (`config::tantivy::query::topn_collector`), per
file:
1. read each group field's term ordinal per doc, fetched in blocks via
`Column::first_vals` (vectorized for full columns)
2. count into a **dense per-ordinal array** indexed by the mixed-radix
ordinal when the fields' dictionary-size product fits 4M cells (no
hashing — ordinals are dense), falling back to a hash map keyed by
bit-packed ordinals (`u64` for 1-2 fields, `u128` for 3-4) beyond that
3. keep groups per the exactness rule below, selecting top-k with a
bounded heap directly off the counting array (no full materialization)
4. resolve only the surviving ordinals to strings, in one sorted forward
pass per field via `Dictionary::sorted_ords_to_term_cb` (point lookups
with `ord_to_term` re-scan a dictionary block per call and were the
dominant cost)
DataFusion re-aggregates the per-file partials and applies the final
ORDER BY + LIMIT.
## Exactness contract
Per-file top-K partials are only exact when nothing is truncated; once
truncation kicks in, a group's total can be spread thin across files and
the merged top-N is approximate. Measured on near-uniform data, no
practical keep size restores accuracy (keeping 100k of 1.4M groups per
file still matched 0/5 of the true top-5), so the rule is bimodal,
decided per file at harvest where the distinct count is known for free:
- **distinct groups <= max(limit-derived top-k,
`ZO_INVERTED_INDEX_TOPN_MAX_GROUP_NUM`)** (env default 1000): the file
returns all groups — its contribution is **exact**
- **beyond**: keep only the limit-derived top-k — **approximate but
fast**; correct for the skewed distributions top-n queries target (true
heavy hitters survive), and the env lets deployments trade speed for
accuracy (raising it above the per-file cardinality reproduces exact
counts; verified)
## Wire format
`SimpleTopN` now carries `repeated string fields` (wire-compatible with
the former singular `field`: same tag, length-delimited) and
`SimpleTopNMulti` is removed with its oneof tag reserved — it only
existed unreleased since #12348.
## Benchmarks
112.7M rows, 72 files, single node, mimalloc build,
`ZO_QUERY_THREAD_NUM=12`:
Group By | Cardinality | Limit | Parquet | this PR | Winner
-- | -- | -- | -- | -- | --
path | low (16) | 1000 | 259ms | 48ms | 🟢 Tantivy 5.4×
trace_id | high (112M) | 1000 | 2459ms | 245ms | 🟢 Tantivy 10.0×
trace_id | high (112M) | 10000 | 2450ms | 466ms | 🟢 Tantivy 5.3×
trace_id, path | high × low | 1000 | 2385ms | 610ms | 🟢 Tantivy 3.9×
trace_id, path | high × low | 10000 | 2400ms | 899ms | 🟢 Tantivy 2.7×
trace_id, server_name | high × high | 1000 | 2380ms | 723ms | 🟢 Tantivy
3.3×
trace_id, server_name | high × high | 10000 | 2403ms | 1624ms | 🟢
Tantivy 1.5×
Correctness verified against DataFusion (`sum(1)` baseline): exact count
match for every query within the exact regime, including cross-segment
merging and the legacy-file (missing column) guard.
## Notes
- files whose tantivy index predates a field being added to
`index_fields` log a warning and contribute nothing, same convention as
the histogram handlers
- a debug log marks per-file truncation so operators can see when a
result enters the approximate regime
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Adds support to show tentative cost in usage page in cloud
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
This PR add bloom filter for parquet again under a env
`ZO_BLOOM_FILTER_PARQUET_ENABLED=false`, if you want to generate bloom
filter for parquet set it to `true`.
> We no longer use it even you generate it.
## Summary
Makes the compactor's bloom-filter **build side** enterprise-only by
moving it into the `o2-enterprise` crate (see
openobserve/o2-enterprise#1889). The search-side pruner and shared
`infra::bloom` SBBF support are unchanged.
- Delete `src/service/compact/bloom_build.rs` and
`src/service/tantivy/bloom_builder.rs` — they now live in
`o2_enterprise::enterprise::bloom`.
- Gate the `merge.rs` build hook behind `#[cfg(feature =
"enterprise")]`, calling
`o2_enterprise::enterprise::bloom::compact::build_for_stream`. In
non-enterprise builds the hook is a no-op.
- Drop the now-unused `get_tantivy_directory` re-export from
`grpc::storage`.
## Dependency
Requires openobserve/o2-enterprise#1889 (the enterprise build is broken
until that lands).
## Test plan
- [x] `cargo check --bin openobserve` (OSS, default features) —
warning-clean
- [x] `cargo check --bin openobserve --features enterprise` — gated call
site resolves
- [x] `cargo check -p o2_enterprise`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- close https://github.com/openobserve/openobserve/issues/12533
it will improve all case when we use tantivy to get row_id, then give
parquet to execute.
# Performance
400M records, original size 750GB
## Q1
```sql
SELECT
kubernetes_namespace_name,
avg(code) AS cnt
FROM
default
WHERE
match_all('zinc')
GROUP BY
kubernetes_namespace_name ORDER BY cnt DESC LIMIT 10
```
the `zinc` apprear in 140M records, so after match_all('zinc'), tantivy
will return 140M row_ids.
### main
tantivy time: 3s
totol time: 4.5s
### this pr
tantivy time: 95ms
total time: 1.5s
## Summary
- **Cloud visibility**: `list()` in `src/service/db/alerts/templates.rs`
was gating `DEFAULT_ORG` templates behind `!cfg!(feature = \"cloud\")`,
so any non-default org on cloud received an empty template list. Removed
the flag entirely — all `DEFAULT_ORG` templates (prebuilt and custom)
are now visible to every org, matching the existing ENT behaviour. This
is safe because cloud org identifiers are UUID-assigned; no regular user
can create an org named `default`.
- **Prebuilt templates fully read-only for non-root**: previously only
delete was blocked (403); updates were silently allowed. Now both
`save()` and `delete()` return `PrebuiltReadOnly` (403) for non-root
callers. Root users retain full access via `is_root_user()` bypass
(consistent with existing patterns in the codebase). Create with a
reserved name returns `ReservedName` (400) for everyone including root.
- **Frontend hardcoding removed**: `AddDestination.vue` was using a
`DEFAULT_TEMPLATE_SENTINEL` (`"__default__"`) and constructing prebuilt
template names client-side. Dropdown is now populated from the
`isPrebuilt: true` entries returned by `GET /alerts/templates`.
`TemplateList.vue` edit button now matches delete: shown-but-disabled
with `systemReadOnly` tooltip for prebuilt rows.
## Root cause
The `!cfg!(feature = "cloud")` flag in `list()` was added to prevent
`DEFAULT_ORG` data bleeding across cloud tenants. This correctly scopes
custom templates but also hides system prebuilt templates that every org
needs. Since prebuilt templates are now enforced as read-only
(non-root), exposing all DEFAULT_ORG templates to all orgs carries no
cross-tenant mutation risk.
## Test plan
- [ ] Non-default org on cloud: `GET /api/{org}/alerts/templates`
returns `prebuilt_slack`, `prebuilt_email`, etc. with `isPrebuilt: true`
- [ ] Non-root user: `PUT /api/{org}/alerts/templates/prebuilt_slack`
returns 403
- [ ] Non-root user: `DELETE /api/{org}/alerts/templates/prebuilt_slack`
returns 403
- [ ] Root user: `PUT /api/{org}/alerts/templates/prebuilt_slack`
succeeds
- [ ] Root user: `DELETE /api/{org}/alerts/templates/prebuilt_slack`
succeeds
- [ ] Any user: `POST /api/{org}/alerts/templates` with name
`prebuilt_slack` returns 400
- [ ] Destinations page: prebuilt destination template dropdown shows
`prebuilt_<type>` sourced from API
- [ ] `TemplateList`: edit button disabled (not hidden) for prebuilt
rows, with "System Read Only" tooltip
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
## Problem
Closes#12491 — a quick refresh (several queries in a row) occasionally
results in `wait in queue` for ~**10s**, then a retry works.
Intermittent, not frequent.
## Root cause
The search-queue dist-lock (`/search/cluster_queue/<work_group>`,
`work_group.rs`) is a NATS KV key whose value embeds an expiration of
`now + LOCKER_WATCHER_UPDATE_TTL` (**10s**). A detached **keep-alive
task** re-`put`s a fresh `now + 10s` value every 3s (`keep_alive_lock`).
`unlock()` did:
```
state = 2 -> tx.send(()) (signal keep-alive to stop) -> bucket.purge(key)
```
but the stop signal and the purge were **not synchronized** with the
keep-alive task. A keep-alive `put` that was already in flight (or fired
from the same `select!` tick as the stop signal) could land **after**
`purge`, recreating the key with a 10s-future expiration.
Because the locker bucket TTL is `lock_max_age` (2h), nothing deletes
that orphan. The next waiter spins in `wait_for_delete` — no delete
event ever comes, and the 1s fallback `check_exist_lock` keeps seeing
"not expired" — until the embedded **10s** expiration passes and
`check_exist_lock` finally purges it. Hence the exact 10s wait.
### Why it's intermittent and tied to "quick refresh"
The keep-alive only `put`s if the lock is held **≥3s**. That happens
under contention: in enterprise the dist-lock is held across the whole
`work_group_need_wait` polling loop, and in OSS for the query duration.
Quick refresh → many concurrent queries → work-group saturated → lock
held >3s → keep-alive fires → narrow race window (≈ put-latency / 3s) →
orphan that clears itself after 10s ("retry works").
## Fix
- `unlock()` now awaits the keep-alive task's `JoinHandle` **before**
`purge`, so any in-flight `put` completes first and the purge is
guaranteed to be the last write.
- The keep-alive `select!` is `biased` toward the stop signal and
re-checks the lock `state` before each `put`, so it never issues a `put`
once unlock has begun.
## Testing
- `cargo check` / `cargo clippy` / `cargo fmt` on `infra` pass.
- The race is timing-dependent and needs a live NATS server, so it isn't
covered by a deterministic unit test; the reasoning above and the
existing locker unit tests stand.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Inferred services for traces
Design at: #12515
Uninstrumented dependencies (databases, redis, kafka, external APIs) are
invisible in trace views today because OpenObserve only surfaces the
instrumented `service_name`. Users ask "why can't you see we call the db
or redis?". This adds **inferred services**: we derive a minimal
identity for those dependencies from CLIENT/PRODUCER span peer
attributes and surface them in the traces list and the service map.
### What's included
- **Ingestion** — derive three fields on CLIENT(3)/PRODUCER(4) spans
following OTel semantic conventions (current + legacy names):
`infer_service_name`, `infer_service_type`
(`database`/`queue`/`rpc`/`external`), `infer_service_system` (e.g.
`postgresql`/`kafka`/`grpc`/`http`). `peer.service` wins naming; ports
are stripped and bare IPs redacted to avoid one node per pod IP. Wired
into both `handle_otlp_request` and `ingest_json`; the three field names
are added to `BLOCK_FIELDS` so a user attribute of the same name can't
shadow them.
- **Traces list** — the per-service time/cost breakdown now groups by
`COALESCE(infer_service_name, service_name)` (schema-gated) and returns
`service_type`, so dependencies appear in the distribution and the UI
can render them dotted.
- **Service graph** — a second, schema-gated SQL over CLIENT/PRODUCER
spans (anti-joined against instrumented services so a dep that *is*
instrumented elsewhere isn't double-counted) emits inferred edges with
`connection_type`; `ServiceNode.service_type` /
`ServiceEdge.connection_type` carry the inferred category for dotted
nodes/edges. The dependency has no server span, so its latency is the
client span's own duration.
### Backward compatibility
Every new SQL path is gated on the field's presence in the stream
schema, so streams that predate the feature (or never call an
uninstrumented dependency) generate byte-identical SQL — no "field not
found". Old parquet reads the field as NULL → `COALESCE` falls back to
`service_name` gracefully.
### Companion PR
Service-graph topology tagging (`build_topology`) lives in the
enterprise repo: **openobserve/o2-enterprise#1876**. Build/merge the two
together.
### Testing
- `cargo check` / clippy / fmt clean
- unit tests: inferred derivation (20), traces-list breakdown
serialization, config `service_graph` types; full traces suite (417)
passes
- verified end-to-end locally: an OTLP trace with a redis CLIENT span
shows up in `GET /traces/latest` as `service_type: database` and as a
dotted node on the service map
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes#12461
## Problem
PromQL queries in a super-cluster returned data **only from the hub
cluster** — remote (spoke) clusters were silently skipped. SQL queries
across the same super-cluster worked correctly.
**Root cause:** The `data` gRPC handler on the spoke never registered
the incoming `trace_id` in `SEARCH_SERVER`. When the PromQL engine
called `insert_sender(trace_id)` to attach its abort channel, it found
no entry and immediately returned `SearchCancelQuery` (error code
20009). The spoke returned empty data with no visible error — the hub
received nothing from the spoke and returned only its local results.
The `query` gRPC handler already had the correct registration. The
`data` handler was missing it.
## Fix
Before spawning the data fetch task on the spoke, the `data` gRPC
handler now registers the incoming `trace_id` in `SEARCH_SERVER` as a
follower task — exactly the same registration the `query` handler
already performed. This allows the PromQL engine to successfully attach
its abort sender and proceed with execution. After the task completes,
the trace_id is removed from `SEARCH_SERVER` to avoid memory leaks.
## Test Queries
Ingest a metric with a distinguishing label to both hub and spoke, wait
~30s for WAL flush, then run:
**PromQL — before fix:** only hub cluster returned, spoke missing
**PromQL — after fix:** both hub and spoke returned
```bash
curl -s "http://<hub-host>/api/<org>/prometheus/api/v1/query_range" \
-u '<user>:<pass>' \
--data-urlencode "query=sum by (cluster) (total_payment_req)" \
--data-urlencode "start=$(($(date +%s) - 3600))" \
--data-urlencode "end=$(date +%s)" \
--data-urlencode "step=60"
```
**SQL — works both before and after (baseline check):**
```bash
curl -s -X POST "http://<hub-host>/api/<org>/_search?type=metrics" \
-H "Authorization: Basic <token>" \
-H "Content-Type: application/json" \
-d '{
"query": {
"sql": "SELECT cluster, SUM(value) as total FROM \"total_payment_req\" GROUP BY cluster",
"start_time": '"$(($(date +%s) - 3600))"'000000,
"end_time": '"$(date +%s)"'000000,
"size": 10
}
}'
```
- close https://github.com/openobserve/o2-enterprise/issues/1865
## TODO
- [x] make split_batch_for_grpc_response based on the real memory sizew
- [x] coalesce and gc view before send to leader
```
SELECT trace_id, count(*) as cnt, max(_timestamp) as ts FROM "k8s_logs" group by trace_id order by cnt desc limit 10
utf8view enable:
main: 30s (15k batches send to leader)
this branch: 15s (1.5k batches send to leader)
utf8view disable:
main: 13s (42k batches send to leader)
this branch: 12s (1.5k batches send to leader)
```
one found is the AggregateExec's time_calculating_group_ids, utf8 is
faster than utf8view
## Summary
Closes#12426.
In cloud-native deployments (ECS/Fargate, Kubernetes) secrets managers
inject individual values, not composed strings. That makes it impossible
to cleanly provide a full `ZO_META_POSTGRES_DSN` without custom tooling
(init containers, Lambda functions, etc.).
This PR adds five individual connection vars as an alternative:
| Variable | Default |
|---|---|
| `ZO_META_POSTGRES_HOST` | `""` |
| `ZO_META_POSTGRES_PORT` | `5432` |
| `ZO_META_POSTGRES_USER` | `""` |
| `ZO_META_POSTGRES_PASSWORD` | `""` |
| `ZO_META_POSTGRES_DBNAME` | `""` |
**`ZO_META_POSTGRES_DSN` takes full precedence when set, no breaking
change.**
When the DSN is empty, `check_common_config()` composes it from the
individual vars at startup. User, password and dbname are
percent-encoded so credentials with special characters (common with
secrets-manager generated passwords) survive sqlx's percent-decoding
when the DSN is parsed. The connection code itself is untouched, every
consumer of `meta_postgres_dsn` (including the migration adapter) just
sees a normal DSN.
## Changes
- `src/config/src/config.rs`: adds 5 new `env_config` fields, validates
that either the DSN or all individual vars are set, and composes the DSN
from the individual vars when it is empty
---------
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
## Problem
`query_by_ids` runs `SELECT ... WHERE id IN (...)` with **no partition
key filter**. On the partitioned `file_list` table this means:
- **planning** must open and lock **every** partition (~101ms on 384
partitions)
- **execution** probes **every** partition's id index (~10ms)
Measured on production-scale data (2.7B rows / 1.2TB / 384 daily
partitions, PostgreSQL 17): **~110ms per call**, vs ~1.5ms on a
non-partitioned table. `query_by_ids` is on the search hot path (id →
FileKey reverse lookup), so every search pays this cost.
## Fix
The ids passed to `query_by_ids` are produced by `query_ids` **using a
known time range** — the search path already holds it
(`service::file_list::query_by_ids` already takes `time_range` but did
not pass it down). This PR passes the range down to the infra layer and
derives a `date` filter via the existing `derive_date_range`, enabling
plan-time partition pruning:
```sql
SELECT ... FROM file_list WHERE id IN (...)
AND date >= '2026/05/06/00' AND date < '2026/05/14/00' -- new
```
## Benchmark
From the file_list partitioning performance test (2.7B rows, 384 daily
partitions, c6i.8xlarge, PG 17.10,
jit=off/work_mem=128MB/rpc=1.1/ANALYZEd). Same in-window batch of 1000
ids, median of 3 runs, total = execution + planning:
| Query window | Before (total) | After (total) | Improvement |
Partitions scanned |
|---|---:|---:|---|---|
| 1h | 112.7ms (11.7 + 101.0) | **1.18ms** (0.82 + 0.36) | **95x** | 384
→ 1 |
| 1d | 110.3ms (9.4 + 100.9) | **1.60ms** (0.97 + 0.62) | **69x** | 384
→ 2 |
| 7d | 109.4ms (8.4 + 100.9) | **2.93ms** (0.79 + 2.14) | **37x** | 384
→ 8 |
With the filter, the partitioned table matches or **beats** the
non-partitioned baseline (execution 0.8~1.0ms vs 1.2~1.3ms — per-day
partition id indexes are shallower than one monolithic 2.7B-row index).
## Safety
- **Semantically safe**: the ids were selected by `query_ids` with
exactly the same derived date range, so the added filter cannot drop
rows.
- **No side effect on non-partitioned tables**: measured 1.5ms → 1.7ms
(within noise) on the single-table baseline.
- **Guarded**: the filter is only applied when `time_range` is
`Some((start, end))` with `start > 0 && end >= start`; all other callers
pass `None` and keep the previous behavior.
- SQLite backend ignores the parameter (not partitioned; already a
single index probe).
## Changes
- `infra::file_list::FileList::query_by_ids` trait: add `time_range:
Option<(i64, i64)>`
- postgres impl: derive and append the date filter when a valid range is
given
- sqlite impl: accept and ignore the parameter
- `service::file_list::query_by_ids`: pass its existing `time_range`
down (db + local cache)
- `cli::basic::bloom`: pass `None`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- close https://github.com/openobserve/openobserve/issues/11956
- [x] check how PartitialReduce work with AggregateTopk
- [x] check how PartitialReduce work with StreamingAgg
# New Env
```
ZO_FEATURE_PARTIAL_REDUCE_ENABLED=true (default is true)
```
# Performance
## First datasets
80M records
single node optimzier disable(to simulate the distribute query)
### Q1
```sql
SELECT count(distinct request_id) FROM "k8s_logs"
```
80M request_id unique value
with partial reduce: 61s (80M records send from follower to leader)
without partial reduce: 54s (80M records send from follower to leader)
### Q2
```sql
SELECT count(distinct client_ip) FROM "k8s_logs"
```
10M client_ip unique request_id
with partial reduce: 5s (10M records send from follower to leader)
without partial reduce: 54s (80M records send from follower to leader)
### Q3
```sql
SELECT
host,
COUNT(DISTINCT client_ip) AS cnt
FROM
k8s_logs
GROUP BY
host ORDER BY cnt DESC
```
2000 host unique value, 10M client_ip unique value
with partial reduce: 41s (75M records send from follower to leader)
without partial reduce: 26s (80M records send from follower to leader)
### Q4
```sql
SELECT
kubernetes_namespace_name,
COUNT(DISTINCT client_ip) AS cnt
FROM
k8s_logs
GROUP BY
kubernetes_namespace_name ORDER BY cnt DESC
```
10 kubernetes_namespace_name unique value, 10M client_ip unique value
with partial reduce: 14s (38M records send from follower to leader)
without partial reduce: 17s (80M records send from follower to leader)
### Q5
```sql
SELECT
histogram(_timestamp, '5 minute') AS ts,
host,
approx_distinct(client_ip) AS cnt,
ROW_NUMBER() OVER (
PARTITION BY histogram(_timestamp, '5 minute')
ORDER BY approx_distinct(client_ip) DESC
) AS rn
FROM
k8s_logs
GROUP BY
ts, host
QUALIFY rn <= 10
ORDER BY ts DESC, cnt DESC
```
2000 host unique value, 10M client_ip unique value
with partial reduce: 3.2s (15k records send from follower to leader)
without partial reduce: 4.2s (46k records send from follower to leader)
## Second datasets
100M records, 250GB original size, 37GB compression size
### Q6
```sql
SELECT
client_ip,
count(*) AS cnt,
max(_timestamp) AS ts
FROM
"k8s_logs"
GROUP BY
client_ip ORDER BY cnt DESC LIMIT 100
```
0.5M client ip unique value
with partial reduce: 5s (517k records send from follower to leader)
without partial reduce: 19s (100M records send from follower to leader)
### Q7
```sql
SELECT
trace_id,
count(*) AS cnt,
max(_timestamp) AS ts
FROM
"k8s_logs"
GROUP BY
trace_id ORDER BY cnt DESC LIMIT 100
```
12M client ip unique value
with partial reduce: 13s (12M records send from follower to leader)
without partial reduce: 23s (100M records send from follower to leader)
**for streaming aggreagte cache genearte**
with partitial reduce first run: 23s
without partial reduce first run: 37s
# Design
## Background
OpenObserve uses a distributed query architecture where a **leader
(querier)** node dispatches physical execution plans to **follower
(ingester/compactor)** nodes via Arrow Flight RPC. The `RemoteScanExec`
node represents work pushed to remote followers; results stream back to
the leader for final processing.
For aggregation queries, DataFusion produces a two-phase plan:
1. **`Partial`** — each follower computes partial aggregates (partial
SUM, COUNT, group keys, etc.)
2. **`Final`** — the leader merges all partial results into the final
answer
**Problem:** All partial aggregate rows from every follower are sent to
the leader unconditionally. When there are many unique group-by keys or
large datasets, this generates significant network traffic even when
many follower partitions produce overlapping keys that could be
pre-merged locally.
---
## Solution
DataFusion supports a third aggregate mode, `PartialReduce`, which sits
between `Partial` and `Final`. It takes the output of a `Partial`
aggregate and applies a further reduction pass — merging rows with the
same group keys — before they leave the node.
Insert a `PartialReduce` node on the **follower side** (between the
`Partial` aggregate and the `RemoteScanExec`). Followers send fewer rows
over the network; the leader's `Final` phase still produces a correct
result.
---
## Execution Plan Transformation
**Before:**
```
FinalAggregateExec (leader)
└─ RepartitionExec (leader)
└─ RemoteScanExec (network — all partial rows)
└─ PartialAggregateExec (follower, N partitions)
└─ ScanExec
```
**After:**
```
FinalAggregateExec (leader)
└─ RepartitionExec (leader)
└─ RemoteScanExec (network — reduced rows)
└─ PartialReduceAggregateExec (follower)
└─ CoalescePartitionsExec (follower — merges N partitions into 1)
└─ PartialAggregateExec (follower, N partitions)
└─ ScanExec
```
`CoalescePartitionsExec` merges all follower partitions before the
`PartialReduce` pass to maximize the reduction ratio.
---
## Trade-offs
**Pros:**
- Reduces network bytes transferred from followers to leader for
aggregation queries
- Zero correctness risk for non-aggregate queries (passthrough)
- Runtime feature flag, safe to disable without redeployment
**Cons:**
- Adds a CPU pass on followers; worthwhile only when network is the
bottleneck
- Queries with high-cardinality GROUP BY and no key overlap between
partitions get no reduction, only extra follower CPU
- `CoalescePartitionsExec` serializes follower partitions before the
reduce pass, reducing follower-side parallelism
## Issue
When `_search_multi_stream` processes a histogram query, streaming
execution splits the time range into partitions. Each partition calls
`do_search()` with its own (smaller) time range, causing
`Sql::new_with_options()` to recompute `histogram_interval` via
`generate_histogram_interval()` based on the **partition's** time range
duration instead of the **original query's** time range.
Since different partitions have different time range durations, each
gets a different auto-generated histogram interval (e.g. 10s for a small
partition, 30s for the full range). This produces inconsistent bucket
boundaries across partitions in the streaming response.
The non-streaming `_search_multi` endpoint does not have this issue
because it searches the entire time range as a single unit.
### Root cause
In `src/service/search/streaming/mod.rs`,
`process_search_stream_request()` enters either:
- the **cache path**: `handle_cache_responses_and_deltas()` →
`process_delta()` → per-partition `do_search()`
- the **no-cache path**: `do_partitioned_search()` → per-partition
`do_search()`
In both paths, each partition clones `req` with a narrowed
`start_time`/`end_time`, so when `do_search()` creates a fresh `Sql`
struct, `histogram_interval` is 0 and `Sql::new_with_options()`
recomputes it from the partition's (shorter) time range.
### Fix
Before either path is entered, parse the SQL once with
`HistogramIntervalVisitor` using the **full query time range** and set
the resulting interval on `req.query.histogram_interval`. All subsequent
partition clones inherit this value, and `Sql::new_with_options()` takes
the `query.histogram_interval > 0` branch — using the preset value
instead of recomputing from the partition range.
## Summary
Fixes#11904 — streaming search kept running on the queriers even after
the user closed the browser tab.
The SSE search endpoints spawn the query in a detached task and stream
results back over `text/event-stream`. When the client disconnects, the
response stream is dropped and we stop *sending* results, but the
distributed query continued executing on the queriers, wasting resources
on results nobody will read.
This wires client disconnect into the existing query-cancellation
machinery.
## What changed
- **`SearchStreamGuard`** (`search/utils.rs`) — moved into the response
stream's closure so it shares the stream's lifetime. It marks itself
finished when a terminal event (`Done` / `Error` / `Cancelled`) is
produced; if the stream is dropped before that (client gone), its `Drop`
fires the cancel for the parent `trace_id`.
- **`cancel_query_internal`** (`search/query_manager.rs`) — best-effort
cancel for internal callers (no HTTP response), reusing the existing
super_cluster/regular split.
- Wired the guard into **`search_http2_stream`**,
**`values_http2_stream`**, and **`search_multi_stream`**.
Cancellation is keyed by the parent `trace_id`; the query manager
removes entries by prefix, so multi-stream sub-queries
(`{trace_id}-q{n}`) are covered by a single cancel.
## Notes
- Distributed cancellation is an **enterprise** feature (the whole
`cancel_query` / `QueryManager` path is enterprise-gated). On the
open-source build the guard only logs the disconnect; OSS already stops
sending once the channel closes.
- Normal completion never triggers a cancel (terminal-event →
`mark_finished`). Worst case on an edge race is a harmless no-op cancel
for an already-finished `trace_id`.
- Complements the existing manual "Cancel" button — same underlying
machinery, triggered automatically on disconnect.
This branch also includes an unrelated config tweak (`fix: enable
compact fast mode for local`) enabling compact `fast_mode` automatically
when running in local mode.
## Testing
- `cargo check` (OSS) and `cargo check --features enterprise` both clean
- Unit test for the guard state machine passes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Hengfei Yang
Huaijin
Subhra264
Loakesh Indiran
udev
Rohith MatamandHengfei Yang
Harsh Mahajan
Manan Shukla
Abhay PadamaniandHengfei Yang
mac119
Derk SchooltinkandHengfei Yang