## Problem
Closes#12491 — a quick refresh (several queries in a row) occasionally
results in `wait in queue` for ~**10s**, then a retry works.
Intermittent, not frequent.
## Root cause
The search-queue dist-lock (`/search/cluster_queue/<work_group>`,
`work_group.rs`) is a NATS KV key whose value embeds an expiration of
`now + LOCKER_WATCHER_UPDATE_TTL` (**10s**). A detached **keep-alive
task** re-`put`s a fresh `now + 10s` value every 3s (`keep_alive_lock`).
`unlock()` did:
```
state = 2 -> tx.send(()) (signal keep-alive to stop) -> bucket.purge(key)
```
but the stop signal and the purge were **not synchronized** with the
keep-alive task. A keep-alive `put` that was already in flight (or fired
from the same `select!` tick as the stop signal) could land **after**
`purge`, recreating the key with a 10s-future expiration.
Because the locker bucket TTL is `lock_max_age` (2h), nothing deletes
that orphan. The next waiter spins in `wait_for_delete` — no delete
event ever comes, and the 1s fallback `check_exist_lock` keeps seeing
"not expired" — until the embedded **10s** expiration passes and
`check_exist_lock` finally purges it. Hence the exact 10s wait.
### Why it's intermittent and tied to "quick refresh"
The keep-alive only `put`s if the lock is held **≥3s**. That happens
under contention: in enterprise the dist-lock is held across the whole
`work_group_need_wait` polling loop, and in OSS for the query duration.
Quick refresh → many concurrent queries → work-group saturated → lock
held >3s → keep-alive fires → narrow race window (≈ put-latency / 3s) →
orphan that clears itself after 10s ("retry works").
## Fix
- `unlock()` now awaits the keep-alive task's `JoinHandle` **before**
`purge`, so any in-flight `put` completes first and the purge is
guaranteed to be the last write.
- The keep-alive `select!` is `biased` toward the stop signal and
re-checks the lock `state` before each `put`, so it never issues a `put`
once unlock has begun.
## Testing
- `cargo check` / `cargo clippy` / `cargo fmt` on `infra` pass.
- The race is timing-dependent and needs a live NATS server, so it isn't
covered by a deterministic unit test; the reasoning above and the
existing locker unit tests stand.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Inferred services for traces
Design at: #12515
Uninstrumented dependencies (databases, redis, kafka, external APIs) are
invisible in trace views today because OpenObserve only surfaces the
instrumented `service_name`. Users ask "why can't you see we call the db
or redis?". This adds **inferred services**: we derive a minimal
identity for those dependencies from CLIENT/PRODUCER span peer
attributes and surface them in the traces list and the service map.
### What's included
- **Ingestion** — derive three fields on CLIENT(3)/PRODUCER(4) spans
following OTel semantic conventions (current + legacy names):
`infer_service_name`, `infer_service_type`
(`database`/`queue`/`rpc`/`external`), `infer_service_system` (e.g.
`postgresql`/`kafka`/`grpc`/`http`). `peer.service` wins naming; ports
are stripped and bare IPs redacted to avoid one node per pod IP. Wired
into both `handle_otlp_request` and `ingest_json`; the three field names
are added to `BLOCK_FIELDS` so a user attribute of the same name can't
shadow them.
- **Traces list** — the per-service time/cost breakdown now groups by
`COALESCE(infer_service_name, service_name)` (schema-gated) and returns
`service_type`, so dependencies appear in the distribution and the UI
can render them dotted.
- **Service graph** — a second, schema-gated SQL over CLIENT/PRODUCER
spans (anti-joined against instrumented services so a dep that *is*
instrumented elsewhere isn't double-counted) emits inferred edges with
`connection_type`; `ServiceNode.service_type` /
`ServiceEdge.connection_type` carry the inferred category for dotted
nodes/edges. The dependency has no server span, so its latency is the
client span's own duration.
### Backward compatibility
Every new SQL path is gated on the field's presence in the stream
schema, so streams that predate the feature (or never call an
uninstrumented dependency) generate byte-identical SQL — no "field not
found". Old parquet reads the field as NULL → `COALESCE` falls back to
`service_name` gracefully.
### Companion PR
Service-graph topology tagging (`build_topology`) lives in the
enterprise repo: **openobserve/o2-enterprise#1876**. Build/merge the two
together.
### Testing
- `cargo check` / clippy / fmt clean
- unit tests: inferred derivation (20), traces-list breakdown
serialization, config `service_graph` types; full traces suite (417)
passes
- verified end-to-end locally: an OTLP trace with a redis CLIENT span
shows up in `GET /traces/latest` as `service_type: database` and as a
dotted node on the service map
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
- Adds **239 new tests** covering the vortex columnar file format across
78 scenarios from the test plan
- **`tests/api-testing/tests/vortex/`** — 9 test modules (83 tests):
format conversion, schema evolution, data integrity, FTS, downsampling,
API format awareness, server config, and performance/stress
- **`tests/api-testing/tests/query_agent/test_3_vortex.py`** — 156
tests: the full 155-query SQL correctness suite + scenario 39
(self-join) run post-flush against vortex files
- **`api-testing.yml`** — adds `--ignore=tests/vortex` to the OSS
default-set run; vortex tests require the enterprise binary with
`ZO_FILE_FORMAT=vortex` and are gated to enterprise CI only
## Test plan coverage
| Module | Scenarios |
|--------|-----------|
| `test_format_conversion.py` | 1–13, 7, 8, 9, 10 |
| `test_schema_evolution.py` | 14–25 |
| `test_data_integrity.py` | 55–62 |
| `test_fts.py` | 51–54 |
| `test_downsampling.py` | 47–50 |
| `test_api_format.py` | 71–73 |
| `test_config.py` | 66–67 |
| `test_performance.py` | 74–78 |
| `test_3_vortex.py` | 26–46 (155 queries + JOIN) |
## Enterprise CI
Two dedicated jobs added in the companion `openobserve/o2-enterprise` PR
on the same branch:
- **`api_query_agent_vortex`** — runs `test_3_vortex.py` with
`ZO_FILE_FORMAT=vortex`
- **`api_vortex_format_tests`** — runs the full `tests/vortex/` suite
with 4-phase server lifecycle (parquet→vortex→parquet→vortex+compaction)
OSS CI is unchanged except for the ignore rule — no vortex tests run
against the OSS binary.
## Test checklist
- [ ] `tests/vortex/` collects cleanly: `83 tests collected`
- [ ] `tests/query_agent/test_3_vortex.py` collects cleanly: `156 tests
collected`
- [ ] Enterprise CI `api_query_agent_vortex` job passes
- [ ] Enterprise CI `api_vortex_format_tests` job passes
🤖 Generated with [Claude Code](https://claude.ai/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
- **Streams-Regression** — `isQueryEditorExpanded()` used
`container.locator('.editor-fullscreen')` which searches *descendants*,
but Vue adds `editor-fullscreen` directly to the container element
itself. Fixed by using combined CSS selector
`.query-editor-container.editor-fullscreen`.
- **Logs-Regression** — `enableHistogram()` checked canvas visibility to
detect histogram state. Canvas is absent before the first query runs, so
it always assumed histogram was OFF and toggled it OFF when it was
already ON. This removed `[data-test="logs-search-result-bar-chart"]`
from the DOM (it is `v-if`-gated), breaking
`expectBarChartHasContent()`. Fixed by using
`logsQueryPage.isHistogramOn()` (reads button `data-state` attribute),
consistent with how `ensureHistogramState()` already works.
## Failing tests fixed
- `should allow query expand icon to work after navigating from stream
explorer (#9337)` — Streams-Regression
- `should not show blank histogram after cancelling query @bug-4315` —
Logs-Regression
- `collapsing/expanding index list should redraw histogram chart` —
Logs-Regression
## Test plan
- [ ] Ran `streams-regression.spec.js --grep "9337"` locally — passes
- [ ] Ran `logs-regression-bugs.spec.js --grep "blank histogram after
cancelling"` locally — passes
- [ ] Ran `logs-regression-bugs-2.spec.js --grep "collapsing/expanding
index list"` locally — passes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
- Adds `|| echo '{"jobs":[]}'` fallback on the `curl` call so a network
error or GitHub API rate-limit response never produces an empty/invalid
variable
- Adds `2>/dev/null || echo "0"` on both `jq` calls so a malformed API
response (non-JSON error body) falls back to `"0"` instead of
propagating a non-zero exit under `-eo pipefail`
- Changes the arithmetic comparison to `[ "${CURRENT_ATTEMPT_JOBS:-0}"
-eq "${TOTAL_MATRIX_JOBS:-0}" ]` with `2>/dev/null` so a `null` jq
output can't trigger "integer expression expected" and kill the step
**Failure mode fixed:** On re-runs, multiple matrix jobs query the same
GitHub API endpoint concurrently with the shared `GITHUB_TOKEN`. A
rate-limit `403` or transient network hiccup returns non-JSON, `jq`
exits non-zero, and with `set -eo pipefail` the whole "Check if this is
a rerun" step fails — blocking the job before any tests run.
**Degraded behaviour:** When the API is unavailable, both counts fall to
`0`, `0 -eq 0` → `rerun_type=all` → step succeeds → all tests run (same
as a normal rerun-all). The TestDino optimisation is skipped but nothing
is blocked.
Companion PR: openobserve/o2-enterprise#<ENT_PR>
## Test plan
- [ ] Trigger a re-run of a workflow run — "Check if this is a rerun"
step should pass even if GitHub API is slow/rate-limited
- [ ] Verify `rerun_type` is still correctly set to `failed` when the
API responds normally
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
Two high-value improvements to the query agent regression test data:
1. **Data variety** — Per-field rotation offsets (`qi*13 + i*7`) replace
the old uniform `idx % len(pool)` approach. Previously all fields picked
from the same pool position, creating only 5 fixed row shapes per query.
Now field-value combinations vary independently per field.
2. **NULL stress** — 8 fields (`bot_flag`, `info_tag`,
`upstream_error_code`, `page_url`, `endpoint_path`, `resource_path`,
`variant_tag`, `threat_flag`) get `None` for 40% of records per query.
This exercises `IS NULL` / `IS NOT NULL` / `COALESCE` paths that were
previously untested.
## Changes
- `data_gen.py`: Rewrote `make_record()` with per-field rotation and
NULL injection
- 24 queries: Adjusted WHERE/HAVING filters to match new data
distribution
- All 12 query JSONs: Regenerated expected results via
`compute_counts.py`
## Impact
- 400 queries all have results (18 pre-existing legacy/skip queries
unchanged)
- No change to record count (still 5/query, 2000 total) — test time
unaffected (~7 min)
## Test plan
- [x] `compute_counts.py` regenerates all expected results
- [ ] Run `pytest tests/query_agent/ -v --tb=short` against dev
environment
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
- Expands the SQL regression catalog from **155 to 200 queries** across
all 12 categories
- Adds **7 queries** extracted from abc.sql (Q156-Q162) — synthesized
from real-world OpenObserve SQL patterns with all data/fields/streams
generated from scratch
- Adds **230 auto-generated queries** (Q163-Q392) from templates to
cover query format diversity
- Both test phases pass: **200 memtable + 200 parquet = 400 total**
## Key changes
- **data_gen.py**: 43 new fields (record_type, site_name, latency_ms,
payload_size, alert_tags, phase_data, db_info, etc.), generates 3920
deterministic records
- **compute_counts.py**: new translations for regexp_like,
str_match_ignore_case, approx_distinct, json_get_str/json_get_json
chaining; empty-aggregate result detection (OO returns 0 rows vs
DuckDB's 1 NULL row)
- **conftest.py**: PID-based stream name suffix for cross-invocation
isolation (prevents data accumulation)
## Test plan
- [x] test_1_memtable.py — 392 passed
- [x] test_2_parquet.py — 392 passed
- [x] All 7 abc.sql queries pass (Q156-Q162)
- [x] No real stream names, field names, or data copied from external
sources
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Fixes#12461
## Problem
PromQL queries in a super-cluster returned data **only from the hub
cluster** — remote (spoke) clusters were silently skipped. SQL queries
across the same super-cluster worked correctly.
**Root cause:** The `data` gRPC handler on the spoke never registered
the incoming `trace_id` in `SEARCH_SERVER`. When the PromQL engine
called `insert_sender(trace_id)` to attach its abort channel, it found
no entry and immediately returned `SearchCancelQuery` (error code
20009). The spoke returned empty data with no visible error — the hub
received nothing from the spoke and returned only its local results.
The `query` gRPC handler already had the correct registration. The
`data` handler was missing it.
## Fix
Before spawning the data fetch task on the spoke, the `data` gRPC
handler now registers the incoming `trace_id` in `SEARCH_SERVER` as a
follower task — exactly the same registration the `query` handler
already performed. This allows the PromQL engine to successfully attach
its abort sender and proceed with execution. After the task completes,
the trace_id is removed from `SEARCH_SERVER` to avoid memory leaks.
## Test Queries
Ingest a metric with a distinguishing label to both hub and spoke, wait
~30s for WAL flush, then run:
**PromQL — before fix:** only hub cluster returned, spoke missing
**PromQL — after fix:** both hub and spoke returned
```bash
curl -s "http://<hub-host>/api/<org>/prometheus/api/v1/query_range" \
-u '<user>:<pass>' \
--data-urlencode "query=sum by (cluster) (total_payment_req)" \
--data-urlencode "start=$(($(date +%s) - 3600))" \
--data-urlencode "end=$(date +%s)" \
--data-urlencode "step=60"
```
**SQL — works both before and after (baseline check):**
```bash
curl -s -X POST "http://<hub-host>/api/<org>/_search?type=metrics" \
-H "Authorization: Basic <token>" \
-H "Content-Type: application/json" \
-d '{
"query": {
"sql": "SELECT cluster, SUM(value) as total FROM \"total_payment_req\" GROUP BY cluster",
"start_time": '"$(($(date +%s) - 3600))"'000000,
"end_time": '"$(date +%s)"'000000,
"size": 10
}
}'
```
## What changed
- Added a wrap/unwrap button (toggle) in the CorrelatedLogsTable header
toolbar, matching the pattern from `logs/SearchResult.vue` (`wrap-text`
icon, `OTooltip` with `search.messageWrapContent` i18n key)
- Replaced the hard-coded 800px column `maxSize` with a dynamic cap
computed from the table container's actual width, tracked via
`ResizeObserver`
- Last-column fill: if total column widths are less than the container
and the last column is a `longTextFields` type, the remaining space is
added to it
- close https://github.com/openobserve/o2-enterprise/issues/1865
## TODO
- [x] make split_batch_for_grpc_response based on the real memory sizew
- [x] coalesce and gc view before send to leader
```
SELECT trace_id, count(*) as cnt, max(_timestamp) as ts FROM "k8s_logs" group by trace_id order by cnt desc limit 10
utf8view enable:
main: 30s (15k batches send to leader)
this branch: 15s (1.5k batches send to leader)
utf8view disable:
main: 13s (42k batches send to leader)
this branch: 12s (1.5k batches send to leader)
```
one found is the AggregateExec's time_calculating_group_ids, utf8 is
faster than utf8view
## Summary
Closes#12426.
In cloud-native deployments (ECS/Fargate, Kubernetes) secrets managers
inject individual values, not composed strings. That makes it impossible
to cleanly provide a full `ZO_META_POSTGRES_DSN` without custom tooling
(init containers, Lambda functions, etc.).
This PR adds five individual connection vars as an alternative:
| Variable | Default |
|---|---|
| `ZO_META_POSTGRES_HOST` | `""` |
| `ZO_META_POSTGRES_PORT` | `5432` |
| `ZO_META_POSTGRES_USER` | `""` |
| `ZO_META_POSTGRES_PASSWORD` | `""` |
| `ZO_META_POSTGRES_DBNAME` | `""` |
**`ZO_META_POSTGRES_DSN` takes full precedence when set, no breaking
change.**
When the DSN is empty, `check_common_config()` composes it from the
individual vars at startup. User, password and dbname are
percent-encoded so credentials with special characters (common with
secrets-manager generated passwords) survive sqlx's percent-decoding
when the DSN is parsed. The connection code itself is untouched, every
consumer of `meta_postgres_dsn` (including the migration adapter) just
sees a normal DSN.
## Changes
- `src/config/src/config.rs`: adds 5 new `env_config` fields, validates
that either the DSN or all individual vars are set, and composes the DSN
from the individual vars when it is empty
---------
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
## Problem
`query_by_ids` runs `SELECT ... WHERE id IN (...)` with **no partition
key filter**. On the partitioned `file_list` table this means:
- **planning** must open and lock **every** partition (~101ms on 384
partitions)
- **execution** probes **every** partition's id index (~10ms)
Measured on production-scale data (2.7B rows / 1.2TB / 384 daily
partitions, PostgreSQL 17): **~110ms per call**, vs ~1.5ms on a
non-partitioned table. `query_by_ids` is on the search hot path (id →
FileKey reverse lookup), so every search pays this cost.
## Fix
The ids passed to `query_by_ids` are produced by `query_ids` **using a
known time range** — the search path already holds it
(`service::file_list::query_by_ids` already takes `time_range` but did
not pass it down). This PR passes the range down to the infra layer and
derives a `date` filter via the existing `derive_date_range`, enabling
plan-time partition pruning:
```sql
SELECT ... FROM file_list WHERE id IN (...)
AND date >= '2026/05/06/00' AND date < '2026/05/14/00' -- new
```
## Benchmark
From the file_list partitioning performance test (2.7B rows, 384 daily
partitions, c6i.8xlarge, PG 17.10,
jit=off/work_mem=128MB/rpc=1.1/ANALYZEd). Same in-window batch of 1000
ids, median of 3 runs, total = execution + planning:
| Query window | Before (total) | After (total) | Improvement |
Partitions scanned |
|---|---:|---:|---|---|
| 1h | 112.7ms (11.7 + 101.0) | **1.18ms** (0.82 + 0.36) | **95x** | 384
→ 1 |
| 1d | 110.3ms (9.4 + 100.9) | **1.60ms** (0.97 + 0.62) | **69x** | 384
→ 2 |
| 7d | 109.4ms (8.4 + 100.9) | **2.93ms** (0.79 + 2.14) | **37x** | 384
→ 8 |
With the filter, the partitioned table matches or **beats** the
non-partitioned baseline (execution 0.8~1.0ms vs 1.2~1.3ms — per-day
partition id indexes are shallower than one monolithic 2.7B-row index).
## Safety
- **Semantically safe**: the ids were selected by `query_ids` with
exactly the same derived date range, so the added filter cannot drop
rows.
- **No side effect on non-partitioned tables**: measured 1.5ms → 1.7ms
(within noise) on the single-table baseline.
- **Guarded**: the filter is only applied when `time_range` is
`Some((start, end))` with `start > 0 && end >= start`; all other callers
pass `None` and keep the previous behavior.
- SQLite backend ignores the parameter (not partitioned; already a
single index probe).
## Changes
- `infra::file_list::FileList::query_by_ids` trait: add `time_range:
Option<(i64, i64)>`
- postgres impl: derive and append the date filter when a valid range is
given
- sqlite impl: accept and ignore the parameter
- `service::file_list::query_by_ids`: pass its existing `time_range`
down (db + local cache)
- `cli::basic::bloom`: pass `None`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
- **OTextarea resize**: Changed default resize behaviour from `resize-y`
to `resize-none`. `OTextarea` uses `inheritAttrs: false`, so
`style`/`class` attrs passed by consumers land on the outer wrapper
`<div>`, not the native `<textarea>`. The `resize-y` class was allowing
users to manually drag the textarea to grow it indefinitely — this
didn't exist with the previous `q-input` implementation.
- **License.vue style cleanup**: Removed `style="height: 200px"` and
`style="min-height: 150px"` from the two `OTextarea` usages. These were
being applied to the outer wrapper div (wrong element) and had no effect
on the actual textarea size. The `rows` prop already controls height
correctly.
- **Button spacing**: Added `tw:mt-3` gap between the license key
textarea and the "Update License" button.
## Test plan
- [ ] Navigate to Settings → License page with no license installed
- [ ] Verify the "Enter License Key" textarea renders at a fixed height
and cannot be resized by dragging
- [ ] Verify the "Update License" button has visible spacing below the
textarea
- [ ] Verify pasting a long license key causes scroll within the
textarea (not expansion)
- [ ] Trigger the update form (on a page with an active license) and
verify the same resize-none behaviour
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: ktx-vaidehi <vaidehi.akhani@kiara.tech>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
## Summary
- ENT CI runners are slower than local/OSS runners — the
`ingest_query_agent_data` fixture was hitting the 120s timeout before
1550 records became searchable, causing all 310 query_agent tests to
fail at setup
- Bumped `wait_until` timeout to 300s
## Test plan
- [ ] OSS `api_query_agent_tests` passes as before
- [ ] ENT `api_query_agent_tests` no longer hits WaitTimeout on ingest
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
- close https://github.com/openobserve/openobserve/issues/11956
- [x] check how PartitialReduce work with AggregateTopk
- [x] check how PartitialReduce work with StreamingAgg
# New Env
```
ZO_FEATURE_PARTIAL_REDUCE_ENABLED=true (default is true)
```
# Performance
## First datasets
80M records
single node optimzier disable(to simulate the distribute query)
### Q1
```sql
SELECT count(distinct request_id) FROM "k8s_logs"
```
80M request_id unique value
with partial reduce: 61s (80M records send from follower to leader)
without partial reduce: 54s (80M records send from follower to leader)
### Q2
```sql
SELECT count(distinct client_ip) FROM "k8s_logs"
```
10M client_ip unique request_id
with partial reduce: 5s (10M records send from follower to leader)
without partial reduce: 54s (80M records send from follower to leader)
### Q3
```sql
SELECT
host,
COUNT(DISTINCT client_ip) AS cnt
FROM
k8s_logs
GROUP BY
host ORDER BY cnt DESC
```
2000 host unique value, 10M client_ip unique value
with partial reduce: 41s (75M records send from follower to leader)
without partial reduce: 26s (80M records send from follower to leader)
### Q4
```sql
SELECT
kubernetes_namespace_name,
COUNT(DISTINCT client_ip) AS cnt
FROM
k8s_logs
GROUP BY
kubernetes_namespace_name ORDER BY cnt DESC
```
10 kubernetes_namespace_name unique value, 10M client_ip unique value
with partial reduce: 14s (38M records send from follower to leader)
without partial reduce: 17s (80M records send from follower to leader)
### Q5
```sql
SELECT
histogram(_timestamp, '5 minute') AS ts,
host,
approx_distinct(client_ip) AS cnt,
ROW_NUMBER() OVER (
PARTITION BY histogram(_timestamp, '5 minute')
ORDER BY approx_distinct(client_ip) DESC
) AS rn
FROM
k8s_logs
GROUP BY
ts, host
QUALIFY rn <= 10
ORDER BY ts DESC, cnt DESC
```
2000 host unique value, 10M client_ip unique value
with partial reduce: 3.2s (15k records send from follower to leader)
without partial reduce: 4.2s (46k records send from follower to leader)
## Second datasets
100M records, 250GB original size, 37GB compression size
### Q6
```sql
SELECT
client_ip,
count(*) AS cnt,
max(_timestamp) AS ts
FROM
"k8s_logs"
GROUP BY
client_ip ORDER BY cnt DESC LIMIT 100
```
0.5M client ip unique value
with partial reduce: 5s (517k records send from follower to leader)
without partial reduce: 19s (100M records send from follower to leader)
### Q7
```sql
SELECT
trace_id,
count(*) AS cnt,
max(_timestamp) AS ts
FROM
"k8s_logs"
GROUP BY
trace_id ORDER BY cnt DESC LIMIT 100
```
12M client ip unique value
with partial reduce: 13s (12M records send from follower to leader)
without partial reduce: 23s (100M records send from follower to leader)
**for streaming aggreagte cache genearte**
with partitial reduce first run: 23s
without partial reduce first run: 37s
# Design
## Background
OpenObserve uses a distributed query architecture where a **leader
(querier)** node dispatches physical execution plans to **follower
(ingester/compactor)** nodes via Arrow Flight RPC. The `RemoteScanExec`
node represents work pushed to remote followers; results stream back to
the leader for final processing.
For aggregation queries, DataFusion produces a two-phase plan:
1. **`Partial`** — each follower computes partial aggregates (partial
SUM, COUNT, group keys, etc.)
2. **`Final`** — the leader merges all partial results into the final
answer
**Problem:** All partial aggregate rows from every follower are sent to
the leader unconditionally. When there are many unique group-by keys or
large datasets, this generates significant network traffic even when
many follower partitions produce overlapping keys that could be
pre-merged locally.
---
## Solution
DataFusion supports a third aggregate mode, `PartialReduce`, which sits
between `Partial` and `Final`. It takes the output of a `Partial`
aggregate and applies a further reduction pass — merging rows with the
same group keys — before they leave the node.
Insert a `PartialReduce` node on the **follower side** (between the
`Partial` aggregate and the `RemoteScanExec`). Followers send fewer rows
over the network; the leader's `Final` phase still produces a correct
result.
---
## Execution Plan Transformation
**Before:**
```
FinalAggregateExec (leader)
└─ RepartitionExec (leader)
└─ RemoteScanExec (network — all partial rows)
└─ PartialAggregateExec (follower, N partitions)
└─ ScanExec
```
**After:**
```
FinalAggregateExec (leader)
└─ RepartitionExec (leader)
└─ RemoteScanExec (network — reduced rows)
└─ PartialReduceAggregateExec (follower)
└─ CoalescePartitionsExec (follower — merges N partitions into 1)
└─ PartialAggregateExec (follower, N partitions)
└─ ScanExec
```
`CoalescePartitionsExec` merges all follower partitions before the
`PartialReduce` pass to maximize the reduction ratio.
---
## Trade-offs
**Pros:**
- Reduces network bytes transferred from followers to leader for
aggregation queries
- Zero correctness risk for non-aggregate queries (passthrough)
- Runtime feature flag, safe to disable without redeployment
**Cons:**
- Adds a CPU pass on followers; worthwhile only when network is the
bottleneck
- Queries with high-cardinality GROUP BY and no key overlap between
partitions get no reduction, only extra follower CPU
- `CoalescePartitionsExec` serializes follower partitions before the
reduce pass, reducing follower-side parallelism
- Add condition layout fix
- drilldown close button alignment and size sm
- OCombobox item align center
- Crosslink combobox and input based on condition
- Actions timezone alignment
- Ai toogle expand shortcut enhance
PR #11691 introduces org-level ingestion tokens with o2oi_ prefix (37
chars). This commit adds comprehensive automated test coverage:
## API Tests (35 tests in test_ingestion_tokens.py)
### Token CRUD (9 tests)
- Create token returns o2oi_ prefix and 37-char length
- Create token persists name, created_by, enabled=true, is_default=false
- Create token with name only (no description) — description is empty
- Create token with hyphens and underscores allowed
- Duplicate name returns 400 with "already exists" message
- Empty name returns 400 with "cannot be empty" message
- Whitespace-only name returns 400 with "cannot be empty" message
- Name too long (>256 chars) returns 400 with non-empty error message
- Parametrized: special chars (spaces, @, ., /) rejected with
"alphanumeric" message
### List Tokens (3 tests)
- GET returns data array
- Created token appears in list
- Token values are unmasked in list response
### Enable/Disable (5 tests)
- Disable token via PATCH {enabled: false}
- Enable disabled token via PATCH {enabled: true}
- Double disable is a no-op (idempotent)
- Toggle nonexistent token returns 404
- DELETE returns 405 (not supported)
### Ingestion Auth — Org Token (6 tests)
- Ingest with org token (org_id:o2oi_token) succeeds
- Ingest with disabled org token fails (401)
- Ingest with nonexistent o2oi_ token fails (401)
- Ingest to wrong org with valid token fails (401)
- Org token blocked on search endpoint (401)
- Org token blocked on dashboards endpoint (401)
### Ingestion Auth — Backward Compat (6 tests)
- User password ingestion still works (pre-PR behavior)
- Non-o2oi_ service account token ingestion works (falls through to user
lookup)
- User password on search succeeds (200) — mirrors org token blocking
tests
- User password on dashboards succeeds (200) — mirrors org token
blocking tests
- Non-o2oi_ SA token NOT blocked by o2oi_ prefix check on search (≠401)
- Non-o2oi_ SA token NOT blocked by o2oi_ prefix check on dashboards
(≠401)
### Multi-token and Security (6 tests)
- Create 5 tokens, all visible in list
- Disable one token, others remain enabled
- SQL injection in token name rejected (special chars → 400)
- Empty name, whitespace name, too-long name (with message assertions)
- Special chars parametrized: spaces, @, ., /
## Playwright UI Tests (7 tests in ingestionTokens.spec.js)
- Ingestion Tokens tab renders with table and columns (P0)
- Create token via UI dialog with name and description (P0)
- Create token with empty name — button disabled, enabled on fill,
disabled on clear
- Revealed token dialog — copy token with clipboard verification
- Toggle token enable/disable from table actions (P0)
- Manage Tokens button on ingestion page navigates to IAM
- Ingestion page token selector dropdown shows available tokens
## Page Object (ingestionTokensPage.js)
- data-test selectors for IAM nav, token list, create dialog, toggle,
toast, revealed dialog
- Registered in centralized PageManager
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Issue
When `_search_multi_stream` processes a histogram query, streaming
execution splits the time range into partitions. Each partition calls
`do_search()` with its own (smaller) time range, causing
`Sql::new_with_options()` to recompute `histogram_interval` via
`generate_histogram_interval()` based on the **partition's** time range
duration instead of the **original query's** time range.
Since different partitions have different time range durations, each
gets a different auto-generated histogram interval (e.g. 10s for a small
partition, 30s for the full range). This produces inconsistent bucket
boundaries across partitions in the streaming response.
The non-streaming `_search_multi` endpoint does not have this issue
because it searches the entire time range as a single unit.
### Root cause
In `src/service/search/streaming/mod.rs`,
`process_search_stream_request()` enters either:
- the **cache path**: `handle_cache_responses_and_deltas()` →
`process_delta()` → per-partition `do_search()`
- the **no-cache path**: `do_partitioned_search()` → per-partition
`do_search()`
In both paths, each partition clones `req` with a narrowed
`start_time`/`end_time`, so when `do_search()` creates a fresh `Sql`
struct, `histogram_interval` is 0 and `Sql::new_with_options()`
recomputes it from the partition's (shorter) time range.
### Fix
Before either path is entered, parse the SQL once with
`HistogramIntervalVisitor` using the **full query time range** and set
the resulting interval on `req.query.histogram_interval`. All subsequent
partition clones inherit this value, and `Sql::new_with_options()` takes
the `query.histogram_interval > 0` branch — using the preset value
instead of recomputing from the partition range.
## Summary
Follow-up to #12380 (feat: super org).
- `plans.vue` and `BillingGroup.vue` were using `--o2-primary-color`
(hardcoded iris/blue `#575FC5`) for the new managed-billing and
org-group hero UI introduced in the super org feature
- `--o2-primary-color` is a static constant — it is never updated by the
theme system
- Replace all 13 occurrences with `--color-primary-600`, which is
dynamically regenerated by `syncO2LibraryTokens()` in `utils/theme.ts`
whenever the user selects a theme color
**Affected elements:**
- `plans.vue` — managed-billing empty state (bank icon circle + icon
color), 4 spots
- `BillingGroup.vue` — standalone invite empty state icon, "Active"
eyebrow badge, and the three membership fact cards (Invited By /
Accepted By / Member Since) icon backgrounds + colors, 9 spots
## Test plan
- [ ] Apply a non-default theme color (e.g. green) and navigate to
Billing → Plans as a child org — bank building icon and circle ring
should match the theme color
- [ ] Navigate to Billing → Organization Group as a child org — "Active"
eyebrow badge and the three sidebar fact-card icons should match the
theme color
- [ ] Verify same views look correct with the default theme and in dark
mode
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
Fixes#11904 — streaming search kept running on the queriers even after
the user closed the browser tab.
The SSE search endpoints spawn the query in a detached task and stream
results back over `text/event-stream`. When the client disconnects, the
response stream is dropped and we stop *sending* results, but the
distributed query continued executing on the queriers, wasting resources
on results nobody will read.
This wires client disconnect into the existing query-cancellation
machinery.
## What changed
- **`SearchStreamGuard`** (`search/utils.rs`) — moved into the response
stream's closure so it shares the stream's lifetime. It marks itself
finished when a terminal event (`Done` / `Error` / `Cancelled`) is
produced; if the stream is dropped before that (client gone), its `Drop`
fires the cancel for the parent `trace_id`.
- **`cancel_query_internal`** (`search/query_manager.rs`) — best-effort
cancel for internal callers (no HTTP response), reusing the existing
super_cluster/regular split.
- Wired the guard into **`search_http2_stream`**,
**`values_http2_stream`**, and **`search_multi_stream`**.
Cancellation is keyed by the parent `trace_id`; the query manager
removes entries by prefix, so multi-stream sub-queries
(`{trace_id}-q{n}`) are covered by a single cancel.
## Notes
- Distributed cancellation is an **enterprise** feature (the whole
`cancel_query` / `QueryManager` path is enterprise-gated). On the
open-source build the guard only logs the disconnect; OSS already stops
sending once the channel closes.
- Normal completion never triggers a cancel (terminal-event →
`mark_finished`). Worst case on an edge race is a harmless no-op cancel
for an already-finished `trace_id`.
- Complements the existing manual "Cancel" button — same underlying
machinery, triggered automatically on disconnect.
This branch also includes an unrelated config tweak (`fix: enable
compact fast mode for local`) enabling compact `fast_mode` automatically
when running in local mode.
## Testing
- `cargo check` (OSS) and `cargo check --features enterprise` both clean
- Unit test for the guard state machine passes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## What
Re-enables the `aarch64-unknown-linux-musl` release target so we publish
an `openobserve-<tag>-linux-arm64-musl.tar.gz` artifact. Closes#12369
(OSS portion).
## Why
A user requested an arm64 + musl build. The target was commented out in
`release.yml` (since #1909, late 2023). This brings it back.
## The fix
The old commented entry ran on an **x86 runner** (`ubicloud-standard-8`)
and cross-compiled. On that host the `Install Protoc for linux ARM` step
(`contains(arch, 'aarch64-unknown-linux')`) installs an **aarch64
protoc** binary, which can't execute on x86 — `prost-build` invokes
protoc at build time, so the build dies. That mismatch is the likely
original "build trouble."
This PR instead builds arm64-musl on the **native ARM runner**
(`ubicloud-standard-16-arm`), the same setup the working
`aarch64-unknown-linux-gnu` artifact already uses, which sidesteps the
host/target protoc problem.
Everything else is already in place:
- `.cargo/config.toml` already configures
`[target.aarch64-unknown-linux-musl]` (linker + `+aes,+neon`)
- the `linux-musl` deps step already installs `musl-dev`/`musl-tools` +
toolchain
- the ARM protoc step matches and runs natively
- OpenSSL-on-musl is already proven by the working `amd64-musl` build
## Verification
This only takes effect on the next `v*.*.*` tag, so it can't be
exercised until a release runs the workflow. The matrix and YAML parse
correctly and the new entry mirrors the arm64-gnu config.
## Note
This covers **OSS** only. The Enterprise arm64-musl build needs a
separate change in the o2-enterprise pipeline.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Fixes#11364.
Decide **client-side** gRPC TLS from the connection address scheme
(`https://` vs `http://`) instead of the global `ZO_GRPC_TLS_ENABLED`
toggle. This decouples client TLS from server TLS, which lets a super
cluster sit behind a TLS-terminating load balancer (AWS ALB/NLB) without
the deadlock described in the issue — **no new env vars and no separate
client/server toggle**.
All gRPC client connections funnel through
`infra::client::grpc::create_channel` (search/metrics/node/cluster-info
clients, the flight client, file downloader, broadcast), so this is a
single-spot change.
### Behaviour
| Config | Server (still on `ZO_GRPC_TLS_ENABLED`) | Advertised addr |
Client (now scheme-based) |
|---|---|---|---|
| `tls_enabled=true` (today) | TLS | `https://…` | TLS ✓ |
| `tls_enabled=false` (today) | h2c | `http://…` | h2c ✓ |
| **Super cluster behind LB:** `tls_enabled=false` +
`O2_SUPER_CLUSTER_PUBLIC_ADDR=https://lb:443` | h2c (ALB terminates TLS)
| intra: `http://`, cross-cluster: `https://lb` | intra h2c,
cross-cluster TLS ✓ |
The server keeps running h2c behind the ALB, intra-cluster pods stay
h2c, and only the cross-cluster connection (which sees `https://`)
negotiates TLS on the client side. Fully backward compatible.
### Notes
- The client SNI/domain is now only overridden when
`ZO_GRPC_TLS_CERT_DOMAIN` is set; otherwise tonic derives it from the
URI host, which matches the load balancer's certificate. (Previously
`.domain_name("")` was always called, which would send an empty SNI in
the new flow.)
- With `ZO_GRPC_TLS_ENABLED=false`, `check_grpc_config` does not require
cert/key files, and the default `webpki` roots cover a public ACM cert —
so **no cert files need to be mounted** for this setup, side-stepping
the Helm `extraVolumes` concern in the issue.
- Relies on the enterprise `O2_SUPER_CLUSTER_PUBLIC_ADDR` being settable
to a full `https://host:port` URL and passed through verbatim to
`create_channel`.
## Test plan
- [x] `cargo test -p infra --lib client::grpc::` — added unit tests
covering the scheme-based decision (`https://` → TLS,
`http://`/`grpc://`/bare/empty → h2c, case-sensitive scheme match).
- [x] `cargo check -p infra`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
- Adds `retry_on_db_lock()` to `tests/api-testing/support/wait.py`
- Companion to the enterprise-side fix (openobserve/o2-enterprise#TBD)
that applies the helper across all RBAC entity base tests
## Root cause
The `ent_rbac` CI shard runs with `-n 2` (two pytest-xdist workers).
SQLite's single-writer model means parallel workers racing on write
operations produce transient `500 {"message": "database is locked"}`
responses (SQLite error codes 5 / 517). These surface as flaky assertion
failures across many PRs.
## Fix
`retry_on_db_lock(fn)` wraps an HTTP call and retries up to 3 times with
exponential backoff (0.5 → 1 → 2 s) only when the response is `500` with
`"database is locked"` in the body. Parallelism is preserved; the brief
retry window absorbs the time the other writer needs to commit and
release the lock.
## Test plan
- [ ] OSS CI passes with this branch
- [ ] Enterprise PR with the same branch name picks up the new utility
and applies it in all RBAC entity base tests
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Abhay Padamani
Huaijin
Hengfei Yang
Sai Nikhilandudev
Neha PandClaude Sonnet 4.6
Harsh Mahajan
Shrinath RaoandClaude Sonnet 4.6
ktx-mihir
Loakesh Indiran
Subhra264
Omkar Kesarkhane
Derk SchooltinkandHengfei Yang
ktx-riya
ktx-vaidehiandktx-mihir
dependabot[bot]andHengfei Yang
