## Problem
`ORDER BY <non-timestamp col>` (e.g. `ORDER BY duration DESC`) gave
wrong results and risked OOM because per-partition \`hits_to_skip\`
assumed rows are time-ordered — but the globally k-th row can be in any
partition.
## Solution
Each partition fetches its local top-`k = from+size` rows. A
**TopKHeap** on the leader merges them globally, skips `from`, returns
`size`. Heap is bounded to `k` elements at all times — O(k) memory
regardless of partition count.
**Multi-column ORDER BY** (`ORDER BY duration DESC, _timestamp DESC`) is
fully honored: heap comparison and final sort respect all columns in
order, using secondary columns as tiebreakers.
## Scenario behavior
| Query | Behavior |
|---|---|
| `ORDER BY _timestamp DESC` | Unchanged — per-partition path |
| `ORDER BY duration DESC` | Heap merge path — correct pagination |
| `ORDER BY duration DESC, _timestamp DESC` | Heap merge; ties in
`duration` broken by `_timestamp` |
| `ORDER BY duration DESC` page 2 | Heap collects global top-k, skips
`from`, returns `size` |
| JOIN / subquery / GROUP BY | `is_aggregate=true` → single partition
(unchanged) |
| `ZO_DISABLE_PARTITIONS_FOR_NON_TS_ORDER_BY=true` | Forces 1 partition
(circuit breaker preserved) |
| Partial cache hit + non-ts ORDER BY | TODO — pre-existing bug, no
regression |
## What is NOT changed
- `ORDER BY _timestamp` execution path
- `_search` (non-streaming) — DataFusion handles it correctly
- Aggregate / histogram queries
This adds support for current cycle for stripe users as period for usage
page
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
## Summary
Cherry-picks two correlation fixes from `branch-v0.90.0` to `main`:
- `a077b760c` fix: corelation custom groups (#11662) — adds `group`
field to `FoundGroup` and threads `semantic_groups` through
`calculate_dimension_analytics`
- `211876a97` fix: corelation settings fixes (#11666) — validates
tracked alias IDs against bundled defaults + custom DB groups; sanitizes
payload (drops `service` and orphan IDs) before save
Conflicts resolved by taking v0.90.0 versions (the fixes are the point
of the cherry-pick). Companion PR on o2-enterprise:
openobserve/o2-enterprise#1648 — that PR is currently red because it
depends on the new `FoundGroup.group` field landing here first.
## Test plan
- [ ] CI green (build, unit, playwright)
- [ ] Verify save Identity Config rejects invalid alias IDs but accepts
default + custom IDs
- [ ] Verify `_analytics` endpoint returns `group` category on
FoundGroup
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
- **fix: llm insights tab**
- **remove custom attributes and use gen ai spec**
---------
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
## Summary
- Replace `SELECT *` Q2 with grouped aggregations so per-trace
enrichment is bounded by N rows instead of total span count.
- Q2a aggregates per `trace_id`: span count, error count, min/max times,
max duration, root span labels (with first-by-timestamp fallback), and
`approx_distinct(service_name)` to gate Q2b.
- Q2b runs only for multi-service traces, returning per-`(trace_id,
service_name)` count and max duration.
## Root cause
When a user filters Traces by Duration, Q2 ran `SELECT * FROM stream
WHERE trace_id IN (...)` and aggregated spans in Rust. For traces with
tens of thousands of spans, the search-service row cap was hit before
every `trace_id` got at least one span back. The un-enriched entries
were still sent to the FE with `spans=[0,0]`, `duration=0`,
`first_event=null` — the blank rows in the screenshot on the issue.
The new Q2a returns one row per `trace_id` (≤ N rows total), so it
cannot trip the row cap. Single-service traces are fully populated from
Q2a alone; only multi-service traces require Q2b. Response shape
(`TraceResponseItem`) is unchanged — `first_event` is now built as a
synthetic `{service_name, operation_name}` object so the FE in
`web/src/composables/useTraces.ts` keeps working unchanged.
Applied to both `get_latest_traces` and `process_latest_traces_stream`.
Closes#11505
## Test plan
- [ ] Open Traces, pick a stream with traces that have many spans (≥10K
spans/trace).
- [ ] Filter by Duration `<= Max value`. Confirm no blank rows; every
row has a service icon, service name, non-zero duration, and a span
count.
- [ ] Single-service stream: confirm the services list under each trace
is correct.
- [ ] Multi-service trace (e.g., a trace that crosses several services):
confirm the per-service breakdown still appears (Q2b path).
- [ ] Sort by duration ASC and DESC.
- [ ] Sort by start_time ASC and DESC.
- [ ] LLM stream: confirm LLM token / cost columns still populate.
- [ ] Pagination: page 1, page 2, page N — every page has fully
populated rows.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Hengfei Yang <undefined@users.noreply.github.com>
- Earlier when map_group_to_role env is enabled, the code flow was
ignoring the map_group_to_role flow when the custom_role was empty
(basically there is no custom role configured in the dex group claim).
This pr fixes this - so no matter whether the custom role is empty or
not, it will go to the map_group_to_role flow (this behaviour was there
earlier, but got broken when the sso claim parser was implemented)
- In the map_group_to_role flow, if the custom role array is empty it
was not adding the sso user to the DEX_DEFAULT_ORG at all even though
the ofga tuple mapping was being done. There is one issue in this pr
though that I just understood, it should add the user to dex_default_org
only when the organizations in the db user create payload is empty
(meaning, if the dex token contains nothing, the user should be added to
dex default org with default dex role). Will fix this shortly.
- When there is new custom org as per the token, it only adds the new
org, but does not remove existing org if there are not present in the
custom group claim returned by the sso claim parser.
1. search across folders is not working
2. when enabling search across or disabling it calling an api which is
not needed
3. while creating report the add folder is not there
4. while edigint the report user is able to change the folder which is
not consistent with alerts
5. when clicking on any folder if it still loading and user is moved
another folder it shoudl display that folder but it is redirecting user
to the previous folder that was clicked
## Bug
Alert queries using SQL that joins two streams (CTE, direct JOIN,
subquery, UNION) fail
validation with a false `SearchFieldNotFound (20004)` error — even
though the same query
runs fine in Logs UI.
**Example error:**
```
Search field not found: body_type. Field not found in stream schema.
```
The query works in `_search` / `_search_stream` but fails when alerts
calls
`_search?validate=true` or `_search_multi?validate=true` before saving.
---
## Root Cause
`validate_query_fields` in `src/handler/http/request/search/utils.rs`
flattened fields
from **all** streams into one set, then validated every field against
the schema of the
**single primary stream** passed in (`stream_name`).
For a query joining `default` + `k8s_events`, `body_type` (which lives
only in
`k8s_events`) was checked against `default`'s schema — where it doesn't
exist — producing
a false negative.
---
## Behaviour Diagram
### Before (broken)
```
SQL joining "default" + "k8s_events"
│
▼
Sql::new_with_options() — parses SQL, loads both schemas
│
▼
ColumnVisitor assigns fields per stream:
┌──────────────────────────────────────┐
│ sql.columns │
│ "default" → {k8s_pod_name, ...} │
│ "k8s_events" → {body_type, ...} │
└──────────────────────────────────────┘
│
▼ ← BUG
flatten ALL streams into one set
{k8s_pod_name, body_type, event_name, ...}
│
▼
validate every field against "default" schema only
│
┌───────┴────────┐
│ │
k8s_pod_name ✅ body_type ❌ ← only in k8s_events
│
▼
SearchFieldNotFound (20004)
```
### After (fixed)
```
SQL joining "default" + "k8s_events"
│
▼
Sql::new_with_options() — parses SQL, loads both schemas
│
▼
ColumnVisitor assigns fields per stream:
┌──────────────────────────────────────┐
│ sql.columns │
│ "default" → {k8s_pod_name, ...} │
│ "k8s_events" → {body_type, ...} │
└──────────────────────────────────────┘
│
▼
find entry where stream_name == "default"
target_fields = {k8s_pod_name, ...}
│
▼
validate ONLY "default" fields against "default" schema
│
┌───────┴────────┐
│ │
k8s_pod_name ✅ body_type → not in target_fields, skipped ✅
│
▼
Ok(())
```
---
## Affected Query Shapes
All four patterns hit the same bug — any SQL that produces `sql.columns`
with keys for
more than one stream:
```sql
-- CTE + JOIN
WITH x AS (SELECT ... FROM "default") SELECT b.body_type FROM "k8s_events" b JOIN x ...
-- Direct JOIN
SELECT a.col, b.body_type FROM "default" a JOIN "k8s_events" b ON ...
-- Subquery
SELECT col FROM "default" WHERE id IN (SELECT id FROM "k8s_events")
-- UNION
SELECT col FROM "default" UNION SELECT body_type FROM "k8s_events"
```
---
## How `validate_query_fields` is meant to behave
The function validates **one stream at a time**. The handler calls it
once per stream in
the request — full coverage comes from all the calls combined, not from
a single call.
### `_search_multi` (multi-stream, correct path)
```
_search_multi?validate=true with streams: ["default", "k8s_events"]
│
├── validate_query_fields(org_id, "default", sql)
│ → checks default's fields ✅
│ → body_type skipped (belongs to k8s_events) ✅
│
└── validate_query_fields(org_id, "k8s_events", sql)
→ checks k8s_events' fields including body_type ✅
```
Every stream's fields are validated against their own schema. No false
rejections, no gaps.
### `_search` (single-stream endpoint)
```
_search?validate=true (primary stream: "default")
│
└── validate_query_fields(org_id, "default", sql)
→ checks default's fields ✅
→ body_type skipped (belongs to k8s_events) ✅
→ k8s_events fields: not checked ⚠️
```
`k8s_events` fields go unvalidated at pre-check time. This is a
**pre-existing
limitation**, not introduced by this fix. Runtime execution (DataFusion)
still validates
them — if `body_type` doesn't exist in `k8s_events` at query time, the
query fails then
with a clear error. `_search_multi` is the correct API for multi-stream
queries and
provides full pre-validation coverage.
### `_search_stream`
Never calls `validate_query_fields` — goes straight to execution.
Multi-stream joins
work because DataFusion resolves all streams at runtime.
---
## Fix
Look up the `sql.columns` entry that matches `stream_name` and validate
only that
stream's fields against its own schema. Fields from other streams in the
query are
validated separately when the handler calls `validate_query_fields` for
each stream.
```rust
// Before
let used_fields: HashSet<String> = sql.columns.values().flatten().cloned().collect();
// After
let (target_stream, target_fields) = sql
.stream_names
.iter()
.find(|r| r.stream_name() == stream_name)
.map(|r| {
let fields = sql.columns.get(r).cloned().unwrap_or_default();
(stream_name.to_string(), fields)
})
.unwrap_or_else(|| {
// fallback: aliased or unresolved table — original behaviour
let fields = sql.columns.values().flatten().cloned().collect();
(stream_name.to_string(), fields)
});
```
**File:** `src/handler/http/request/search/utils.rs` —
`validate_query_fields()`
---
## Tests
Added `test_cte_multi_stream_regression` module in `utils.rs`:
- `test_cte_join_cross_stream_field_not_rejected` — regression test for
the exact bug:
sets up `default` schema (no `body_type`) + `k8s_events` schema (has
`body_type`),
runs a CTE join query, asserts `validate_query_fields` returns `Ok(())`
for `default`.
Fails on old code, passes after fix.
This fixes issues
1. on cloud FE would not do case insensitive comparison when checking
user's role
2. in invites flow BE would not do case insensitive comparison for
checking user's invites
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Updated the logic for ordering partitions based on the SQL order
expression. If the expression is "zo_sql_timestamp DESC", partitions
remain unchanged. For "zo_sql_timestamp ASC", partitions are reversed.
For other fields, a single partition is returned, ensuring compatibility
with the new ordering requirements.
- fix incidents not correlating for alert ID
- SRE SA no more visible in other screens than main SA list
- SreAgent role cannot be assigned to any other user now
- regular user now working for MCP
When validating access token, we should use the email returned from db
query to do `check_permissions` rbac function call. Reason is, in
openfga db, we store the same user email as stored in the openobserve
db. Hence, all the permission is supposed to be evaluated against the
same email stored in the db.
The email decoded from token can have any case (upper/lower) as per
user's sso config, so it is important to follow the case stored in the
db created when the user logged in for the first time.
## Summary
Design at: https://github.com/openobserve/o2-enterprise/issues/1413
- **MCP 2025-11-25 Streamable HTTP transport**: validate
`MCP-Protocol-Version` header (400 on mismatch), return HTTP 202 for
JSON-RPC notifications, add `DELETE /api/{org_id}/mcp` for session
termination
- **Pinned tools**: mark `StreamList`, `StreamSchema`, `SearchSQL`,
`PrometheusRangeQuery`, `GetIncident` with `"pinned": true` in
`x-o2-mcp` extension so they appear directly in `tools/list`
## Test plan
- [x] POST with no `id` field (notification) → 202 Accepted, no body
- [x] POST with wrong `MCP-Protocol-Version` header → 400 Bad Request
- [x] DELETE `/api/{org_id}/mcp` → 204 No Content
- [x] `tools/list` response includes the 5 pinned tools directly
- [x] Pinned tools callable directly via `tools/call name=StreamList`
without `tool_search` first
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Huaijin <haohuaijin@gmail.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Following fixes and enhancements have been made:
**Service Discovery**:
- default field tracker is now used to distinguish services as well in
absence of a user config, the system works out of box
- UI issues
- incorrect field was displayed when a card in recommendation was
clicked
- scope/workload selectors were leftover
- stream based service search
**Incidents**:
- AI Analysis failure now emits a failure event correctly, avoiding
false-positive and preventing UI from getting stuck when a prior
`AIAnalysisBegin` event remained in place
- correlate alerts by alert ID, this was a regression after correlation
revamp, useful for correlation in absence of alert ID
- fix sqlx issue wherein `u64` wasn't supported for sqlite, now turned
into `i64`
**Service graph**:
- now ingests using HTTP bulk API. Was failing since recently moved
service graph daemon to alertmanager, which used internal ingest call,
which was unsupported on alertmanager
- Also add Python 2.0 license support for JS
---------
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: ByteBaker <undefined@users.noreply.github.com>
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Fixes#11070
When `ZO_BASE_URI` is set, the rum endpoints get broken. This is because
when we strip prefix in the `validator_rum` functiom, it checks for
`{base_uri}/v1` endpoint. But axum already strips the nesting path when
the `req_data.uri()` is called.
The routing tree for a sub-path deployment `(base_uri = "/openobserve")`
is:
```
nest("/openobserve", app) ← outer: strips /openobserve
└── merge(other_service_routes())
└── nest("/rum", rum_routes) ← inner: strips /rum
└── layer(rum_auth_middleware) ← middleware runs here
```
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
## Summary
Implement automatic provisioning and lifecycle management of
per-organization system service accounts for the AI-powered SRE Agent.
These accounts serve as the identity for RCA (Root Cause Analysis) agent
API interactions and are created automatically for each organization.
## Key Features
### Service Account Provisioning
- Database migration creates SRE Agent accounts for all existing
organizations
- Deterministic email format:
`o2-sre-agent.org-<ORG_ID>@openobserve.internal`
- System accounts are automatically created on organization creation
- Idempotent operations ensure safe re-runs and self-healing
### System Account Protection
- API endpoints prevent modification/deletion of system service accounts
(403 Forbidden)
- User responses include `is_system` flag to identify system accounts
- Protection enforced at HTTP handler layer
### RCA Agent Client Refactoring
- Singleton pattern: global client initialized once at startup
- Per-request user credential forwarding for better audit trails and
scoping
- Lazy initialization: returns None if agent service not configured
- Updated all query methods to accept and forward auth credentials
### Incident Correlation Enhancement
- Query `ServiceIdentityConfig` to get configured semantic dimension
groups
- Extract only `distinguish_by` fields from matched identity sets
- Use parallel correlation execution (Service Discovery + semantic
extraction)
- Improved accuracy by using configured groups instead of fallback label
extraction
### Improved Token Redaction
- Consistent 12-character format: first 4 visible characters + 8
asterisks
- Better security for token display in UI
- Updated tests for new redaction format
## Technical Details
### Database Changes
- New migration:
`m20260331_000001_create_sys_rca_agent_service_accounts`
- Creates accounts with organization and RUM tokens
- Safe pagination (100 orgs at a time) with idempotency checks
- System accounts use role type 6 (SreAgent, distinct from
ServiceAccount)
### Authorization
- System accounts added to OpenFGA with standard org membership tuples
- SRE-specific read-only role for permission control
- Self-healing: runtime ensures OpenFGA state is consistent with
database state
### API Changes
- GET /users: Returns `is_system` and `description` fields
- PUT/DELETE /users: Reject modification/deletion of system accounts
## Test Coverage
- Token redaction tests updated for new format
- System service account protection verified
- User list pagination tested with system accounts
## Feature Flag
This is an enterprise feature gated behind `#[cfg(feature =
"enterprise")]`. System accounts are always created, but RCA integration
only works if the agent service is configured.
## Deployment
- Zero-downtime: migration is additive
- Opt-in for OSS deployments
- Automatic provisioning on upgrade and org creation
---------
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
**Issue**
Logs query generation used raw field names in SELECT [FIELD_LIST] and
non-SQL WHERE filters. If a field name is a SQL reserved word (for
example user, role), the generated SQL could be invalid or fail to
parse.
**Fix**
Field names are now consistently wrapped in double quotes when building
SQL:
- Quick mode field list ([FIELD_LIST])
- Non-SQL filter tokens converted into WHERE clauses
Tests were updated to assert quoted-field behavior for both quick mode
and filter conversion.
---------
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
## Summary
Design at:
https://github.com/openobserve/designs/blob/main/infrastructure/work-group/work_group_v2_impl.md
This PR adds slot-based distributed admission control for search queries
and fixes file dispatch consistency when a subset of querier nodes is
selected per query.
To enable this feature we need to set:
```
O2_WORK_GROUP_MAX_NODES_PER_QUERY > 0
O2_WORK_GROUP_NODE_SELECTION_STRATEGY = org | stream
```
### Slot-based admission
- Add `TryAcquire / StartQuery / ReleaseQuery` gRPC RPCs — Follow nodes
reserve slots before execution starts and self-release via
`FlightEncoderStream::drop`
- Wire slot admission in both Arrow Flight search and PromQL paths,
co-located with the existing org/user concurrency queue check
- Delegate DataFusion `target_partitions` / `memory_size` to enterprise
`get_resource_limit()`, which applies per-query slot fraction with
optional dynamic expansion
- TTL sweep interval derived from `slot_reserved_ttl_ms / 4` (min 100
ms) instead of a hardcoded 500 ms
- Move `QUERY_PENDING_NUMS` / `QUERY_RUNNING_NUMS` metric updates to
after slot admission completes (a blocked request is no longer counted
as "running")
### Consistent-hash file dispatch fix (`max_nodes_per_query > 0`)
- Add `get_node_from_consistent_hash_within(key, role, group, allowed)`
to `infra::cluster`: walks the global ring clockwise from `hash(key)`
but skips nodes outside `allowed`. Preserves all consistent-hash
properties (even distribution via vnodes, minimal remapping on
membership change) while confining dispatch to the selected subset
- `partition_file_by_hash` now passes only the selected querier nodes as
the `allowed` set, so every file maps to one of the N selected nodes —
no more fallback-to-index-0 imbalance when `strategy=org/stream`
- File-list broadcast (enterprise) pre-computes per-item target nodes
using the same filtered ring walk via `broadcast_item_targets()` in
`o2-enterprise`, so broadcast and dispatch always agree on which node
owns each file
### Proto fix
- Reserve field 10 in `QueryStatus` (was `WorkGroup` enum / varint wire
type) and move `work_group` string to field 13 — prevents wire-type
mismatch during rolling upgrades
### Other fixes from code review
- `get_first_stream_key()` now returns `"{stream_type}/"` sentinel
instead of `""` for multi-stream / cross-index queries, so
`select_nodes` always receives a non-empty deterministic key
- PromQL node selection uses `"metrics/"` stream key (consistent with
the sentinel format) instead of bare `"metrics"`
- Move `_orig_trace_id` binding inside `#[cfg(feature = "enterprise")]`
block in `FlightEncoderStream::drop`
## Key config (all default to safe/off values)
| Env var | Default | Description |
|---|---|---|
| `O2_WORK_GROUP_MAX_NODES_PER_QUERY` | `0` | Nodes per query; 0 = all
nodes (disables slot admission) |
| `O2_WORK_GROUP_NODE_SELECTION_STRATEGY` | `all` | `all` / `org` /
`stream` |
| `O2_WORK_GROUP_SHORT_SLOT_RATIO` | `0.8` | Max fraction of node slots
for short queries |
| `O2_WORK_GROUP_DYNAMIC_RESOURCE` | `true` | Expand DataFusion
allocation on idle nodes |
## Test plan
- [x] Default (`max_nodes_per_query=0`): behavior identical to before,
zero overhead
- [x] `strategy=org`, `max_nodes_per_query=3`, 10-node cluster: each org
consistently uses the same 3 nodes; files distributed evenly across
those 3 nodes (no node-0 pile-up)
- [x] Broadcast routes each file to the same node that
`partition_file_by_hash` selects at query time → cache hits on
re-queries
- [x] Slot admission: 3rd request waits when budget=2 and 2 queries
running; slots released on completion
- [x] PromQL queries respect slot budget
- [x] Rolling upgrade: old node sends `QueryStatus` without field 10
conflict
## Related
Requires companion PR in `o2-enterprise`:
https://github.com/openobserve/o2-enterprise/pull/1378🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Loakesh Indiran
Ashish Kolhe
Harsh MahajanandShrinath Rao
ktx-kirtan
Subhra264
HuaijinandHengfei Yang
Bhargav
ByteBakerandAshish Kolhe
漂流
GobinathALandHengfei Yang
