## Summary
- Change `detect_llm_stream` to accept `impl Fn(&str) -> bool` instead
of `&HashMap<String, json::Value>`, so the same helper works for both
`HashMap` (span attributes path) and `json::Map` (JSON ingest path)
without cloning or wrapping.
- Update the two callers in `src/service/traces/mod.rs` to pass `|k|
map.contains_key(k)`.
- Add unit tests covering empty input, no-match, and each LLM attribute
family (GenAi, O2, Langfuse) for both `HashMap` and `json::Map`.
## Test plan
- [x] `cargo check -p openobserve`
- [x] `cargo test -p openobserve --lib
service::traces::tests::test_detect_llm_stream` (7 passed)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- When auto-UDS is triggered, the persisted `defined_schema_fields`
includes `_timestamp`. The UI hides `_timestamp` from the UDS selector
so the user can never deselect it.
- A "remove all UDS fields" action from the frontend therefore leaves
`defined_schema_fields = [\"_timestamp\"]` on the backend, which is
still treated as UDS enabled — making it impossible to disable UDS
through the UI even after raising `ZO_SCHEMA_MAX_FIELDS_TO_ENABLE_UDS`.
- Fix: in `update_stream_settings`, after applying the remove diff, if
only `_timestamp` is left, clear `defined_schema_fields` so UDS is
disabled.
Refs: openobserve/o2-enterprise#1570
## Test plan
- [x] On a stream where auto-UDS is enabled, open Schema settings and
switch to the User Defined Schema tab.
- [x] Select all visible UDS fields and delete them, then save.
- [x] Verify the response/stored settings have `defined_schema_fields =
[]` and the stream falls back to all fields (no `_timestamp`-only UDS
lingering).
- [x] Re-add fields to UDS and confirm the normal add path still works.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Currently, the v2 list api for reports with `cache=true` uri params
fails with the following error -
```
Query Error: error returned from database: operator does not exist: json = jsonb
```
## Summary
- Reverts the three behavioral changes from #8744 in
`src/infra/src/cluster/mod.rs` (the file has since moved from
`src/common/infra/cluster/mod.rs`).
- Keeps the `heatbeat` → `heartbeat` comment typo fix.
## Why
The original `continue` on a missing `NODES_HEALTH_CHECK` entry was
correct self-healing behavior. The replacement in #8744 introduced three
problems:
1. **Cluster-wide failure from one stale entry.** `check_nodes_status`
iterates all online nodes, so `return Err(...)` on a missing uuid aborts
the entire loop. Every node not yet checked in that tick is silently
skipped.
2. **The proactive init on node add is redundant.** The success path in
`check_nodes_status` already runs
`w.entry(node.uuid.clone()).or_insert(0)`, so a node with a missing
entry was never actually unreachable by the health checker.
3. **There's a race window.** `NODES.write` and
`NODES_HEALTH_CHECK.write` in `Event::Put` are two separate lock
acquisitions. A concurrent `check_nodes_status` reading `NODES` can
observe the node present there but not yet in `NODES_HEALTH_CHECK`.
Pre-#8744 that just `continue`d; post-#8744 it aborted the whole tick.
PR-Agent's own reviewer flagged (1) and (3); those concerns weren't
addressed before merge.
## Test plan
- [ ] `cargo check -p infra` passes (confirmed locally)
- [ ] Manual / staging: bring up a multi-node cluster, bounce one node,
confirm health checks for the remaining nodes continue through the tick
where the bounced node becomes briefly inconsistent.
- [ ] Confirm the failure-threshold eviction path (`times >=
failed_times`) still works for a node that stops responding.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Earlier when map_group_to_role env is enabled, the code flow was
ignoring the map_group_to_role flow when the custom_role was empty
(basically there is no custom role configured in the dex group claim).
This pr fixes this - so no matter whether the custom role is empty or
not, it will go to the map_group_to_role flow (this behaviour was there
earlier, but got broken when the sso claim parser was implemented)
- In the map_group_to_role flow, if the custom role array is empty it
was not adding the sso user to the DEX_DEFAULT_ORG at all even though
the ofga tuple mapping was being done. There is one issue in this pr
though that I just understood, it should add the user to dex_default_org
only when the organizations in the db user create payload is empty
(meaning, if the dex token contains nothing, the user should be added to
dex default org with default dex role). Will fix this shortly.
- When there is new custom org as per the token, it only adds the new
org, but does not remove existing org if there are not present in the
custom group claim returned by the sso claim parser.
When ORDER BY is on a non-timestamp column (e.g. duration), each
partition returns its own local top-N rows. Merging those produces
incorrect global sort order since globally highest values may not appear
in any partition's local top-N.
Add `ZO_DISABLE_PARTITIONS_FOR_NON_TS_ORDER_BY` env var (default false).
When true, non-aggregate non-histogram queries with ORDER BY on a
non-timestamp field return a single full-range partition instead.
1. search across folders is not working
2. when enabling search across or disabling it calling an api which is
not needed
3. while creating report the add folder is not there
4. while edigint the report user is able to change the folder which is
not consistent with alerts
5. when clicking on any folder if it still loading and user is moved
another folder it shoudl display that folder but it is redirecting user
to the previous folder that was clicked
## Bug
Alert queries using SQL that joins two streams (CTE, direct JOIN,
subquery, UNION) fail
validation with a false `SearchFieldNotFound (20004)` error — even
though the same query
runs fine in Logs UI.
**Example error:**
```
Search field not found: body_type. Field not found in stream schema.
```
The query works in `_search` / `_search_stream` but fails when alerts
calls
`_search?validate=true` or `_search_multi?validate=true` before saving.
---
## Root Cause
`validate_query_fields` in `src/handler/http/request/search/utils.rs`
flattened fields
from **all** streams into one set, then validated every field against
the schema of the
**single primary stream** passed in (`stream_name`).
For a query joining `default` + `k8s_events`, `body_type` (which lives
only in
`k8s_events`) was checked against `default`'s schema — where it doesn't
exist — producing
a false negative.
---
## Behaviour Diagram
### Before (broken)
```
SQL joining "default" + "k8s_events"
│
▼
Sql::new_with_options() — parses SQL, loads both schemas
│
▼
ColumnVisitor assigns fields per stream:
┌──────────────────────────────────────┐
│ sql.columns │
│ "default" → {k8s_pod_name, ...} │
│ "k8s_events" → {body_type, ...} │
└──────────────────────────────────────┘
│
▼ ← BUG
flatten ALL streams into one set
{k8s_pod_name, body_type, event_name, ...}
│
▼
validate every field against "default" schema only
│
┌───────┴────────┐
│ │
k8s_pod_name ✅ body_type ❌ ← only in k8s_events
│
▼
SearchFieldNotFound (20004)
```
### After (fixed)
```
SQL joining "default" + "k8s_events"
│
▼
Sql::new_with_options() — parses SQL, loads both schemas
│
▼
ColumnVisitor assigns fields per stream:
┌──────────────────────────────────────┐
│ sql.columns │
│ "default" → {k8s_pod_name, ...} │
│ "k8s_events" → {body_type, ...} │
└──────────────────────────────────────┘
│
▼
find entry where stream_name == "default"
target_fields = {k8s_pod_name, ...}
│
▼
validate ONLY "default" fields against "default" schema
│
┌───────┴────────┐
│ │
k8s_pod_name ✅ body_type → not in target_fields, skipped ✅
│
▼
Ok(())
```
---
## Affected Query Shapes
All four patterns hit the same bug — any SQL that produces `sql.columns`
with keys for
more than one stream:
```sql
-- CTE + JOIN
WITH x AS (SELECT ... FROM "default") SELECT b.body_type FROM "k8s_events" b JOIN x ...
-- Direct JOIN
SELECT a.col, b.body_type FROM "default" a JOIN "k8s_events" b ON ...
-- Subquery
SELECT col FROM "default" WHERE id IN (SELECT id FROM "k8s_events")
-- UNION
SELECT col FROM "default" UNION SELECT body_type FROM "k8s_events"
```
---
## How `validate_query_fields` is meant to behave
The function validates **one stream at a time**. The handler calls it
once per stream in
the request — full coverage comes from all the calls combined, not from
a single call.
### `_search_multi` (multi-stream, correct path)
```
_search_multi?validate=true with streams: ["default", "k8s_events"]
│
├── validate_query_fields(org_id, "default", sql)
│ → checks default's fields ✅
│ → body_type skipped (belongs to k8s_events) ✅
│
└── validate_query_fields(org_id, "k8s_events", sql)
→ checks k8s_events' fields including body_type ✅
```
Every stream's fields are validated against their own schema. No false
rejections, no gaps.
### `_search` (single-stream endpoint)
```
_search?validate=true (primary stream: "default")
│
└── validate_query_fields(org_id, "default", sql)
→ checks default's fields ✅
→ body_type skipped (belongs to k8s_events) ✅
→ k8s_events fields: not checked ⚠️
```
`k8s_events` fields go unvalidated at pre-check time. This is a
**pre-existing
limitation**, not introduced by this fix. Runtime execution (DataFusion)
still validates
them — if `body_type` doesn't exist in `k8s_events` at query time, the
query fails then
with a clear error. `_search_multi` is the correct API for multi-stream
queries and
provides full pre-validation coverage.
### `_search_stream`
Never calls `validate_query_fields` — goes straight to execution.
Multi-stream joins
work because DataFusion resolves all streams at runtime.
---
## Fix
Look up the `sql.columns` entry that matches `stream_name` and validate
only that
stream's fields against its own schema. Fields from other streams in the
query are
validated separately when the handler calls `validate_query_fields` for
each stream.
```rust
// Before
let used_fields: HashSet<String> = sql.columns.values().flatten().cloned().collect();
// After
let (target_stream, target_fields) = sql
.stream_names
.iter()
.find(|r| r.stream_name() == stream_name)
.map(|r| {
let fields = sql.columns.get(r).cloned().unwrap_or_default();
(stream_name.to_string(), fields)
})
.unwrap_or_else(|| {
// fallback: aliased or unresolved table — original behaviour
let fields = sql.columns.values().flatten().cloned().collect();
(stream_name.to_string(), fields)
});
```
**File:** `src/handler/http/request/search/utils.rs` —
`validate_query_fields()`
---
## Tests
Added `test_cte_multi_stream_regression` module in `utils.rs`:
- `test_cte_join_cross_stream_field_not_rejected` — regression test for
the exact bug:
sets up `default` schema (no `body_type`) + `k8s_events` schema (has
`body_type`),
runs a CTE join query, asserts `validate_query_fields` returns `Ok(())`
for `default`.
Fails on old code, passes after fix.
This fixes issues
1. on cloud FE would not do case insensitive comparison when checking
user's role
2. in invites flow BE would not do case insensitive comparison for
checking user's invites
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Updated the logic for ordering partitions based on the SQL order
expression. If the expression is "zo_sql_timestamp DESC", partitions
remain unchanged. For "zo_sql_timestamp ASC", partitions are reversed.
For other fields, a single partition is returned, ensuring compatibility
with the new ordering requirements.
Sub-cluster internal node failures (h2 errors) were silently dropped —
partial_err was never transmitted to the super cluster leader, causing
degraded results to be cached as if they were complete.
Adds PreCustomMessage::PartialErrRefEarly + PartialErrRef to carry
partial_err over the flight wire independently of ScanStats (which can
be disabled at deployment level).
Sent twice for reliability: once before first batch (catches
fast-failing
nodes), once after all batches (final accumulated value). Decoder
dedupes
on receive.
Also elevates node gRPC error log from info → error.
Service graph daemon used to look at stream stats to determine whether
data was ingested in a stream since last service graph aggregation. This
was extremely flaky as the stream stats may face sync issues and may not
be realtime.
We've now switched to using the `usage` stream wherein we get the last
ingestion time for all the trace streams in all the orgs using gRPC
search.
In a recent change, service graph ingestion was moved from ingesters to
alert managers. Since alertmanager cannot run bulk::ingest, we switched
to HTTP API using root credentials.
This created a side effect wherein environments without root
credentials, such as cloud started throwing 401.
To mitigate this, we switched to gRPC API, which is also used for
various other internal calls.
- also fix dark mode on service graph's stream selector
Hengfei Yang
Huaijin
Subhra264
Harsh MahajanandShrinath Rao
ktx-kirtan
vnghiaandHengfei Yang
Bhargav
ByteBakerandAshish Kolhe
