## Summary
Follow-up to #12249. The cross-type collision check in the file upload
handler passes the formatted `stream_name` to `get_url_jobs_for_table`,
but URL jobs are stored with the raw `table_name` from the HTTP path.
When names contain uppercase or special characters,
`format_stream_name("My_Table")` produces `"my_table"`, which won't
match the stored URL job name `"My_Table"` — causing the cross-type
protection to be silently bypassed.
## Fix
Changed `get_url_jobs_for_table(&org_id, &stream_name)` to
`get_url_jobs_for_table(&org_id, table_name.trim())` — matching how the
URL handler itself queries URL jobs at line 299.
## Before/After
```rust
// Before: formatted name passed to URL jobs lookup (WRONG)
get_url_jobs_for_table(&org_id, &stream_name)
// After: raw table_name passed (CORRECT, matches how URL handler queries)
get_url_jobs_for_table(&org_id, table_name.trim())
```
## Notes
- `ENRICHMENT_TABLES` cache key continues to use `stream_name`
(formatted) — that's correct
- `get_url_jobs_for_table` now uses `table_name.trim()` — matching the
URL handler at line 299
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
Bug fix for - https://github.com/openobserve/openobserve/issues/12247
- **Bug:** Creating a URL enrichment table with name `X`, then creating
a file upload enrichment also named `X` (or vice versa), silently
overwrites the first one at every storage layer.
- **Fix:** Added cross-type name collision checks at the API boundary.
The file upload handler now rejects if a URL enrichment already exists
with the same name, and the URL handler rejects if a file upload
enrichment already exists. Same-type updates (append/retry/replace) are
unaffected.
## Root Cause
Enrichment tables are uniquely identified by `(org_id, table_name)`
across all storage layers — database, caches, object store, and
coordinator keys. Both URL-based and file-upload enrichments share the
same `StreamType::EnrichmentTables` and the same key space, so creating
one with the same name as another silently clobbers the previous one.
## Changes
**1 file changed** —
[src/handler/http/request/enrichment_table/mod.rs](src/handler/http/request/enrichment_table/mod.rs)
### `save_enrichment_table` (file upload handler)
Added check after input validation: if the table name already exists in
`ENRICHMENT_TABLES` and has URL jobs, return `400` — "already exists as
a URL enrichment."
### `save_enrichment_table_from_url` (URL handler)
Added check after fetching existing jobs: if the table name exists in
`ENRICHMENT_TABLES` but has no URL jobs, return `400` — "already exists
as a file upload enrichment."
### Behavior matrix
| | Existing: URL | Existing: File Upload | Existing: None |
|---|---|---|---|
| **Create File Upload** | ❌ Reject (400) | ✅ Allow (update) | ✅ Allow |
| **Create URL** | ✅ Allow (update) | ❌ Reject (400) | ✅ Allow |
## Test Plan
- [ ] Manually verify: create a URL enrichment named `test_cross_type` →
attempt file upload with same name → should get 400 error
- [ ] Manually verify: create a file upload enrichment named
`test_cross_type2` → attempt URL creation with same name → should get
400 error
- [ ] Manually verify: same-type updates still work (file→file append,
URL→URL retry/replace)
- [ ] Playwright automation will be added in a follow-up PR in the
`RegressionSet` folder
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Fixes https://github.com/openobserve/o2-enterprise/issues/1796 (item 1:
"Reload Existing URLs" returns error).
## Summary
Commit
[`85d261b5d0`](https://github.com/openobserve/openobserve/commit/85d261b5d0e4b68dbc79ca18b844fc53e2368ee9)
(vulnerability scan fixes, Apr 14 2026) removed the `!retry` guard from
enrichment table URL validation. This broke the **reload** update mode
because the frontend sends an empty URL when retrying/reprocessing
existing URLs — the backend already has them stored and just needs to
re-trigger processing.
## Root Cause
Before the breaking commit, URL validation was guarded by `!retry`:
```rust
if !retry && let Err(err_msg) = validate_enrichment_url(&request_body.url) {
return MetaHttpResponse::bad_request(err_msg);
}
```
The security fix removed the guard under the assumption that skipping
validation on retry was an SSRF bypass vector. However, the retry flow
sends `url=""` intentionally — the existing URLs are in the database.
## Fix
Re-adds the `!retry` guard around URL validation. In retry mode, URL
validation is skipped entirely (the URLs come from DB). All other modes
(create, append, replace, replace_failed) still require a non-empty,
valid URL with full SSRF protection.
## Test Plan
- [ ] Create enrichment table from URL
- [ ] Edit table → Reload existing URLs → Should succeed (no "URL cannot
be empty")
- [ ] Edit table → Add new URL → Should still validate URL format
- [ ] Edit table → Replace all URLs → Should still validate URL format
- [ ] Edit table → Replace failed URL → Should still validate URL format
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
## Problem
`sort_by=duration` (and any non-timestamp sort) in the
`/traces/latest_stream` endpoint had three bugs:
1. All partitions collapsed into one full-range scan
2. Q1 results re-sorted by `trace_start_time`, discarding SQL engine
order
3. `fetch_size` capped at `query_default_limit` (1000), so top-k ranked
on a truncated sample
## Fix
Mirrors the approach from #11654 (_search_stream):
- Detect non-ts ORDER BY generically via \`sql_order_expr\` prefix — no
hardcoded field names
- Run Q1 across **all** partitions; feed deduped hits into a
\`TopKHeap\` bounded to \`k = from + size\`
- Q2 (span details) runs **once** after all partitions complete, over
the narrowed time window of the deliverable traces
- Final hits emitted in heap order — correct global rank by ORDER BY
column
## Refactor
Extracted `run_q2_for_traces` async helper to eliminate ~300 lines of
duplicated code between the ts-path and non-ts-path. Both paths now call
the same helper for the `TraceResponseItem` build + Q2a (per-trace
aggregates) + Q2b (per-service breakdown). Also removed the dead
`"duration"` sort arm from the ts-path and the now-unused `sort_by`
parameter from `process_latest_traces_stream`.
## Known Limitation
`zo_sql_duration` is computed per-partition, so traces whose spans
straddle a partition boundary get an underestimated duration. Rare in
practice; a 2-pass global aggregation would be needed to fix.
## Related
Companion to #11654 — correct pagination for ORDER BY non-timestamp
column in `_search_stream`
related to
- https://github.com/openobserve/openobserve/pull/8003
- https://github.com/openobserve/openobserve/issues/7986
# Why We Can Delete `enable_align_histogram`
## What It Was
`enable_align_histogram` was a query parameter introduced in [PR
#8003](https://github.com/openobserve/openobserve/pull/8003) to fix
[Issue #7986](https://github.com/openobserve/openobserve/issues/7986).
The bug was that in the old two-step search flow (`_search_partition` →
`_search`), the two APIs independently generated histogram intervals,
leading to mismatched time buckets and inaccurate chart rendering in
dashboards.
The flag told `_search_partition` to:
1. Align partition boundaries to histogram interval boundaries (e.g.,
round to the start of a minute/hour/day).
2. Return the computed `histogram_interval` in the response so the
frontend could pass it to `_search`.
## Why It Is Now Dead Code
### 1. The frontend no longer calls `_search_partition`
All frontend search now uses `_search_stream` (HTTP/2 streaming), which
internally calls `search_partition()` with `enable_align_histogram =
false`. The old `_search_partition` endpoint is no longer consumed by
the UI. The only frontend caller (`getValuesPartition` in
`IndexList.vue`) is defined but never invoked.
### 2. `search_stream` handles alignment inherently
In the streaming search flow, partitions themselves define the search
time ranges. Each partition search runs on the exact aligned time
boundaries returned by the partition step. There is no separate
histogram query that needs an externally provided interval — alignment
is "baked into" the partition boundaries.
### 3. Histogram queries are already covered by `is_complex_query`
For histogram queries (which have `GROUP BY`), `is_complex_query`
returns `true`. Combined with `has_ts_column = true`, this triggers
`calculate_partition_settings()` to align `min_step` to the histogram
interval — the same alignment behavior that `enable_align_histogram`
previously provided, but without the extra flag.
### 4. All backend callers pass `false`
Of the 5 callers of `search_partition()`:
| Caller | Value |
|--------|-------|
| gRPC handler | hardcoded `false` |
| streaming execution | hardcoded `false` |
| traces handler | hardcoded `false` |
| HTTP `_search_partition` | from query param (frontend always passed
`true`, but frontend no longer calls this) |
| HTTP `_search_partition_multi` | from query param (frontend always
passed `true`, but frontend no longer calls this) |
No active code path ever sets `enable_align_histogram = true`.
## What Was Removed
- `get_enable_align_histogram_from_request()` — the HTTP query param
parser
- The `enable_align_histogram` parameter from:
- `search_partition()` function signature
- `search_partition_multi()` function signature
- `calculate_partition_settings()` function signature
- The alignment logic in `search_partition()`:
- `add_mini_partition` (always `false` without the flag)
- `PartitionGenerator` align parameter (now driven by `is_histogram`
alone)
- `resp.histogram_interval` override block
- The `|| enable_align_histogram` branch in
`calculate_partition_settings`
- The query parameter from both HTTP handler `utoipa` specs
- The `false` argument from all 5 call sites
## Problem
`ORDER BY <non-timestamp col>` (e.g. `ORDER BY duration DESC`) gave
wrong results and risked OOM because per-partition \`hits_to_skip\`
assumed rows are time-ordered — but the globally k-th row can be in any
partition.
## Solution
Each partition fetches its local top-`k = from+size` rows. A
**TopKHeap** on the leader merges them globally, skips `from`, returns
`size`. Heap is bounded to `k` elements at all times — O(k) memory
regardless of partition count.
**Multi-column ORDER BY** (`ORDER BY duration DESC, _timestamp DESC`) is
fully honored: heap comparison and final sort respect all columns in
order, using secondary columns as tiebreakers.
## Scenario behavior
| Query | Behavior |
|---|---|
| `ORDER BY _timestamp DESC` | Unchanged — per-partition path |
| `ORDER BY duration DESC` | Heap merge path — correct pagination |
| `ORDER BY duration DESC, _timestamp DESC` | Heap merge; ties in
`duration` broken by `_timestamp` |
| `ORDER BY duration DESC` page 2 | Heap collects global top-k, skips
`from`, returns `size` |
| JOIN / subquery / GROUP BY | `is_aggregate=true` → single partition
(unchanged) |
| `ZO_DISABLE_PARTITIONS_FOR_NON_TS_ORDER_BY=true` | Forces 1 partition
(circuit breaker preserved) |
| Partial cache hit + non-ts ORDER BY | TODO — pre-existing bug, no
regression |
## What is NOT changed
- `ORDER BY _timestamp` execution path
- `_search` (non-streaming) — DataFusion handles it correctly
- Aggregate / histogram queries
This adds support for current cycle for stripe users as period for usage
page
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
## Summary
Cherry-picks two correlation fixes from `branch-v0.90.0` to `main`:
- `a077b760c` fix: corelation custom groups (#11662) — adds `group`
field to `FoundGroup` and threads `semantic_groups` through
`calculate_dimension_analytics`
- `211876a97` fix: corelation settings fixes (#11666) — validates
tracked alias IDs against bundled defaults + custom DB groups; sanitizes
payload (drops `service` and orphan IDs) before save
Conflicts resolved by taking v0.90.0 versions (the fixes are the point
of the cherry-pick). Companion PR on o2-enterprise:
openobserve/o2-enterprise#1648 — that PR is currently red because it
depends on the new `FoundGroup.group` field landing here first.
## Test plan
- [ ] CI green (build, unit, playwright)
- [ ] Verify save Identity Config rejects invalid alias IDs but accepts
default + custom IDs
- [ ] Verify `_analytics` endpoint returns `group` category on
FoundGroup
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
- **fix: llm insights tab**
- **remove custom attributes and use gen ai spec**
---------
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
## Summary
- Replace `SELECT *` Q2 with grouped aggregations so per-trace
enrichment is bounded by N rows instead of total span count.
- Q2a aggregates per `trace_id`: span count, error count, min/max times,
max duration, root span labels (with first-by-timestamp fallback), and
`approx_distinct(service_name)` to gate Q2b.
- Q2b runs only for multi-service traces, returning per-`(trace_id,
service_name)` count and max duration.
## Root cause
When a user filters Traces by Duration, Q2 ran `SELECT * FROM stream
WHERE trace_id IN (...)` and aggregated spans in Rust. For traces with
tens of thousands of spans, the search-service row cap was hit before
every `trace_id` got at least one span back. The un-enriched entries
were still sent to the FE with `spans=[0,0]`, `duration=0`,
`first_event=null` — the blank rows in the screenshot on the issue.
The new Q2a returns one row per `trace_id` (≤ N rows total), so it
cannot trip the row cap. Single-service traces are fully populated from
Q2a alone; only multi-service traces require Q2b. Response shape
(`TraceResponseItem`) is unchanged — `first_event` is now built as a
synthetic `{service_name, operation_name}` object so the FE in
`web/src/composables/useTraces.ts` keeps working unchanged.
Applied to both `get_latest_traces` and `process_latest_traces_stream`.
Closes#11505
## Test plan
- [ ] Open Traces, pick a stream with traces that have many spans (≥10K
spans/trace).
- [ ] Filter by Duration `<= Max value`. Confirm no blank rows; every
row has a service icon, service name, non-zero duration, and a span
count.
- [ ] Single-service stream: confirm the services list under each trace
is correct.
- [ ] Multi-service trace (e.g., a trace that crosses several services):
confirm the per-service breakdown still appears (Q2b path).
- [ ] Sort by duration ASC and DESC.
- [ ] Sort by start_time ASC and DESC.
- [ ] LLM stream: confirm LLM token / cost columns still populate.
- [ ] Pagination: page 1, page 2, page N — every page has fully
populated rows.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Hengfei Yang <undefined@users.noreply.github.com>
- Earlier when map_group_to_role env is enabled, the code flow was
ignoring the map_group_to_role flow when the custom_role was empty
(basically there is no custom role configured in the dex group claim).
This pr fixes this - so no matter whether the custom role is empty or
not, it will go to the map_group_to_role flow (this behaviour was there
earlier, but got broken when the sso claim parser was implemented)
- In the map_group_to_role flow, if the custom role array is empty it
was not adding the sso user to the DEX_DEFAULT_ORG at all even though
the ofga tuple mapping was being done. There is one issue in this pr
though that I just understood, it should add the user to dex_default_org
only when the organizations in the db user create payload is empty
(meaning, if the dex token contains nothing, the user should be added to
dex default org with default dex role). Will fix this shortly.
- When there is new custom org as per the token, it only adds the new
org, but does not remove existing org if there are not present in the
custom group claim returned by the sso claim parser.
1. search across folders is not working
2. when enabling search across or disabling it calling an api which is
not needed
3. while creating report the add folder is not there
4. while edigint the report user is able to change the folder which is
not consistent with alerts
5. when clicking on any folder if it still loading and user is moved
another folder it shoudl display that folder but it is redirecting user
to the previous folder that was clicked
## Bug
Alert queries using SQL that joins two streams (CTE, direct JOIN,
subquery, UNION) fail
validation with a false `SearchFieldNotFound (20004)` error — even
though the same query
runs fine in Logs UI.
**Example error:**
```
Search field not found: body_type. Field not found in stream schema.
```
The query works in `_search` / `_search_stream` but fails when alerts
calls
`_search?validate=true` or `_search_multi?validate=true` before saving.
---
## Root Cause
`validate_query_fields` in `src/handler/http/request/search/utils.rs`
flattened fields
from **all** streams into one set, then validated every field against
the schema of the
**single primary stream** passed in (`stream_name`).
For a query joining `default` + `k8s_events`, `body_type` (which lives
only in
`k8s_events`) was checked against `default`'s schema — where it doesn't
exist — producing
a false negative.
---
## Behaviour Diagram
### Before (broken)
```
SQL joining "default" + "k8s_events"
│
▼
Sql::new_with_options() — parses SQL, loads both schemas
│
▼
ColumnVisitor assigns fields per stream:
┌──────────────────────────────────────┐
│ sql.columns │
│ "default" → {k8s_pod_name, ...} │
│ "k8s_events" → {body_type, ...} │
└──────────────────────────────────────┘
│
▼ ← BUG
flatten ALL streams into one set
{k8s_pod_name, body_type, event_name, ...}
│
▼
validate every field against "default" schema only
│
┌───────┴────────┐
│ │
k8s_pod_name ✅ body_type ❌ ← only in k8s_events
│
▼
SearchFieldNotFound (20004)
```
### After (fixed)
```
SQL joining "default" + "k8s_events"
│
▼
Sql::new_with_options() — parses SQL, loads both schemas
│
▼
ColumnVisitor assigns fields per stream:
┌──────────────────────────────────────┐
│ sql.columns │
│ "default" → {k8s_pod_name, ...} │
│ "k8s_events" → {body_type, ...} │
└──────────────────────────────────────┘
│
▼
find entry where stream_name == "default"
target_fields = {k8s_pod_name, ...}
│
▼
validate ONLY "default" fields against "default" schema
│
┌───────┴────────┐
│ │
k8s_pod_name ✅ body_type → not in target_fields, skipped ✅
│
▼
Ok(())
```
---
## Affected Query Shapes
All four patterns hit the same bug — any SQL that produces `sql.columns`
with keys for
more than one stream:
```sql
-- CTE + JOIN
WITH x AS (SELECT ... FROM "default") SELECT b.body_type FROM "k8s_events" b JOIN x ...
-- Direct JOIN
SELECT a.col, b.body_type FROM "default" a JOIN "k8s_events" b ON ...
-- Subquery
SELECT col FROM "default" WHERE id IN (SELECT id FROM "k8s_events")
-- UNION
SELECT col FROM "default" UNION SELECT body_type FROM "k8s_events"
```
---
## How `validate_query_fields` is meant to behave
The function validates **one stream at a time**. The handler calls it
once per stream in
the request — full coverage comes from all the calls combined, not from
a single call.
### `_search_multi` (multi-stream, correct path)
```
_search_multi?validate=true with streams: ["default", "k8s_events"]
│
├── validate_query_fields(org_id, "default", sql)
│ → checks default's fields ✅
│ → body_type skipped (belongs to k8s_events) ✅
│
└── validate_query_fields(org_id, "k8s_events", sql)
→ checks k8s_events' fields including body_type ✅
```
Every stream's fields are validated against their own schema. No false
rejections, no gaps.
### `_search` (single-stream endpoint)
```
_search?validate=true (primary stream: "default")
│
└── validate_query_fields(org_id, "default", sql)
→ checks default's fields ✅
→ body_type skipped (belongs to k8s_events) ✅
→ k8s_events fields: not checked ⚠️
```
`k8s_events` fields go unvalidated at pre-check time. This is a
**pre-existing
limitation**, not introduced by this fix. Runtime execution (DataFusion)
still validates
them — if `body_type` doesn't exist in `k8s_events` at query time, the
query fails then
with a clear error. `_search_multi` is the correct API for multi-stream
queries and
provides full pre-validation coverage.
### `_search_stream`
Never calls `validate_query_fields` — goes straight to execution.
Multi-stream joins
work because DataFusion resolves all streams at runtime.
---
## Fix
Look up the `sql.columns` entry that matches `stream_name` and validate
only that
stream's fields against its own schema. Fields from other streams in the
query are
validated separately when the handler calls `validate_query_fields` for
each stream.
```rust
// Before
let used_fields: HashSet<String> = sql.columns.values().flatten().cloned().collect();
// After
let (target_stream, target_fields) = sql
.stream_names
.iter()
.find(|r| r.stream_name() == stream_name)
.map(|r| {
let fields = sql.columns.get(r).cloned().unwrap_or_default();
(stream_name.to_string(), fields)
})
.unwrap_or_else(|| {
// fallback: aliased or unresolved table — original behaviour
let fields = sql.columns.values().flatten().cloned().collect();
(stream_name.to_string(), fields)
});
```
**File:** `src/handler/http/request/search/utils.rs` —
`validate_query_fields()`
---
## Tests
Added `test_cte_multi_stream_regression` module in `utils.rs`:
- `test_cte_join_cross_stream_field_not_rejected` — regression test for
the exact bug:
sets up `default` schema (no `body_type`) + `k8s_events` schema (has
`body_type`),
runs a CTE join query, asserts `validate_query_fields` returns `Ok(())`
for `default`.
Fails on old code, passes after fix.
This fixes issues
1. on cloud FE would not do case insensitive comparison when checking
user's role
2. in invites flow BE would not do case insensitive comparison for
checking user's invites
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Updated the logic for ordering partitions based on the SQL order
expression. If the expression is "zo_sql_timestamp DESC", partitions
remain unchanged. For "zo_sql_timestamp ASC", partitions are reversed.
For other fields, a single partition is returned, ensuring compatibility
with the new ordering requirements.
Sub-cluster internal node failures (h2 errors) were silently dropped —
partial_err was never transmitted to the super cluster leader, causing
degraded results to be cached as if they were complete.
Adds PreCustomMessage::PartialErrRefEarly + PartialErrRef to carry
partial_err over the flight wire independently of ScanStats (which can
be disabled at deployment level).
Sent twice for reliability: once before first batch (catches
fast-failing
nodes), once after all batches (final accumulated value). Decoder
dedupes
on receive.
Also elevates node gRPC error log from info → error.
Harsh Mahajan
yujinqiu
Ashish Kolhe
Neha PandClaude Opus 4.7
Loakesh Indiran
Huaijin
udev
ktx-kirtan
Bhargav
ByteBakerandAshish Kolhe