Fixes#3764
Supersedes #13195 (stale base, unsigned CLA, and non-rendering — see
Units below). The event-parsing and exception-detection approach from
that PR is reused, credited via `Co-authored-by`.
## What this adds
- **Trace waterfall** — a marker per span event, positioned against the
trace window alongside the span bar. Clicking one selects its span.
- **Sidebar mini-timeline** — the same events re-plotted against the
*span's own* duration, above the Events tab. Answers "where within the
span did this fire?", which the waterfall can't show for short spans and
the table only shows as absolute timestamps. Clicking a marker expands
the matching row.
- **Exceptions stand out** — OTel semconv detection (`exception` name or
any flattened `exception.*` attribute) renders in the error token; other
events in amber. Both are existing registered tokens, so theming is
automatic and no new tokens were added.
Issue #3764 asks for "dots for events **and** exceptions", so all events
are marked with exceptions emphasised, rather than filtering to
exceptions only.
## Units — why the previous attempt rendered nothing
A span event's `_timestamp` is raw OTLP `time_unix_nano`
(**nanoseconds**), stored unconverted:
Frontend span timing is **microseconds**. The same field name therefore
means different units on a span than on its events. #13195 compared the
two directly, so every marker landed ~1000x outside the window and was
silently dropped by the range filter — matching the "I couldn't see the
UI changes" report on that PR. The `ns → µs` conversion is now confined
to one composable and pinned by a regression test using nanosecond
fixtures.
Corroboration that events are nanoseconds: the sidebar's own events
table already divides the same field by `1e6` to render milliseconds.
## Structure
`web/src/composables/traces/useSpanEvents.ts` owns event semantics —
parse (array or JSON string), convert, classify — while each caller
supplies its own window. The two surfaces differ only in that window.
Events with no usable timestamp, or outside the window, are skipped
rather than pinned to an invented position.
## Verification
- `npm run type-check` — clean
- ESLint on changed files — 0 errors, 0 warnings
- `prettier --check` — conforms
- Vitest — **184 passed**: 27 new (`useSpanEvents.spec.ts`), 7 new
(`SpanBlock.spec.ts`), 7 new (`TraceDetailsSidebar.spec.ts`), rest
pre-existing
Pre-existing and untouched: three `no-dupe-keys` errors in
`eslint.config.js`, and a `vue/no-dupe-keys` warning for `activeTab` in
`TraceDetailsSidebar.vue` (confirmed present on unmodified `main`).
## Reviewer notes
- **Not yet validated against a live ingested trace.** That step is what
would have caught #13195, and it is the one gap here.
- Specifically worth checking: `gen_ai.*` events are constructed with
`_timestamp: 0` in the OTel processor's test fixtures. If real SDK data
does the same, LLM spans will show no markers (dropped rather than
mispositioned, by design). If that turns out to be real, the follow-up
is an explicit "N events without timestamps" affordance, not a
fabricated position.
- Marker clustering for very dense event sets is deliberately deferred
until real traces show it is needed.
---------
Co-authored-by: coyaSONG <66289470+coyaSONG@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Paired with o2-enterprise `hf/issue-2491-admission-slot-leak` (same
branch name).
**Stacked on #13953** — base is that branch, so the diff here is just
this change.
Retargets to `main` automatically once #13953 merges.
## Problem
#13953 fixed the wal file locks. `do_get` touches two more trace-id
keyed
resources, and both had the identical defect: release depended on
control flow
reaching a call, inside the window where gRPC cancellation drops the
future.
**The node's admission slot.** Released from three hand-written sites —
the two
error arms in `do_get` and `FlightEncoderStream::drop`. All three are
skipped
when the request is cancelled during plan generation, because the
encoder
stream does not exist yet.
**The follower task status.** In the super-cluster path the entry was
inserted
before plan generation and removed after it:
```rust
if is_super_cluster && !SEARCH_SERVER.contain_key(&trace_id).await {
SEARCH_SERVER.insert(trace_id.clone(), TaskStatus::new_follower(vec![], false)).await;
}
let result = get_ctx_and_physical_plan(&trace_id, &req).await; // <-- cancellable
if is_super_cluster && !SEARCH_SERVER.is_leader(&trace_id).await {
SEARCH_SERVER.remove(&trace_id, false).await;
}
```
Cancel in the middle and the entry stays in the map for the life of the
process. Nothing else removes it: an explicit `cancel_query` would, but
a
timeout or a client disconnect would not.
## Approach
Both move onto the same ownership chain #13953 established.
`SessionGuard` — already armed on the first line of `do_get` and already
carried into `FlightEncoderStream` — now releases the admission slot
too, so
the three call sites collapse into its drop.
The follower task status is registered through an `AsyncDefer` that
lives in
the same scope as plan generation, so it is removed on every exit
including a
cancelled one. It is now removed only when this node registered it,
rather than
whenever the entry is not a leader's: a request can no longer clear an
entry it
did not create. In practice these coincide, since the composite
`{trace_id}-{job_id}`
is unique per request.
### Drive-by
The `physical plan generate error` and `physical plan execution error`
logs sat
*inside* `if get_o2_config().work_group.max_nodes_per_query > 0`, so a
plan
failure was silent unless work groups happened to be configured — and
silent in
every OSS build. They are unconditional now.
## Enterprise side
The paired PR gives `Running` reservations a TTL backstop. They had no
expiry
edge at all, so a slot whose release was lost — this cancellation path,
or a
leader that dies before sending `Release` — was held until the process
restarted. This PR closes the local path; the backstop covers the remote
one,
which no local guard can.
## Verification
- `cargo fmt --all` and the CI clippy command: clean
- Enterprise verified via the `Cargo.toml.openobserve` swap against the
paired
branch: `cargo check --all-targets` clean, and `cargo clippy --no-deps
-p openobserve-api-grpc --all-targets` clean. The workspace-wide
enterprise
clippy still reports the two pre-existing `cognitive_complexity` errors
in
`core/src/alerts/scheduler/handlers.rs` and `core/src/ofga/mod.rs`,
untouched
by this change
- `api-grpc` flight tests 21/21
## Also here
Three zh-CN strings whose English is *events* said 活动 (activity).
Unrelated to
the rest of the PR, carried along at review request.
| key | en-US | was | now |
|---|---|---|---|
| `home.docsCountLbl` | Events | 活动 | 事件 |
| `rum.searchEvents` | Search Events | 搜索活动 | 搜索事件 |
| `logs.noEvents.title` | No events found | 未找到任何活动 | 未找到任何事件 |
zh-TW already had 事件 for all three, and every other locale translates
them as
events. `logs.noEvents` also contradicted itself — `descNoFilters`
directly
below the title already said 未找到任何事件.
## Not in this PR
The `QueryManager` map has a `created_at` on every entry but no reaper.
With
this change its entries are removed on cancellation, so it is no longer
known
to grow — but it has no backstop the way the ledger now does.
## Problem
RUM-to-traces correlation stopped working after the browser SDK 0.4.x
line. The SDK's identifiers became plain `BigInt`, so the unchanged
`toString(16)` serialization silently started dropping leading zeros.
Trace ids are UUIDv7-shaped (48-bit ms timestamp in the top bits), so
the leading hex nibble is always zero today — effectively every RUM
event stored a 31-char `_oo_trace_id`/`_o2_trace_id` while the traces
stream stores the padded 32-char `trace_id` (the `traceparent` header
was padded correctly, so backend traces have the full id). Exact-match
correlation could never hit, in either direction.
Verified against live data: for sampled traced requests, `'0' +
stored_id` found the backend trace every time; the stored id never did.
## Fix (frontend compatibility layer)
Works for the already-ingested short ids **and** for correct ids from
the fixed SDK:
- `utils/rum/fields.ts`: `normalizeTraceId` (pads to canonical 32-char
hex), `traceIdLookupVariants` (padded + legacy zero-stripped forms),
`rumFieldEqualsAnySql` (matches both namespace spellings x both id
forms), and a shared `RUM_CORRELATION_TRACES_STREAM` constant replacing
four scattered "default" literals.
- `useRumSpanBuilder`: trace→RUM lookup queries both id forms; canonical
id stamped on generated RUM spans.
- `useTraceCorrelation`: backend-span query now targets the real
`default` traces stream (was a nonexistent `_traces` stream) as
`page_type: "traces"` (was `"logs"`), with the padded id.
- `EventDetailDrawerContent`: trace navigation pads the id before
building the route.
- `PlayerTracesTab`: the session Traces tab canonicalizes ids before the
traces-stream metadata join (rows were silently dropped).
- `ErrorViewer`: error correlation card receives the padded id;
implausible (non-hex) ids hide the card instead of showing a misleading
empty state.
- `package.json`: browser SDK bumped to 0.4.2-beta.3, which serializes
trace/span ids as fixed-width canonical hex at the source (resolved
`browser-rum-core` verified at 0.4.2-beta.3).
## Testing
- New unit coverage: `fields.spec.ts` (16 tests) plus regression tests
in each touched spec (legacy 31-char id fixtures asserting padded
32-char behavior); full RUM/traces suites green (53 files, 1971 passed).
- Live-verified against real data: a trace from the broken window
renders the full RUM hierarchy (Session→View→Action→Resource + backend
spans) where the old code showed backend spans only; pre-regression
sessions still correlate (no behavior change for correct ids).
- `npm run lint` and `npm run type-check` clean.
Design At: designs/dashboards/interactive-table-cell-drilldown
What changed
New component: DashboardLogDrawer.vue
Clicking a cell in a dashboard table panel now opens a side drawer
scoped to that field/value filter. The drawer shows filtered log events
with a SQL editor (Ctrl+Enter to run, Reset to restore default),
server-side pagination with configurable page size (50/100/200/500), and
an event detail sub-drawer on row click.
Event detail has three tabs:
Insights — Field Anomaly Profile (rarity scoring vs current page), Event
Timeline (SVG histogram of event frequency across the selected time
range), Log Patterns (grouped by structure via /patterns/extract, each
with "Open in Logs" that prefills the log explorer), and Surrounding
Events (events from the same stream in a configurable ±1–10 min window,
paginated 20/page with per-row expand)
Details — key/value field list with copy buttons
JSON — raw event JSON
Modified: PanelSchemaRenderer.vue
Fixed type mismatch: route.query.cell_panel returns string; panel schema
IDs may be number — comparison now uses String() on both sides
Added cell_event_ts to the URL param cleanup list so it's cleared when
the outer log drawer closes
Modified: en-US.json — added complete panel.logExplorer key tree
including the new insights subsection
Modified: design-debt-baseline.json — updated baseline after audit (new
file's rawVarInComponent uses are all in sanctioned color-mix()
positions)
URL persistence
All drawer state is encoded in URL params (cell_panel, cell_field,
cell_value, cell_stream, cell_stype, cell_t0, cell_t1, cell_event_ts).
Sharing the URL reopens both drawers and restores the exact event — on
mount it does a WHERE _timestamp = {ts} targeted fetch (size 1) rather
than searching through paginated results.
Zero backend changes
Uses only existing endpoints: search (SQL), search (histogram via
histogram() function), and patterns/extract.
---------
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech>
Co-authored-by: Abhay Padamani <141122205+ktx-abhay@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Shrinath Rao
Abhay PadamaniandHengfei Yang
ktx-vaidehi
Omkar Kesarkhane
Benziza
Prabhat Sharma
Bhargav
sidharth-sinhasaneandCursor
Loakesh Indiran
Ashish Kolhe
Vaibhav Mittal
