6995 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Hengfei Yangandgithub-actions[bot]	94a0a58866	fix(i18n): backfill mistranslated locale keys via context-aware regeneration (#13968) ... Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2026-08-26 13:29:11 +00:00
Bhargav	a108194a7e	fix(i18n): correct Spanish product terminology (Logs, Trazas, Spans, … (#13973)	2026-08-26 12:19:53 +00:00
Hengfei Yang	b40094e8eb	fix(search): scan the exact requested range for histogram queries (#13969) ... ## Problem Fixes #13946. The sum of histogram bucket counts exceeded `count(*)` over the same time range (e.g. 2744 vs 2704). #12313 made the leader snap the follower scan `start_time` **down** to the histogram bucket boundary in `RemoteScanNodes::get_remote_node()`, to keep the first chart bucket fully populated. But that widens the actual scan window: records in `[bucket_start, requested_start)` — data the user explicitly excluded — get counted into the first bucket, while `count(*)` (a separate query without `histogram_interval`) scans the exact requested range. The reporter confirmed the diagnosis: aligning `count(*)`'s start to the bucket grid makes the two numbers match. #13896 was the same root cause surfacing on sub-10s ranges (worked around in #13916 by dropping to 1s buckets there, without touching the snap). ## Fix Revert the snap: the scan window sent to followers now equals the requested range exactly. - `distributed_plan/node.rs`: `start_time` passes through unchanged; a comment pins the invariant and a regression test (`test_get_remote_node_keeps_raw_start_time_for_histogram`) asserts the window is not widened when `histogram_interval > 0`. - `sql/histogram.rs`: delete the now-unused `histogram_bucket_start()` (zero remaining references, including in the enterprise repo). - `sql/visitor/histogram_interval.rs`: reword two comments that were written in terms of the snap. The 1s interval floor for sub-10s ranges from #13916 is kept — a bucket wider than the whole range is bad charting regardless of the snap. Histogram semantics are now the same as Elasticsearch/Kibana `date_histogram`: buckets only count in-range records, so `sum(buckets) == count(*)` always holds, and the first bucket may be partial. Safety notes: - The tantivy `SimpleHistogram` fast path is unaffected: it aligns bucket *labels* itself (`start_time - start_time % rounding_by`) and only counts docs matched by the query's own time filter, consistent with the `date_bin` path. - The results cache is unaffected: `write_results` already aligns the cache-accepted window *inward* and drops edge buckets, so partial (and previously over-counted) first buckets never entered the cache. ## Trade-off / follow-up for the UI The left-edge display artifact that motivated #12313 (first bar looks lower because its bucket is cut by the range start) returns by design. Suggested frontend handling, either: - align the picker's `start_time` down to the returned `histogram_interval` and issue **all** page queries (histogram + logs + count) over that aligned range — Grafana's approach; totals stay consistent and no partial bucket exists, or - detect `buckets[0].key < start_time` and dim/annotate the first bar as partial. ## Test plan - `search` crate: 88 histogram tests + 9 node tests pass (includes the new regression test); workspace clippy with CI flags passes. - Single-node E2E: ingested 240 events at 500ms spacing over 120s; queried with mid-bucket start/end (+17s/+107s), interval 30s. `count(*)` = 180; buckets 26+60+60+34 = 180 (match). First bucket correctly holds 26 in-range records instead of the full 60 (old behavior summed to 214).	2026-08-26 10:51:36 +00:00
ktx-akshay	33f0a23f96	test(dashboards): fix broken and false-green E2E specs on alpha (#13912)	2026-08-26 10:28:37 +00:00
Hengfei Yang	7789e90118	fix(i18n): send key-path context and a term glossary to the translation model (#13967) ... Addresses the mechanism half of #13966 — auto-translations were context-blind because `translate_batch()` sent bare string arrays with a one-line system prompt. ## What changed - **Key-path context**: each string is now sent as `{"key": "alerts.incidents.statusOpen", "text": "Open"}` and the prompt tells the model to use the key to disambiguate ("Open" there is an incident status, not the verb). Keys are context only and never echoed into the output; the response format is unchanged. - **Product background + `TERMS` glossary** (`translator.py`): the prompt now introduces OpenObserve and carries one term table — an English `sense` line per ambiguous term that steers all 14 languages at once (an "event" is a record, not a party; a "stream" is a data stream, not a brook; "live" is real-time, not a broadcast), plus optional per-locale pinned renderings (zh-CN/zh-TW seeded: event → 事件, stream → 数据流, live → 实时, trace → 追踪, report → 报表, …). - **Module vocabulary mined from the locale itself**: `build_module_glossary()` extracts the committed `menu.*` translations (Alerts → 告警, Pipelines → 流水线, Dashboards → 仪表盘) and merges them with the pinned terms into a single conflict-free renderings table — pinned terms win, so the prompt can never mandate two renderings for one word. Works for every locale with no hand-maintained tables. - The system prompt is built once per locale and is byte-stable (sorted renderings), so DeepSeek's prefix caching keeps the added prompt tokens cheap. Change detection, validation, batching/split-retry, and the state file are untouched. ## Testing - Stubbed the API client and verified: payload shape, batch/index alignment (string + array leaves), failed-item isolation via split retry, prompt assembly for locales with and without pinned terms, byte-stability of the prompt, and the pinned-vs-module dedup (`Streams` module entry absorbed by pinned `stream`). - `python3 main.py --check` (the pure comparison the merge-queue gate runs) is unaffected. ## Not in this PR Existing wrong translations are not re-sent (state hashes unchanged); the backfill pass and the bootstrap-run gap are tracked as follow-ups in #13966.	2026-08-26 10:20:34 +00:00
3 people	e48d9b2b39	feat(traces): annotate span events and exceptions on the trace timeline (#13715) ... Fixes #3764 Supersedes #13195 (stale base, unsigned CLA, and non-rendering — see Units below). The event-parsing and exception-detection approach from that PR is reused, credited via `Co-authored-by`. ## What this adds - **Trace waterfall** — a marker per span event, positioned against the trace window alongside the span bar. Clicking one selects its span. - **Sidebar mini-timeline** — the same events re-plotted against the *span's own* duration, above the Events tab. Answers "where within the span did this fire?", which the waterfall can't show for short spans and the table only shows as absolute timestamps. Clicking a marker expands the matching row. - **Exceptions stand out** — OTel semconv detection (`exception` name or any flattened `exception.*` attribute) renders in the error token; other events in amber. Both are existing registered tokens, so theming is automatic and no new tokens were added. Issue #3764 asks for "dots for events **and** exceptions", so all events are marked with exceptions emphasised, rather than filtering to exceptions only. ## Units — why the previous attempt rendered nothing A span event's `_timestamp` is raw OTLP `time_unix_nano` (**nanoseconds**), stored unconverted: Frontend span timing is **microseconds**. The same field name therefore means different units on a span than on its events. #13195 compared the two directly, so every marker landed ~1000x outside the window and was silently dropped by the range filter — matching the "I couldn't see the UI changes" report on that PR. The `ns → µs` conversion is now confined to one composable and pinned by a regression test using nanosecond fixtures. Corroboration that events are nanoseconds: the sidebar's own events table already divides the same field by `1e6` to render milliseconds. ## Structure `web/src/composables/traces/useSpanEvents.ts` owns event semantics — parse (array or JSON string), convert, classify — while each caller supplies its own window. The two surfaces differ only in that window. Events with no usable timestamp, or outside the window, are skipped rather than pinned to an invented position. ## Verification - `npm run type-check` — clean - ESLint on changed files — 0 errors, 0 warnings - `prettier --check` — conforms - Vitest — **184 passed**: 27 new (`useSpanEvents.spec.ts`), 7 new (`SpanBlock.spec.ts`), 7 new (`TraceDetailsSidebar.spec.ts`), rest pre-existing Pre-existing and untouched: three `no-dupe-keys` errors in `eslint.config.js`, and a `vue/no-dupe-keys` warning for `activeTab` in `TraceDetailsSidebar.vue` (confirmed present on unmodified `main`). ## Reviewer notes - **Not yet validated against a live ingested trace.** That step is what would have caught #13195, and it is the one gap here. - Specifically worth checking: `gen_ai.*` events are constructed with `_timestamp: 0` in the OTel processor's test fixtures. If real SDK data does the same, LLM spans will show no markers (dropped rather than mispositioned, by design). If that turns out to be real, the follow-up is an explicit "N events without timestamps" affordance, not a fabricated position. - Marker clustering for very dense event sets is deliberately deferred until real traces show it is needed. --------- Co-authored-by: coyaSONG <66289470+coyaSONG@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2026-08-26 10:12:35 +00:00
Hengfei Yang	ebdc2f0a4e	fix(search): release the admission slot and follower task status on cancellation (#13954) ... Paired with o2-enterprise `hf/issue-2491-admission-slot-leak` (same branch name). **Stacked on #13953** — base is that branch, so the diff here is just this change. Retargets to `main` automatically once #13953 merges. ## Problem #13953 fixed the wal file locks. `do_get` touches two more trace-id keyed resources, and both had the identical defect: release depended on control flow reaching a call, inside the window where gRPC cancellation drops the future. **The node's admission slot.** Released from three hand-written sites — the two error arms in `do_get` and `FlightEncoderStream::drop`. All three are skipped when the request is cancelled during plan generation, because the encoder stream does not exist yet. **The follower task status.** In the super-cluster path the entry was inserted before plan generation and removed after it: ```rust if is_super_cluster && !SEARCH_SERVER.contain_key(&trace_id).await { SEARCH_SERVER.insert(trace_id.clone(), TaskStatus::new_follower(vec![], false)).await; } let result = get_ctx_and_physical_plan(&trace_id, &req).await; // <-- cancellable if is_super_cluster && !SEARCH_SERVER.is_leader(&trace_id).await { SEARCH_SERVER.remove(&trace_id, false).await; } ``` Cancel in the middle and the entry stays in the map for the life of the process. Nothing else removes it: an explicit `cancel_query` would, but a timeout or a client disconnect would not. ## Approach Both move onto the same ownership chain #13953 established. `SessionGuard` — already armed on the first line of `do_get` and already carried into `FlightEncoderStream` — now releases the admission slot too, so the three call sites collapse into its drop. The follower task status is registered through an `AsyncDefer` that lives in the same scope as plan generation, so it is removed on every exit including a cancelled one. It is now removed only when this node registered it, rather than whenever the entry is not a leader's: a request can no longer clear an entry it did not create. In practice these coincide, since the composite `{trace_id}-{job_id}` is unique per request. ### Drive-by The `physical plan generate error` and `physical plan execution error` logs sat *inside* `if get_o2_config().work_group.max_nodes_per_query > 0`, so a plan failure was silent unless work groups happened to be configured — and silent in every OSS build. They are unconditional now. ## Enterprise side The paired PR gives `Running` reservations a TTL backstop. They had no expiry edge at all, so a slot whose release was lost — this cancellation path, or a leader that dies before sending `Release` — was held until the process restarted. This PR closes the local path; the backstop covers the remote one, which no local guard can. ## Verification - `cargo fmt --all` and the CI clippy command: clean - Enterprise verified via the `Cargo.toml.openobserve` swap against the paired branch: `cargo check --all-targets` clean, and `cargo clippy --no-deps -p openobserve-api-grpc --all-targets` clean. The workspace-wide enterprise clippy still reports the two pre-existing `cognitive_complexity` errors in `core/src/alerts/scheduler/handlers.rs` and `core/src/ofga/mod.rs`, untouched by this change - `api-grpc` flight tests 21/21 ## Also here Three zh-CN strings whose English is *events* said 活动 (activity). Unrelated to the rest of the PR, carried along at review request. | key | en-US | was | now | |---|---|---|---| | `home.docsCountLbl` | Events | 活动 | 事件 | | `rum.searchEvents` | Search Events | 搜索活动 | 搜索事件 | | `logs.noEvents.title` | No events found | 未找到任何活动 | 未找到任何事件 | zh-TW already had 事件 for all three, and every other locale translates them as events. `logs.noEvents` also contradicted itself — `descNoFilters` directly below the title already said 未找到任何事件. ## Not in this PR The `QueryManager` map has a `created_at` on every entry but no reaper. With this change its entries are removed on cancellation, so it is no longer known to grow — but it has no backstop the way the ledger now does.	2026-08-26 09:09:35 +00:00
3 people	b09b56e25a	feat(llm): add Phase 3.0 experiment workflows (#13778) ... ## Summary - allow dataset rows without an expected output across management APIs and UI flows - add immutable experiment preview, creation, listing, persistence, RBAC, and super-cluster synchronization - add Experiment as an evaluation target with slot hydration, scoring, and result indexing - harden idempotent creation, scorer-version pinning, score-index setup, and upgrade migrations ## Tickets - Related to [openobserve/o2-enterprise#2398](https://github.com/openobserve/o2-enterprise/issues/2398) - Related to [openobserve/o2-enterprise#2399](https://github.com/openobserve/o2-enterprise/issues/2399) - Related to [openobserve/o2-enterprise#2400](https://github.com/openobserve/o2-enterprise/issues/2400) ## Paired Enterprise PR - [openobserve/o2-enterprise#2413](https://github.com/openobserve/o2-enterprise/pull/2413) ## Validation - focused Rust and frontend tests for the affected dataset and experiment seams - experiment migration test passes - upgraded local environment applies schema version 68 and returns 200 from GET /api/team3/experiments --------- Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2026-08-26 05:47:37 +00:00
Omkar Kesarkhane	2c17c07a11	fix(rum): restore RUM-traces correlation for zero-stripped trace ids (#13959) ... ## Problem RUM-to-traces correlation stopped working after the browser SDK 0.4.x line. The SDK's identifiers became plain `BigInt`, so the unchanged `toString(16)` serialization silently started dropping leading zeros. Trace ids are UUIDv7-shaped (48-bit ms timestamp in the top bits), so the leading hex nibble is always zero today — effectively every RUM event stored a 31-char `_oo_trace_id`/`_o2_trace_id` while the traces stream stores the padded 32-char `trace_id` (the `traceparent` header was padded correctly, so backend traces have the full id). Exact-match correlation could never hit, in either direction. Verified against live data: for sampled traced requests, `'0' + stored_id` found the backend trace every time; the stored id never did. ## Fix (frontend compatibility layer) Works for the already-ingested short ids **and** for correct ids from the fixed SDK: - `utils/rum/fields.ts`: `normalizeTraceId` (pads to canonical 32-char hex), `traceIdLookupVariants` (padded + legacy zero-stripped forms), `rumFieldEqualsAnySql` (matches both namespace spellings x both id forms), and a shared `RUM_CORRELATION_TRACES_STREAM` constant replacing four scattered "default" literals. - `useRumSpanBuilder`: trace→RUM lookup queries both id forms; canonical id stamped on generated RUM spans. - `useTraceCorrelation`: backend-span query now targets the real `default` traces stream (was a nonexistent `_traces` stream) as `page_type: "traces"` (was `"logs"`), with the padded id. - `EventDetailDrawerContent`: trace navigation pads the id before building the route. - `PlayerTracesTab`: the session Traces tab canonicalizes ids before the traces-stream metadata join (rows were silently dropped). - `ErrorViewer`: error correlation card receives the padded id; implausible (non-hex) ids hide the card instead of showing a misleading empty state. - `package.json`: browser SDK bumped to 0.4.2-beta.3, which serializes trace/span ids as fixed-width canonical hex at the source (resolved `browser-rum-core` verified at 0.4.2-beta.3). ## Testing - New unit coverage: `fields.spec.ts` (16 tests) plus regression tests in each touched spec (legacy 31-char id fixtures asserting padded 32-char behavior); full RUM/traces suites green (53 files, 1971 passed). - Live-verified against real data: a trace from the broken window renders the full RUM hierarchy (Session→View→Action→Resource + backend spans) where the old code showed backend spans only; pre-regression sessions still correlate (no behavior change for correct ids). - `npm run lint` and `npm run type-check` clean.	2026-08-26 04:24:16 +00:00
Hengfei Yang	429e5148fe	fix(search): tie wal search file locks to guards so cancelled queries release them (#13953) ... Fixes o2-enterprise#2491. OSS-only — see "Enterprise side" below. ## Problem Local wal parquet files that had already been merged and uploaded to object storage stayed on disk forever, because their `SEARCHING_FILES` refcount never returned to zero. The post-merge cleanup in `jobs/src/job/files/parquet.rs` checks `wal::lock_files_exists(file_key)`, defers the deletion when a search holds the file, and re-checks every `ZO_FILE_PUSH_INTERVAL` — forever, if the lock is never released. Queries stayed correct (`pending_delete::filter` excludes those files from the search set); the symptom was disk. The proximate cause is a missing decrement. The root cause is that **release depended on control flow reaching a line, while the only thing that can cancel the request guarantees no further line runs**. ### The unowned window `SEARCHING_FILES` and `SEARCHING_REQUESTS` are global maps. Nothing in the type system points at an entry, so nothing releases one automatically: ``` lock_files() search_service/src/grpc/wal.rs, in get_file_list | match_source, parquet footer reads, calculate_files_size, table build lock_request() search_service/src/grpc/wal.rs | pack segments, union table, plan rewrite, execute_stream_partitioned FlightEncoderStreamBuilder::build() api/grpc .../flight/mod.rs <-- first owner with a Drop ``` `lock_request()` is not a handoff to an owner — `SEARCHING_REQUESTS` is removed from only by `clear_session_data`, itself reached by control flow. So the whole of plan generation was unprotected, not just the gap between the two calls. ### Why that window is exactly the window cancellation lives in Both gRPC cancellation paths can only fire *before* `do_get` returns a response: - `hyper-1.9.0/src/proto/h2/server.rs` polls `poll_reset` only on the `Poll::Pending` branch — "check if the client has sent a RST_STREAM frame which would cancel the current request" — and drops the service future. - `tonic-0.14.6/src/transport/service/grpc_timeout.rs` reads the client's `grpc-timeout` header and returns `Poll::Ready(Err(TimeoutExpired))`, dropping the inner future. Every flight request sets that header (`make_flight_client` → `req.set_timeout(grpc_timeout)`). Once the response stream exists, `FlightEncoderStream::drop` handles teardown and neither path applies. So the set of cancellable moments and the set of unowned moments coincided. Routine triggers, all of which drop in-flight `do_get` calls to ingesters: `query_task.abort()` on leader timeout or user cancel (`search_service/src/cluster/flight.rs`), `SearchStreamGuard::drop` cancelling on client disconnect, and the ingester `grpc-timeout` itself. None of them are visible: `remote_scan_exec.rs` treats `Cancelled` / `DeadlineExceeded` from a node as an empty partial result, so the query still succeeds. ## Approach Give the locks an owner for every instant they exist. - **`wal::SearchingFiles`** owns the raw locks from acquisition until `into_request` hands them to `SEARCHING_REQUESTS`; `Drop` releases whatever is left. `get_file_list` returns it alongside the file list, so the type forces every early return to account for it. - **`flight::SessionGuard`** is armed on the first line of `do_get` — before anything can be locked — owns `clear_session_data` for that trace id, and moves into `FlightEncoderStream`, whose `Drop` already performed it. It is a required argument of `FlightEncoderStreamBuilder::new`, not an optional setter, so a future call site cannot forget it. `lock_files`, `release_files` and `lock_request` are module-private now: the guard is the only way to take a wal lock. The eight hand-written `release_files` call sites are gone, and `create_tables_from_files` loses the `on_error: F` generic that existed solely to run one of them — `storage.rs` was already passing `|| {}`. Cancellation at any `.await` in the chain now releases: | cancelled during | released by | |---|---| | `get_file_list`, metadata reads, table build | `SearchingFiles::drop` | | after `into_request`, before the stream exists | `SessionGuard::drop` | | streaming | `FlightEncoderStream::drop` → `SessionGuard::drop` | ### Others on the same chain - `release_request` no longer early-returns on non-ingesters — the last step of an ownership chain should not be conditional — and logs only a release that actually happened, so `lock_request` / `release_request` now pair one-to-one per trace id and can be reconciled from logs. - `lock_request` merges instead of overwriting; a second handoff for one trace id used to silently drop the first file list. - `FlightEncoderStream::drop` releases session data unconditionally. Letting "am I a super-cluster follower leader?" decide whether wal locks are released was the same bug in miniature; the deferred lock only covers the work-group slot. ### Observability Two gauges, both O(1) reads, published by the existing 60s ingester collector `update_parquet_metrics` (no stream label): - `ingest_wal_searching_files` — size of the lock table - `ingest_wal_pending_delete_files` — uploaded files still awaiting deletion A pending-delete backlog that rises and never drains is the leak, plotted directly. Previously the only signal was an *absent* `debug!` line. ## Enterprise side **No paired PR — the fix is entirely in this repo.** The issue suspected the super-cluster follower-leader path, but `super_cluster/follower.rs` only builds a `RemoteScanExec` and never touches wal locks; `wal::lock_files` has exactly one call site, in `grpc/wal.rs`; and o2-enterprise contains no reference to `lock_files`, `release_files`, `lock_request` or `release_request`. Verified by a full enterprise build after making those three private. ## Verification - `cargo fmt --all`, and the CI clippy command (`--workspace --all-targets -W too_many_lines -W cognitive_complexity -W excessive_nesting -D warnings`): clean - Enterprise verified via the `Cargo.toml.openobserve` swap, `cargo check --all-targets`: clean. Clippy on that build reports two pre-existing `cognitive_complexity` errors in `core/src/alerts/scheduler/handlers.rs` and `core/src/ofga/mod.rs`, untouched by this change - Unit tests: `common` wal 10/10 (4 new, covering drop-release, duplicate collapse, `release_one`, and the `into_request` handoff), `api-grpc` flight 21/21, `search` file_list 4/4 ## Not in this PR - **A timestamped reaper for stale locks.** Redundant once the chain is unbroken, but it is the lowest-risk first step when backporting to `branch-v0.40.0` and `branch-v0.80.0`, which carry the same defect. - **`admission::ledger` and the `SEARCH_SERVER` task status.** Both are trace-id keyed and both release only by reaching a call, so a cancellation in the same window leaks a work-group slot the same way. Same bug class, different resource, enterprise-gated — worth its own issue. - **`clean_lock_files()`** (`/node/flush`, `/node/enable`) wipes the whole refcount table, including locks held by running queries. It was survivable while locks leaked; now it turns a leak into deleting a file a query is reading. Retiring it is a product call.	2026-08-25 14:03:09 +00:00
Hengfei Yang	663e2f75da	fix(api): stop unprintable error messages collapsing responses into a bare 500 (#13952) ... ## Problem `api_fuzz_tests` (schemathesis) intermittently reported a 500 on `POST /api/{org_id}/streams/{stream_name}` with the body `failed to parse header value`. That string is not ours — it is the `http` crate's `InvalidHeaderValue` display. The chain: - ~20 sites return `(StatusCode, [(ERROR_HEADER, format!(...))], Json(...))`, where the message interpolates user-supplied names (org, stream, **field type**). - The failing payload carried `"type": "v^0K…"`. `HeaderValue` accepts only `b >= 32 && b != 127 || b == b'\t'`, so the control byte makes header construction fail. - axum's `TryIntoHeaderError` then **discards the whole response** and substitutes `(500, inner.to_string())`. So the handler's real 400 and its JSON body never reached the client. The status code was not chosen by the handler at all — axum overwrote it. The org name was never the trigger; it only had to exist so the request got past the auth check. The field type was the poison. A second, quieter bug sits in the same place: non-ascii bytes *are* accepted into a header, but `HeaderValue::to_str()` only reads back `32..127`, and the audit middleware reads this header with exactly `to_str().ok()`. Any error mentioning a non-ascii stream name was being silently dropped from the audit log. ## Approach Route every `X-Error-Message` write through one chokepoint in `common::meta::http`: - `error_header_value()` escapes anything outside visible ascii via `char::escape_default`, which fixes both failure modes at once — the header always builds, and it always reads back. - `HttpResponse::error_with_header()` sets the header and the JSON body from a single message, replacing the hand-built tuples. Most sites were formatting the identical message twice (once for the header, once for the body), so this removes that duplication as well. `enrichment_table` additionally had four `HeaderValue::from_str(..).unwrap()` calls — outright panics, not 500s, on the same class of input. Those are gone. Other `HeaderValue` construction sites (router `user_email`, `x-api-node`) already guard with `unwrap_or_else` and are untouched. ## Organization identifier validation Investigating how such an org existed at all: `create_org` validates the display name and assigns a uuid identifier, but `core::organization::check_and_create_org` takes the identifier **verbatim from the URL path** with no validation. It is reachable from any root-authenticated ingestion write (`ZO_CREATE_ORG_THROUGH_INGESTION`, default true), which is how the fuzzer provisioned it. `validate_auto_created_org_id` (ascii alphanumeric plus `-` and `_`) now guards both auto-create entry points. It runs **after** the existing-org lookup, so it only gates new creation: no deployment loses an org it already has, and SSO auto-provisioning from IdP group names keeps working for normal identifiers. ## Verification Against a local server with the CI payload: - stream create → `400 {"code":400,"message":"invalid data type: v^0K"}` instead of the 500 - ingestion to the exotic org → 404; `autoorg-1` still auto-creates and ingests - valid create, duplicate create, settings update, settings on a missing stream, delete, enrichment upload → correct codes, no panics Two edits sit under `#[cfg(feature = "cloud")]` and cannot be compiled here: swapping in `Cargo.toml.openobserve` fails inside o2-enterprise itself, whose `main` has not yet picked up the `get_orm_client_rw` split from #13927. They were verified by temporarily un-gating them and stubbing the cloud-only predicate.	2026-08-25 11:14:03 +00:00
5 people	91e524ce2d	feat(dashboards): interactive table cells with in-place Logs Explorer drawer (#13675) ... Design At: designs/dashboards/interactive-table-cell-drilldown What changed New component: DashboardLogDrawer.vue Clicking a cell in a dashboard table panel now opens a side drawer scoped to that field/value filter. The drawer shows filtered log events with a SQL editor (Ctrl+Enter to run, Reset to restore default), server-side pagination with configurable page size (50/100/200/500), and an event detail sub-drawer on row click. Event detail has three tabs: Insights — Field Anomaly Profile (rarity scoring vs current page), Event Timeline (SVG histogram of event frequency across the selected time range), Log Patterns (grouped by structure via /patterns/extract, each with "Open in Logs" that prefills the log explorer), and Surrounding Events (events from the same stream in a configurable ±1–10 min window, paginated 20/page with per-row expand) Details — key/value field list with copy buttons JSON — raw event JSON Modified: PanelSchemaRenderer.vue Fixed type mismatch: route.query.cell_panel returns string; panel schema IDs may be number — comparison now uses String() on both sides Added cell_event_ts to the URL param cleanup list so it's cleared when the outer log drawer closes Modified: en-US.json — added complete panel.logExplorer key tree including the new insights subsection Modified: design-debt-baseline.json — updated baseline after audit (new file's rawVarInComponent uses are all in sanctioned color-mix() positions) URL persistence All drawer state is encoded in URL params (cell_panel, cell_field, cell_value, cell_stream, cell_stype, cell_t0, cell_t1, cell_event_ts). Sharing the URL reopens both drawers and restores the exact event — on mount it does a WHERE _timestamp = {ts} targeted fetch (size 1) rather than searching through paginated results. Zero backend changes Uses only existing endpoints: search (SQL), search (histogram via histogram() function), and patterns/extract. --------- Co-authored-by: Shrinath Rao <shnath@openobserve.ai> Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech> Co-authored-by: Abhay Padamani <141122205+ktx-abhay@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2026-08-25 10:31:13 +00:00
Hengfei Yang	ccd254ad00	refactor(infra): split ORM read/write clients, retire the sqlite write lock (#13927) ... Paired with o2-enterprise `refactor/orm-rw-ro-client` (same branch name). ## Problem Every ORM caller went through a single client backed by the sqlite **write** pool, so writes were kept apart only by `table::get_lock()` — a convention 365 call sites had to remember. Forgetting it surfaced as `(code: 517) database is locked`: a deferred transaction that reads and then upgrades to a write finds the snapshot moved under it, and sqlite reports that **without ever consulting `busy_timeout`**, which is why the failures were immediate rather than retried. ## Approach Serialization now comes from the pool, not from a lock. The sqlite write pool holds **one connection** — all sqlite allows anyway — so writers queue on acquire and there is nothing to forget. That is only affordable because reads move to the read-only pool. Measured, same load, fresh data dir each run: | | reads on the write connection | reads on the RO pool | |---|---|---| | 200 concurrent reads | p50 **135 ms** | p50 **1 ms** | | writes during mixed load | p50 **189 ms** | p50 **14 ms** | Both were 100% success with zero lock errors, so this is purely throughput — but without the split, reads queue behind the single writer and drag writes down with them. ## API `infra::db` now exposes three functions and nothing else: ```rust pub async fn get_orm_client_ro() -> &'static DatabaseConnection pub async fn get_orm_client_rw() -> &'static DatabaseConnection pub async fn get_orm_client_ddl() -> &'static DatabaseConnection ``` The statics behind them, every `connect_to_orm_*`, and the raw sqlite/postgres pools are now private or `pub(crate)`. `ORM_CLIENT`, `connect_to_orm` and `table::get_lock()` are gone, as is the raw `CLIENT_RW` mutex (56 lock acquisitions, plus the 52 guard derefs that existed only to reach through it). Final split: **RO 247 / RW 360 / DDL 9**; `get_lock` 0, `ORM_CLIENT` 0, `CLIENT_RW.lock()` 0. ## How read/write was decided Per call site, then verified by resolving each call through **its own file's imports** (bare-name matching is useless here — `get_by_id`, `list` and `update` collide across modules) and propagating write-ness along the edges that hand a connection to a callee. A wrong write client only costs throughput; a wrong read client fails in production. So the audit is zero-miss in the read direction, and a site is only downgraded when *every* connection-passing callee resolves. ## Behaviour changes on sqlite All three make sqlite match what postgres already does today — postgres has never had this lock: - check-then-write pairs like `folders::put` are no longer atomic; they rely on their unique index (`folders_org_folder_id_idx`) - `db::put` runs two sequential transactions, so concurrent puts to one key are last-write-wins - `db::delete` runs several statements outside a transaction `write_pool_is_single_connection_on_sqlite` is now the **only** guard against reintroducing 517 — please don't drop it. ## Review guide The mutex removal is separable if you want to look at it on its own: `src/infra/src/{db,file_list,scheduler,pipeline,schema/history}/sqlite.rs`. Everything else is the client split. ## Verification - `cargo check --workspace --all-targets` and the CI clippy command: clean - Tests match a HEAD baseline built with the same `.env`, exactly: infra 1410/2, core 1629/1, db 431/0, api-management 179/0. The three failures are pre-existing and environmental (two need a postgres that isn't running locally) - Static scans, all zero: queries on the pool inside an open transaction; transactions spanning a second connection acquisition; ro/rw misclassification - Runtime smoke on a fresh data dir: 200 concurrent reads, then 100 reads + 100 writes simultaneously — 400/400 HTTP 200, rows persisted, **0** lock errors, 0 ERROR/WARN lines - Enterprise verified via the `Cargo.toml.openobserve` swap (`cargo check --all-targets`): clean Also fixes `users::tests::set_up`, which truncated shared tables but released its lock before the test body ran, letting a concurrent test clear rows mid-test. https://claude.ai/code/session_016ouf1RxiMpiRw5W7GvCUDh	2026-08-25 08:37:50 +00:00
3 people	abc0fdc42d	feat: model pricing time-based conditonal pricing (#13903) ... Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-25 07:04:58 +00:00
Huaijin	d87170ce0e	chore: upgrade build image to ubicloud-standard-16 (#13950)	2026-08-25 13:36:49 +08:00
dependabot[bot]andHengfei Yang	3c2640b1dc	chore(deps): bump actions/cache from 4 to 6 (#13940) ... Bumps [actions/cache](https://github.com/actions/cache) from 4 to 6. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/actions/cache/releases">actions/cache's releases</a>.</em></p> <blockquote> <h2>v6.0.0</h2> <h2>What's Changed</h2> <ul> <li>Update packages, migrate to ESM by <a href="https://github.com/Samirat"><code>@​Samirat</code></a> in <a href="https://redirect.github.com/actions/cache/pull/1760">actions/cache#1760</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/actions/cache/compare/v5...v6.0.0">https://github.com/actions/cache/compare/v5...v6.0.0</a></p> <h2>v5.1.0</h2> <h2>What's Changed</h2> <ul> <li>Bump <code>@​actions/cache</code> to v5.1.0 - handle read-only cache access by <a href="https://github.com/jasongin"><code>@​jasongin</code></a> in <a href="https://redirect.github.com/actions/cache/pull/1775">actions/cache#1775</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/actions/cache/compare/v5...v5.1.0">https://github.com/actions/cache/compare/v5...v5.1.0</a></p> <h2>v5.0.5</h2> <h2>What's Changed</h2> <ul> <li>Update ts-http-runtime dependency by <a href="https://github.com/yacaovsnc"><code>@​yacaovsnc</code></a> in <a href="https://redirect.github.com/actions/cache/pull/1747">actions/cache#1747</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/actions/cache/compare/v5...v5.0.5">https://github.com/actions/cache/compare/v5...v5.0.5</a></p> <h2>v5.0.4</h2> <h2>What's Changed</h2> <ul> <li>Add release instructions and update maintainer docs by <a href="https://github.com/Link"><code>@​Link</code></a>- in <a href="https://redirect.github.com/actions/cache/pull/1696">actions/cache#1696</a></li> <li>Potential fix for code scanning alert no. 52: Workflow does not contain permissions by <a href="https://github.com/Link"><code>@​Link</code></a>- in <a href="https://redirect.github.com/actions/cache/pull/1697">actions/cache#1697</a></li> <li>Fix workflow permissions and cleanup workflow names / formatting by <a href="https://github.com/Link"><code>@​Link</code></a>- in <a href="https://redirect.github.com/actions/cache/pull/1699">actions/cache#1699</a></li> <li>docs: Update examples to use the latest version by <a href="https://github.com/XZTDean"><code>@​XZTDean</code></a> in <a href="https://redirect.github.com/actions/cache/pull/1690">actions/cache#1690</a></li> <li>Fix proxy integration tests by <a href="https://github.com/Link"><code>@​Link</code></a>- in <a href="https://redirect.github.com/actions/cache/pull/1701">actions/cache#1701</a></li> <li>Fix cache key in examples.md for bun.lock by <a href="https://github.com/RyPeck"><code>@​RyPeck</code></a> in <a href="https://redirect.github.com/actions/cache/pull/1722">actions/cache#1722</a></li> <li>Update dependencies &amp; patch security vulnerabilities by <a href="https://github.com/Link"><code>@​Link</code></a>- in <a href="https://redirect.github.com/actions/cache/pull/1738">actions/cache#1738</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/XZTDean"><code>@​XZTDean</code></a> made their first contribution in <a href="https://redirect.github.com/actions/cache/pull/1690">actions/cache#1690</a></li> <li><a href="https://github.com/RyPeck"><code>@​RyPeck</code></a> made their first contribution in <a href="https://redirect.github.com/actions/cache/pull/1722">actions/cache#1722</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/actions/cache/compare/v5...v5.0.4">https://github.com/actions/cache/compare/v5...v5.0.4</a></p> <h2>v5.0.3</h2> <h2>What's Changed</h2> <ul> <li>Bump <code>@actions/cache</code> to v5.0.5 (Resolves: <a href="https://github.com/actions/cache/security/dependabot/33">https://github.com/actions/cache/security/dependabot/33</a>)</li> <li>Bump <code>@actions/core</code> to v2.0.3</li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/actions/cache/compare/v5...v5.0.3">https://github.com/actions/cache/compare/v5...v5.0.3</a></p> <h2>v.5.0.2</h2> <h1>v5.0.2</h1> <h2>What's Changed</h2> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/actions/cache/blob/main/RELEASES.md">actions/cache's changelog</a>.</em></p> <blockquote> <h1>Releases</h1> <h2>How to prepare a release</h2> <blockquote> <p>[!NOTE] Relevant for maintainers with write access only.</p> </blockquote> <ol> <li>Switch to a new branch from <code>main</code>.</li> <li>Run <code>npm test</code> to ensure all tests are passing.</li> <li>Update the version in <a href="https://github.com/actions/cache/blob/main/package.json"><code>https://github.com/actions/cache/blob/main/package.json</code></a>.</li> <li>Run <code>npm run build</code> to update the compiled files.</li> <li>Update this <a href="https://github.com/actions/cache/blob/main/RELEASES.md"><code>https://github.com/actions/cache/blob/main/RELEASES.md</code></a> with the new version and changes in the <code>## Changelog</code> section.</li> <li>Run <code>licensed cache</code> to update the license report.</li> <li>Run <code>licensed status</code> and resolve any warnings by updating the <a href="https://github.com/actions/cache/blob/main/.licensed.yml"><code>https://github.com/actions/cache/blob/main/.licensed.yml</code></a> file with the exceptions.</li> <li>Commit your changes and push your branch upstream.</li> <li>Open a pull request against <code>main</code> and get it reviewed and merged.</li> <li>Draft a new release <a href="https://github.com/actions/cache/releases">https://github.com/actions/cache/releases</a> use the same version number used in <code>package.json</code> <ol> <li>Create a new tag with the version number.</li> <li>Auto generate release notes and update them to match the changes you made in <code>RELEASES.md</code>.</li> <li>Toggle the set as the latest release option.</li> <li>Publish the release.</li> </ol> </li> <li>Navigate to <a href="https://github.com/actions/cache/actions/workflows/release-new-action-version.yml">https://github.com/actions/cache/actions/workflows/release-new-action-version.yml</a> <ol> <li>There should be a workflow run queued with the same version number.</li> <li>Approve the run to publish the new version and update the major tags for this action.</li> </ol> </li> </ol> <h2>Changelog</h2> <h3>6.1.0</h3> <ul> <li>Bump <code>@actions/cache</code> to v6.1.0 to pick up <a href="https://redirect.github.com/actions/toolkit/pull/2435">actions/toolkit#2435 Handle cache write error due to read-only token</a></li> <li>Switch redundant &quot;Cache save failed&quot; warning to debug log in save-only</li> </ul> <h3>6.0.0</h3> <ul> <li>Updated <code>@actions/cache</code> to ^6.0.1, <code>@actions/core</code> to ^3.0.1, <code>@actions/exec</code> to ^3.0.0, <code>@actions/io</code> to ^3.0.2</li> <li>Migrated to ESM module system</li> <li>Upgraded Jest to v30 and test infrastructure to be ESM compatible</li> </ul> <h3>5.0.4</h3> <ul> <li>Bump <code>minimatch</code> to v3.1.5 (fixes ReDoS via globstar patterns)</li> <li>Bump <code>undici</code> to v6.24.1 (WebSocket decompression bomb protection, header validation fixes)</li> <li>Bump <code>fast-xml-parser</code> to v5.5.6</li> </ul> <h3>5.0.3</h3> <ul> <li>Bump <code>@actions/cache</code> to v5.0.5 (Resolves: <a href="https://github.com/actions/cache/security/dependabot/33">https://github.com/actions/cache/security/dependabot/33</a>)</li> <li>Bump <code>@actions/core</code> to v2.0.3</li> </ul> <h3>5.0.2</h3> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/actions/cache/commit/55cc8345863c7cc4c66a329aec7e433d2d1c52a9"><code>55cc834</code></a> Merge pull request <a href="https://redirect.github.com/actions/cache/issues/1768">#1768</a> from jasongin/readonly-cache</li> <li><a href="https://github.com/actions/cache/commit/d8cd72f230726cdf4457ebb61ec1b593a8d12337"><code>d8cd72f</code></a> Bump <code>@​actions/cache</code> to v6.1.0 - handle cache write error due to RO token</li> <li><a href="https://github.com/actions/cache/commit/2c8a9bd7457de244a408f35966fab2fb45fda9c8"><code>2c8a9bd</code></a> Merge pull request <a href="https://redirect.github.com/actions/cache/issues/1760">#1760</a> from actions/samirat/esm_migration_and_package_update</li> <li><a href="https://github.com/actions/cache/commit/e9b91fdc3fea7d79165fceb79042ef45c2d51023"><code>e9b91fd</code></a> Prettier fixes</li> <li><a href="https://github.com/actions/cache/commit/e4884b8ff7f92ef6b52c79eda480bbc86e685adb"><code>e4884b8</code></a> Rebuild dist</li> <li><a href="https://github.com/actions/cache/commit/10baf0191a3c426ea0fa4a3253a5c04233b6e18f"><code>10baf01</code></a> Fixed licenses</li> <li><a href="https://github.com/actions/cache/commit/e39b386c9004d72a15d864ade8c0b3a702d47a37"><code>e39b386</code></a> Fix test mock return order</li> <li><a href="https://github.com/actions/cache/commit/b6928203372a8571ff984c0c883ef3a1adfb0c06"><code>b692820</code></a> PR feedback</li> <li><a href="https://github.com/actions/cache/commit/60749128a44d25d3c520a489e576380cf00ff3f1"><code>6074912</code></a> Rebuild dist bundles as ESM to match type:module</li> <li><a href="https://github.com/actions/cache/commit/5a912e8b4af820fa082a0e75cfd2c782f8fbfe0e"><code>5a912e8</code></a> Fix lint and jest issues</li> <li>Additional commits viewable in <a href="https://github.com/actions/cache/compare/v4...v6">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>	2026-08-25 05:07:50 +00:00
Amey Pawar	9a44af980e	fix(logs): stop quick mode rewriting SELECT inside subqueries (#13787) ... ## Summary - move the quick-mode field-list rewrite out of `Index.vue` into `replaceSelectFieldList` - return the query untouched when it nests a SELECT, instead of corrupting it - mask string literals and comments before matching, so neither can be rewritten or hide a subquery - add `quickModeFieldList.spec.ts` covering the reported query and the literal/comment cases Fixes #8542. ## Root cause Enabling quick mode replaced the search bar's SELECT list via `query.replace(/SELECT\s+(.*?)\s+FROM/gi, ...)`. The global flag applied that to *every* `SELECT ... FROM` pair, so a subquery was rewritten as readily as the outer query, and `(.*?)` swallowed the projection including any `DISTINCT`: ```sql where trace_id in (select distinct trace_id from trace_list_index ...) where trace_id in (SELECT trace_id,_timestamp FROM trace_list_index ...) ``` The `IN (...)` is left comparing against two columns and the `DISTINCT` is gone. Dropping the global flag is **not** enough: `.` does not match a newline, so in a query formatted over several lines the outer pair never matches and the subquery is the *first* match, not the second. The rewrite has to bail out instead. The check and the rewrite both run over a masked copy of the query, so `msg = 'select id from users'` is not rewritten and `in (/* c */ select ...)` cannot hide a subquery. Masking is a single pass because a quote can appear inside a comment and a `--` inside a quoted value. The global flag is kept — a multi-stream search is a UNION of one SELECT per stream and each needs the field list. `quick_mode` still goes out on the request, so field trimming for `SELECT *` is unaffected. When the query is skipped the toggle stays silent rather than notifying — happy to add a message if you'd prefer one. ## Testing - `npx vitest run src/utils/query/quickModeFieldList.spec.ts` — 18 passing - mutation-tested: removing the subquery guard turns 7 red, removing the masking turns 5 red - `src/plugins/logs/Index.spec.ts` — 65 passing, unchanged - `npm run type-check:app`, `lint:ci`, `format:check` — clean Two nearby things I noticed but did not touch: disabling quick mode does not restore the previous query, and the field list joins without a space after the comma.	2026-08-25 05:05:23 +00:00
Vamsi Karthik R	9a0a272103	test(mobile): P0 session-replay upload assertion (#13945) ... ## What Adds a **P0 data-layer assertion** to the mobile RUM masking suite: after driving the replay flow, it queries the **`_sessionreplay`** stream and requires that segments actually landed. A rendered dashboard is no longer accepted as proof that Session Replay works. This is the check that catches **#13942** — on iOS the RN Session-Replay bridge posts segments to Datadog's `/api/v2/replay` instead of OpenObserve's `/replay`, so they're rejected (401, no error surfaced) and **zero** segments reach `_sessionreplay`. RN iOS session replay silently records nothing. ## RC The masking suite verified Session Replay **only through the dashboard UI** (does the player render + is PII masked). It had no data-layer check that segments actually reached OpenObserve. So a broken upload showed only as "the replay didn't render" — ambiguous (a benign dashboard-render flake *or* a real non-upload) — and that non-render had been scoped to a skip. The bug slipped through the ambiguity. ## The fix - **P0 (data):** `maskingSuite` asserts `_sessionreplay` received segments for the session (`> 0`). Non-upload → hard fail. Platform-agnostic; catches this whole class of bug. - **P1 (privacy):** the dashboard render + PII scan stays, run only when the player paints (segments-landed-but-player-didn't = dashboard flake, not an SDK bug). - Removed the `requireReplay` skip hack. **`ios-native` masking unskipped** → runs as hard P0 (native iOS SDK uses the correct `/replay` endpoint). - **`rn-ios` masking is a plain test that hard-FAILS until #13942 is fixed** — **no skip, no expected-fail marker.** The P0 assertion sees 0 segments → red; it **passes on its own** the moment the SDK starts uploading. The fix lives in `openobserve-react-native-rum`. ## Expected result per platform | Platform | Masking (P0 upload + PII) | Why | |---|:-:|---| | rn-android | ✅ pass | RN replay works on Android | | **rn-ios** | ❌ **fail** | **#13942** — 0 replay segments | | android-native | ✅ pass | native SDK, correct endpoint | | ios-native | ✅ pass | native SDK, correct endpoint | So whenever iOS runs (the `mobile-e2e` label / an SDK dispatch), the iOS job is **red until the SDK fix lands** — an intentional, self-healing signal. ## Validation (local, real device/data runs) - `android-native` + `ios-native` masking **pass** the new P0 (segments present in `_sessionreplay`). - A non-uploading session returns **0 segments → P0 correctly fails** — the exact rn-ios symptom. Scope: only `tests/mobile-testing/` — 3 files.	2026-08-25 04:36:18 +00:00
Huaijin	b0841df2a3	feat(metrics): support Vortex hash ordering (#13931) ... ## Summary - enable the TSID-major hash-ordered layout for configured Vortex compactor output while keeping Metrics ingester output in Parquet - dispatch the original DataFusion batch channel directly to separate Metrics Parquet/Vortex writers; Vortex consumes it on the dedicated runtime without a nested per-file channel or Arrow IPC staging - keep Metrics Vortex files free of custom o2_file_meta metadata; exact rotated-file metadata remains authoritative in file-list - translate .midx RowRanges into VortexAccessPlan IncludeRoaring selections so Indexed Vortex queries are pruned instead of full-scanned - cover Parquet and Vortex naming, size splitting, metadata behavior, row ordering, ordered merge, and Vortex range-plan attachment ## Validation - cargo test -p metrics_index --lib - cargo test -p search test_size_split_metrics_rotates_at_batch_boundary --lib - cargo test -p search test_access_plan_from_metrics_index_ranges --lib - cargo test -p search test_generate_access_plan_attaches_vortex_metrics_ranges --lib - cargo test -p search test_hash_sorted_files_merge_without_sort_exec --lib - cargo fmt --all --check - cargo clippy --workspace --all-targets -- -D warnings - cp ../o2-enterprise/Cargo.toml.openobserve Cargo.toml - enterprise manifest: cargo check --workspace --all-targets - enterprise manifest: cargo clippy --workspace --all-targets -- -D warnings	2026-08-25 04:31:48 +00:00
sidharth-sinhasaneandHengfei Yang	acfcd59376	fix(promql): preserve mixed-case WAL metric names (#13937) ... Fixes #13926 ## Problem With `ZO_FORMAT_STREAM_NAME_TO_LOWERCASE=false`, ingestion keeps the original metric name, but the PromQL WAL distributed plan sent it as an unquoted SQL table name. DataFusion lowercased it in the Flight receiver before schema lookup, so an exact mixed-case query returned no data. ## Fix Represent the metric as a case-preserving `TableReference::bare` and serialize its quoted form in the placeholder physical plan. The existing receiver and normal SQL paths remain unchanged. Regression coverage now checks physical-plan creation and distributed codec round-tripping. ## Verification - `cargo fmt --package promql-service --package search -- --check` - `cargo test -p promql-service` (18 passed) - `cargo test -p search datafusion_codec` (8 passed) - `cargo build` Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>	2026-08-25 01:56:05 +00:00
dependabot[bot]andHengfei Yang	4cbb25c823	chore(deps): bump taiki-e/install-action from 2.85.10 to 2.86.5 (#13939) ... Bumps [taiki-e/install-action](https://github.com/taiki-e/install-action) from 2.85.10 to 2.86.5. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/taiki-e/install-action/releases">taiki-e/install-action's releases</a>.</em></p> <blockquote> <h2>2.86.5</h2> <ul> <li> <p>Update <code>zola@latest</code> to 0.23.4.</p> </li> <li> <p>Update <code>wasm-tools@latest</code> to 1.257.1.</p> </li> <li> <p>Update <code>protoc@latest</code> to 3.36.0.</p> </li> <li> <p>Update <code>mise@latest</code> to 2026.8.10.</p> </li> <li> <p>Update <code>cargo-dinghy@latest</code> to 0.8.6.</p> </li> <li> <p>Update <code>wasmtime@latest</code> to 48.0.0.</p> </li> </ul> <h2>2.86.4</h2> <ul> <li> <p>Update <code>oxfmt@latest</code> to 1.79.0.</p> </li> <li> <p>Update <code>mise@latest</code> to 2026.8.8.</p> </li> <li> <p>Update <code>martin@latest</code> to 1.14.0.</p> </li> <li> <p>Update <code>dprint@latest</code> to 0.56.0.</p> </li> <li> <p>Update <code>cargo-about@latest</code> to 0.9.2.</p> </li> <li> <p>Update <code>biome@latest</code> to 2.5.9.</p> </li> </ul> <h2>2.86.3</h2> <ul> <li> <p>Update <code>prek@latest</code> to 0.4.14.</p> </li> <li> <p>Update <code>osv-scanner@latest</code> to 2.5.1.</p> </li> <li> <p>Update <code>cargo-llvm-cov@latest</code> to 0.9.0.</p> </li> </ul> <h2>2.86.2</h2> <ul> <li> <p>Update <code>uv@latest</code> to 0.12.5.</p> </li> <li> <p>Update <code>mise@latest</code> to 2026.8.6.</p> </li> <li> <p>Update <code>cargo-tarpaulin@latest</code> to 0.37.2.</p> </li> </ul> <h2>2.86.1</h2> <ul> <li>Fix an issue where <code>oxfmt</code> was　accidentally installed as <code>oxfmt-{target}{exe}</code>. (<a href="https://redirect.github.com/taiki-e/install-action/pull/1969">#1969</a>)</li> </ul> <h2>2.86.0</h2> <ul> <li>Support <code>oxfmt</code>. (<a href="https://redirect.github.com/taiki-e/install-action/pull/1967">#1967</a>, thanks <a href="https://github.com/rami3l"><code>@​rami3l</code></a>)</li> </ul> <h2>2.85.14</h2> <ul> <li> <p>Update <code>uv@latest</code> to 0.12.4.</p> </li> <li> <p>Update <code>trivy@latest</code> to 0.74.0.</p> </li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md">taiki-e/install-action's changelog</a>.</em></p> <blockquote> <h2>[2.86.5] - 2026-08-21</h2> <ul> <li> <p>Update <code>zola@latest</code> to 0.23.4.</p> </li> <li> <p>Update <code>wasm-tools@latest</code> to 1.257.1.</p> </li> <li> <p>Update <code>protoc@latest</code> to 3.36.0.</p> </li> <li> <p>Update <code>mise@latest</code> to 2026.8.10.</p> </li> <li> <p>Update <code>cargo-dinghy@latest</code> to 0.8.6.</p> </li> <li> <p>Update <code>wasmtime@latest</code> to 48.0.0.</p> </li> </ul> <h2>[2.86.4] - 2026-08-20</h2> <ul> <li> <p>Update <code>oxfmt@latest</code> to 1.79.0.</p> </li> <li> <p>Update <code>mise@latest</code> to 2026.8.8.</p> </li> <li> <p>Update <code>martin@latest</code> to 1.14.0.</p> </li> <li> <p>Update <code>dprint@latest</code> to 0.56.0.</p> </li> <li> <p>Update <code>cargo-about@latest</code> to 0.9.2.</p> </li> <li> <p>Update <code>biome@latest</code> to 2.5.9.</p> </li> </ul> <h2>[2.86.3] - 2026-08-18</h2> <ul> <li> <p>Update <code>prek@latest</code> to 0.4.14.</p> </li> <li> <p>Update <code>osv-scanner@latest</code> to 2.5.1.</p> </li> <li> <p>Update <code>cargo-llvm-cov@latest</code> to 0.9.0.</p> </li> </ul> <h2>[2.86.2] - 2026-08-17</h2> <ul> <li> <p>Update <code>uv@latest</code> to 0.12.5.</p> </li> <li> <p>Update <code>mise@latest</code> to 2026.8.6.</p> </li> <li> <p>Update <code>cargo-tarpaulin@latest</code> to 0.37.2.</p> </li> </ul> <h2>[2.86.1] - 2026-08-15</h2> <ul> <li>Fix an issue where <code>oxfmt</code> was　accidentally installed as <code>oxfmt-{target}{exe}</code>. (<a href="https://redirect.github.com/taiki-e/install-action/pull/1969">#1969</a>)</li> </ul> <h2>[2.86.0] - 2026-08-15</h2> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/taiki-e/install-action/commit/ba47c86ac325773530516bb756137ac718732518"><code>ba47c86</code></a> Release 2.86.5</li> <li><a href="https://github.com/taiki-e/install-action/commit/5e0b59bc928f672ccd1072a19b6815d0cf5b21cb"><code>5e0b59b</code></a> Update <code>zola@latest</code> to 0.23.4</li> <li><a href="https://github.com/taiki-e/install-action/commit/60dfdd7edfbb113f6254c9b3889a61fb3e20baf4"><code>60dfdd7</code></a> Update <code>wasm-tools@latest</code> to 1.257.1</li> <li><a href="https://github.com/taiki-e/install-action/commit/c262502059d0289897f7720971cb62c8b83217f1"><code>c262502</code></a> Update <code>protoc@latest</code> to 3.36.0</li> <li><a href="https://github.com/taiki-e/install-action/commit/4985cbc05f60164fac55ad0f7e788c0a97fd2aed"><code>4985cbc</code></a> Update <code>mise@latest</code> to 2026.8.10</li> <li><a href="https://github.com/taiki-e/install-action/commit/e782b6cb9189134d57cdae9cdf8ac0ae4c62384c"><code>e782b6c</code></a> Update <code>cargo-dinghy@latest</code> to 0.8.6</li> <li><a href="https://github.com/taiki-e/install-action/commit/e891b49f0d0c25fc74d5bf0f29a720e43c063750"><code>e891b49</code></a> Update biome manifest</li> <li><a href="https://github.com/taiki-e/install-action/commit/394bb5ddf8b2bba382638dadf8c93ff59b9b3fb5"><code>394bb5d</code></a> Update wasmtime manifest</li> <li><a href="https://github.com/taiki-e/install-action/commit/a2a5f6e99e1a31540baa0468acfa302cff0f359f"><code>a2a5f6e</code></a> Release 2.86.4</li> <li><a href="https://github.com/taiki-e/install-action/commit/5afbb1a197d47fef2a5793727b494b64e09ac265"><code>5afbb1a</code></a> Update wasm-tools manifest</li> <li>Additional commits viewable in <a href="https://github.com/taiki-e/install-action/compare/v2.85.10...v2.86.5">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>	2026-08-25 01:45:31 +00:00
dependabot[bot]andHengfei Yang	322037bdd7	chore(deps): bump actions/setup-java from 4 to 5 (#13941) ... Bumps [actions/setup-java](https://github.com/actions/setup-java) from 4 to 5. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/actions/setup-java/releases">actions/setup-java's releases</a>.</em></p> <blockquote> <h2>v5.0.0</h2> <h2>What's Changed</h2> <h3>Breaking Changes</h3> <ul> <li>Upgrade to node 24 by <a href="https://github.com/salmanmkc"><code>@​salmanmkc</code></a> in <a href="https://redirect.github.com/actions/setup-java/pull/888">actions/setup-java#888</a></li> </ul> <p>Make sure your runner is updated to this version or newer to use this release. v2.327.1 <a href="https://github.com/actions/runner/releases/tag/v2.327.1">Release Notes</a></p> <h3>Dependency Upgrades</h3> <ul> <li>Upgrade Publish Immutable Action by <a href="https://github.com/HarithaVattikuti"><code>@​HarithaVattikuti</code></a> in <a href="https://redirect.github.com/actions/setup-java/pull/798">actions/setup-java#798</a></li> <li>Upgrade eslint-plugin-jest from 27.9.0 to 28.11.0 by <a href="https://github.com/dependabot"><code>@​dependabot</code></a>[bot] in <a href="https://redirect.github.com/actions/setup-java/pull/730">actions/setup-java#730</a></li> <li>Upgrade undici from 5.28.5 to 5.29.0 by <a href="https://github.com/dependabot"><code>@​dependabot</code></a>[bot] in <a href="https://redirect.github.com/actions/setup-java/pull/833">actions/setup-java#833</a></li> <li>Upgrade form-data to bring in fix for critical vulnerability by <a href="https://github.com/gowridurgad"><code>@​gowridurgad</code></a> in <a href="https://redirect.github.com/actions/setup-java/pull/887">actions/setup-java#887</a></li> <li>Upgrade actions/checkout from 4 to 5 by <a href="https://github.com/dependabot"><code>@​dependabot</code></a>[bot] in <a href="https://redirect.github.com/actions/setup-java/pull/896">actions/setup-java#896</a></li> </ul> <h3>Bug Fixes</h3> <ul> <li>Prevent default installation of JetBrains pre-releases by <a href="https://github.com/priyagupta108"><code>@​priyagupta108</code></a> in <a href="https://redirect.github.com/actions/setup-java/pull/859">actions/setup-java#859</a></li> <li>Improve Error Handling for Setup-Java Action to Help Debug Intermittent Failures by <a href="https://github.com/gowridurgad"><code>@​gowridurgad</code></a> in <a href="https://redirect.github.com/actions/setup-java/pull/848">actions/setup-java#848</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/gowridurgad"><code>@​gowridurgad</code></a> made their first contribution in <a href="https://redirect.github.com/actions/setup-java/pull/848">actions/setup-java#848</a></li> <li><a href="https://github.com/salmanmkc"><code>@​salmanmkc</code></a> made their first contribution in <a href="https://redirect.github.com/actions/setup-java/pull/888">actions/setup-java#888</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/actions/setup-java/compare/v4...v5.0.0">https://github.com/actions/setup-java/compare/v4...v5.0.0</a></p> <h2>v4.9.1</h2> <p>Adds a deprecation warning for setup-java v4.</p> <h2>v4.9.0</h2> <h2>What's Changed</h2> <ul> <li>Fix v4 dependency audit failures by <a href="https://github.com/brunoborges"><code>@​brunoborges</code></a> in <a href="https://redirect.github.com/actions/setup-java/pull/1188">actions/setup-java#1188</a></li> <li>Deprecate legacy Adopt distributions in v4 by <a href="https://github.com/brunoborges"><code>@​brunoborges</code></a> in <a href="https://redirect.github.com/actions/setup-java/pull/1187">actions/setup-java#1187</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/actions/setup-java/compare/v4.8.0...v4.9.0">https://github.com/actions/setup-java/compare/v4.8.0...v4.9.0</a></p> <h2>v4.8.0</h2> <h2>What's Changed</h2> <ul> <li>License and Audit Fixes by <a href="https://github.com/HarithaVattikuti"><code>@​HarithaVattikuti</code></a> in <a href="https://redirect.github.com/actions/setup-java/pull/960">actions/setup-java#960</a></li> <li>Update SapMachine URLs by <a href="https://github.com/RealCLanger"><code>@​RealCLanger</code></a> in <a href="https://redirect.github.com/actions/setup-java/pull/965">actions/setup-java#965</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/actions/setup-java/compare/v4...v4.8.0">https://github.com/actions/setup-java/compare/v4...v4.8.0</a></p> <h2>v4.7.1</h2> <h2>What's Changed</h2> <h3>Documentation changes</h3> <ul> <li>Add Documentation to Recommend Using GraalVM JDK 17 Version to 17.0.12 to Align with GFTC License Terms by <a href="https://github.com/aparnajyothi-y"><code>@​aparnajyothi-y</code></a> in <a href="https://redirect.github.com/actions/setup-java/pull/704">actions/setup-java#704</a></li> <li>Remove duplicated GraalVM section in documentation by <a href="https://github.com/Marcono1234"><code>@​Marcono1234</code></a> in <a href="https://redirect.github.com/actions/setup-java/pull/716">actions/setup-java#716</a></li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/actions/setup-java/commit/b6effb05e454b25005698d916606bdc6ffcbf961"><code>b6effb0</code></a> Deprecate legacy Adopt distributions in v5 (<a href="https://redirect.github.com/actions/setup-java/issues/1186">#1186</a>)</li> <li><a href="https://github.com/actions/setup-java/commit/e498d2a66a953492f322542257b22125c989b422"><code>e498d2a</code></a> Backport <a href="https://redirect.github.com/actions/setup-java/issues/1151">#1151</a>: Fix missing wrapper cache distributions (<a href="https://redirect.github.com/actions/setup-java/issues/1153">#1153</a>)</li> <li><a href="https://github.com/actions/setup-java/commit/6a3384db745932178632d0e22b2bd28cad1678e6"><code>6a3384d</code></a> Fix npm audit failures on releases/v5 (<a href="https://redirect.github.com/actions/setup-java/issues/1154">#1154</a>)</li> <li><a href="https://github.com/actions/setup-java/commit/03ad4de0992f5dab5e18fcb136590ce7c4a0ac95"><code>03ad4de</code></a> Backport <a href="https://redirect.github.com/actions/setup-java/issues/1097">#1097</a>/<a href="https://redirect.github.com/actions/setup-java/issues/1098">#1098</a>: cache Maven and Gradle wrapper distributions separately...</li> <li><a href="https://github.com/actions/setup-java/commit/d229d2e858d9137cc0b3f118fa5184b9f0a44ac4"><code>d229d2e</code></a> Backport <a href="https://redirect.github.com/actions/setup-java/issues/1111">#1111</a>: Preserve Maven toolchains across repeated setup-java runs (<a href="https://redirect.github.com/actions/setup-java/issues/1">#1</a>...</li> <li><a href="https://github.com/actions/setup-java/commit/bbf0f6967066506f72571a96d5d6c67ca42ab460"><code>bbf0f69</code></a> dist: Cover Tencent Kona JDK 25 (<a href="https://redirect.github.com/actions/setup-java/issues/1110">#1110</a>)</li> <li><a href="https://github.com/actions/setup-java/commit/513edc4f8710565e4ad696f3b7d8e3bda584a46c"><code>513edc4</code></a> feat: expose cache-primary-key output (<a href="https://redirect.github.com/actions/setup-java/issues/597">#597</a>) [v5 backport] (<a href="https://redirect.github.com/actions/setup-java/issues/1089">#1089</a>)</li> <li><a href="https://github.com/actions/setup-java/commit/62df799a9c6e3022bb466697c66c36e9a2dbf347"><code>62df799</code></a> Add Maven compiler problem matcher for javac diagnostics (<a href="https://redirect.github.com/actions/setup-java/issues/1087">#1087</a>)</li> <li><a href="https://github.com/actions/setup-java/commit/176156a187714aaf460b0a3c8f21e8b4f784b978"><code>176156a</code></a> chore: bump version to 5.6.0 for v5 release line</li> <li><a href="https://github.com/actions/setup-java/commit/bf7b8deac240b9cee05eb15ccdb1d2f424a54b9f"><code>bf7b8de</code></a> build: rebuild dist for backported changes (<a href="https://redirect.github.com/actions/setup-java/issues/1079">#1079</a>, <a href="https://redirect.github.com/actions/setup-java/issues/1083">#1083</a>, <a href="https://redirect.github.com/actions/setup-java/issues/1084">#1084</a>)</li> <li>Additional commits viewable in <a href="https://github.com/actions/setup-java/compare/v4...v5">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>	2026-08-25 01:42:01 +00:00
Hengfei Yang	022492fbde	fix: file list timestamp maybe zero (#13936) ... improved on #13925 - [x] Improve db check for timestamp zero - [x] Improve ingestion for timestamp zero	2026-08-25 00:24:58 +00:00
Shrinath Rao	f5b17c98db	test: stabilize logoManagement _meta org navigation on alpha1 (#13934) ... Fixes intermittent alpha1 failure of `GeneralTests/logoManagement.spec.js` ("Logo Upload on Management"). **Cause:** the `_meta` general-settings navigations raced the org-switch redirect to `/web/`, failing three different ways across the 3 retries — home-icon timeout, missing logo upload field, and `goto` interrupted by nav to `/web/`. Not a regression (spec unchanged since #13391); ~50% flake on alpha. **Fix:** `openMetaGeneralSettings()` retries the goto until the URL sticks on `_meta` settings and the page renders; dropped the redundant `navigateToManagement()` menu click that competed with the goto.	2026-08-24 12:05:37 +00:00
Ashish Kolhe	b2639c4f9f	feat(dbm): Database Monitoring (#13918) ... ## What this is Database Monitoring: an aggregation and presentation layer over telemetry OpenObserve already ingests. It reads three vantages — client spans, server-vantage logs, and collector metrics — and turns them into per-statement, per-instance and per-database views. No new storage engine and no new ingest path. Statements are normalized and rolled up into a stats stream, and the read API queries that stream through the existing search path. ## Shape of the change | | | |---|---| | Read endpoints | 17 | | UI pages | 8 (plus a shell) | | Commits | 141 | **Backend** — `src/core/src/db_monitoring/` holds the non-HTTP half (normalization, rollup, server-vantage parsing). The read API lives in `src/api/management/src/request/db_monitoring/` as a handler / models / service trio; the service layer is split one file per feature. **Frontend** — `web/src/views/DatabaseMonitoring/`, with shared logic in `web/src/composables/dbm/`. ## Feature availability Databases, queries, query detail and history, endpoints, samples, activity, insights, plans, server metrics, server queries, server samples, instances, instance metrics, badges — is available in OSS. ## Reviewing this It is a large diff, and most of it is additive: 327 files, and the two largest non-test files are the shared service module and the deadlocks service. Suggested order: 1. `src/api/management/src/request/db_monitoring/handler.rs` — the 17 entry points and their gating 2. `src/api/management/src/request/db_monitoring/service/common.rs` — auth, SQL building, the shared read path 3. `src/core/src/db_monitoring/rollup.rs` — how the stats stream is produced 4. `web/src/views/DatabaseMonitoring/` — the pages Time semantics are the subtlest part. Rollup rows are stamped at the window **end**, so reads are exclusive at both edges; `stats_read_range` carries the reasoning and a test pins it. ## Known follow-ups - **The server-vantage stream named as `_o2_dbm_server`.** ## Testing - Rust: unit tests across the DBM core and read service, including SQL-building, auth-gating and normalizer cases - Web: unit tests across the DBM pages and composables - Playwright: DBM suite covering the pages and cross-page navigation Verified before opening: OSS and enterprise builds both compile warning-free, `cargo fmt` clean, `vue-tsc` clean, and the full web unit suite passes (44,704 tests).	2026-08-24 11:08:00 +00:00
4 people	5f034c6666	feat: folder icons (#13895) ... Design at: https://github.com/openobserve/designs/blob/main/ux/folder-icons/design.md --------- Co-authored-by: Kirtan Thakkar <kirtan@kiara.tech> Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech> Co-authored-by: Abhay Padamani <141122205+ktx-abhay@users.noreply.github.com>	2026-08-24 08:56:02 +00:00
Huaijin	81ffa4b8e2	fix(metrics): bound indexed merge output memory (#13928) ... ## Summary - spool finalized indexed metrics Parquet and midx output to temporary files, materializing one data file only when it is uploaded - keep MetricsFileLayout limited to metrics-specific HashSorted and Indexed layouts while standard files carry no metrics layout - apply one StorageTier consistently to compacted Parquet/Vortex data, midx sidecars, and Tantivy indexes - preserve the ingester upload tier behavior as Default ## Validation - cargo fmt --check - git diff HEAD --check - cargo check -p infra -p tantivy_utils -p compaction -p openobserve-jobs - cargo test -p tantivy_utils index_builder::tests - cargo test -p compaction merge:: - cargo test -p openobserve-jobs job::files - cargo clippy -p infra -p tantivy_utils -p compaction -p openobserve-jobs --all-targets -- -D warnings Follow-up to #13882.	2026-08-24 08:49:48 +00:00
Prabhat Sharma	fea71e02d4	feat(alerts): alert library — browse, preview and install curated alerts (#13922) ... ## Alert Library A curated catalog at `/alert-library`: browse packs of ready-made alerts, see which ones can actually run against the streams this org already receives, preview a verdict against real data, and install one or many. The library is a **peer page** of Alerts / Notification Destinations / Templates — flat routing, presented as tabs via a shared `AlertSectionTabs` in `OPageLayout #header-tabs` (the pattern `PipelineSectionTabs` already ships). ### What landed, by phase | Phase | What | | --- | --- | | Data layer | S3 manifest fetch, cache, readiness resolution (`useAlertLibrary`, `constants/alertLibrary`, `types/alertLibrary`) | | Gallery | Rail (packs / categories, multi-select, segmented switch), stat-strip availability facets, cards | | Drawer | Detail view with query preview and customizable tunables | | Install | Bulk-install wizard — destination selection, per-alert tunables, stale-session hardening | | Prefill | Carry a library alert (query, trigger, threshold, severity) into the alert editor | ### Design decisions worth knowing - **Rail navigates, strip filters.** Availability is deliberately not in the rail — carrying both overflowed it and pushed a facet group below the fold. - **Colour marks the exception only.** No "Ready" pill (it would describe ~75 of 86 cards and signal nothing); only unusable cards carry a label. - **A failed `/streams` call leaves readiness unknown**, rather than claiming "none of your data matches" — that claim would be about the request, not the org, and it is the most alarming thing this page can say. Counts read `—` until there is an answer, and a failed *refresh* keeps the verdict it already had. - **No "Installed" tile** — installed detection doesn't exist yet, and a tile reading a permanent zero is dishonest UI. - **`severity: warning` renders as `medium`** — the badge group's scale is critical/high/medium/low/info, and medium is the amber it also gives P3, which is exactly the priority a warning alert installs as. - Pack and category names come from the manifest, so they render as `raw()` data — a pack added upstream needs no frontend release. ### Design-system changes (used by the library, applied app-wide) - **`warning-quiet` badge tone** — one variant whose values differ by theme: the soft amber wash in light, a neutral chip in dark. Neither the filled amber chip nor the neutral one was right in both, because the two ramps aren't symmetric. - **Border tokens now meet WCAG 2.1 SC 1.4.11.** Every border token measured under 3:1 (`border-strong` was 2.52:1). `border-default` firms to grey-300 and controls get a token that clears 3:1. - Nav: Reliability's flyout groups its four alerting entries under one **Alerts** header, declared in the same order as the tab strip those pages carry. Flyout levels are sized by depth. - `sensors-off` icon added to the `OIcon` registry. ### Fixes that rode along - **metrics**: a shared URL whose `metrics_data` blob omits panel query fields no longer blanks the whole metrics page — blob queries are filled out to the panel schema before being laid over. - **dashboards**: a PromQL series with no legend template was named `JSON.stringify(metric)`, so fifty series read identically for the width of the chip. Series are now named by the labels that actually differ, preferring the ones that say *which workload* it is (namespace / service / pod / host). Correlation's semantic groups refine that when loaded, but nothing is fetched to render a chart — that endpoint is enterprise-only and a chart must render the same on OSS. - **alerts**: preview evaluates against the alert's real trigger; deleting a row no longer reloads the list behind it; peer tabs no longer shift when switching alerting pages. - **ui**: badge-only select options no longer render bold. ### Tests - `test(search)`: pins the streaming-aggregate **wire contract** at the HTTP boundary. Under streaming aggregates each partition carries the cumulative merged state (last frame is the answer) while ordinary partitions are disjoint and must be summed. No pytest had ever received `streaming_aggs: true` — the emitter could have flipped to per-partition deltas with the whole suite green while dashboards, Logs and the alert preview all silently over-counted. - `test(e2e)`: four helpers that passed only while an org stayed small — an assertion reading past a 20-row page, an unfiltered `OSelect` that virtualises past 50 entries, a tab click landing on the rename pencil, and two builder tests asserting on whichever metric sorted first. - New unit specs across the library components, composable, facets, tunables and install path. ### Notes - **i18n**: en-US only; CI generates the other 14 locales. - **No backend changes** — this branch is entirely `web/` plus test helpers.	2026-08-23 20:32:28 +00:00
Ashish Kolhe	029fbbd3d5	fix: stop serving deployment internals on the unauthenticated /config (#13921) ... ## Problem `GET /config` is unauthenticated by design ## Fix Split the endpoint: - **`GET /config` (unauthenticated)** now returns only the login-page bootstrap: `sso_enabled`, `native_login_enabled`, the four custom-logo branding fields, the `rum` init block, `telemetry_enabled`, `build_type`, and `commit_hash` (kept so the UI's stale-build detection keeps working; version, license, billing orgs, storage region, etc. no longer leak). An **exact-key-set test** pins this surface so a new field cannot land on the unauthenticated endpoint unreviewed. - **`GET /api/{org_id}/config` (authenticated)** serves the full configuration, minus `instance`, which is dropped entirely. It goes through `auth_middleware` like every service route and is classified as a querier route, matching where `/config` was served before. - The web app fetches the bootstrap at boot and the full config from MainLayout once an org is known (with a watcher for fresh sessions), and the logo-settings / Usage-tab refresh paths use the authenticated endpoint. ## Paired PR `o2-enterprise` branch `fix/config-api-exposure` adds the `ROUTE_PERMISSIONS` entry (`bypass: true`, like the marketplace routes) so any authenticated org member can fetch `GET /{org}/config` past OpenFGA. ## Verification - Router tests: bootstrap exact-key-set, `/config/reload` still 401, `/{org_id}/config` dispatch + no-`instance` (3/3) - `cargo test` for config / api-http / api-management crates: green - Targeted web unit tests (config service, buildVersionChecker, General settings): green - Enterprise build verified via the `Cargo.toml.openobserve` swap: `cargo check --all-targets` clean - `o2_openfga` route-resolution tests for the new entry: green - `cargo fmt --check` clean; CI clippy (`--workspace --all-targets -- -D warnings`) clean	2026-08-23 04:47:45 +00:00
Hengfei Yang	fbab9b80a2	fix: reduce db calls for realtime alerts (#13920) ... Closes #13910 ## What changed Realtime alerts made a burst of metadata db calls on every firing — and with `silence = 0`, on every matching ingestion request: - short URL creation: 1 SELECT (always a miss, since the alert URL embeds row timestamps and is unique per firing) + 1 INSERT + a coordinator event whose empty body forced a SELECT on **every** watcher node - silence update: 1 UPDATE + a coordinator event with an empty body (Postgres only) forcing a `scheduler::get` SELECT per watcher node - with `silence = 0`, nothing throttled at all, so a high-volume stream repeated all of this per matching request This PR cuts a firing down to 1 INSERT + 1 UPDATE with zero watcher fan-out, and bounds the firing rate: 1. **Coordinator events carry the body.** The Postgres scheduler backend now serializes the trigger into the event (SQLite already did — the two backends had silently drifted). The five copy-pasted emit blocks across both backends are collapsed into one `emit_realtime_trigger_event` helper in `infra::scheduler`, so the event-body wire contract lives in one place. Short URL put events likewise carry the `ShortUrlRecord`. Both watchers use the body when non-empty and keep the db-read fallback for empty-body events from old nodes, so rolling upgrades are safe. 2. **Short URL uses `INSERT … ON CONFLICT DO NOTHING`** instead of select-then-insert; `add` reports whether a row was written, no-op writes emit no events, and the redundant `contains()` pre-check in the super-cluster queue (plus the now-dead helper) is gone. The old `UniqueViolation` downcast in `shorten()` was dead code — `add` never produced that variant, so an md5 collision used to fail the notification; collisions are now genuinely handled. 3. **Fixed 30s minimum silence for realtime alerts.** The effective silence is `max(alert's silence, 30s)`, computed by `TriggerCondition::effective_silence_micros`. A realtime alert can no longer fire once per matching request, even with its silence set to 0. 4. **Bug fix in `get_stream_alerts`:** an early `return` (instead of `continue`) when one stream's alerts all filtered out silently dropped alerts for the remaining streams of the request — this hit the multi-stream metrics/traces callers. 5. **Traces and metrics ingestion spawn `evaluate_trigger`** like the logs path already did, instead of awaiting notification + db work inline on the ingest request path. ## Notes for review - The event-body watcher fallback means mixed-version clusters degrade to today's behavior (one SELECT per event) rather than breaking. - Enterprise verified: `cargo check --all-targets` passes with the enterprise feature (via `Cargo.toml.openobserve` swap); no enterprise-side changes needed. - `cargo fmt`, the CI clippy command, and unit tests for the touched crates pass. - Known remaining gap, out of scope here: silence takes effect on a node only after the coordinator event round-trips into the `REALTIME_ALERT_TRIGGERS` cache, so requests landing in that (now shorter) window can still fire duplicates. A synchronous local cache write in `evaluate_trigger` would close it and composes cleanly with this PR.	2026-08-21 14:36:19 +00:00
Hengfei Yang	efdbd45f84	fix(search): clamp auto histogram interval to 1s for sub-10s ranges (#13916) ... ## What For time ranges narrower than 10 seconds, the auto-generated histogram interval now drops to `1 second` instead of the previous `10 second` floor. Implemented as a new `s` unit in the `intervals!` ladder in `generate_histogram_interval` plus a `(s, 10, "10 second")` tier, with the fall-through returning `1 second`. All tiers from 10s upward are unchanged, and intervals written explicitly in user SQL are untouched. ## Why Fixes #13896. With the 10s floor, a 1-second query range got a bucket wider than the whole range. The scan-start snap to the bucket boundary (`histogram_bucket_start`, introduced by #12312 to keep the first chart bucket complete) then pulled in up to 10s of data from before the selected range, so histogram bucket counts dwarfed the log search total — the UI event counter and pagination are derived from the bucket sum, producing e.g. "1 to 50 out of 195 events" with only 8 real rows and empty trailing pages. With 1s buckets, whole-second range starts sit exactly on the bucket grid, so the snap no longer widens the scan and the bucket sum matches the log search total. ## Verification - Unit tests added for the 1s/10s boundary; full `search` crate suite passes. - E2E reproduction (single node, 20 events/sec ingested, 1s mid-bucket range `[hh:mm:s4, hh:mm:s5)`): - before: log search `total: 20`, histogram bucket sum **100** (follower plan showed `FilterExec: _timestamp >= …s0` — start snapped back 4s) - after: log search `total: 20`, histogram bucket sum **20**, `FilterExec` covers exactly the requested 1s range ## Notes for reviewers - The snap behavior from #12312 itself is intentionally left untouched; this fix only ensures the derived bucket is never wider than the range. - The clamp is deliberately **not** added to `validate_and_adjust_histogram_interval`: that path validates caller-supplied intervals (explicit SQL intervals, and the per-partition re-validation of the streaming pre-computed interval), where range-dependence would make partitions disagree — see `test_preset_interval_is_consistent_across_partitions`. - Millisecond-precision range starts (relative time pickers) can still be snapped back by up to 1s; that residue is bounded by one 1s bucket versus up to 10s before.	2026-08-21 13:31:02 +00:00
Huaijin	17cbe709a1	fix(metrics): preserve timestamp sorting for hash-ordered queries (#13919) ... ## Summary - keep the required `_timestamp DESC` result order independent from the physical input order for metrics queries - preserve the existing `TimestampDesc` source-order optimization for non-metrics streams, including join right-side limits - disable Tantivy `SimpleSelect` optimization for metrics while leaving other stream types and index modes unchanged - add explicit, implicit, projected-field, and join regression coverage for hash-ordered metrics input ## Context Follow-up to the query-correctness TODO in #13882. This PR does not include the separate TSID-major merge-memory or WAL-ordering follow-ups. ## Testing - `cargo test -p search --lib -- datafusion::optimizer::` (152 passed) - `cargo test -p search_service --lib -- --test-threads=1` (200 passed) - `cargo clippy -p search -p search_service --tests -- -D warnings` - `cargo fmt --all -- --check`	2026-08-21 13:23:14 +00:00
Hengfei Yang	e148433fc6	fix(metrics): match __name__ selectors for metrics with uppercase original names (#13917) ... ## Problem Fixes #13892. Stream names are lowercased by `format_stream_name` at ingestion, but the Prometheus remote-write and JSON ingestion paths stored the **original (mixed-case) metric name** in the `__name__` column. A `{__name__="..."}` selector resolves the stream by its lowercase name and then pushed the same matcher down as a **column filter**, which matched zero rows. The metrics explorer builds every card query and has-data probe in the `{__name__="..."}` form, so any metric whose original name contains uppercase letters (e.g. ClickHouse's `ClickHouseAsyncMetrics_*`) showed **no data** there — while the visualize page's `name{}` form (where the name never enters the matcher list) returned data. That is exactly the 8 → 0 behavior in the issue screenshots; the `pod`/`status` label filters shown there were not the cause (matchers on columns absent from the schema are skipped). ## Fix **Query side** - `engine.rs`: when the `__name__` matcher is lifted into `selector.name`, remove it from the matcher list. This single point also stops the matcher leaking into partition-pruning filters, and fixes the self-inconsistent selector whose re-serialized PromQL text (`foo{__name__="foo",...}`) is rejected by the parser when sent to super-cluster peers — un-upgraded peers now behave correctly for this selector form too. - Defensive `NAME_LABEL` skips in `matcher_predicates` (shared by the parquet and WAL scan paths), the tantivy index-condition conversion, and `get_series` (aligning it with the existing skip in `get_label_values`). **Write side** - Remote-write and JSON ingestion now store the formatted stream name in the `__name__` column, matching what the OTLP writer already does. Other label values are untouched. ## Verified - Reproduced on a local build before the fix: `count({__name__="clickhouseasyncmetrics_cpufrequencymhz_test"})` returned empty while `avg(clickhouseasyncmetrics_cpufrequencymhz_test{})` returned data; stored `__name__` was `ClickHouseAsyncMetrics_CPUFrequencyMHz_Test`. - After the fix, against the same data dir: the `{__name__="..."}` count/bare/`avg_over_time(...[1m])` forms and the series API all return the pre-fix (uppercase-stored) rows; other matchers (`job="nope"`) are still enforced; newly ingested records store the lowercase name. - New regression tests: `matcher_predicates` skips `__name__`; the engine strips the consumed `__name__` matcher before handing matchers to storage. Full `promql` (380), `openobserve-core` `metrics::` (173) and `promql-service` (17) suites pass; clippy clean. ## Reviewer note `__name__` participates in the series hash, so rows ingested **before** this change keep the original-case value and will render as a separate series alongside new rows in windows spanning the deployment, until the old data ages out. This is a one-time transition cost of normalizing the stored value.	2026-08-21 13:03:19 +00:00
Hengfei Yang	f3287659e6	chore: add project coding rules and tidy lint/gitignore config (#13913) ... ## What - **CLAUDE.md** (new, tracked): project-level rules for AI coding sessions — build/workflow basics, Rust code organization (item order `mod → use → const/static → types → impls → fns`, pub/private function grouping, `#[cfg(test)] mod tests` always last, structural thresholds are ratchet-only), comment discipline (one line, only for non-obvious constraints), PR conventions, and enterprise-gated code verification (including keeping `Cargo.toml.openobserve` in the o2-enterprise repo in sync with this repo's root `Cargo.toml`). Replaces `.claude/.clauderc`, which is not a file Claude Code reads. - **Structural clippy ratchets in CI**: the clippy step now also enables `too_many_lines`, `cognitive_complexity`, and `excessive_nesting`, with clippy.toml thresholds set just above today's worst offenders (1024 lines / complexity 100 / nesting 10), so the current tree passes clean. Measured across the workspace with `--all-targets`: max function 1019 lines, max complexity 94, max nesting 10. The thresholds only ratchet down as code is cleaned up — CI failing on one of these means split the function, not raise the number. - **clippy.toml**: add `source-item-ordering = ["module"]` so the opt-in `clippy::arbitrary_source_item_ordering` lint checks module-level item order only (its default also forces alphabetical struct fields/enum variants). - **.gitignore**: reorganized into commented sections and slimmed from 116 to 75 lines — removed duplicate entries, patterns shadowed by other patterns, one dead rule, and 14 stale one-off entries with no matching files; simplified the `.vscode`/`.opencode`/`.claude` blocks. ## Why Coding sessions kept placing functions in arbitrary positions (above constants, mixed into the wrong visibility group, tests at the top of files) and writing multi-line comment essays, and nothing bounded function size/complexity growth. CLAUDE.md states the conventions once for every contributor's sessions; enforcement stays with the standard Rust toolchain (rustfmt, clippy thresholds, CI's `-D warnings`). ## Notes for reviewers - The enterprise CI clippy step runs its own command without these `-W` flags, so it is unaffected. - An earlier revision of this branch added a PostToolUse hook script for style checking; it was dropped in favor of toolchain + prompt rules only, and `.claude/settings.json` stays gitignored. - The gitignore rewrite was verified behavior-preserving at each step: the untracked-file list and an 86-path rule battery were compared before/after every change; the only intentional semantic changes are the removals listed above.	2026-08-21 10:21:48 +00:00
Hengfei Yang	cf40d3c7f0	fix(infra): fail startup when the NATS queue backend is unreachable (#13914) ... ## What With `ZO_QUEUE_STORE=nats` and no reachable NATS server, the process used to boot successfully. The first NATS connection happened lazily inside background tasks (pipeline execution etc.), whose panics tokio swallows, so the node kept running but could not serve queue-backed features — an error state users easily miss. Now an unreachable NATS fails boot with a clear error and exit code 1: ``` infra init failed: Error# NATS connect failed for address(es): [...], err: IO error: Connection refused (os error 61) Error: backend job init failed, exiting ``` ## How - Implement the missing queue backend init hook: `queue::init()` dispatches by `queue_store` — `nats::init()` establishes the connection at startup and fails boot on error, `memory::init()` is a no-op. - Call `queue::init()` from `infra::init()` as well: `db_init()` runs only when migrations are needed, so on a normal boot (schema version matches) it is skipped entirely and cannot carry the check alone. - Make `db::nats::connect()` return `Result` so the startup path gets a clean error instead of a panic. `get_nats_client()` keeps its `&'static Client` panic-on-failure contract for existing lazy callers (with the full error now in the panic message); after this change every NATS-configured boot connects during startup, so that panic path is effectively dead. The meta store already had an equivalent startup check: `migration::init_db()` reads the schema version on every boot with a 5-attempt retry and aborts on an unreachable database. ## Verified Ran the debug binary in local mode against each scenario: | scenario | result | |---|---| | fresh db, `queue_store=nats`, NATS down | exit 1 with clear error | | migrated db (skips `db_init`), NATS down | exit 1 with clear error | | NATS up (docker, JetStream) | boots, healthz ok, no panics | | `queue_store=memory` | boots, never touches NATS |	2026-08-21 10:14:04 +00:00
Huaijin	37e5f55684	feat(metrics): .midx series index for TSID-major files and PromQL pruning through it (#13899) ... The query half of the TSID-major layout: #13882 wrote `tsid-major-v3-*` files ordered by `(__hash__, _timestamp)`; this PR adds the `.midx` series index next to them and makes PromQL prune through it. Supersedes the remaining scope of #13839. ## Write side - Every `tsid-major-v3-*` file written by the hour-end merge gets a `.midx` series index (Arrow IPC + zstd), stored like the Tantivy index under a derived root distinct from the data tree: `files/{org}/metrics/{stream}/{date}/{hour}/tsid-major-v3-{id}.parquet` → `files/{org}/midx/{stream}/{date}/{hour}/tsid-major-v3-{id}.midx` (Tantivy: `files/{org}/index/{stream}_{type}/…/{id}.ttv`): **one row per contiguous TSID run** — its `row_count: u32` plus the label columns of the run's first row (source types preserved). Runs tile the data file, so each run's starting row is the prefix sum of the preceding counts and is not stored; `__hash__` is excluded (matchers never reference it). The whole index is one ZSTD-compressed IPC batch. Runs crossing an input RecordBatch boundary appear as two adjacent entries; query-time range coalescing makes that representation equivalent without buffering samples. File and row-group boundaries stay independent of TSID boundaries. - `TsidSeriesIndexWriter` lives inside `search/datafusion/merge/tsid_major.rs` and collects run entries alongside the Parquet writer (one buffered row per run), encoding and compressing once at finish. `MergedFile` carries `series_index: Option<Vec<u8>>`; the compactor uploads it to the `midx/` tree (compliance mode included). - `.midx` is intentionally **not** tracked in `file_list`; its path is derived from the data file's key (`MetricsFileLayout::series_index_path`) and it is deleted together with the parent in the delayed-deletion pass (`deleted.rs`); like the `index/` tree, the `midx/` day directories are covered by retention's local-disk deletion and the gc tool. ## Query side (PromQL) - `promql_service::…::tsid_series_index::search` runs where Tantivy used to, before the metrics table is registered: for each TSID-major file it loads the `.midx` (concurrent, `2×target_partitions ≤ 64`), projects the row-range columns plus the matched labels **by name against each series index's own schema** (label sets / column orders differ between files), builds the matcher filter **per series index** with `physical_expr::create_physical_expr` — the exact same predicates as `apply_matchers` (`matcher_predicates`) — evaluates it over the run rows, coalesces adjacent/overlapping ranges and attaches them as `FileSelection::RowRanges` (converted into a Parquet access plan by the generic helper from #13861, no per-row bitmap). - **Applied per file**: files of any other layout in the same query (still-open hours, legacy data, late arrivals) are left in place and scanned in full. A matched label missing from a series index over-selects that file; a query whose matchers cannot be evaluated at all selects the whole file; any series index failure logs and falls back to a full scan. The final PromQL matchers are always applied, so results are exact in every case. - Evaluated selections are cached (LRU, 256 MiB) keyed by account + series index path + the full matcher set. - The Tantivy path in the PromQL storage scan is removed (`convert_matchers_to_index_condition`, index-field plumbing, `is_add_filter_back` bookkeeping). SQL search is untouched. ## Benchmark (from the #13839 POC, same layout + series index) 478,144,896 rows, 7 CPUs, one-hour PromQL range, step 15s, result cache off, release + mimalloc: | Layout | Filter steady median | No-filter steady median | Data files | Total storage | |---|---:|---:|---:|---:| | main, 2 GiB | 695.709 ms | 4.638 s | 88 | 4,925.81 MiB | | TSID-major + `.midx` | 269.496 ms | 2.793 s | 80 | 1,294.61 MiB | ## Validation - `cargo fmt --check`; `cargo check` / `clippy` on config, search, compaction, promql-service, jobs, search_service (`--tests`) clean - `config tsid_layout` 3 passed (incl. `series_index_path`); `search tsid_major mode:: merge::` 10 passed (rotation test now also asserts the series index, exact-run and coalescing series index tests); `compaction merge deleted` 33 passed; `promql-service` 20 passed + 1 pre-existing wall-clock flake (`cache::test_promql_cache_set_and_get`, fails intermittently on main too) - End-to-end on the #13839 branch (same code paths): hour-end merge writes N `tsid-major-v3-*` + `.midx`; PromQL takes the series index path (`selected 39 ranges across 4/4 files`), counts/sums match the synthetic data, cold-cache identical after restart; a mixed hour (3 major + 1 late `tsid-sorted`) reports `1 files without a series index left to tantivy/full scan` with unchanged results; delayed deletion removes `.parquet` **and** `.midx`	2026-08-21 10:05:46 +00:00
ktx-vaidehi	7e0bdec6fd	fix: png to svg chart types (#13911)	2026-08-21 06:54:16 +00:00
Omkar Kesarkhaneandgithub-actions[bot]	fac24d59bb	fix(traces): trace-log correlation fixes and Trace Graph height (#13901) ... ## Problem Four defects around traces and the trace↔log correlation flows: 1. **Trace Graph is clipped.** The service map renders inside a 200px band, vertically centred in an otherwise empty pane, instead of using the available height. 2. **A span row's View Logs goes nowhere.** In OSS builds it navigates to `/logs?stream=&from=NaN&to=NaN`; the logs page drops the filter and falls back to its default window, so the user gets an unfiltered result set instead of that span's logs. 3. **The log detail drawer has no View Trace button.** Opening the trace for a log record only works from the inline expanded row. 4. **View Trace lived inside the JSON tab**, so it disappeared when switching to Table. ## Root causes **Trace Graph.** The ancestor chain was already correct — the sibling Waterfall tab fills the same box. Only the leaf was clamped: the chart carried `h-50! min-h-50!` (12.5rem, `!important`), which beat its own root `h-full w-full`. That clamp dates to when the map lived in a small collapsible timeline strip, where 200px was the intended design. When it was promoted to a full-height tab, the scoped rule `.trace-details .trace-chart-height { height: 12.5rem }` was left behind and its two-class specificity silently defeated the `h-full!` added at the time. The later design-token migration deleted the rule but preserved the rendered result by translating it into utilities, which made the clamp look deliberate. **Span View Logs.** `buildQueryDetails()` reads the raw API shape (`start_time`/`end_time`, nanoseconds), but the waterfall passes formatted nodes, which `getFormattedSpan()` rebuilds as a fresh object literal — camelCase, times as `startTimeUs`/`endTimeUs` in microseconds, snake_case originals dropped. Reading `start_time` off one yields `undefined`, hence `NaN`. The helper already had a `spanId || span_id` fallback for the id, which is why the query string was correct while the window was not. **Drawer View Trace.** `JsonPreview` decides whether to offer the action once, in `onBeforeMount`. `DetailTable` fills `rowData` in an `async created()` — the `await` defers assignment past the child's mount even though the work is synchronous — so the preview mounted with an empty record, the check failed, the traces streams were never fetched, and nothing recomputed it. ## Changes - Trace Graph chart uses `h-full`; the orphaned `trace-chart-height` class is dropped (its rule no longer exists) and two nested wrappers collapse into one. Matches the full-bleed pattern in `ServiceGraph.vue`. - The span row resolves the raw span from `spanMap` before calling the helper — the same lookup this component already does for span data, HTTP status and events — rather than teaching the shared helper to accept two shapes. Guarded, since a span missing `span_id` gets a synthetic id absent from the map. - The span row also warns instead of navigating when no logs stream is selected. The header's own View Logs button is disabled in that state but the row-level one is not, so it was a silent dead end. - `JsonPreview` re-evaluates the View Trace gate when the previewed record changes, so the flag tracks its input rather than a single snapshot. - The action moves to the drawer's header row beside the wrap toggle, reachable from every tab, with `hide-view-trace` on `JsonPreview` (mirroring the existing `hide-view-related`). The gate, stream fetch and filter move into `useViewTraceAction` rather than being copied into `DetailTable` — `service_streams_enabled` is already hand-rolled in three files and drifted. `JsonPreview` keeps exposing the same names, so its specs are untouched. Two latent issues surfaced while moving the action: - `hiddenMenus` is seeded as an array and only becomes a `Set` once `MainLayout` has read `custom_hide_menus`, so `.has()` is not always available. Both shapes are now accepted. - `OSelect`'s root sets `w-full`, which won a plain `w-40` and pushed the button outside the drawer — the same class-collision shape as defect 1. Pinned with `w-40!`. ## Tests No behaviour is asserted by new specs; existing suites cover the touched paths and pass unmodified: - `JsonPreview.spec.ts` — 91 passed (unchanged, including the assertions on `tracesStreams`) - `DetailTable.spec.ts` — 70 passed, 3 skipped - `TraceTree.spec.ts`, `useTraces.spec.ts`, `TraceDetailsSidebar.spec.ts` — 297 passed - `TraceDetails` suites — 234 passed - `SearchResult.viewTrace.spec.ts`, `CorrelatedLogsTable.viewTrace.spec.ts` — passed Prettier and ESLint are clean on every touched file. ## Verification Checked against live trace and log data sharing a trace id: - Trace Graph fills its pane (measured 549px where it previously measured 200px), survives a tab round-trip, and reflows on window resize. - A span row's View Logs lands on `/logs` with the span+trace filter and a ±60s window whose bounds match the span's real timestamps exactly; with no stream selected it warns and does not navigate. - The drawer's header shows the picker and button on both the JSON and Table tabs with no overflow, and opens the trace with the correct id and a ±15 min window. - Console clean across every navigation. --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2026-08-21 05:58:02 +00:00
Hengfei Yang	e28de6c6c3	fix(infra): serialize every remaining sqlite write path (#13905) ... ## Problem In local mode with the default sqlite meta store, concurrent metadata writes fail with: ``` Execution Error: error returned from database: (code: 517) database is locked ``` 517 is `SQLITE_BUSY_SNAPSHOT`: a deferred sqlite transaction pins its read snapshot at the first SELECT, and when it later performs its first write the upgrade fails immediately if any other connection committed in between — `busy_timeout` never retries it. The codebase convention is to serialize sqlite writes with `infra::table::get_lock()`, but a full audit found ~100 write functions that never take it, concentrated in the newer subsystems. ## What changed (31 files) - **Table layer, previously zero locks**: `alert_states`, `alert_incidents`, `incident_events`, `dashboards`, `folders`, `ratelimit`, `sessions`, `system_settings`, `alert_composites`, `slo` / `slos` / `slo_budget` / `slo_backfill_jobs`, `synthetics_jobs` / `synthetics_runs`. - **Partially covered modules**: the specific unlocked writers in `users`, `org_users`, `organizations`, `org_ingestion_tokens`, `synthetics_agents`, `synthetics_checks`, `compactor_manual_jobs`, `trial_quota_usage`, `alert_eval_intervals`, `search_job/search_jobs`. - **Service layer**: `db/pipeline_errors` (self-reporting flush), alert deduplication state, the composite-alert scheduler transaction in `core/alerts/scheduler/handlers.rs`, raw anomaly-detection config updates, and two direct writes in `core/alerts/composite/service.rs`. - **Bonus fixes**: `search_job_partitions::submit_partitions` ran its `FOR UPDATE` probe on the pool instead of its open transaction (on Postgres that row lock was not tied to the transaction at all); a ratelimit unit test raced the global `ORM_CLIENT` OnceCell and failed depending on suite ordering. ## Lock-placement rules (why some functions deliberately do NOT lock) The mutex is not reentrant, so the diff enforces **exactly one acquisition per call chain**: - The function that owns the transaction locks before `begin()`; helpers that receive an open transaction (`persist_update_in_transaction`, `apply_status_in_txn`, `record_evaluation_with`, the `alert_states` public wrappers, …) never lock. - Guards are dropped before coordinator emits, `db::scheduler` calls, and helpers that lock internally (e.g. `dashboards::delete_all` locks only after `distinct_values::batch_remove`, which takes the lock itself). ## Verification - Full workspace `cargo check` + clippy clean; 928 infra table tests pass (run twice for ordering stability). - Combined build with the enterprise crate passes; **companion enterprise PR uses the same branch name** `fix/sqlite-lock-remaining` per paired-CI convention. - Live smoke on a sqlite local-mode instance: 40 concurrent folder creations (previously an unlocked path) plus mixed ingestion/metadata load — all requests succeed, zero locked errors in the log. ## Note for reviewers This is the per-call-site fix consistent with the existing convention. A structural alternative (single-connection sqlite RW pool, which makes 517 impossible and would let the ~330 manual lock sites retire) was prototyped separately and can follow as its own change.	2026-08-21 05:38:34 +00:00
Huaijin	08d31371cc	refactor(metrics): prepare indexed metrics groundwork (#13907) ... ## Summary - rename the existing TSID-major config, modules, types, merge modes, and persisted file prefixes to indexed-metrics terminology - add the promql-service dependency declarations needed by the follow-up metrics-index work - carry the exact Cargo.lock content from #13899, including h2 0.4.16 and the promql-service dependency entries ## Scope This is no-new-feature groundwork split from #13899. It retains the existing metrics layout mechanics under the new names and does not add .midx sidecar writing, index lookup/query pruning, or index lifecycle handling. The old environment variable and file prefixes are intentionally not kept as compatibility aliases because this experimental path has not been used in a deployment. ## Validation - cargo fmt --all -- --check - cargo metadata --locked --no-deps --format-version 1 - cargo check -p promql-service - cargo test -p config --lib -- layout (5 passed) - cargo test -p search --lib -- datafusion::merge (8 passed) - cargo test -p compaction --lib -- merge (32 passed) - cargo check -p openobserve-jobs	2026-08-21 04:58:50 +00:00
Huaijin	8ef1fd4cbb	refactor(compaction): split merge/mod.rs by function into job/stream/file modules (#13900) ... ## Summary Pure code-move refactor: the 1795-line `src/compaction/src/merge/mod.rs` is split by responsibility into three modules under `merge/`: - **`job.rs`** — job generation: `generate_job_by_stream`, `generate_old_data_job_by_stream`, `generate_downsampling_job_by_stream_and_rule`, plus the shared `job_range_end` helper (now `pub(super)`) and its test - **`stream.rs`** — per-stream merge orchestration: `merge_by_stream` with its private helpers `write_file_list` and `sort_by_time_range`, plus all `sort_by_time_range` tests - **`file.rs`** — file-level merge execution (called by the worker): `merge_files` with its private helpers `cache_remote_files` and `generate_inverted_index` - **`mod.rs`** — now only module declarations and `pub use` re-exports; `tsid_major.rs` untouched No logic changes — only `use` blocks and relative paths were adjusted (`super::is_past_hour` → `crate::is_past_hour`, etc.). The public API paths are unchanged, so the call sites in `worker.rs` and `lib.rs` need no edits. ## Testing - `cargo check -p compaction` and `cargo clippy -p compaction` clean, zero warnings - All 30 merge-module tests pass (`cargo test -p compaction merge::`)	2026-08-20 12:11:22 +00:00
Huaijin	1dbf32baa2	fix(metrics): preserve numeric JSON series hash (#13893) ... ## Summary - exclude the generated `__hash__` field from JSON metric label stringification - normalize metrics `__hash__` schemas inferred as `Int64` or `Float64` to the canonical `UInt64` type - keep the existing hash calculation order and the rest of the JSON ingestion flow unchanged ## Why The JSON metrics path inserted the numeric series hash and then stringified every label, including `__hash__`. This left JSON-ingested streams with an `Utf8` hash and prevented the UInt64-only TSID-major compaction path from being selected. For hashes above `i64::MAX`, Arrow text JSON schema inference reports the field as `Float64`. The source `serde_json::Number` still retains the exact `u64`, so normalizing only the metrics system hash schema back to `UInt64` preserves the full hash range without changing ordinary floating-point fields. ## Validation - `cargo fmt --all -- --check` - `cargo test -p config --lib utils::schema::tests -- --nocapture` (19 passed) - `cargo test -p openobserve-core --lib metrics::json::tests -- --nocapture` (41 passed) - `cargo clippy -p config -p openobserve-core --lib --tests -- -D warnings` ### E2E Built the PR with `cargo build --features mimalloc` and started OpenObserve with an isolated local data directory. A JSON gauge was ingested through `/api/default/ingest/metrics/_json`, then verified through the stream schema and SQL search APIs: - schema: `__hash__ = UInt64` - schema: `value = Float64` - queried value: `1.25` - queried hash: `14531083140834660755` (greater than `i64::MAX`, returned exactly) - repository metrics API suite: `tests/ingest/test_metrics.py` — 4 passed	2026-08-20 10:35:03 +00:00
ktx-akshay	aa00c5528e	test: improve Alpha automation test stability (#13841) ... …color-by-series features	2026-08-20 10:10:43 +00:00
Huaijin	789eee463f	feat(metrics): TSID-major Parquet layout behind ZO_METRICS_TSID_MAJOR_ENABLED (#13882) ... The storage-layout half of #13839, without the `.sidx` series index and its PromQL query path (those stay in #13839, which will rebase on this). ## What Behind `ZO_METRICS_TSID_MAJOR_ENABLED` (default off), Parquet metrics files of a stream whose `__hash__` column is `UInt64` are written ordered by `(__hash__, _timestamp)` so every series is contiguous: - **`config::meta::promql::tsid_layout`** — `metrics_tsid_major_enabled(stream_type)` (flag + configured Parquet), `metrics_tsid_major_stream(stream_type, schema)` (also `__hash__: UInt64`; JSON-ingested metrics store `__hash__` as a string and keep the classic layout), and `MetricsFileLayout { Legacy, HashSorted, TsidMajor }` encoded in the file-name prefix (`tsid-sorted-*`, `tsid-major-v3-*`) so readers and later merges know the physical order without opening the file. - **`MergeMode::HashSorted`** (open hour): the ingester movers and incremental compactor merges write one hash-ordered file named `tsid-sorted-*`. In pack mode the single-segment upload-as-is fast path is skipped for such streams (segments are in arrival order). - **`MergeMode::TsidMajor`** (closed hour): `merges_whole_batch()` — every file of the hour, regardless of size — into size-split `tsid-major-v3-*` files written at RecordBatch boundaries using `ZO_COMPACT_MAX_FILE_SIZE` as logical original size (`search/datafusion/merge/tsid_major.rs`). `MergeMode::for_compactor / for_ingester` take the schema and pick the mode via `metrics_tsid_major_stream`; downsampling keeps precedence. The compactor skips a closed hour that is already all `tsid-major-v3-*`. - **`MergedFile.layout`** decides the output file name (`layout.file_name` / `mark_file_key`) in the compactor and the ingester movers. - **Merging pre-sorted inputs**: `MergeMode::input_sort_order(&files)` declares `FileSortOrder::HashTimestampAsc` when every input is hash-ordered (`tsid-sorted-*` / `tsid-major-v3-*`). Overlapping hash ranges make DataFusion drop the order for its size-based groups; the listing adapter regroups one file per partition (single-file groups are trivially ordered) and keeps the declared `output_ordering`, so `EnforceSorting` turns the `SortExec` into a `SortPreservingMergeExec` over N pre-sorted streams (memory N × batch instead of the whole hour). A batch that mixes layouts declares no order and falls back to the full sort. Two file states only: `tsid-sorted-*` (ordered, not finalized) and `tsid-major-v3-*` (finalized). Nothing changes for streams without the flag, non-Parquet metrics, or a non-UInt64 `__hash__`. ## Known limitation The SQL query path still derives `sorted_by_time` from the SQL shape only, so with the flag on a `SELECT … FROM <metrics> ORDER BY _timestamp DESC LIMIT n` may assume a timestamp order that hash-sorted files do not have. PromQL is unaffected (no ordering declared). To be addressed with the query-side work. ## Validation - `cargo fmt --check`; `cargo check` / `cargo clippy` on config, search, compaction, jobs, promql-service, search_service (`--tests`) clean - `cargo test -p config --lib -- tsid_layout` 2 passed; `cargo test -p search --lib -- merge:: mode:: tsid_major listing_adapter` 9 passed (mode selection, per-layout input order, hash `ORDER BY`, size-split rotation, SPM plan shape); `cargo test -p compaction --lib -- merge` 31 passed - End-to-end on #13839 (debug build, local disk, OTLP gauge 300 series): ingester uploads `tsid-sorted-*` verified `(__hash__, _timestamp)` ordered; hour-end `merge [tsid_major] input sort order: hash_timestamp_asc` → plan `SortPreservingMergeExec` ← `DataSourceExec{N single-file groups}` (no `SortExec`) → `tsid-major-v3-*`; incremental `merge [hash_sorted]` 2 → 1 `tsid-sorted-*`; JSON-ingested stream (Utf8 `__hash__`) keeps the classic layout with no compactor errors. ## TODO (follow-up, not in this PR) **Sort TSID-major streams by `(__hash__, _timestamp)` already at WAL dump time.** `Partition::persist` full-sorts every dumped chunk anyway (today by `_timestamp DESC` in `merge_record_batches`); switching the sort key for TSID-major streams is near-free at dump time and removes the biggest ingester-side cost of this layout — the full `SortExec` over the merge input: 1. **persist**: lexsort the chunk by `(__hash__ ASC, _timestamp ASC)` (same gate as the writers, `metrics_tsid_major_stream`) and stamp a `sort_order: hash_timestamp_asc` key-value into the Parquet footer. A file-name marker is not an option for WAL files — `parse_time_range_from_filename` splits on `.` and a prefix would break WAL time pruning. 2. **ingester mover**: it already reads every input's footer for the schema; declare `HashTimestampAsc` on the table only when *all* inputs carry the footer marker (else `FileSortOrder::None`, full sort — handles mixed files during rolling upgrades / flag flips, same role `MetricsFileLayout::of` plays for the compactor). The merge then becomes a `SortPreservingMergeExec` over pre-sorted files instead of a full sort. 3. **pack, single segment**: the as-is fast path (skipped for TSID-major streams in this PR because segments were time-sorted) can come back — the segment is already hash-sorted, upload it unchanged and mark it `tsid-sorted-*`. 4. **pack, multiple segments**: one `MemTable` partition per segment with a declared sort order, so the in-memory merge is also an SPM instead of a full sort. Query-side is already safe for this change: metrics SQL never declares `sorted_by_time` and PromQL declares no order, so hash-sorted WAL files cannot mislead a reader. The cost is slightly coarser time clustering inside WAL files, negligible at WAL file sizes. Steps 1–2 carry most of the win; 3–4 are incremental.	2026-08-20 09:20:00 +00:00
Vamsi Karthik RandShrinath Rao	6930a9b724	test(mobile): RUM SDK end-to-end suite across 4 platforms + self-contained CI (#13628) ... ## What this PR adds end-to-end test suite for the **Mobile RUM SDK** under `tests/mobile-testing/`, plus a **self-contained CI workflow** that runs it on real emulators/simulators against an OpenObserve **built from this PR's source**. Test / fixture / CI only — no product-code/existing test changes ## How each test works (3 layers) Every scenario is a real end-to-end journey, so a failure points at the actual broken layer: 1. **Drive** — a Maestro flow taps through the app like a user (`maestro/<track>/*.yaml`). 2. **API** — query the `_rumdata` stream and assert the telemetry is correct (`utils/ooClient.js`, poll-and-retry for async ingestion). 3. **UI** — open the RUM dashboard and assert the sessions ## Coverage (42 tests across 4 platforms) The **native platforms (android-native, ios-native) are now at parity with React Native** — each new native test (masking, network, bg/fg, user-identity, no-phone-home) was built + validated on a real emulator/simulator before landing. | Capability | Layer | RN-Android | Android-native | RN-iOS | iOS-native | |---|---|:-:|:-:|:-:|:-:| | Core telemetry (views/actions/handled-error/crash → `_rumdata`) | API | ✅ | ✅ | ✅ | ✅ | | Crash session viewable in dashboard | UI | ✅ | ✅ | ✅ | ✅ | | Attributes: env / service / version tagging | API | ✅ | ✅ | ✅ | ✅ | | User identity (`setUser`) | API | ✅ | ✅ | ✅ | ✅ | | Network: fetch → resource + 4xx/5xx | API | ✅ | ✅ | ✅ | ✅ | | Session-replay PII masking | UI | ✅ | ✅ | ✅ | ✅ | | Background / foreground view transitions | API | ✅ | ✅ | — | ✅ | | No-phone-home (no egress to Datadog hosts) | security | ✅ | ✅ | — | ✅ | - Shared per-capability factories keep the specs DRY (`utils/rumChecks.js`, `utils/coreRumSpec.js`). - Negatives are covered: 4xx capture, PII-must-not-leak (masking), no phone-home, handled-error ≠ crash. - `o2-enterprise#2289` dashboard bugs are kept as inline skips that flip green once fixed. - Full capability × platform detail + on-device status: `docs/COVERAGE-MATRIX.md`. ## Fixture apps + build-time config - Four vendored apps under `apps/`: `o2-rum-tester` (React Native — drives both RN platforms), `o2-native-android` (Kotlin), `o2-native-ios` (Swift / xcodegen). - Each reads its RUM target (endpoint / org / token) and SDK version from a **build-time-overridable** config. The committed default is the current pin, so local dev is unchanged; CI regenerates it to point at the runner's OpenObserve and can pin the latest SDK on release. ## CI — `.github/workflows/mobile-sdk-e2e.yml` (self-contained, no secrets) - **Build once per OS from source.** `build_oo_linux` / `build_oo_macos` compile OpenObserve (binary + embedded web UI) from this PR's code and cache it. Device jobs download and run it **natively**, and the app posts real RUM to it (Android reaches the host at `10.0.2.2`, the iOS simulator via `localhost`). No Docker, no shared cluster, no secrets. - **Device jobs run in parallel.** `android` (Linux + KVM emulator → rn-android + android-native) and `ios` (macOS simulator, pinned Xcode 16.4 → rn-ios + ios-native). - **`merge_reports`** combines both platforms into one browsable HTML report — even when a device job fails. - **`decide` picks which platform runs, to keep scarce macOS runners in check:** - PR touching Android or shared mobile-RUM code → **Android auto-runs**; iOS is gated behind the `mobile-e2e` label. - `mobile-e2e` label → both platforms (full pre-merge gate). - `repository_dispatch(mobile-sdk-release)` from an SDK repo → runs that SDK's platform (companion PRs wire this up: openobserve-react-native-rum#3, openobserve-sdk-android#1, openobserve-sdk-ios#1). - `workflow_dispatch` → manual, choose platforms. *Separate workflow :* it needs macOS **and** Android emulators, builds 4 phone apps, and is dispatch-driven — matching the repo's existing per-suite split ## Status Green end-to-end on GitHub runners (Android + iOS device jobs + merge) : https://github.com/openobserve/openobserve/actions/runs/31733717758/job/94565520891 --------- Co-authored-by: Shrinath Rao <shnath@openobserve.ai>	2026-08-20 08:13:35 +00:00
ktx-akshay	6fe56575e2	test(datetime-picker): E2E cover for copy/paste in the date-time picker (#13894) ... Adds E2E coverage for the date-time picker copy/paste feature from #13374, which shipped with no E2E tests. **14 tests — 3 P0, 9 P1, 2 P2** Logs picker (auto-apply, 12): controls render with aria-labels; copy resolves a *relative* range to absolute epoch µs; full copy→change→paste round-trip; paste of epoch µs / epoch seconds / ISO-with-Z / slash-absolute / human-log formats; single-value proximity rule on both boundaries; unparseable text and empty clipboard rejected without side effects. Metrics picker (manual-apply, 2): paste stages until Apply commits; closing without Apply reverts the trigger label — guards the `triggerLabel` change that #13374's own review flagged as a behavioural warning. **Notes** - The paste grammar is already exhaustively unit-tested in `web/src/utils/dateTimeRangeParse.spec.ts` (497 lines), so this suite covers the UI wiring those can't reach plus one representative case per format family. - Assertions use a copy-back round-trip rather than wall-clock strings: `getConsumableDateTime()` resolves through `store.state.timezone`, so pasting epoch µs and re-reading via Copy is exact and timezone-independent. Wall-clock assertions are used only for the two timezone-naive formats. - `navigator.clipboard.readText()` requires a secure context. localhost and https qualify; a plain-HTTP host does not, and paste has no `execCommand` fallback (copy does). CI targets localhost, so this is satisfied there. - Registered in the `GeneralTests` CI shard.	2026-08-20 07:29:14 +00:00
Omkar Kesarkhaneandgithub-actions[bot]	cb1b064a5d	fix(synthetics): support to read a run's evidence without leaving the step (#13879) ... ## What Makes the evidence a synthetic browser check captures readable **where you already are**, instead of one tab away. Two related changes on the run-detail view: **1. An evidence row opens to its full record.** The table renders three fields the probe already captures and nothing displayed: the stack trace on a page error, the resource type, and the first-party flag. Plus the full URL — the row strips the host and then truncates, so a login URL died mid-query-string with no way to see the rest. **2. The step expansion's "Page activity" card now hosts that same table.** It used to show five severity-ranked rows and a `5 of 136` count — a card whose headline metric was its own incompleteness. It now renders the step's whole bucket with the same header, sorting, row expansion, filters, wrap and paging the Evidence tab has. Supporting work: the filter row is extracted into one component shared by both surfaces, so they cannot drift onto different controls; and the Evidence tab's step scope is a select you can change rather than a banner you can only dismiss. ## Why Every question past "was there anything?" cost a tab switch, and the correlation you arrived with — this step, this failure — had to be rebuilt by hand against a step filter. ## Notable fixes found along the way - **The Step column was sortable but sorted nothing.** It had no accessor, so the comparator read a field the row does not have, compared `undefined` to `undefined`, and left every row in place while the header showed a working arrow. - **The step expansion's content column had no width clamp.** As a flex item it kept its content's intrinsic width, so a long URL pushed it past the container and every card in it overflowed — and `truncate` and `overflow-x-auto` both went inert, since each needs a bounded parent. Measured at 1487px inside a 1000px container. - **Expand chevrons did not line up.** The row status rail's class also insets the first cell, so rows without a rail sat half a rem left. Benign rows now reserve the slot transparently. - **"Took" was never right-aligned** — the class sat on an inline span, where it aligns nothing. ## Testing `components/synthetics/`, `views/synthetics/`, `composables/synthetics/`, `utils/synthetics/` — **1398 passing**. `type-check`, `lint:design:strict` and `format:check` all clean. New behavioural coverage where it was thin: sorting is driven through the header and asserted on rendered row order rather than on column config, which is what let the non-functional Step sort ship. ## Reviewer notes - `EvidenceEvents.vue` is shared by both surfaces. The only differences left are the Step column and the page size — everything else is deliberately identical. - The step select is gated on being passed options, which is how it stays off the step card, where the step is already the scope. A test on each side pins that. - The per-step card's five-row cap is deliberately removed; pagination bounds the height instead, which bounds the card *and* keeps every event reachable in place. --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>	2026-08-20 07:05:05 +00:00
udev	a6af96289f	fix(ofga): migrate annotation queue and dataset permissions (#13888) ... ## What Adds the OpenFGA migration for the `annotation_queues` and `datasets` resources, gated on model version `0.0.39`, following the pattern used for every other resource in `init()`: - a `need_annotation_queues_datasets_migration` flag - a `v0_0_39` version constant - an `existing_model_version < v0_0_39` gate - `get_ownership_all_org_tuple(org_name, "annotation_queues", …)` and `…"datasets"…` in the per-org loop Those two strings are the `OFGA_MODELS` map keys added in o2-enterprise, which resolve to the `annotation_queue` and `dataset` object types. ## Why `annotation_queue` and `dataset` were added to the OpenFGA model without a migration, so existing deployments never receive the `_all_<org>` ownership tuples for them. ## Depends on o2-enterprise#2477, which adds the two types to `model.json` and bumps `o2_version` to `0.0.39`. Without it this migration never fires, since the version gate is never crossed. ## Verification Verified on a local enterprise instance running both changes: ``` [OFGA:Local] model version changed: 0.0.38 -> 0.0.39 [OFGA:Local] annotation queues and datasets permissions migration needed [OFGA:Local] Data migrated to openfga ``` - `org:<org> owningOrg annotation_queue:_all_<org>` and `… dataset:_all_<org>` are written, with no errors in the batch. - Creating a role and saving `annotation_queue` / `dataset` permissions returns 200 and reads back correctly. - OpenFGA `check`, with the `org_context` contextual tuple the app sends, resolves correctly: `AllowAll` on `annotation_queue` permits GET/POST/DELETE; `AllowList` only on `dataset` permits LIST and denies DELETE. - `cargo check -p openobserve-core --features enterprise` is clean. No frontend change is needed — the roles edit page builds its resource list from the `/resources` endpoint, and the entity fetchers for both resources already shipped in #13719.	2026-08-20 05:42:47 +00:00
3 people	0f4b606e8e	feat(synthetics): super-cluster support, then move synthetics to OSS (#13795) ... > ### ⚠️ This PR carries four changes — review alongside openobserve/o2-enterprise#2420 > > 1. **Super-cluster support (consumer half)** — `8ceb041c43 … 5885c57b3a`. > 2. **Synthetics OSS migration** — `3a380087a3 … 2596300dbf`, nine commits — one per build-order step, plus a ninth for fixes found by running the binary. Moves synthetics out of `o2_enterprise` into `src/synthetics` (`openobserve-synthetics`), leaving only the private-VPC-agent path enterprise: `job_api::lease`, `agent/*`, private locations and the agent-setup install flow. Paired with openobserve/o2-enterprise#2420. > > 3. **Last-check columns read from the results stream** — `6af411e808`, `e7a3f2c759`. Outside the region whose scheduler runs a check, the list showed a stale status and a blank Last check / response time: `last_check_status` replicates only on change, `last_triggered_at` is a runtime column that does not replicate, and `last_response_ms` was never populated anywhere. The `synthetics_results` stream already carries all three and is federated, so the list API reads it instead of replicating the columns — no queue traffic, no schema change. Best-effort: any miss keeps the stored DB values, so search being unavailable cannot take the list down. OSS-only; no enterprise counterpart. > > 4. **Check secrets are readable in every region** — `d47a6b2345` (+ o2-enterprise `da71d7ec`). The per-org DEK is minted by each region for itself, so a check created outside the scheduler region replicated fine and then silently never ran — `resolve` could not decrypt its credentials (o2-enterprise#2451). Secrets now cross the queue **in clear** and the applier re-encrypts them under its own DEK on arrival, fixing the class at the replication boundary with no key to distribute or keep in sync. > > ⚠️ **This puts customer secrets in plaintext on the inter-region bus** — a deliberate, recorded trade-off (`o2oi_` ingest tokens and probe tokens already travel it in clear). Securing the bus itself (TLS on the gateway links, short retention on secret-carrying subjects) is the real fix and is **not** in this PR. > > ⚠️ **Deploy consumers before producers.** The payload's bytes did not change, only the meaning of the secret strings, so an un-upgraded consumer handed the new payload would write customer passwords to disk in clear. The plaintext-carrying messages therefore use new variant names (old ones retired as `CreateEncrypted`/`UpdateEncrypted`), so an old region fails serde and redelivers until upgraded instead of silently persisting plaintext. > > An OSS build can now create, schedule, dispatch to Lambda, take `resolve`/`ack` and report a check. 7 393 lines moved into the new crate; 679 lines stay enterprise. > > **🛑 Deployment prerequisite (2 of 2).** `ZO_SYNTHETICS_ENCRYPTION_KEY` must be set to the **same** base64 64-byte value in every region of a super cluster. Without it synthetics falls back to the per-org DEK and checks carrying secrets silently fail to run outside the scheduler region. It is read once at startup and pinned — changing it later requires a restart, and changing it at all orphans every secret already encrypted under the old value. > > **🛑 Deployment prerequisite (1 of 2).** 14 `O2_SYNTHETICS_*` env vars are hard-renamed to `ZO_SYNTHETICS_*` with **no fallback read**, and `ZO_SYNTHETICS_ENABLED` defaults to `false`. A deployment still setting the old names does not lose tuning — **synthetics stops running entirely, and the first symptom is checks silently not firing.** Update env vars *before* rolling the binary. > > #### Things a reviewer should look at first > > - **`super_cluster_queue` does not depend on `openobserve-synthetics`** (`cargo tree` → 0), which is what keeps the applier's loop-prevention guarantee a property of the crate graph rather than a convention. `a7d5fdd969` adds the assertion that fires if that ever changes. > - **`job_upload` was skipping authorization in OSS builds** (`4c3a0b8f0d`). Unreachable before, because no synthetics route was registered in OSS; registering it would have made it an unauthenticated artifact upload. `authorize_probe` is ungated and the bypass is gone. > - **The dependency edge had to reverse in a single commit** (`3247d4093f`). Cargo rejects an `o2_enterprise` ↔ OSS package cycle at the manifest level — `optional` and feature gating do not help — so the two edges could never coexist, and everything needed for the reversal lands together. This is why the spec's steps 4 and 5 are one commit and the config moved earlier than planned. > > #### Accepted OSS limitations, all documented in the commits > > - No notifications: `notify_check_result` ends in `alerts::alert::dispatch_notification`, which is enterprise. An OSS build runs the check and records the run; it has nowhere to send an alert, exactly as for alerts themselves. > - No per-resource RBAC on synthetics, and no OpenFGA ownership tuples — both are enterprise capabilities across the platform. > - Private locations are refused with a message rather than created. > - Agent-served public `net-*` pools are unavailable, so an OSS user needs a Lambda for every public pool (spec §1). --- Consumer half of synthetics super-cluster support. **Depends on openobserve/o2-enterprise#2420** (producer, message types, queue struct) — and **this side must deploy first**: `MessageType` bytes 83–85 hit a panicking `unreachable!()` in `From<u8>` on a consumer that does not know them, which takes down the subscriber task rather than failing one message. Based on `fix/lease-claimed-by-2270` (#13785), not `main`, so the diff is only this work. ## What this fixes Synthetics writes are not querier routes, so a create lands on ingesters in whichever region received the request and commits to that region's Postgres. There is no synthetics topic on the super-cluster queue, so a check created in a region that runs no scheduler is never seen by the region that does: the UI shows it enabled, the scheduler has never heard of it, and nothing reports the gap. ## Commits | commit | what | | --- | --- | | `8ceb041c43` | orphan-detection query + metrics | | `1c3e10da48` | apply replicated check config | | `79e6abc052` | apply replicated locations and probe tokens | | `16d4fa0fde` | render unknown live status instead of a false pending | | `5885c57b3a` | share the job-cluster predicate | ## Design notes worth reviewing **Config columns only, and the list is not restated.** `synthetics_checks::update` rebuilds its `ActiveModel` from the row already in this region, and `update_mutable_fields` writes only user-editable columns — so `next_run_at` and the alerting counters keep local values for free. Hand-listing them in the consumer would duplicate a list that already exists and let the two drift. A test asserts every runtime column comes back `Unchanged`. **`put()` is deliberately unused.** Its insert branch builds from `build_active_model`, which ends in `..Default::default()` and never assigns `owner`, `tz_offset` or `next_run_at` — a replicated create through it would drop the owner and fire a scheduled-later check immediately. **`create()` gains `use_given_id`**, matching `table::alerts::create`. It generated its own uuid, which would have made every region invent a different primary key for the same check. **Consumers apply through `infra::table` only.** The publish lives one layer up in the enterprise service, and `infra` cannot reach the enterprise crate — that layering is what stops a region re-broadcasting what it just applied. `source_cluster` does not cover this: it only stops a region re-consuming its *own* message. Each consumer greps its own source to keep this true. **Runtime tables are never replicated** — `synthetics_jobs`, `synthetics_runs`, `synthetics_agents`, and the locations `down_notified_at` claim. Asserted at both producer and consumer. ## Known gaps, named not hidden - If two regions mint a token (or location) with the same name for one org before either message crosses, the UNIQUE constraint rejects the arriving row. Local row wins, logged loudly. Erroring would redeliver to dead-letter, which is worse. - Pre-existing and untouched: `write_triggers_stream`/`write_results_stream` in the enterprise reaper post to `_meta` with the *org's* ingest token, which 401s, and neither checks response status. Orphan detection fixes this in its own path only. - Pre-existing: `create` never sets `tz_offset`, so a new check gets DB default 0 until its first edit. Replicates identically everywhere, so it causes no divergence. ## Spec corrections The spec (`docs/synthetics-lcl/2026-08-11-p1-3-super-cluster-spec.md`, local notes) is wrong in five places, all found by reading the tree: §1's invariant evidence (see the enterprise PR), §10's never-run guard, §4's column list (and it lists six runtime columns — there are seven, `alerting` is missing), §3's "private rows read-only when remote", and §7's claim that non-scheduler regions render OFFLINE (they render `pending`). ## Verification infra 42, super_cluster_queue 32, jobs 21, web 111/111 vitest + type-check + design lint, `cargo fmt --check` and clippy clean, full `--features enterprise` build. Pre-existing failures on this branch, unrelated and untouched: `super_cluster_queue::schemas::tests::test_parse_key_{valid,metrics}`, `infra::table::ratelimit::tests::test_add_batch_missing_rule_id`, and 8 web vitest failures in `buildPayload.spec.ts`/`payloadContract.spec.ts`. --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-08-20 05:40:34 +00:00
ktx-akshay	78998b5fa3	test(logs): E2E cover for Build tab stream list with no stream selected (#13884) ... Adds E2E regression cover for the fix in #13812. The Logs → Build toggle mounts PanelFieldList with pageKey "build" in edit mode, which deferred its stream-list fetch until a stream was already set. Since ZO_QUERY_ON_STREAM_SELECTION defaults to true, the logs page selects nothing, so Build mounted blank and its Stream dropdown was left enabled, openable and permanently empty on "No options found" — no way to pick a stream, no way out. ## Tests (Logs-SelectStar shard) - **P0** — with no stream selected on the logs page, the Build tab's Stream dropdown is populated and a stream can actually be picked: no empty state, options render, `e2e_automate` is offered, selecting it lands, and the field list loads that stream's schema. - **P1** — changing Stream Type with a blank stream refetches the list: logs → traces drops `e2e_automate` (a logs stream) and loads a real traces list, switching back restores it. Covers the stream_type watcher, which is gated on the same `initialStreamsLoaded` flag the fix arms. `beforeEach` asserts the logs page starts with no stream selected — without that precondition both tests would pass against the buggy build. ## Notes - Page-object locators are scoped to `[data-test="logs-build-query-page"]`: the Visualize tab mounts a second PanelEditor that stays in the DOM under `v-show`, so an unscoped `index-dropdown-stream` is a strict-mode violation. - Verified 6/6 green over 3 repeats locally. The shard runs the backend with ZO_QUICK_MODE_ENABLED=true, which the local instance does not — assertions avoid named fields so quick mode's field narrowing can't affect them.	2026-08-20 05:38:27 +00:00