mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-05 16:10:43 +00:00
fac24d59bb15fd3932b6bc1fd19b54b0e8a9f22e
87
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  
|
0f4b606e8e |
feat(synthetics): super-cluster support, then move synthetics to OSS (#13795)
> ### ⚠️ This PR carries four changes — review alongside openobserve/o2-enterprise#2420 > > 1. **Super-cluster support (consumer half)** — `8ceb041c43 … 5885c57b3a`. > 2. **Synthetics OSS migration** — `3a380087a3 … 2596300dbf`, nine commits — one per build-order step, plus a ninth for fixes found by running the binary. Moves synthetics out of `o2_enterprise` into `src/synthetics` (`openobserve-synthetics`), leaving only the private-VPC-agent path enterprise: `job_api::lease`, `agent/*`, private locations and the agent-setup install flow. Paired with openobserve/o2-enterprise#2420. > > 3. **Last-check columns read from the results stream** — `6af411e808`, `e7a3f2c759`. Outside the region whose scheduler runs a check, the list showed a stale status and a blank Last check / response time: `last_check_status` replicates only on change, `last_triggered_at` is a runtime column that does not replicate, and `last_response_ms` was never populated anywhere. The `synthetics_results` stream already carries all three and is federated, so the list API reads it instead of replicating the columns — no queue traffic, no schema change. Best-effort: any miss keeps the stored DB values, so search being unavailable cannot take the list down. OSS-only; no enterprise counterpart. > > 4. **Check secrets are readable in every region** — `d47a6b2345` (+ o2-enterprise `da71d7ec`). The per-org DEK is minted by each region for itself, so a check created outside the scheduler region replicated fine and then silently never ran — `resolve` could not decrypt its credentials (o2-enterprise#2451). Secrets now cross the queue **in clear** and the applier re-encrypts them under its own DEK on arrival, fixing the class at the replication boundary with no key to distribute or keep in sync. > > ⚠️ **This puts customer secrets in plaintext on the inter-region bus** — a deliberate, recorded trade-off (`o2oi_` ingest tokens and probe tokens already travel it in clear). Securing the bus itself (TLS on the gateway links, short retention on secret-carrying subjects) is the real fix and is **not** in this PR. > > ⚠️ **Deploy consumers before producers.** The payload's bytes did not change, only the meaning of the secret strings, so an un-upgraded consumer handed the new payload would write customer passwords to disk in clear. The plaintext-carrying messages therefore use new variant names (old ones retired as `CreateEncrypted`/`UpdateEncrypted`), so an old region fails serde and redelivers until upgraded instead of silently persisting plaintext. > > An OSS build can now create, schedule, dispatch to Lambda, take `resolve`/`ack` and report a check. 7 393 lines moved into the new crate; 679 lines stay enterprise. > > **🛑 Deployment prerequisite (2 of 2).** `ZO_SYNTHETICS_ENCRYPTION_KEY` must be set to the **same** base64 64-byte value in every region of a super cluster. Without it synthetics falls back to the per-org DEK and checks carrying secrets silently fail to run outside the scheduler region. It is read once at startup and pinned — changing it later requires a restart, and changing it at all orphans every secret already encrypted under the old value. > > **🛑 Deployment prerequisite (1 of 2).** 14 `O2_SYNTHETICS_*` env vars are hard-renamed to `ZO_SYNTHETICS_*` with **no fallback read**, and `ZO_SYNTHETICS_ENABLED` defaults to `false`. A deployment still setting the old names does not lose tuning — **synthetics stops running entirely, and the first symptom is checks silently not firing.** Update env vars *before* rolling the binary. > > #### Things a reviewer should look at first > > - **`super_cluster_queue` does not depend on `openobserve-synthetics`** (`cargo tree` → 0), which is what keeps the applier's loop-prevention guarantee a property of the crate graph rather than a convention. `a7d5fdd969` adds the assertion that fires if that ever changes. > - **`job_upload` was skipping authorization in OSS builds** (`4c3a0b8f0d`). Unreachable before, because no synthetics route was registered in OSS; registering it would have made it an unauthenticated artifact upload. `authorize_probe` is ungated and the bypass is gone. > - **The dependency edge had to reverse in a single commit** (`3247d4093f`). Cargo rejects an `o2_enterprise` ↔ OSS package cycle at the manifest level — `optional` and feature gating do not help — so the two edges could never coexist, and everything needed for the reversal lands together. This is why the spec's steps 4 and 5 are one commit and the config moved earlier than planned. > > #### Accepted OSS limitations, all documented in the commits > > - No notifications: `notify_check_result` ends in `alerts::alert::dispatch_notification`, which is enterprise. An OSS build runs the check and records the run; it has nowhere to send an alert, exactly as for alerts themselves. > - No per-resource RBAC on synthetics, and no OpenFGA ownership tuples — both are enterprise capabilities across the platform. > - Private locations are refused with a message rather than created. > - Agent-served public `net-*` pools are unavailable, so an OSS user needs a Lambda for every public pool (spec §1). --- Consumer half of synthetics super-cluster support. **Depends on openobserve/o2-enterprise#2420** (producer, message types, queue struct) — and **this side must deploy first**: `MessageType` bytes 83–85 hit a panicking `unreachable!()` in `From<u8>` on a consumer that does not know them, which takes down the subscriber task rather than failing one message. Based on `fix/lease-claimed-by-2270` (#13785), not `main`, so the diff is only this work. ## What this fixes Synthetics writes are not querier routes, so a create lands on ingesters in whichever region received the request and commits to that region's Postgres. There is no synthetics topic on the super-cluster queue, so a check created in a region that runs no scheduler is never seen by the region that does: the UI shows it enabled, the scheduler has never heard of it, and nothing reports the gap. ## Commits | commit | what | | --- | --- | | `8ceb041c43` | orphan-detection query + metrics | | `1c3e10da48` | apply replicated check config | | `79e6abc052` | apply replicated locations and probe tokens | | `16d4fa0fde` | render unknown live status instead of a false pending | | `5885c57b3a` | share the job-cluster predicate | ## Design notes worth reviewing **Config columns only, and the list is not restated.** `synthetics_checks::update` rebuilds its `ActiveModel` from the row already in this region, and `update_mutable_fields` writes only user-editable columns — so `next_run_at` and the alerting counters keep local values for free. Hand-listing them in the consumer would duplicate a list that already exists and let the two drift. A test asserts every runtime column comes back `Unchanged`. **`put()` is deliberately unused.** Its insert branch builds from `build_active_model`, which ends in `..Default::default()` and never assigns `owner`, `tz_offset` or `next_run_at` — a replicated create through it would drop the owner and fire a scheduled-later check immediately. **`create()` gains `use_given_id`**, matching `table::alerts::create`. It generated its own uuid, which would have made every region invent a different primary key for the same check. **Consumers apply through `infra::table` only.** The publish lives one layer up in the enterprise service, and `infra` cannot reach the enterprise crate — that layering is what stops a region re-broadcasting what it just applied. `source_cluster` does not cover this: it only stops a region re-consuming its *own* message. Each consumer greps its own source to keep this true. **Runtime tables are never replicated** — `synthetics_jobs`, `synthetics_runs`, `synthetics_agents`, and the locations `down_notified_at` claim. Asserted at both producer and consumer. ## Known gaps, named not hidden - If two regions mint a token (or location) with the same name for one org before either message crosses, the UNIQUE constraint rejects the arriving row. Local row wins, logged loudly. Erroring would redeliver to dead-letter, which is worse. - Pre-existing and untouched: `write_triggers_stream`/`write_results_stream` in the enterprise reaper post to `_meta` with the *org's* ingest token, which 401s, and neither checks response status. Orphan detection fixes this in its own path only. - Pre-existing: `create` never sets `tz_offset`, so a new check gets DB default 0 until its first edit. Replicates identically everywhere, so it causes no divergence. ## Spec corrections The spec (`docs/synthetics-lcl/2026-08-11-p1-3-super-cluster-spec.md`, local notes) is wrong in five places, all found by reading the tree: §1's invariant evidence (see the enterprise PR), §10's never-run guard, §4's column list (and it lists six runtime columns — there are seven, `alerting` is missing), §3's "private rows read-only when remote", and §7's claim that non-scheduler regions render OFFLINE (they render `pending`). ## Verification infra 42, super_cluster_queue 32, jobs 21, web 111/111 vitest + type-check + design lint, `cargo fmt --check` and clippy clean, full `--features enterprise` build. Pre-existing failures on this branch, unrelated and untouched: `super_cluster_queue::schemas::tests::test_parse_key_{valid,metrics}`, `infra::table::ratelimit::tests::test_add_batch_missing_rule_id`, and 8 web vitest failures in `buildPayload.spec.ts`/`payloadContract.spec.ts`. --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
 udev
|
84b666ce0b |
fix: annotation_queues item hydration intermittent 500s (#13881)
## Summary
RCA (2026-08-19, 08:20-08:22 UTC): `GET
/api/{org}/annotation_queues/{queue_id}/items/{item_id}` (and its
`/reviews` sibling) intermittently returned 500. Originally attributed
to search work-group admission capacity exhaustion; tracing the actual
code found a different root cause.
**Real mechanism**: `eval_jobs::search::execute()`'s registered
executors (in `src/jobs/src/job/mod.rs`) called
`search_service::search("", ...)` with a hardcoded empty `trace_id`.
With no propagated OTel span in that detached task,
`search_service::search()` falls back to OpenTelemetry's *invalid* trace
ID for every such call - the same `000...000` seen in every failing log
line. Every concurrent internal search therefore shared one
`QueryManager` map key: one query's cleanup (`SEARCH_SERVER.remove`)
deleted another's still in-flight registration, so the next
`insert_sender` failed with "trace_id not found" -> `SearchCancelQuery`
-> 500. `work_group_checking` never reached the real concurrency-wait
branch in this failure mode - "work-group capacity" was never the actual
mechanism.
## Changes (3 commits)
1. **`fix(annotation-queues): merge item detail and reviews into one
call`** - the Queue Workbench always fetches item detail + reviews
together (confirmed: no caller ever needs one without the other,
including after review submission); fold them into one backend call,
cutting the per-item load from 2 HTTP round trips to 1. See companion PR
openobserve/o2-enterprise#2473 for the response-shape change this
depends on.
2. **`fix(annotation-queues): map search admission cancellation to
429`** - `annotation_queue_error_response` unconditionally 500'd on
`AnnotationQueueError::Search`/`Hydration`, even when the underlying
cause already has a defined 429 `http_status()` (`SearchCancelQuery` /
`RatelimitExceeded`). Downcast and map correctly instead.
3. **`fix(search): stop colliding on trace_id in internal eval
searches`** - the actual root-cause fix: generate a real per-call
`trace_id` via `config::ider::generate_trace_id()` at all three internal
`search_service::search("", ...)` call sites instead of passing `""`.
## Dependency
This PR's enterprise build depends on openobserve/o2-enterprise#2473
(same branch name `ud/fix-annotation-queue-hydration-500s`) via the path
dependency in `Cargo.toml.openobserve` - that PR adds the `reviews`
field on `AnnotationQueueItemDetail` this PR's merge (commit 1)
consumes.
## Test plan
- [x] `cargo build --features enterprise`
- [x] `cargo test -p openobserve-api-management --lib --features
enterprise annotation_queue` - 11 passed (incl. 2 new tests for the 429
mapping), no regressions
- [x] `cargo test -p o2_enterprise --lib annotation_queues` (companion
repo) - 16 passed, no regressions
- [x] `cargo clippy --features enterprise -p openobserve-jobs -p
search_service -p openobserve-api-management --no-deps` - clean (no
warnings in touched files)
- [x] `cargo fmt --check` - clean
- [x] Frontend: `npx vitest run` on `llm-queues.service.spec.ts` +
`QueueWorkbenchPage.backTarget.spec.ts` - 9 passed
- [x] `vue-tsc --noEmit` (`type-check:app`) - clean
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
  
|
71ac1071be |
fix: draft workflows (#13666)
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yashodhan@openobserve.ai> |
||
  
|
69bd367d9b |
feat: notification dependency view — where-used for templates, destinations & alerts (#13820)
## What Adds a **"View dependencies"** affordance to the **Templates**, **Notification Destinations**, and **Alerts** lists. A small `graph-1` icon on each row opens a compact popover that shows everything that entity connects to along the **Template → Destination → Alert** chain, with search, pagination, and per-row **Open** / **Delete**. ## Why The linkage between alert templates, destinations, and alerts is invisible today. There is **no reverse-lookup API** — the only way to learn "which alerts use this destination" is to try deleting it and hit the backend's `409 "Destination is currently used by alert: X"`. That's a dead end, not a map. This surfaces the dependency proactively so you can see the blast-radius *before* you change or delete something, and act on it in place (open or delete a related entity, with the same `409` guard still protecting in-use items). ## Design The three entities are linked **by name** (there are no ID foreign keys): - `Alert.destinations: string[]` → `Destination.name` - `Destination.template` → `Template.name` (a destination's default) - `Alert.template` → `Template.name` (an alert-level override) ```mermaid flowchart LR T["Template"] -->|"Destination.template"| D["Destination"] D -->|"Alert.destinations[]"| A["Alert"] T -. "Alert.template (override)" .-> A ``` Because the alerts **list** DTO previously omitted `destinations`, the whole graph would have needed N per-alert detail fetches. Instead a small **opt-in additive field** on the list item makes the entire graph **three list calls**, cross-referenced client-side and cached per-org: ```mermaid flowchart TD Row["List row · graph-1 icon"] --> Pop["DependencyChainPopover<br/>(OPopover + ConfirmDialog)"] Pop --> Panel["DependencyUsagePanel<br/>search · collapsible sections · pager"] Panel --> Rows["DependencyUsageRow ×N<br/>Open · Delete"] Panel --> Load["useDependencyGraph.loadGraph()<br/>per-org cache"] Load -->|"alerts?include_dependencies=true<br/>+ destinations + templates (3 list calls)"| Build["buildGraph() → nodes + edges"] Panel --> Focus["buildFocusChain(graph, focus)<br/>directional: a shared template never<br/>drags in sibling destinations"] Rows -. "requestDelete(node)" .-> Pop Rows -. "open(node) — same-kind, in-place" .-> Pop Pop -. "ConfirmDialog → service.delete()<br/>(409 'used by' → toast)" .-> Done["emit deleted → list refreshes"] ``` Why the ConfirmDialog lives in the popover **wrapper**, not the panel: a modal can't render inside the popover without dismissing it, so the row/panel only *request* a delete and the wrapper owns the real confirmation. **Open** works the same way for the in-place editors: templates/destinations edit on their own list route (which only opens from the page's mount-time handler), so a same-page open is bubbled to the host page to invoke its native editor rather than a no-op `router.push` to the same route. ## UX / behaviour - **Where-used, grouped:** collapsible **Templates · Destinations · Alerts** sections, each a clickable row (kind-coloured icon, name, and for destinations an "N alerts" **total-usage** badge). - **Search-first:** one search box filters all groups — the key affordance when a destination feeds thousands of alerts. - **Paged:** the alerts group renders **50 at a time** with a pager pinned beside its (sticky) header, so 4,000-alert destinations stay responsive. - **Act in place:** every row has **Open** (templates/destinations → their editor; alerts → the alert's detail page) and **Delete** (the same `ConfirmDialog` the list pages use; the backend `409` still blocks in-use items). - Consistent trigger on all three lists; **purely additive** — no existing button, icon, or column was moved or replaced. ## Backend change `ListAlertsResponseBodyItem` (`src/api/management/src/models/alerts/responses.rs`) gains two additive fields — the linkage the dependency view cross-references: - `destinations: Vec<String>` - `template: Option<String>` (the alert-level override) They serialize **only when the caller passes `?include_dependencies=true`** (which just the dependency popover does); the default paginated alerts-list path is unchanged, so ordinary consumers pay no extra bytes and receive no module-scoped destination/template names. Both are also `skip_serializing_if`-empty. Backward-compatible, OSS, no new endpoint. ## Testing - **Unit (vitest) — 17 tests:** - `useDependencyGraph.spec.ts` — `buildGraph` (linkage, unused/orphan, dangling refs, override edges, shared destinations) and directional `buildFocusChain` per focus kind (incl. the alert-focus total-usage count). - `DependencyUsagePanel.spec.ts` — sections render every kind, search + 50/page paging, collapse, `requestDelete`, the same-kind `open` bubble, and the `include_dependencies` opt-in. - `DependencyChainPopover.spec.ts` — trigger + lazy panel mount, `requestDelete` → ConfirmDialog → API + `deleted`, and the `open` bubble + close. - **Backend:** serialization test asserts `destinations`/`template` are omitted when empty. - `type-check`, `lint:design:strict`, and `prettier` all clean. - A popover e2e (Playwright) is a follow-up in a separate PR — the earlier spec targeted an interim dedicated page that this PR does not ship, so it was removed. ## Notes - Net change is popover-only: an earlier iteration added a dedicated Reliability page + route + rail entry; those were dropped in favour of the on-the-fly popover, so `router.ts` / `navGroups.ts` are untouched vs `main`. - Icon `graph-1` was added to the icon registry (additive) for the popover trigger. - **Deferred (see design doc Non-goals):** a server-side where-used / `?destinations=` endpoint so the popover fetches only the neighbourhood instead of the full alert list; and enterprise RBAC name-filtering on the gated fields. Design at: https://github.com/openobserve/designs/blob/main/alerts/notification-dependencies/design.md (designs#8: `alerts/notification-dependencies/design.md`) --------- Co-authored-by: Abhay Padamani <141122205+ktx-abhay@users.noreply.github.com> Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech> |
||
 Hengfei Yang
|
9d76505882 |
feat(traces): resolve session time ranges through the trace time index (#13837)
Design at: [https://github.com/openobserve/designs/blob/main/infrastructure/file-list/session-time-index.md](https://github.com/openobserve/designs/blob/main/infrastructure/file-list/session-time-index.md) ## Summary - carry an optional `session_id` attribute on trace time index records, resolved from span attributes by the existing `SESSION_ID_COLUMNS` precedence; the first non-empty value in the request wins for the whole trace, stamped onto all of its hour-bucket records - keep the aggregation key `(hour, trace_id)` unchanged; hourly compaction picks the attribute up with `MAX(session_id)`, which ignores NULLs - reuse the locate/expand timer for sessions with a 7-day expand window (traces keep 24h); a UUID v7 session id doubles as a probe anchor and tightens the scan's left bound to the session's lifetime neighborhood - aggregate session lookups per trace, so a lookup resolves the session's range plus every member trace's real range - reshape the time-range endpoint to query parameters so one endpoint serves both lookup kinds - make the session-details caller range optional, union it with the indexed range (never narrowed), take trace ids from the index, and keep the legacy source-stream scan for pre-coverage data - add `session_id` to the default bloom filter fields (gated by the existing `ZO_FEATURE_DEFAULT_INDEX_FIELDS_ENABLED`, folds into the secondary-index field set like `trace_id`); streams without the column build nothing ## Why Session detail lookup depends on a caller-provided time range: the sessions list passes its own picker window into the details request, so an LLM session that started before the window silently loses its earlier turns, and a deep link without a range is a hard 400. The trace time index already turns a `trace_id` into a time range at bounded cost; a session is the same problem one level up — `session_id` → range plus member traces. ## Environment variables No new variables. The existing `ZO_TRACE_TIME_INDEX_ENABLED` (default `true`) gates the feature, session column included. The existing coverage marker is reused; there is no historical backfill. This change must ship in the same release as #13827: reusing the coverage marker is valid only while no release separates the two. ## API changes All timestamp parameters and returned ranges use microseconds. ### Time-range endpoint reshaped ```http GET /api/{org_id}/{stream_name}/traces/time_range?trace_id=xx&hint_ts=... GET /api/{org_id}/{stream_name}/traces/time_range?session_id=xx&hint_ts=... ``` Exactly one of `trace_id` / `session_id` is required; zero or both is `400`. This replaces `GET /{stream_name}/traces/{trace_id}/time_range`; the old path form is removed without a compatibility alias since #13827 has not shipped in a release. ### Session details: range becomes optional `GET /api/{org_id}/{stream_name}/traces/session/details` now accepts `start_time` / `end_time` as an optional pair plus an optional `hint_ts` (defaulting to the caller range's midpoint). The effective range is the union of the caller range and the indexed session range, so it never narrows what the caller asked for; `400` only when neither side yields a range. When the caller range lies entirely inside index coverage the source-stream trace-id scan is skipped; otherwise it still runs over the caller range and its results are merged, so behavior is never worse than before. No frontend changes are required — the frontend keeps sending its list window and the union completes the session. ## Validation - `cargo test -p openobserve-core traces::time_index --lib` — 13 passed - `cargo test -p search datafusion::merge --lib` — 5 passed, including NULL-mixed session aggregation and pre-column file compatibility - `cargo test -p openobserve-api-search --lib` — 149 passed - `cargo test -p openobserve-api-http --lib` — 53 passed, including OpenAPI/MCP initialization - `cargo test -p config router:: --lib` — 8 passed - `cargo test -p infra db::trace_time_index --lib` — passed - `cargo clippy` on all touched crates (`--all-targets`) and `cargo check --workspace` — clean - the enterprise repository has no references to any changed API and no enterprise-gated code was modified |
||
|
udev
|
35860aff9d |
fix: route annotation queue score lookups to queriers (#13801)
## Summary - route Discovery and annotation queue endpoints to queriers so score lookups, item hydration, and review reads do not fall through to ingesters - make Discovery source_stream required at the API response boundary - cover Discovery, queue item detail, and review routing with regression tests The QueueItem remains reference-only. Workbench and enqueue validation derive source_stream from timestamp-bounded _llm_scores queries in the enterprise implementation. Depends on https://github.com/openobserve/o2-enterprise/pull/2423 ## Testing - 5 querier routing tests - Discovery response serialization test with the enterprise feature - 16 enterprise annotation queue tests - 14 enterprise Discovery tests |
||
|
Hengfei Yang
|
41ff3cd9b6 |
feat: replace distinct_values_* streams with tantivy TopK for the _values API (#13834)
Design at: #13822 ## Summary `distinct_values_*` metadata streams existed to serve one reader: the `_values` API. They are a redundant — and lossier — materialization of what the tantivy secondary index already holds, and the index has the collectors needed to replace them outright (`SimpleTopN` / `SimpleDistinct`). This PR removes the distinct streams from the write and read paths and lets the `_values` API be served by the tantivy index optimizer. ### 1. Treat distinct value fields as secondary index fields (read-time union) - `get_stream_setting_index_fields()` — the single accessor used by both the index build side (ingester parquet/pack jobs, compactor merge) and the query side (leader/follower index optimizers) — now unions `distinct_value_fields` into the effective index field set, gated on `enable_distinct_fields`. Fields that cannot be secondary-indexed (full text search keys, partition keys, reserved columns) are skipped; the values API falls back to a normal scan for those. - The distinct list is deliberately **not merged into `settings.index_fields`**: the two stay independent, so removing a distinct value field (or disabling `enable_distinct_fields`) cleanly stops its indexing with no residue, and the user-managed `index_fields` list is never mutated behind their back. - Correctness gate: a distinct-only field is considered indexed since its `added_ts` (`get_stream_setting_index_updated_at_for_fields`), so parquet files older than that are served by DataFusion scan instead of the index (`split_file_list_by_time_range`). `added_ts` is set on add and reset when `enable_distinct_fields` is re-enabled (existing behavior). For streams configured before this release the fields only start being indexed at upgrade time, so `added_ts` is additionally clamped by a release-wide cutoff `/tantivy/distinct_fields/updated_at` — a lazy marker initialized to the first access after upgrade, following the existing `ttv_secondary_index_updated_at` pattern. No data migration and no `DB_SCHEMA_VERSION` bump needed. ### 2. `_values` API always queries the original stream - Removed `can_use_distinct_stream()` and the `distinct_values_*` / `SUM(count)` / `StreamType::Metadata` rewrite from `values_inner` (GET `/_values`, formerly `values_v1`) and `build_search_request_per_field` (POST `/_values_stream`). - The generated SQL shapes are exactly what the index optimizer detects: - `SELECT f AS zo_sql_key, COUNT(*) AS zo_sql_num FROM s GROUP BY zo_sql_key ORDER BY zo_sql_num DESC, zo_sql_key ASC` + limit → `SimpleTopN` - `no_count`: `SELECT f AS zo_sql_key FROM s GROUP BY zo_sql_key ORDER BY zo_sql_key ASC` + limit → `SimpleDistinct` - The `keyword` filter (`str_match_ignore_case`) is a valid index condition, so keyword-filtered values queries stay on the index path too. ### 3. Remove distinct values generation - Dropped the per-record distinct value collection in logs and traces ingestion and the whole `core::metadata` module (channel, mem-table, flush job, shutdown hook). - Removed the now-unused `DISTINCT_FIELDS` static and the `ZO_FEATURE_DISTINCT_EXTRA_FIELDS` / `ZO_DISTINCT_VALUES_INTERVAL` / `ZO_DISTINCT_VALUES_HOURLY` config knobs, including the `SUM(count)` merge rollup the last one gated — legacy `distinct_values_*` streams now compact as ordinary metadata streams. The default fields `DISTINCT_FIELDS` carried (`service_name`, `operation_name`) are already default secondary index fields via `SQL_SECONDARY_INDEX_SEARCH_FIELDS`, so traces `_values` queries on them remain index-served. - Renamed `values_v1` to `values_inner`: the `v2` variant was removed long ago (#5087) and the distinct stream switch inside `v1` is gone now too, so the version suffix was pure history. ### 4. Default fields after this change - `service_name`, `operation_name`: default secondary index fields (unchanged, via `SQL_SECONDARY_INDEX_SEARCH_FIELDS`). - `trace_id`: now also a built-in default **bloom filter** field (`BLOOM_FILTER_DEFAULT_FIELDS`, gated on `ZO_FEATURE_DEFAULT_INDEX_FIELDS_ENABLED`), in addition to being a default index field — bloom sits on top of the secondary index and prunes files for high-cardinality equality lookups. - There is no "default distinct fields" concept anymore. ### 5. Fix surfaced by e2e: SimpleDistinct returned null values as `""` The tantivy index encodes null/missing field values as empty strings (`index_builder` adds `""` for null docs), so the `SimpleDistinct` term-dictionary walk returned `""` as a distinct value — sorted first in ascending order, it surfaced as a blank first option in dashboard variable dropdowns, which the DataFusion `GROUP BY` path never returns (its null group sorts last). This was exposed once distinct fields became index-served; `handle_simple_distinct` now skips empty terms (with a unit test). ### 6. Drive-by: fix the failing `api_fuzz_tests` CI job `CreateAlertRequestSchema` was an untagged documentation union with two identical variants, producing `oneOf: [CreateAlertRequestBody, CreateAlertRequestBody]` in the OpenAPI spec. Under `oneOf` semantics (exactly one branch must match) no instance can ever validate, so schemathesis filtered out every generated example and `api_fuzz_tests` failed on `POST /api/v2/{org_id}/alerts` on every CI run since that schema landed. The handler deserializes `CreateAlertRequestBody` directly, so the annotation now documents exactly that type; runtime behavior is unchanged. ## What stays (intentionally) - `distinct_value_fields` / `enable_distinct_fields` in stream settings: still the user-facing way to declare values-API fields; the list now drives the read-time index-field union described above. - The `distinct_value_fields` DB table that tracks dashboard/report usage of these fields (still gates removal in the settings API). - `DISTINCT_STREAM_PREFIX` helpers, retention sync and stream-deletion cleanup for existing `distinct_values_*` streams — legacy stream data keeps compacting (as ordinary metadata streams) and ages out via retention; no new records are written to them. ## Testing - `cargo check --workspace --all-targets` and `cargo clippy --workspace --all-targets` clean. - Enterprise build verified with `Cargo.toml.openobserve` (`cargo check --all-targets`). - New unit tests for the index-field union (skip rules for FTS/partition/reserved fields, `enable_distinct_fields` gating) and the `added_ts` / release-cutoff indexed-since resolution; schema, infra, config and api-search unit tests pass. A few pre-existing failures on a clean tree (config `ssrf_guard`, one flaky sqlite-backed destinations test, one `.env`-dependent index-fields test) are local-environment issues and unrelated. Closes #13822 |
||
 Harsh Mahajan
|
7f74925605 | feat: banner as notification (#13667) | ||
 Prabhat Sharmaandgithub-actions[bot]
|
0dba67d638 |
feat: composite alerts (#13797)
Design at : alerts/composite_alerts/composite_alerts.md ## Summary - add composite alert graph persistence, evaluation, scheduling, lifecycle APIs, and deletion safeguards - add composite alert UI forms, preview, references, list/detail integration, and localization - add Rust, API, Vue, and Playwright contracts ## Verification - cargo fmt --all -- --check - cargo check -p openobserve-core - cargo check -p openobserve-api-management - npm run type-check:composite-alerts - npm run type-check:app - npm run lint:ci -- --quiet - design/token/style checks - targeted Rust/Vitest suites - browser QA against local alerts UI ## Follow-ups (intentionally deferred, documented in the spec) - `RunOutcome::Error`/`NotifyFailed` outcome records on composite delivery failure (§11.5) - Incident detail live-definition resolver for composites + "definition unavailable" retention (§9.6) - `alert_composite_*` observability metrics (§16) - SLO/stream delete-conflict 409 bodies with readable-vs-hidden parent split (§10.3) - Super-cluster startup preflight (§18) <img width="1650" height="979" alt="image" src="https://github.com/user-attachments/assets/3def5246-7bf7-48fe-9a42-c0ac08271ea9" /> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Hengfei Yang
|
3fda083f51 |
feat(traces): add trace time index (#13827)
Design at: [https://github.com/openobserve/designs/blob/main/infrastructure/file-list/trace-time-index.md](https://github.com/openobserve/designs/blob/main/infrastructure/file-list/trace-time-index.md) ## Summary - add an hourly Metadata time index per trace stream to resolve a trace ID to its full time range - add external time-range and trace-details APIs, including bounded locate/expand behavior and standard search responses - integrate index generation into trace ingestion and preserve query visibility through the synchronous Metadata write path - synchronize retention and deletion for associated Metadata streams - reuse the shared trace stream permission check across trace APIs ## Why Trace detail lookup currently depends on a caller-provided time range, which can omit spans outside that window. The time index provides a bounded, best-effort range lookup for `stream + trace_id` without scanning the full trace stream. Closes #13823 ## Environment variables | Variable | Default | Effect | | --- | --- | --- | | `ZO_TRACE_TIME_INDEX_ENABLED` | `true` | Enables per-trace-stream Metadata indexes, index writes during trace ingestion, and indexed time-range lookup. Only traces ingested after the feature is enabled are indexed; there is no historical backfill. | | `ZO_QUERY_TIMEOUT` | 600 seconds | Existing setting reused as the cumulative Trace Timer locate/expand deadline and as the default timeout for the trace-details search. | When enabled, each traces stream can create a related Metadata stream named `trace_time_index_{stream}`. Its retention settings follow the source stream, and it is removed with the source stream. ## API changes All timestamp parameters and returned time ranges use microseconds. ### Get a trace time range ```http GET /api/{org_id}/{stream_name}/traces/{trace_id}/time_range ``` Query parameters: - `hint_ts` (optional): timestamp hint used to start the index lookup Returns the source stream, trace ID, and the resolved `start_time` / `end_time`. The `range` field is `null` when the index cannot resolve the trace. ### Get trace details ```http GET /api/{org_id}/{stream_name}/traces/{trace_id}/details ``` Query parameters: - `start_time` and `end_time` (optional, but must be supplied together): caller-provided range - `hint_ts` (optional): timestamp hint for the time-index lookup - `timeout` (optional): trace-details search timeout in seconds The endpoint unions the caller range with the indexed range, so it never narrows a caller-provided range. If neither the caller range nor the index can provide a range, it returns `400`. The response uses the standard search response and includes the effective range in `new_start_time` and `new_end_time`. ## Validation - `cargo test -p openobserve-api-search --lib` — 145 passed - `cargo test -p openobserve-core traces::time_index::tests --lib` — 7 passed - `cargo test -p infra db::trace_time_index::tests::test_coverage_start_cache -- --exact --nocapture` — passed - `cargo check -p openobserve-api-search` - `cd web && npm run type-check:app` - end-to-end ingestion and query validation with the data generator ## Validation notes - targeted ESLint could not start because the local web dependencies are missing `@eslint/css` - enterprise-only compilation could not be completed from this OSS checkout because the local enterprise overlay is unavailable |
||
|
|
dbee572722 |
fix: slo bugs (#13784)
closes https://github.com/openobserve/o2-enterprise/issues/2397 --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Harsh Mahajan
|
0ed5336bf1 | chore: adding HA mode for o2-ai agent (#13525) | ||
 Huaijin
|
3901c548a2 |
perf(promql): use Utf8View for metric labels (#13796)
## Summary
- read PromQL metric label columns as Arrow `Utf8View` when the existing
`ZO_UTF8_VIEW_ENABLED` setting is enabled
- preserve `__hash__` and exemplars in their original representation
while retaining schema metadata
- make PromQL equality, negative, and regex matchers type-aware for
`Utf8View`; cast regex operands to `Utf8` to avoid the DataFusion 54
mixed-type rewrite issue
- support `StringViewArray` in series-label loading and keep
compatibility with the process-wide series-label cache
No PromQL-specific environment variable is introduced, and metrics
continue to use the existing global `ZO_BATCH_SIZE`.
## Performance
For the filtered histogram query below, a controlled local A/B using the
same release binary and identical settings except Utf8View measured P50
latency improving from 0.964s to 0.677s, about 29.8%:
```promql
histogram_quantile(0.9, sum by(le, path) (rate(codelab_api_request_duration_seconds_bucket{path="/api/service-1"}[5m])))
```
Test range was one hour with `step=15` and `use_cache=false`.
## Testing
- `cargo fmt --all -- --check`
- `cargo test -p promql --lib utils::tests` (10 passed)
- `cargo test -p promql --lib series_labels::tests` (11 passed)
- `cargo test -p search --lib
datafusion::exec::tests::test_metrics_query_schema` (2 passed)
- `cargo check -p promql-service -p promql`
|
||
 Ashish Kolhe
|
de0d37e23d |
fix(alerts): stop losing Slack alerts on undeliverable link URLs (#13742) (#13753)
Fixes #13742 Fixes openobserve/o2-enterprise#2394 ## The bug A content template link with a non-http scheme (`javascript:alert(1)`) made Slack reject the **entire** notification with `400 invalid_attachments`. The alert was lost outright and `triggers` recorded `notify_failed`. ## Root cause `safe_url` neutralises a hostile scheme by substituting `BLOCKED_URL` (`"#"`). That is **safe** but not **deliverable** — Slack, Teams, Discord and PagerDuty validate action URLs server-side and reject the whole payload when one is not an absolute `http(s)` URL. A security substitution was turning into total message loss. The two properties are different, and links need both. So the fix splits by medium: | renderer | link handling | |---|---| | Slack buttons, Teams actions, Discord `embed.url`, PagerDuty `href` | `dispatchable_url` → **drop** | | HTML email, email plaintext, SNS plaintext, Discord description | `safe_url` → substitute `#` | | Webhook envelope | unfiltered — versioned machine contract, documented | Dropping beats rewriting: the alert body is worth more than the button, and a button pointing somewhere misleading is worse than no button. `dispatchable_url` requires a **non-empty authority**, not just an allowlisted scheme — `http:javascript:alert(1)` and `https://` are valid *opaque* RFC 3986 http(s) URLs that Slack still rejects. It also rejects raw spaces and control characters, which variable substitution can introduce. ## Related issues fixed in the same area - **Empty `ZO_WEB_URL`** (not in the report): `alert_url` becomes a root-relative `/web/logs?…`, which `safe_url` passes through by design, and it lands in the always-appended "View in OpenObserve" button. Same `invalid_attachments`, on deployments that never authored a hostile template. - **PagerDuty** validates `links[].href` server-side too — same lost-alert class, converted. - **Save-time rejection** (`ContentSpec::validate`): a clear 400 naming the offending link, instead of a template whose link silently never works. Also wired into **preview** and **test-send**, which previously accepted a draft that couldn't be saved and echoed the raw hostile URL back to the UI. - **Slack image-fallback misdiagnosis**: removing the image proves the image was at fault, but only an image pointing at *our* `web_url` justifies tripping the process-wide chart suppression (1h, across every alert) and telling the operator to fix `ZO_WEB_URL`. A custom template's third-party image no longer does either. - **a11y**: `OInput` links its error message to the control via `aria-describedby`; the textarea branch also gained `aria-invalid`. ## o2-enterprise#2394 (fixed here, not in the enterprise repo) Clearing the Monaco body and saving toasted "Please fill required fields" with no indication of *where*. The schema already produced an error on the `body` path — a bare Monaco bridged in via `setFieldValue` just had no `OFormInput` error slot to render it. Now a red border + message driven by the same field state `OFormInput` reads. The aria wiring goes on the **shell**, not on `<QueryEditor>`: `CodeQueryEditor` sets `inheritAttrs: false` and binds `$attrs` to a non-focusable wrapper, so attributes passed to the component would be inert — false assurance for a screen-reader user. > The issue names the title field too. It's an `OFormInput` and already highlights correctly; I added a regression test pinning that rather than changing it. This fix is entirely in OSS (`web/`) since the template form isn't enterprise-gated, so there's no companion enterprise PR. ## Layering note Save-time validation is a **UX gate, not the security boundary**. Link URLs are templates — a scheme that only materialises after `{variable}` substitution can't be judged at save time. The render layer is what actually stops delivery, and it also covers every template saved before this validation existed. Both layers are needed; neither is sufficient. ## Testing - Every fix written test-first (verified RED) and **mutation-tested** — each new test fails when its fix is reverted in substance. - Golden files unchanged: legitimate URLs render byte-identically, so no over-blocking. - Rust: `alerts::notifications` 159 · `config` 3130 · `api-management` 424 — all passing. `cargo clippy --all-targets` and `cargo fmt --check` clean. - Frontend: full suite **41,597 passing** (run because `OInput` is shared library code). `vue-tsc`, `eslint`, and `npm run format:check` clean. - Two independent cold-start review passes; findings from both are folded in, including a P0 where the first version of `dispatchable_url` still let the original bug through. Pre-existing and unrelated: `alerts::destinations::tests::test_alert_destination_template_is_optional` fails locally on a DB connection reset — verified failing on unmodified `main` too. |
||
|
Hengfei Yang
|
238b0d5262 |
fix: support zstd-compressed HTTP requests (#13728)
## Summary - enable tower-http zstd request decompression - add a router middleware regression test using a real zstd-compressed JSON payload - update decompression documentation to list zstd support ## Root cause The Axum router uses `RequestDecompressionLayer`, but the workspace did not enable tower-http's `decompression-zstd` feature. Requests from clients such as Vector configured with `compression: "zstd"` were therefore rejected with `415 Unsupported Media Type` before reaching the ingestion handler. ## User impact Vector HTTP sinks can continue using zstd compression when sending batches to OpenObserve ingestion endpoints. ## Validation - `cargo fmt --all -- --check` - `cargo test -p openobserve-api-http handler::http::router::decompression::tests` - `git diff --check` Fixes #13726 |
||
|
udevandsai nikhil kethe
|
bbd8601c64 |
feat(llm): add annotation queues and datasets (#13719)
## Summary - add annotation queue and dataset storage, management APIs, and review workflows - add discovery, dataset, queue, and review user interfaces - add RBAC synchronization, super-cluster support, tags, version history, and review progress ## Validation - CI should pair this branch with the identically named Enterprise branch for cumulative compilation testing ## Migration note This is an additional cumulative PR created from the committed ud/ebb3 tip. The existing SPR pull requests and branches are intentionally left unchanged. --------- Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> |
||
|
Ashish Kolhe
|
b3e6a77270 |
fix(correlation): service-streams P2 behavioral fixes + discovered services flat table (#13710)
Correlation P2 round: behavioral/security fixes across the service-streams backend and correlation FE, plus the Discovered Services UX rework and the correlation e2e suites. **Companion enterprise PR (must merge together):** openobserve/o2-enterprise#2381 — the enterprise side calls the new 3-arg `put(org, record, max_streams_per_type)` introduced here. ## Commit 1 — behavioral fixes (F2–F38) **Backend** - F25/F26: RBAC on `list_services` (`service_streams` GET — the one previously-unenforced endpoint in the module) + validate `distinguish_by` ids on identity-config save - F2: validate trace filter param before WHERE-clause interpolation - F6: `_reset` invalidates caches on all nodes via reload event; org deletion clears service-streams caches - F8: shared coverage-based identity-set ranking for ingest and read - F10: purge rows of removed/changed identity sets on config save - F19: `put` returns deleted orphan keys for cache eviction - F28: discriminate no-match from failure; stop aliasing 404 to no-match - F32: cap DB-side stream union at `max_streams_per_type` - `all_dimensions` unions on put so late-tracked dimensions surface - F14: `StreamInfo.dropped_dimensions` contract field; allow empty `tracked_alias_ids` when identity sets exist **Frontend** - F35/F36/F31: per-stream filter-edit resolution via semantic groups; dimension edits in both key spaces; subject overrides never guess fields - F27/F30: echo `source_stream`/`source_type`; stop guessing another stream's filters; chip dedup - F2/F38: SQL field/value escaping via shared `quoteSqlLiteral` helper ## Commit 2 — Discovered Services UX + e2e - Flat table (one row per instance), same-name services merge their name cell (OTable `pivotRowColumns`); optional per-service collapse with aggregated summary row; search/filters override collapse - Fixes an OTable pivot-merge bug (value-keyed merge map blanked single-column runs) + regression test - Semantic Field Groups: category tabs with counts + cross-category search with match highlighting - Correlation Phase-1 API + Phase-2 UI Playwright suites ## ⚠️ Release note (enterprise + RBAC deployments) Viewing Discovered Services now requires the **Service Streams** permission (previously unenforced). Root/admin users are unaffected; custom roles may need a one-time grant via the role editor (resource already exists in the OFGA model). ## Checks - `cargo fmt --all --check` ✅ · `cargo clippy --all-targets` ✅ enterprise + OSS feature sets (0 warnings) - prettier ✅ · eslint 0 errors · 207 unit tests on touched specs ✅ - Compiles with paired enterprise branch ✅ |
||
|
Prabhat Sharma
|
087e1498ab |
feat(slos): add SLO burn-rate alerts (#13704)
design at /slo ## What Adds **SLO burn-rate alerts** — an alert family that fires off an SLO's error budget rather than a raw query — and finishes the SLO surface around it: navigation entry, detail-page fixes, and the flag removal now that SLOs have shipped. ## Changes **SLO burn-rate alerts (backend)** - New burn-rate evaluation (`config/src/meta/slo/burn.rs`, `condition.rs`) with multi-window burn conditions, and `core/src/slo/evaluate.rs` extended to evaluate them on the SLO job. - Alerts carry an `slo_id`; `infra/src/table/alerts` gains the column, filtering, and query support so an SLO's alerts can be listed and deleted with it. - Alert levels (`meta/alerts/level.rs`) and tags extended for the SLO family; scheduler handlers route SLO alerts through the same notification path. **SLO alert UI** - `SloAlertForm` + `AlertDetailSlo`: create, edit and view burn-rate alerts from the SLO detail page and the alert list. - Payload/routing helpers (`utils/alerts/sloAlertPayload.ts`, `sloAlertRouting.ts`, `alertPayload.ts`) with unit tests. - `AlertDestinationsField` extracted as the complete Destination form field — side label, tooltip, combined destinations/workflows picker, inline error, refresh, and create-in-new-tab. It replaces two verbatim copies in the generic alert form and is reused by the SLO form and synthetics `CheckAlerts` (which sets `supportsWorkflows: false`, since a check has no workflow routing). Fixes the SLO form's previously unwired refresh and Add Destination buttons. **SLO detail page fixes** - Burndown and burn-rate charts drew as bare axes when coverage was sparse: `showSymbol: false` means an isolated measured bucket between gaps has no segment and paints nothing. Symbols are now sized per point, and a sparse-coverage note states how many buckets actually recorded events. - Nothing scrolled below ~700px of viewport height — `OPageLayout`'s fixed flex body plus `OTabPanels` `scroll="none"` clipped the panels. Panels now grow and scroll; the stat strip and tab bar stay pinned. Verified at 1024x500, 1280x620, 1512x945. **Other** - `ZO_SLO_ENABLED` removed — SLOs have shipped, so the flag and its status/route gating are gone. - SLOs surfaced in the main nav. - Dashboard chart panel header gets its own tint via new `--color-panel-bar-bg` / `--color-panel-bar-border` tokens, and the bar is extracted into `PanelBar.vue` (it existed as five hand-copied class strings that had already drifted). - `tsconfig.vitest.tsbuildinfo` untracked; third-party vendor name removed from comments and test names. ## Testing - New Rust tests across `meta/slo/burn.rs`, `condition.rs`, `core/src/alerts/alert.rs` and `infra/src/table/alerts`. - New frontend suites: `sloAlertPayload.spec.ts`, `sloAlertRouting.spec.ts`, `alertPayload.spec.ts`, `AlertDetailSlo.spec.ts`, `SloDetailAlerts.spec.ts`, `SloListDelete.spec.ts`, `SloTimeSlicePreview.spec.ts`. - `cargo clippy --all-targets` and `web` `lint:ci` clean. |
||
 Pinguladora
|
575e8ea4d5 |
feat: add zstd compression on OTLP gRPC ingest (#13703)
Design at: #13702 / inline in PR description Add the `zstd` feature to tonic and enable `accept_compressed(Zstd)` on the OTLP logs, metrics and trace gRPC services in both `run_common` and run_router`. Closes #13702 ## Out of scope - Response compression on `send_compressed(Zstd)` - OTLP/HTTP path (tower-http zstd features) - snappy would be the better choice under CPU pressure, but tonic has no snappy support. |
||
|
Yashodhan JoshiandYashodhan Joshi
|
24083c018e |
fix: cache invites (#13613)
This PR introduces cache layer for user invites, as well as removes the ratelimit from org list api --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> |
||
|
21dfaf36cf |
fix: handle auth for invite reject correctly (#13566)
API call for rejecting an invite was not handled properly and would log out an user with unauth. This PR fixes that. --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Subhra264 <chakrabortysubhradeep556@gmail.com> |
||
|
Huaijin
|
aaed33543c |
refactor(search): group DataFusion and Tantivy feature configs (#13664)
## Summary - add ZO_FEATURE_METRICS_PUSHDOWN_FILTER_ENABLED as an opt-in filter pushdown switch for metrics queries - group DataFusion and Tantivy feature configuration under the search config struct - move inverted-index result cache sizing, footer caches, skip threshold, TopN group limit, and bloom footer cache settings from limit to search - preserve the moved cache settings' environment variable names and defaults - apply the metrics-specific setting to object storage, WAL, and Flight query contexts ## Testing - cargo fmt --all -- --check - cargo check for config, infra, search, tantivy_utils, search_service, and promql-service - cargo test -p config every_env_config_default_parses --lib - cargo test -p config inverted_index_config --lib - cargo test -p search test_create_session_config --lib - cargo clippy for all affected crates with -D warnings |
||
|
Ashish Kolhe
|
bafc2be4f0 |
fix(correlation): P0/P1 fixes from service-streams correlation review (#13668)
## Summary OSS half of the correlation P0/P1 fix set from the line-by-line review (`docs/ak_review/correlation-line-by-line-review-2026-08-06.md`). Companion enterprise PR: same branch name in o2-enterprise (required for CI manifest pairing). ### Fixes (by review finding) | Finding | Fix | |---|---| | **F14** | `StreamInfo.dropped_dimensions` contract field (`#[serde(default, skip_serializing_if)]`, backward-compatible wire format) + TS type | | **F8** | Shared coverage-based identity-set ranking (`resolve_best_set_by_coverage`) in `config::meta::correlation` — full coverage > ratio > count, declaration-order ties; ingest and read now rank identically | | **F10** | `save_identity_config` purges rows of removed/changed sets (`obsolete_set_ids`) + cache clear + org reload event | | **F6** | `_reset` routes through enterprise `ServiceStorage::delete_all` (cache clear + super-cluster replication) and emits an org-scope reload; super-cluster Delete consumer emits org reload instead of per-service delete | | **F19** | `infra::table::service_streams::put`/`delete_subset_orphans` return deleted orphan disambiguations so callers can evict cache keys | | **F2** | `config::utils::sql::validate_where_fragment` (sqlparser) applied to the `filter` param at both `traces/latest*` interpolation sites — statement smuggling and comment-truncation payloads → 400 | | **F2/F38** | FE: shared `quoteSqlIdentifier`/`quoteSqlLiteral`/`buildSqlCondition` adopted by all six SQL builders (two trace paths were fully unescaped) | | **F35** | `applyFilterOverlay`: filter edits resolve per-stream via semantic groups instead of literal key identity; dead `removeFilter`/`resetFilters` removed; `semanticGroups` threaded to `CorrelatedLogsTable` parents | | **F36** | `applyDimensionEditsToFilters`: dimension-bar edits work in both key spaces (raw-field and semantic-ID), case-insensitive shared field→group map | | **F31** | `mergeSubjectOverrides`: subject overrides replace the same-group key (never union two aliases into contradictory WHERE), schema-less field guessing removed | | F14 (FE) | Warning banner shipped, then removed by reviewer decision (`dropped_dimensions` stays in the API) | ### Verification - `cargo test -p config` correlation/service_streams/sql: 121 tests green; enterprise-feature and OSS-only cfg builds both compile - FE: 263 vitest tests green across the correlation suites; vue-tsc clean - **Live-verified** against a local build (fresh DB): 19/19 API checks (raw-value emission, set filing, obsolete-set purge, reset cache invalidation, orphan eviction, filter validation) plus browser checks (escaped SQL on the wire: `"environment" = 'Prod-EU'`) Deliberately out of scope (per review decision): F15, F17/F37, F9. |
||
 Omkar KesarkhaneandClaude
|
d171d98873 |
fix(synthetics): recorder extension install link, record/replay setup gate, configurable store URL (#13658)
## Summary The OpenObserve Recorder Chrome extension is now published on the Chrome Web Store. This PR wires the browser-test UI to it: - **Install link + setup checklist** — the CreateBrowserTest gate links to the Web Store listing and walks through install → Allow-in-Incognito → connect, with the connect step verified live via the extension probe. The checklist is extracted into a shared `ExtensionSetupChecklist` component. - **Record/replay setup dialog** — `BrowserJourney.vue` now gates both Record and Replay behind an `ExtensionSetupDialog` when the extension is not detected (previously Replay was ungated and would hang in "running" until the 15-min watchdog). - **Loader on Record journey** — the gate's Record button shows a spinner while the extension probe awaits its ~4.5s worst-case timeout. - **Configurable store URL** — the Web Store URL is served by `/config` as `synthetics_recorder_extension_url`, sourced from the new env var `O2_SYNTHETICS_RECORDER_EXTENSION_URL`. The frontend falls back to the built-in constant when the field is absent or empty, so the UI works against older backends and OSS builds. ## Test plan - `ExtensionSetupChecklist.spec.ts` (8), `ExtensionSetupDialog.spec.ts` (11), `BrowserJourney.spec.ts` dialog gating tests — all passing - `npm run lint` / `npm run type-check` clean (3 pre-existing `eslint.config.js` duplicate-key errors untouched) - Manual: record + replay gating verified against the published extension --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Subhra264
|
6cbb62ffa8 | fix: metric stream name rbac migration (#13521) | ||
|
Ashish Kolhe
|
58c7dafa55 |
fix: notification templates v2 — rich content templates + alert chart images (#13640)
## Summary
Two coupled alerting features:
**1. Notification Templates v2 ("content" templates)**
- New `kind: content` template type: a structured spec (title, body
markdown, fields, links, rows, per-channel title overrides,
severity-filtered visibility) rendered natively per channel — existing
`custom` (raw body) templates are untouched and byte-identical on the
send path.
- Per-channel renderers: Slack Block Kit, Teams (Adaptive Card +
MessageCard), Discord embeds, PagerDuty Events v2, Opsgenie, ServiceNow,
SNS, HTML/text email, and a stable machine `webhook` envelope.
- Escaping is renderer-owned (mrkdwn entities, HTML encoding, markdown
stripping) with a shared URL-scheme allowlist; golden-fixture tests per
channel.
- Template editor UI: guided content form (fields/links row tables,
compact rows config, channel title overrides), live preview panel with
per-channel preview + payload tab, test-send (rate-limited,
`[TEST]`-marked).
- Migration `m20260802_000001_add_template_kind`; prebuilt default
template + org backfill.
**2. Alert chart images (stateless, storage-free)**
- Alerts can attach a metric-history chart PNG rendered from the
triggers self-reporting stream (`actual_value` per evaluation), with
warn/crit threshold lines and the firing point appended.
- Zero storage: the chart URL *is* the data — deflate+base64url payload
signed with truncated HMAC-SHA256 (org-bound, expiry embedded), rendered
on fetch by the HTTP-terminating node (`GET
/api/v2/{org}/alerts/charts/render`). Signature verified before inflate
(decompression-bomb guard), uniform 404s, per-node render semaphore +
per-minute query budget + small LRU.
- Renderer: plotters + embedded DejaVu Sans (Bitstream Vera license
included), gap-broken line segments so sparse evaluation history doesn't
draw false plateaus.
- Channel delivery: Slack image block, email inline CID (no remote
fetch), Discord multipart attachment, Teams/PagerDuty image URLs;
per-destination opt-in via template chart flag.
- Slack layout live-A/B'd on a real channel: content + buttons/chart
split across two same-colored attachments so the severity stripe spans
everything while staying under Slack's "Show more" fold; clickable link
title; "View in OpenObserve" keeps a reserved primary-button slot.
- Slack image-delivery fallback: Slack validates image URLs server-side
and rejects the ENTIRE message with `400 invalid_attachments` when its
proxy can't fetch the chart (VPN-only/private `ZO_WEB_URL`) —
live-verified. On that rejection the send strips the image blocks and
resends once (alert always delivers), sets a process-wide flag so later
sends pre-strip (reachability is deployment-wide, not per-destination),
and re-probes hourly. Zero config; observable via `slack_image_rejected`
/ `slack_image_pre_stripped` metric events.
## Testing
- 147 notification unit tests (resolve/render/preview/test-send incl.
per-channel goldens and hostile-input/injection coverage), 8 chart
payload tests, renderer PNG tests.
- Live-verified end to end: Slack (Slackbot chart fetches ~100ms,
cache-immutable semantics), SES email inline image, multi-severity
firing scenario (warning→critical) with correct threshold lines.
- FE: AddTemplate/ContentTemplateForm/preview specs; UX audit pass with
fixes.
- `cargo clippy -D warnings` clean in both OSS and enterprise
arrangements.
## Enterprise pairing
Must merge together with openobserve/o2-enterprise#2353 (workspace
twin-manifest mirror for the new OSS deps).
|
||
|
Hengfei Yang
|
d5d0b23666 |
fix: accept org ingestion tokens for OTLP gRPC (#13627)
## Summary - accept organization ingestion tokens (`o2oi_`) for OTLP gRPC logs, metrics, and traces - keep internal cluster, search, node, and other non-OTLP gRPC services on the existing authentication path - add coverage for accepted, missing, and internal-RPC token cases ## Root cause The HTTP ingestion authentication path validates organization ingestion tokens, but the global gRPC interceptor only recognized internal gRPC tokens and user credentials. As a result, the same organization token succeeded over HTTP OTLP and failed over gRPC OTLP with `Unauthenticated: No valid auth token[5]`. This change assigns the organization-token-aware interceptor only to external OTLP services. Internal RPCs continue using the original interceptor, preserving the existing security boundary. ## Testing - `cargo test --release -p openobserve-api-grpc handler::grpc::auth::tests --lib` (6 passed) - `cargo build --release --bin openobserve` - live local E2E on latest `main`: created a real organization ingestion token, exported an OTLP log over gRPC, then searched the stream and verified the expected record (`total=1`) Fixes #12540 |
||
|
|
6f0b2dcb07 |
fix: workflow history (#13588)
This enhances the workflow history , stores all runs instead of just failed ones and also some UX improvements on FE --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> |
||
|
Huaijin
|
12dce4c274 |
ci: lint all Rust workspace targets (#13615)
## Summary - run Clippy against every workspace member and target in the unit-test CI - fix the existing test-target Clippy findings, including the non-snake-case test name - keep warnings as hard failures so `unit_tests_summary` can be used as the required check ## Validation - `cargo fmt --all -- --check` - `cargo clippy --workspace --all-targets -- -D warnings` - `cargo test -p search --lib catalog_functions_documents_what_only_the_server_can_supply` - `cargo test -p config --lib` — 2967 passed, 2 ignored - `cargo test -p infra --lib` — 1243 passed, 35 ignored; the full parallel run also exposed two pre-existing local-only failures: an environment-sensitive default-index-field assertion and a shared mock database initialization race. The ratelimit test passes in isolation. |
||
|
Ashish Kolhe
|
b31ff6c7fe |
chore: incident destinations, source rename, and Alert Sources UX redesign (#13612)
## Summary - Adds a typed `destinations` field to incident_integrations (migration + entity + table ops), threaded through create/list/event-ingestion. - Adds a PATCH endpoint to rename an alert source, including the org's `default` source (rename allowed; delete stays blocked for it). - Moves Alert Sources under the Reliability nav menu. - Redesigns the Alert Sources list and Add/Edit drawer: - Table-only landing page; Add Source opens a drawer instead of a route. - Single row-level Edit action (name + destinations together), matching the Destinations/Templates list pattern instead of per-field popovers. - Fixed-layout, horizontally scrollable table with a pinned Actions column; wrapped/truncated cells confined to their own columns. - Separate "copy token" action distinct from "copy URL". - "Never resolves" warning moved from an unbounded banner list into a tooltip on the status tag/icon. - Removed the confusing "Shared token" badge and reworded the no-senders-yet caption. - All 3 setup steps shown together in the drawer (checklist-style) instead of hiding steps 2-3 behind a post-save view. - Destination creation now routes to the real Destinations page instead of a fake local-only option. - Fixed the test curl snippet: no stray `#` comment in the copyable text, and a payload the backend's generic-format detector actually accepts. ## Test plan - `cargo build`/`clippy -D warnings` clean on `infra`, `openobserve-api-management`, `openobserve-api-http` (enterprise feature) and OSS-only workspace check. - 151 frontend vitest tests passing (stable across repeated runs). - Manually verified against a local enterprise build + Vite dev server. |
||
|
Prabhat Sharma
|
624425a08a |
feat(editor): rebuild query autocomplete, and fix the value path behind it (#13586)
Design at autocomplete/autocomplete.md
Rewrites the query editor's autocomplete across every surface that has
one, and
follows the field-value path far enough to fix the bugs it was hiding.
53
commits, in six phases; each phase was written test-first, reviewed
adversarially, and checked in a browser before the next one started.
## What was wrong
The completion providers were wired inconsistently. Six surfaces bound a
resolver that was never defined, so their value completion could not
resolve at
all — SLO, dashboards and anomaly detection among them. Traces never
bound
`:suggestions`. Alerts never called `updateStreamKeywords`. PromQL ran a
second
editor runtime purely for its grammar, and asked
`/prometheus/api/v1/series`
— the heaviest endpoint available — for data the stream schema already
has.
## Phases
**1 — icons, staleness, quoting.** Correct `CompletionItemKind` glyphs,
stream
keywords refreshed when the stream changes, `spath` quoting.
**2 — the function catalog.** Custom functions get one tab stop per
argument,
org VRL functions stop appearing twice, column types read from
`dataType` as
well as `type`.
**3 — signature help, hover, async completion.** Providers return real
`CompletionList`s with `incomplete: true`, so a list that arrives late
is
re-queried rather than pinned.
**4 — numeric ranking.** Numeric columns sort first inside a numeric
aggregate.
**5 — values for streams nobody has searched.** The value cache only
ever held
what a search had already returned, so a stream you had not queried
offered
nothing. Values are now fetched on demand, bounded (10s timeout, 60s
cooldown,
one in-flight request per key), and shared between the editor and the
builder.
PromQL joins the same path: labels from the stream schema, values from
the
shared cache. `/series` now has zero production callers.
**6 — one way to fetch a value.** Item 22 only, in this PR.
## The dashboard filter 400
Clicking the ✕ beside "Select Field" in a filter's dropdown sets the
column to
`{}`, and the watcher on `condition.column` then asks for that column's
values.
`loadFilterItem` built `fields: [row.field]` out of `undefined`,
JSON.stringify
turned that into `null`, and the server answered
```
400 {"code":400,"message":"SerdeJsonError# invalid type: null, expected a string"}
```
`_values_stream` answers 400, not an empty result, for each way a piece
of the
payload can go missing — and every one is reachable: an unresolved
stream alias
(`.find(…)?.stream`), a stream not yet chosen (`""`), a range not yet
set. The
range fails differently: `meta.dateTime` starts as `{start_time: ""}`
and
`""?.toISOString()` does not short-circuit, so it throws and both
callers
surface "Something went wrong!" for a lookup the user never asked to
fail.
Guarded at the two callers, where the payload is built and where the
throw
happens. Not at `fetchFieldValues` — a guard there is too late to stop
the
throw and would duplicate validation the callers need anyway. The
trigger is
left alone: re-picking a field must still load its values.
## Notable removals
CodeMirror is gone — it was still bundled for the PromQL grammar alone.
Bundle 26.3 MB → 25.43 MB; the PromQL chunks 442.7 KB → 20.6 KB. The
PromQL
term catalog went from 7 entries to 113, generated from Prometheus's own
tables
and vendored with its MIT notice.
## Verification
- 76 files, +10.5k/−1.4k, ~9,100 tests passing across the touched areas.
- `vue-tsc` clean, `npm run build` exit 0.
- Every phase checked in a running browser against a live instance,
including
the 400 above: the ✕ now produces no request and no toast, adding a
filter and
re-picking a field each still fetch once, and the value list still
populates.
## Known, deliberately out of scope
- A filter still fetches twice (once on add, once on open) — that is
item 23,
the cache read, not this PR.
- `useMetricsExplorer.loadFilterItem` has the same unguarded
`toISOString()`
pattern on a different surface.
- `services/metrics.ts` and `services/search.ts` still carry dead
`get_promql_series` wrappers now that nothing calls `/series`.
|
||
|
Loakesh Indiran
|
f16e8e53ff |
perf(synthetics): config caching, run-budget validation, HA correctness, step-grain Steps tab (#13580)
Synthetics performance and correctness work. Four areas: caching the
config reads on the job path, a false-alarm bug in validation, four HA
correctness bugs found by auditing what happens with more than one alert
manager, and the Steps tab moving to a step-grain stream.
Pairs with o2-enterprise#2325 — **merge this first**, that one calls
functions introduced here.
---
## Caching
- [x] Probe-token auth cache — 10s TTL, negative results cached so a bad
token can't force a query per request
- [x] Locations whole-table cache — 30s, serves `get` / `find_by_pool` /
`list_visible` from one load
- [x] Check definition cache via `get_cached()` — was read twice per job
(resolve + ack)
- [x] Agent capabilities cache via `get_cached()` — was read on every
lease poll
- [x] Org ingest-token cache — `find_default_enabled()` replaces three
hand-rolled `list_by_org().find(enabled)` copies
- [x] `list_by_locations()` — one batched agents query, replacing one
per location
## Cross-node invalidation
- [x] `coordinator::synthetics` — event invalidation following the
alerts/pipelines pattern
- [x] Write paths call `invalidate_and_publish()`, the watcher calls
plain `invalidate_cache()` so events don't echo
- [x] Deletes emit a delete event, so coordinator keys don't accumulate
one per check ever created
- [x] Ingest-token invalidation reuses the existing
`/org_ingestion_tokens/` watch — no second event stream
- [x] SQLite write mutex released before emitting — `get_lock()` and the
coordinator's `put()` take the same lock, so emitting while holding it
deadlocked the process
## HA — correctness with more than one alert manager
Confirmed with @subhradeep that a region can run multiple alert
managers. The synthetics scheduler, dispatcher, reaper and staleness
watcher all run on **every** one of them.
- [x] **Duplicate runs** — `claim_due()` claims due checks with `SELECT
… FOR UPDATE SKIP LOCKED`, advancing the schedule inside the same
transaction
- [x] **Duplicate "location down" pages** — the one-shot flag moved from
an in-process `HashSet` to `synthetics_locations.down_notified_at`,
claimed with a CAS
- [x] **Duplicate run completion** — `increment_jobs_done` claims
completion via `completed_at IS NULL`; two acks of the same run could
both be told they completed it
- [x] **Inert dedup index** — `scheduled_ts` is now the schedule slot,
not each node's tick clock, so the existing `synthetics_jobs_dedup_uq` +
`ON CONFLICT DO NOTHING` actually fires
- [x] Migration `m20260803_000001` adds `down_notified_at`, defaulted so
no backfill is needed
- [x] `DB_SCHEMA_VERSION` 62 → 63 — without it the migration never runs
on an already-upgraded deployment
## The false-alarm fix
- [x] Validate against the run budget (840s), not the job lease (900s)
- [x] The three bounds are deployment-configurable:
`O2_SYNTHETICS_MAX_CHECK_BUDGET_SECS` / `_JOB_LEASE_SECS` /
`_MAX_NET_TIMEOUT_MS`
- [x] Rejected limits are logged, not fatal — nothing installed means
validation keeps the conservative defaults
- [x] Browser retries capped at 2; protocol types stay at 3; default
stays 0 for every type
## Steps tab — the step-grain stream (B10)
- [x] Three `GROUP BY`s over the step stream replace downloading 5000
execution rows and tallying their JSON in the browser
- [x] Falls back to the old tally when the stream is empty, so the tab
does not blank for checks whose agents have not been upgraded
- [x] Gated on the `step_id` column rather than stream existence — a
partially-created stream would otherwise cause four failed queries
## Overview KPIs
- [x] Count tiles summed from the cached histogram; `buildKpiSql`
deleted
## Rename
- [x] `synthetics_monitors` → `synthetics_checks`, `monitor_type` →
`check_type`
- [x] Three UI call sites that read the old field names, including a
delete guard that silently disabled itself
---
**Why validation moved off the lease.** Three bounds stack:
```
check worst case <= max_check_budget_secs < job_lease_secs
840s 840s 900s
```
Validation used the lease, so the server accepted checks the execution
environment cannot finish. On the managed path the probe is a Lambda,
and a function that hits its timeout is killed mid-run — lease headroom
is irrelevant once the process is gone. The net probe's function timeout
had drifted to **60s** while validation accepted up to 900s, so a plain
`timeout_ms=30s, retries=2` (worst case 100s) was accepted and then
killed, reporting a failure the target never had.
**Why the limits are ENT-declared but OSS-consumed.** Synthetics is
enterprise-only, so the values live in `SyntheticsConfig`. The
validation they bound lives in `config`, which cannot depend on
`o2_enterprise`. `SyntheticsLimits` + `init_limits` is the seam, falling
back to built-in defaults in OSS builds and tests. A bad limit is logged
rather than fatal: refusing to start ingest, search and dashboards
because a probe ceiling is misconfigured would be the worse outage.
**The duplicate-run bug.** `fetch_due` was a plain `SELECT` and the
advance was an unconditional `UPDATE … WHERE id = ?`, so nothing stopped
two alert managers firing the same check. Each inserted its own
`synthetics_runs` row under a fresh KSUID, which made the resulting job
rows genuinely distinct — `lease_batch`'s `status = 0` guard cannot
collapse them, because they are not contending for one row. N alert
managers meant every check ran N times.
`SKIP LOCKED` rather than a compare-and-swap because it *distributes*: a
second scheduler skips the rows the first holds and picks up different
due checks, so N replicas split the backlog. A CAS gives the same
exactly-once guarantee but every node reads the same candidates, one
wins them all, and the losers each issue a doomed `UPDATE`. The lock and
the advance share a transaction because the locks only live until
`COMMIT`; the fan-out stays outside it so config rows are not locked
across N job inserts.
**Duplicate run completion needed no second alert manager.** The
increment was atomic, but the completion decision re-read the row in a
second statement and returned "complete" to whichever caller saw
`jobs_done >= job_count` — which both acks of a 2-job run do. Acks are
served on ingesters, of which every environment runs two, and the probes
of one run finish independently. Effect: `consecutive_failures` counted
one run twice on every check, plus a genuinely duplicated notification
on checks left at the default `cooldown_mins = 0`.
**The Steps tab was truncating silently.** It downloaded up to 5000
execution rows and tallied `last_attempt_steps` in the browser. On the
busiest check measured — 18,079 executions over 7 days — that cap
covered 4 of the 7 days and reported a **56.3% fail rate against a true
33.7%**. The win is correctness; the ~18MB payload reduction is
secondary.
## Verification
`cargo check --workspace`, clippy, fmt clean. Config 98, infra 33
synthetics tests pass. Frontend: **839 synthetics tests pass**. Three
frontend spec files fail to import on `useLocalTimezone` — verified
identical on clean `main`, unrelated to this branch.
|
||
|
Ashish Kolhe
|
23e816b1e4 |
feat: External Alert Sources incident ingestion (#13578)
## Summary Adds a full "External Alert Sources" feature so OpenObserve Incidents can serve as ingestion/correlation layer for third-party alert tools (Grafana, Alertmanager, generic webhooks): - Token-authenticated inbound webhook endpoint (`o2iat_` tokens) that normalizes Grafana/Alertmanager/generic payloads and correlates them into Incidents alongside internally-generated alerts. - Incident integrations CRUD (create, list, enable/disable, rotate token, delete — default integration protected from deletion) with per-sender observed status. - Settings UI: single always-open alert sources table (merged from an earlier two-panel design), shared-token badge, add/rotate/delete flows. - Incident detail UX: contributing-alert Source/Labels columns, "view raw payload" action (surfaces the exact webhook JSON received), and an external-source badge on the incident header so it's obvious at a glance whether an incident has external contributors. - **Auto-resolve**: incidents now close automatically once every contributing external alert reports `status: resolved` — previously required manual resolution even when the source alert cleared. ## Notable follow-up fixes bundled in - Rebased onto latest `main`; repaired two build breaks surfaced by upstream changes that landed after this branch diverged (`CorrelationSubject` refactor's new `eval_level` param, and a new `SeverityEscalated` correlation outcome) — both textually merged clean but were semantically incomplete for the external-event path. - Removed the `O2_INCIDENTS_EXTERNAL_SOURCES_ENABLED` enterprise config flag (companion removal in o2-enterprise) — external sources are now gated solely by `O2_INCIDENTS_ENABLED`, removing a redundant toggle. ## Test plan - [x] `cargo build --no-default-features --features enterprise` — clean, no warnings introduced - [x] `cargo clippy` on touched crates (`infra`, `openobserve-core`, `openobserve-api-management`, `openobserve-api-http`) — no new warnings - [x] `cargo fmt --check` — clean on all touched files - [x] `cargo test` — all tests in touched modules pass (`alert_incidents`, `incident_integrations`, `external_alerts`, `alerts::*` — 274/274); two unrelated flaky DB-pool-timeout tests (`db::tests::test_delete`, `table::ratelimit::tests::test_add_batch_missing_rule_id`) confirmed pre-existing and pass in isolation - [x] Frontend: `vitest run` on all touched components — 156/156 passing; `vue-tsc --noEmit` clean - [x] Manually verified end-to-end against a running dev server: fired/resolved external alerts, confirmed auto-resolve, delete, and raw-payload retrieval all work against real HTTP calls |
||
|
Prabhat Sharma
|
d99ecff3e8 |
feat(nav): add Reliability menu, restore deferred SLO routes (#13577)
Groups the alerting surface under one left-rail tile, and un-defers the SLO backend that shipped disabled. > **Lands with** openobserve/o2-enterprise#2321, which adds `ROUTE_PERMISSIONS` for the restored SLO endpoints — the route-coverage test pairs the two, so this one alone fails it. ## Reliability menu New `reliability` nav group absorbing **Alerts, SLOs, Incidents**, plus **Notification Destinations** and **Templates**, which move out of Settings — they are alerting configuration, not deployment configuration. Settings keeps Pipeline Destinations (that one really is pipeline config); its group is renamed "Destinations". Their routes are **top-level and flat** (`/alert-destinations`, `/alert-templates`) rather than nested under `/alerts`. They are siblings of Alerts, not sub-pages of it — nesting them made the URL claim otherwise and the rail believed it, highlighting Alerts alongside them. - Route **names** are unchanged, so the ~15 call sites that navigate by name are untouched. - Old `/settings/*` paths redirect, with the query preserved — `?action=import` deep links keep working. ## SLO routes `/api/{org}/slos` answered **404**. The routes were deliberately unregistered before the last release (`6977b4020d`, `ac78a622e1`) while the rest of the feature — `core/src/slo/*`, the infra tables, both migrations, the `slo_maintenance` job, all seven handlers — stayed in the tree. This restores the five registrations and the OpenAPI entries, recovered from the pre-strip commit rather than rewritten. `ZO_SLO_ENABLED` still **defaults to false**. It is now published as `slo_enabled` on `/config`, so the menu entry follows the flag instead of offering a page the API answers with 501. ## Nav active-state Active state was decided per child, so any section whose path prefixed the current route lit up alongside it. It is now resolved once per flyout: exact route-name match wins, else the **deepest** path prefix — so a genuine drill-down like `/alerts/detail/:id` is still attributed to Alerts. `placeAfter` also accepts a group key, letting Data anchor on the Reliability tile it follows rather than on an item that tile absorbs. ## Tests `tests/ui-testing` referenced `[data-test="alert-destinations-tab"]` / `alert-templates-tab` in six files — those data-tests were on the Settings rail items removed here, so the locators would have matched nothing and hung until timeout. Those paths now go through the existing `openNavFlyoutChild` helper, extended with the Reliability group. Two URL regexes that would have silently mismatched on the hyphen are fixed (one ended in `.catch(() => {})`, so it would have degraded to a silent 15s stall). ## Verification - 3597 unit tests pass; `vue-tsc` and eslint clean; enterprise build compiles. - `playwright test --list`: all 141 tests across 23 files load, no module errors. - Against a live server: `GET /slos` → 200, `POST /slos/move` → 422 (so the literal wins over the `{slo_id}` catch-all), `GET /slos/x` → handler-level `{"code":404,"message":"SLO not found"}`. - In the running app: `/alert-destinations`, `/alert-templates` and `/alerts` each highlight exactly one row; `/settings/alert_destinations?...&action=import` redirects with the query intact; rail order unchanged. |
||
|
Hengfei Yang
|
72f7bff378 |
perf: optimize AI session list queries (#13574)
## Summary - Page sessions by `session_id` before computing the expensive summary fields. - Aggregate only the selected page of sessions with exact `count(DISTINCT trace_id)`, removing trace ID materialization and Rust-side per-trace rollups. - Generate direct Agent predicates and evaluate them as aggregate membership conditions without a session-membership subquery. - Support efficient server pagination by fetching one extra session to determine `has_more`, without counting every distinct session on every page. ## Why The previous endpoint first materialized `array_agg(DISTINCT trace_id)` for every candidate session and then issued another query using those trace IDs. A single-pass full rollup was also slower because it aggregated every summary field for all matching sessions before applying TopK. This change keeps the inexpensive session page selection separate and bounds the expensive rollup to the requested page size. Page selection and display both use the latest `end_time` across the complete session; Agent predicates only decide whether a session is a member of the result set. The endpoint previously returned `total = hits.len()`. With a 20-row page, OTable therefore calculated one total page and permanently disabled Next even when more sessions existed. An exact `count(DISTINCT session_id)` query scanned the full data set and took about 1.18 seconds by itself on this benchmark. Instead, the page query now requests `size + 1`: while another page exists, `total` is a documented lower bound and the response includes `has_more=true` / `total_is_exact=false`. The final page returns the exact total. OTable displays lower bounds with `+` and hides Last-page navigation until the exact total is known. ## Performance Measured locally against `bench_traces` (167 files, about 99.9 million records), using five runs and ignoring the first two: | Query shape | Mean wall time | | --- | ---: | | Original trace-ID flow | 6.829 s | | Page session IDs + membership subquery | 3.219 s | | Page session IDs + Last activity | 0.850 s | The optimized query shape is about 87.6% faster than the original flow. The second-phase rollup returned 20 rows with about 74 KB of intermediate output and about 2.45 MB peak memory in the query plan. Fetching the extra pagination sentinel did not introduce a meaningful page-size-dependent regression in the final runtime sweep. ## Behavior note The Agent filter controls membership through `HAVING max(CASE WHEN <filter> THEN 1 ELSE 0 END) = 1`, while the final rollup includes all spans for each selected session. Both pagination and display use the complete session's `max(end_time)`, ordered by `session_last_activity DESC, session_id DESC` for stable offset pagination. The UI labels this value `Last activity`; session start and duration remain available from the rollup. Current clients send direct Agent predicates. For compatibility, the endpoint also recognizes the previous narrow same-stream `session_id IN (SELECT session_id ... GROUP BY session_id)` shape and safely extracts its inner WHERE predicate; unrelated subqueries are left unchanged. This assumes every relevant span has a `session_id`. ## Validation - `cargo build --features mimalloc --profile release` - `cargo test -p openobserve-api-search traces::session --lib` (18 passed) - `cargo test -p openobserve-api-search user::tests --lib` (13 passed in the earlier query-change validation) - `cargo fmt --all` - `git diff --check` - targeted Vitest: Sessions composable, SessionsList, and OTable (169 passed) - `vue-tsc --noEmit -p tsconfig.vitest.json --composite false` - targeted ESLint and Prettier checks - runtime pagination check against the supplied legacy-filter curl: - page 1: 20 hits, `total=21`, `has_more=true`, `total_is_exact=false` - page 2: 20 hits, `total=41`, `has_more=true`, `total_is_exact=false` - final page: 3 hits, `total=184443`, `has_more=false`, `total_is_exact=true` - pages 1 and 2 returned 40 unique sessions in descending `end_time` order across the page boundary |
||
|
|
4772df1723 |
feat: per-group and per-series alerting, and SLO measurement (#13547)
Design at: alerts_2.md Per-group and per-series alerting, SLO measurement, and the UI for both. ## Alerts **Multi-alerts (Feature 3).** A grouped alert can opt in to evaluating and paging **per group** rather than collapsing to one verdict. Opt-in only (`multi_alert`), so no existing alert changes behaviour: the flag cannot be present in JSON written before it existed. Per-group state, transitions, silence fingerprints, disappearance/reaping and a cardinality cap, with the count-gate rules that make "most severe group" and the legacy collapsed level provably the same verdict. **Per-series alerting for PromQL.** The same feature for the metrics family, where a group is one returned **series** keyed by its full label set — that is a series' identity in Prometheus, and the expression's own `by (…)` clause is already where the label set is chosen, so a second picker could only disagree with it. One additive nullable column; NULL means off, so stored alerts are untouched. **Alert detail page**, replacing the row-click side panel: stat strip, evaluation chart with threshold marklines, per-group table and history. Simple alerts get a real history too — evaluations (one record per run) by default, level changes alongside. ## SLOs Count / time-slice / alert SLIs, error-budget and burn-rate arithmetic, rolling windows, per-group slices with an exact rollup, coverage gating, backfill, and SLO-based alerts. Two capabilities the model could not express, both added tests-first: - **Metrics-native counting** (`CountSource::PromQl`) — counter metrics have no rows for a predicate to classify, so "good" exists only as arithmetic between series, and correct counter arithmetic is `increase()`. - **Freshness semantics** (`absent_is_bad`) — for a pipeline SLO, silence IS the failure, which the default gap rule can never report. A *failed* query still freezes rather than pages; only a query that PROVED emptiness counts as bad. **SLO form**: live good/bad preview so a wrong predicate is caught before saving rather than days later, real query editors with field autocomplete, folders shared with alerts, and a move endpoint. ## Notes for review - The SLO and composite features are **deferred (TODO)**: `ZO_SLO_ENABLED` stays default **false** and the SLOs menu entry is hidden until they ship; the SLO backend and UI remain in the tree behind the flag, and composites stay pure logic + tests. - One migration (schema 61): a nullable column for the PromQL opt-in. No backfill. - Two fixes outside the feature that it surfaced: query-editor completions were duplicated per mounted editor (Monaco registers providers per *language*), and the alert detail chart had no PromQL branch, so it charted row counts against the series-count gate. - Lands together with the matching o2-enterprise PR — `QueryType::Slo` is new here and the enterprise dedup matches are exhaustive over it. - `ratelimit::test_add_batch_missing_rule_id` fails intermittently on `main` as well; unrelated to this branch. --------- Co-authored-by: Shrinath Rao <shnath@openobserve.ai> Co-authored-by: hengfeiyang <hengfei.yang@gmail.com> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> |
||
|
Huaijin
|
015a20a184 |
refactor: remove HTTP API compatibility re-exports (#13560)
## Summary - remove the HTTP request/model compatibility re-exports introduced by the domain crate split - migrate router, OpenAPI, enterprise action-server, and integration-test callers to the crates that own each API - remove the ingest Content-Type re-export and declare the direct root/test dependencies - delete the pure `handler::http::models` facade module ## Validation - `cargo fmt --all -- --check` - `cargo clippy -p openobserve-api-ingest -p openobserve-api-http -- -D warnings` - `cargo test --test integration_test --no-run` - `cargo check -p openobserve --no-default-features` - enterprise overlay: `cargo test --offline --workspace --no-default-features --features enterprise --no-run` - cloud overlay: `cargo test --offline --workspace --no-default-features --features cloud --no-run` - `git diff --check` |
||
|
Loakesh Indiran
|
a90bdea831 |
fix(synthetics): name the degradation instead of calling it flaky (#13556)
QA reported an alert that contradicts itself: ``` 🟡 EU - TLS cert passed only after retries (flaky) Error: certificate expiring soon ``` Flaky means the check already fixed itself and there is nothing to do. A certificate running out will not fix itself. The headline is the part that gets read first, and it was pointing the reader away from the problem. Notification half. Pairs with synthetic-o2-agent#8 and o2-enterprise#2312 — each is inert without the others, since the reason has to travel probe → control plane → notification. ### What changed `CheckNotification` carries `status_reason`, so the Slack subject and the headline can name the condition: | | before | after | | --- | --- | --- | | subject | `🟡 <name> is DEGRADED` | `🟡 <name> — CERTIFICATE EXPIRING SOON` | | headline | `is reachable but degrading` | `the TLS certificate is expiring soon, renew it before it lapses` | The subject matters most: it is what decides whether anyone opens the alert. `CERTIFICATE EXPIRING SOON` gets renewed; a generic `DEGRADED` gets skimmed. Falls back to the existing generic wording when the probe sends no reason, so an older probe reads exactly as it does today. Covers `sftp_degraded` too — an SSH check that connects and authenticates but whose SFTP probe fails was mislabelled the same way, just never reported. ### Verified live Local o2 built from these branches, against `eu1.openobserve.ai` (28 days left, 30-day window): ``` 🟡 LCL - TLS cert expiring — the TLS certificate is expiring soon, renew it before it lapses Target: eu1.openobserve.ai Error: certificate expiring soon ``` Record: `status=warning`, `attempts=1`. That number is the point — the reported alert required attempts > 1 to claim retries had happened. Binary confirmed to contain the new strings before testing, so the result is not a stale build. `cargo check` clean on both OSS and enterprise. |
||
|
db2f0fb62a |
feat(synthetics): attempts view, evidence panel, alerting, concurrency (#13543)
> Note: this branch carries ~20 earlier synthetics commits. The alerting piece is ~280 lines and is cleanly separable if a smaller review is wanted. ## Backend — alert state - **Four columns on `synthetics`** (`consecutive_failures`, `last_alert_at`, `alerting`, `degraded_notified_at`), all defaulted so no backfill is needed - **`DB_SCHEMA_VERSION` 55 → 56** — without the bump the migrator short-circuits and the columns never appear - Guarded with `has_column` on **both** `up` and `down`: SQLite ignores `IF NOT EXISTS` on `ADD COLUMN`, and sea-query has no `drop_column_if_exists` - **Compare-and-swap** accessor (`update_alert_state_if`), because the caller does a read-modify-write with no transaction - `alerting` is a separate column because without it "recovered" cannot be told from "was never alerting" - **`synthetics_jobs::failing_locations`** — the notification previously had no location field, so a six-location check with one broken region could only say "the check is failing" - Distinct subject, headline, icon and colour for FLAKY vs DEGRADED, from one shared `locations_line` helper so Slack/plain-text/email cannot drift > The table is `synthetics`, not `synthetics_monitors` — the entity module is named after the concept but declares `table_name = "synthetics"`. Recorded at the `DeriveIden` so the next column addition doesn't repeat it. ## Performance — the step-stats panel fetched ~35 MB per load - **`recorded_steps` out of the 5000-row tally** — ~60% of payload; ~4 KB/row and near-identical within a config version, so an *aggregation* re-read the same definitions 5000 times. Still written to every record, because steps are editable and history must not be rewritten - **`retry_history` out too**, replaced by `retry_step_ids` scanned only over rows that actually retried - **Key step stats by `step_id`, not name** — a rename *split* one step into two rows; two steps sharing a name *merged* into one - **Single-pass sparklines** — was O(steps²) per row: ~2M iterations for a 20-step journey, ~50M for a 100-step one - **Flaky Rate could exceed 100%** — the fallback incremented the numerator without the denominator - **Label the real window** — a 1-min check × 2 locations × 4 device combos is 11,520 executions/day, so a 5000-row cap is ~10 hours of a "last 7 days" selection ## UI - **Attempts selector** in the run detail — costs no request, `retry_history` is already on the row. Labels are 1-based while `attempt` is 0-based - Each attempt resolves **its own** screenshots, trace and evidence bundle; a superseded attempt's live under an `attempt-N-` key - **Evidence panel** — parses `evidence.ndjson` per line, grouped by kind in severity order, chips with counts visible at zero, third-party dimmed not dropped, truncation stated, four distinct empty states - **Flaky / Degraded / Unstable tiles.** Unstable is partitioned by `(location, device, engine)` — aggregated, a check solidly broken in one region and healthy in five is indistinguishable from one intermittently broken everywhere. It measures *oscillation*, not badness - All flakiness tiles read **"—" at `retries = 0`**, never `0.0%`: a zero is a measurement, that is the absence of one - **`init_ms` and queue delay shown separately** — init was observed at 113,131 ms on a cold Lambda against a 243 ms check; unsubtracted, Lambda locations look permanently slower at every percentile - Steps and Evidence are sibling `OTabPanel`s — stacking them broke the drawer's scroll ## Bugs fixed during review - **`mapRetryHistory` guarded on `Array.isArray`** but the search API returns blob columns as JSON *strings*, so it returned `[]` for every real record — the attempts strip could never render, whatever the query selected. Same bug in `mapEvidence` and the `attempts` fallback - **`mapRunDetail` read `rawHit.attempt`**; the field is `attempts`. The count was 0 on every run - **`mapRetryHistory` hard-coded `status: "failed"`** — reported a flaky run's *passing* final attempt as a failure - **A superseded attempt's failing step rendered as a pass** — the compact timeline's `passed`/`failed` was never normalised to the `ok`/`fail` vocabulary every consumer tests for - **`failure_detail` is not a column** — OpenObserve flattens nested objects; naming the object is rejected by the search API - **The retry-attribution query named `status_reason` unconditionally.** That field only exists once something has been a `warning`, and the three step queries share a `Promise.all` — so one missing column emptied the *entire* Steps tab - **`fetchSchemaFields` returned an empty Set on failure**, which the gate reads as "the schema has no fields", silently degrading every optional column to a literal ## Lease correctness (added after review of the concurrency docs) Two corruption-class bugs, both of which double-count the alert failure streak this PR introduces. Included here rather than in the concurrency branch because they undermine *this* PR's own feature. - **The browser lease validation was per-device while the work is per-job.** `validate_browser_config` computed `(retries+1) x journey_budget + retries x wait` against the 900s lease, but the probe runs `browser_devices` **sequentially inside** the leased job and nothing multiplied by the combo count (`MAX_BROWSER_DEVICE_COMBOS = 12`). At defaults a plain **desktop + mobile** config is 1210s of work; validation computed 605s and allowed it. It then blew its 900s lease on **every run, forever**, with nothing failing at save. At the cap: 7260s vs 900s. The consequence is verbatim what the `LEASE_SECS` comment in `dispatcher/mod.rs` was written to prevent — *"the reaper would requeue mid-run and the journey would EXECUTE AGAIN — duplicate result records, multiplied browser cost, and false alerts caused by the reliability fix itself."* All three happen at two devices today, on the **managed** path, since browser is Lambda-only. > ⚠️ **Breaking validation change.** An existing 2+ device check will fail to save until its budget, retries or combo count comes down. The error now names the combo count as a lever. **Query dev and prod for `json_array_length(config->'browser_devices') > 1` before rolling out.** - **`ack_complete` had no status guard** — `WHERE id = $4` alone. A duplicate ack succeeded and the caller called `increment_jobs_done` again for one job: `jobs_done` overshot `job_count`, the run was declared complete on a **partial set**, and since this PR it also advanced `consecutive_failures` twice, so `alert_if_fails: 3` fires after two failures. Reachable today with no concurrency work: `aws-sdk-lambda` retries a timed-out `RequestResponse` invoke, re-executing a check already in flight. `AND status = 1` makes it idempotent, and `ack_complete` now returns `None` when it did not apply so the caller skips run accounting. Reaper reassignment is covered too, as of the concurrency work merged below: `AND claimed_by = $5`. ## Verification `vue-tsc` + `type-check:app` clean · Prettier + design ratchet clean · **711 synthetics tests** · `cargo check` clean · infra unit tests pass. Migration applied against a real SQLite metastore: `55 → 56`, all columns correct. Every drawer field verified against live ingested data. > `MonitorResults.spec.ts` fails on a pre-existing `storage.ts` import error — identical on a stashed tree. ## Requires o2-enterprise#2305 — the `AlertDecision` this consumes. Merge together; each is inert alone. --- # Also merged: probe concurrency (#13552) The concurrency branch merged into this one, so this PR now carries both. Everything below is a correctness fix — none of it is throughput work. ### The reaper never completed a run, on any path A run completes when `jobs_done` reaches `job_count`, and only three callers move that counter: the probe ack, the dispatcher's failed invoke, and the reaper. The reaper never did — including in `dead_letter_expired`, whose whole purpose is to report "we gave up after N attempts". It wrote a customer-visible error record into a run that then stayed open forever: no `completed_at`, and since this PR's alerting work, no `resolve_alert` either, so the failure streak never moved. The mechanism was `requeue_expired` ignoring `valid_until`. It reset a job to Pending, and `prune_stale` — later in the **same tick** — deleted it again, because `valid_until` is one interval (60s for a 1-minute check) while a lease is 300s+. `dispatch_attempts` never got past 1, so `MAX_DISPATCH_ATTEMPTS` was unreachable and the retry budget was dead code on the agent path. - requeue only while the job is still inside its window - dead-letter everything that can no longer report, tagged with why (attempts exhausted / window closed / never dispatched), so a dead agent does not read as an unresponsive probe - per-row compare-and-swap instead of SELECT-then-bulk-UPDATE; the reaper runs on every alert_manager node - increment **before** the ingest-token lookup, which returns early when an org has no token ### Every check's retry sequence is bounded by the lease Retries run *inside* the leased job. The browser path has checked this since `journey_budget_ms`; the protocol path never did, and its inputs were unbounded — the `5_000..=300_000` bound on `timeout_ms` is **browser-only**, so a TCP check with `timeout_ms: 3_600_000` saved fine. With the reaper now completing runs, the symptom is a wrong answer rather than a stuck one: the lease expires mid-flight, the reaper completes the run as Error, and the probe's real ack is rejected as stale. A passing check reports Error permanently. `timeout_ms=100s, retries=2, wait=30` needs 360s and hits it. `lease_batch` treats the client's `lease_secs` as a floor request, not the decision — both probes guess 300s, and a probe cannot know how long its own job may take. ### Guards on the ack, and on dispatch failure `AND claimed_by = $5` closes reaper reassignment. `claimed_by` is `Option` during rollout: an older probe acks without it and falls back to the status guard, so nothing is discarded. `fail_dispatch` had no status guard, so a dispatch judged failed *after* the probe acked reset a **completed** job to Pending — leased again, run again, duplicate result, `increment_jobs_done` twice. Third outcome `AlreadySettled` added; the dispatcher matches all three with no wildcard, so handling it is compile-time enforced. `limit` is now clamped before the query. It is cast to `u64`, where a negative wraps to ~1.8e19 and the LIMIT goes unbounded — one agent leasing the entire pending queue for its pool. Reachable from one mistyped env var. ### Queue backlog is observable `synthetics_pending_jobs` and `synthetics_oldest_pending_age_seconds`, per location and pool. `started_ts - scheduled_ts` already covered work that *ran*; a check nobody leased produces no record, so its backlog was invisible by construction. ## Before merging this to main The `timeout_ms` validation is a **breaking change for existing protocol configs**. It runs at create/update only, so nothing stops on deploy — but an over-budget check is rejected the next time someone edits it. Expected result is zero rows: `retries` of 0 or 1 cannot trip the budget rule at all, and the defaults sit at 10,000ms, 1.1% of the lease. ```sql WITH c AS ( SELECT id, org_id, name, synthetics_type, COALESCE((config->>'timeout_ms')::bigint, 10000) AS t, COALESCE((settings->>'retries')::bigint, 0) AS r, COALESCE((settings->>'wait_before_retry_secs')::bigint, 5) AS w FROM synthetics WHERE synthetics_type <> 'browser' ) SELECT id, org_id, name, t AS timeout_ms, r AS retries, w AS wait_s, (r + 1) * t + r * w * 1000 AS worst_case_ms FROM c WHERE t NOT BETWEEN 1000 AND 300000 OR (r + 1) * t + r * w * 1000 > 900000 ORDER BY worst_case_ms DESC; ``` Anything returned is a check that **cannot work today either** — fix the config rather than loosening the rule. ## Not done Wiring the reaper's now-completing runs to a **notification**. The send lives in a crate that depends on `o2_enterprise`, so the reaper cannot call up into it; it needs the send moved into `openobserve_core::synthetics` or a queue. No regression — before this, the run never completed, so there was no notification either. --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: omkark06 <omkar@zinclabs.io> Co-authored-by: Harsh Mahajan <115500013+007harshmahajan@users.noreply.github.com> Co-authored-by: Yashodhan Joshi <yashodhan@openobserve.ai> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> Co-authored-by: Shrinath Rao <shnath@openobserve.ai> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Hengfei Yang
|
044f561506 |
feat(evaluation): eval scheduler metrics, window-split executor, completion-window defaults and guard rails (#13457)
Design at: docs/online-evaluation-scheduler-queue-review.zh-CN.md OSS-side half of the phase-2 eval scheduler rework (paired o2-enterprise PR linked in comments). The enterprise PR depends on the metrics and the executor contract below, so **this PR must merge first**. ## Scheduler metrics - `eval_scheduler_pending_targets` (gauge, org/stream) — in-flight pending targets tracked across scheduler passes - `eval_scheduler_pending_memory_bytes` (gauge, state=used|limit) — accounted pending memory vs the configured budget - `eval_scheduler_forced_ready_total` (counter, org/stream) — targets force-evaluated early by budget enforcement - `eval_scheduler_watermark_lag_seconds` (gauge, org/stream) — scan cursor vs committed (persisted) watermark Registration only; all emission sites live in the enterprise scheduler. ## Detection-search executor: window-split support The scheduler's search executor no longer errors on an oversized scan window. It reports `over_limit` against the shared 100k-row cap (`EVAL_DETECTION_MAX_ROWS`) so the scheduler can split the window in half and re-query, and fetches a capped result set only when the scheduler explicitly allows truncation at the minimum window width. This removes the previous failure mode where a high-throughput window could never be fetched and retried forever. ## Completion-window defaults, floor, and hard caps Eval jobs that don't set their own trace/session completion windows now take deployment-wide defaults, declared in the enterprise `LlmEvaluationConfig` (`O2_EVAL_TRACE_IDLE_TIMEOUT_SECS`=30, `O2_EVAL_TRACE_MAX_AGE_SECS`=1800, `O2_EVAL_SESSION_IDLE_TIMEOUT_SECS`=1800, `O2_EVAL_SESSION_MAX_AGE_SECS`=14400 — the design-spec baselines) and injected into infra at startup via `init_completion_window_defaults` (same registration pattern as the search executor). Sessions span user think-time between traces, hence the much wider session idle window. - **Floor**: the old 45s idle-window minimum is dropped to 1s. Readiness fires only once the observed ingest-time silence reaches the window, so sub-poll-interval windows are correct — their firing is merely quantized to the next scan pass. - **Hard caps** (`validate_trace_completion_window` / `validate_session_completion_window`, applied identically to job-supplied values and env defaults): trace idle ≤ 30m / max age ≤ 2h, session idle ≤ 4h / max age ≤ 24h. Max age bounds pending-target memory residency, committed-watermark lag, and restart rescan, so it is a guard rail, not an operator preference. The enterprise config parser calls the same validators and **fails startup** on out-of-range env values (0 still means built-in default). - **Frontend** mirrors all of it: per-scope defaults and limits in `completionWindow.ts`, zod schema validation, input `max` attributes, and validation messages across all 15 locales. ## Legacy eval-buffer cleanup (OSS side) Removes the pipeline-update eviction call into the legacy trace-buffering LLM evaluation node (`LlmEvaluationNode::remove_buffer`) — a no-op on an always-empty registry since span evaluation moved to the eval-task queue. The paired enterprise PR deletes the legacy module itself. ## Module split `infra::table::online_eval_jobs` splits into submodules — `completion_window.rs` (defaults, guard rails, trace/session configs) and `span_selector.rs` — with re-exports keeping every external path unchanged. |
||
|
Hengfei Yang
|
0150472981 |
fix: default /prometheus/api/v1/series time range to last 24h when omitted (#13550)
Design at: #13120 Fixes #13120 ## What changed - `get_series` (src/core/src/metrics/prom.rs) now normalizes the time range before building the search request: a missing/zero `start` defaults to `end - 24h`, a missing/zero `end` defaults to now. The lookback is a plain constant (`DEFAULT_SERIES_LOOKBACK_MICROS`), no new env knob. - Route docs for `/api/v1/series` (utoipa params + x-o2-mcp description) no longer claim start/end are required. Backend-only fix: the web UI keeps sending explicit `start`/`end` and is not touched. ## Why Per the Prometheus HTTP API spec, `start`/`end` are optional for `/api/v1/series`. The handler defaulted a missing `start` to `0`, which `infra::file_list::validate_time_range` rejects (`start == 0` is its unset sentinel), so every time-range-less `/series` call returned HTTP 500 `[file_list] invalid time range`. 24h was chosen over an unbounded range (Prometheus' own default) because `/series` here is a columnar scan over object storage, not a local TSDB index lookup — full-retention scans are exactly what `validate_time_range` exists to prevent. 24h keeps low-frequency metrics (daily jobs) discoverable while bounding cost, and matches the fallback the dashboard frontend already uses client-side. Scope note: `/labels` and `/label/{name}/values` are intentionally untouched — they resolve via stream-stats intersection where `start=0` already means "all data", which is spec-correct. ## Testing - 3 new unit tests for `normalize_series_time_range` (default lookback, default end, explicit values untouched) — pass. - `cargo check -p openobserve-core -p openobserve-api-search` — clean. |
||
|
|
6d4eb704ed |
fix: allow testing workflow without saving (#13537)
This PR adds a fix so that workflows can be tested without having to save. --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> |
||
|
Hengfei Yang
|
4231ca5833 |
perf(schema): share stream settings via Arc and centralize cache mutation (#13535)
Design at: #13465 PR 2 of the plan in #13465 (finding 1: the settings cache O(S) deep clone per mutation). ## Background The stream settings cache is a locked map (`STREAM_SETTINGS`) plus an ArcSwap read snapshot (`STREAM_SETTINGS_ATOMIC`) introduced in `405794db08` for lock-free ingestion reads. That architecture is kept — the read/write ratio justifies it. The problem was the constant factor: - every mutation republished the snapshot by **deep-cloning every cached `StreamSettings`** (vectors, UDS field lists, maps). The snapshot is republished on every schema change event cluster-wide (any new stream or field, on every node) plus every cache-miss backfill — at 100k+ streams this is a full deep copy of the map at minute-level frequency; - every point read (`get_settings`, called on each ingestion request) deep-cloned the returned `StreamSettings`, including UDS lists that can hold hundreds of strings; - the two maps each held a complete deep copy of all settings. ## Changes **Arc-ified values.** Both maps now store `Arc<StreamSettings>`. Snapshot republish is a shallow clone (String keys + refcount bumps), point reads return an `Arc` clone, and the maps share one set of values, roughly halving the cache's memory. **Centralized mutation.** The map statics are private; all writes go through `put_stream_settings` / `put_stream_settings_batch` / `remove_stream_settings`, which republish the snapshot under the write lock so snapshots cannot be published out of order. Startup load collects settings and publishes once via the batch API. Callers can no longer accidentally rebuild the snapshot — the pattern that caused the original quadratic startup. Lock-free reads stay available via `get_stream_settings_atomic`; the status API uses a new `get_stream_settings_stats()` accessor. **API ripple, kept minimal.** `get_settings()` now returns `Option<Arc<StreamSettings>>`. Field access and `unwrap_or_default()` call sites compile unchanged; the few paths that edit a local copy before persisting (`update_stream_settings`, LLM/gen-ai settings, dashboard distinct fields) do an explicit `(*s).clone()`. The `get_stream_setting_*` helpers are generic over `Borrow<StreamSettings>`, so both `&Option<StreamSettings>` and `&Option<Arc<StreamSettings>>` callers work — including the downstream enterprise call sites, which were checked and need no changes. **`SchemaCache::fields_map` Arc-ified** — cloning a cached schema entry no longer deep-copies the field-name index, which holds thousands of entries for wide streams. ## Measurements (criterion bench used during development, not kept in-tree) | operation | result | |---|---| | bulk load 10k / 50k / 100k streams | 0.80ms / 4.4ms / 8.7ms — linear, ~87ns per stream | | point write against 100k-entry cache | 3.5ms (shallow snapshot republish) | | point read against 100k-entry cache | 7.9ns (lock-free) | For comparison, the field report in #13465 measured ~27ms per stream insert at 58k cached streams under the old per-mutation deep clone; reads previously deep-cloned the settings value on every call. ## Testing - new unit test for the mutation API (put/batch/remove, snapshot sharing via `Arc::ptr_eq`, republish keeps other keys) - full test suites pass for all touched crates (infra, db, schema, stream, core, api-search, api-management, search_service, compaction, enrichment-data); the only failures are pre-existing local-environment ones also failing on main (sqlite-path retention tests, one env-dependent index test, one enterprise doctest) - `cargo clippy --workspace --all-targets`: no findings in touched files Finding 2 (grouped stream enumeration) landed in #13536; this branch has been rebased onto main (the #13531 overlap is resolved). |
||
|
Huaijin
|
3d96c9670c |
chore: regenerate Rust CycloneDX SBOMs (#13534)
## Summary - regenerate Rust CycloneDX SBOMs with cargo-cyclonedx cyclonedx - update 9 existing SBOMs and add 32 workspace crate SBOMs - exclude all generated files under src/enterprise ## Validation - cargo-cyclonedx version: 0.5.9 - all 41 retained XML files pass xmllint --noout - malformed vcs_url PURL pattern from 0.5.7 is absent - git diff --check passes - src/enterprise has no changes |
||
|
Huaijin
|
17ef03b8d6 |
refactor: cut release build time in the web and telemetry paths (#13528)
Two changes that cut build time, both measured on the same machine (M2
Max,
12 cores) with `cargo build --features mimalloc --profile release`.
## 1. The `web` crate no longer depends on `config`
`src/web` is a leaf crate so the RustEmbed expansion over `web/dist`
only
re-runs when the assets change — that expansion is 149.8s, of which
144.1s is
macro expansion (the crate already sets `#[br = false]` / `#[gzip =
false]`, so
this is not compression).
It also depended on `config`, for exactly one value:
`cfg.common.base_uri`. That
put a 50MB embed downstream of a 724-crate subgraph, so **every `config`
edit
paid the 148.5s embed again** — which is precisely what the crate's own
doc
comment says it exists to avoid.
`ui_routes()` now takes the `<base href>` value from the caller and
`base_href_middleware` carries it as middleware state.
`create_app_router()`
already computed that exact string (`"{base_uri}/web/"`) for its own
redirect,
so it is now resolved once and passed in rather than recomputed per
request
from a second `get_config()` call. `base_uri` is env-derived, normalised
once at
config init and never touched by `refresh_config`, so hoisting it out of
the
request path cannot drift.
The web crate's dependency tree goes **724 → 92 crates**, and `cargo
tree -i
config` no longer lists it.
## 2. Logging and tracing setup moved out of `main.rs`
`setup_logs`, `enable_tracing` and their span-processor / exporter types
compose
very large generic trees. Everything `main.rs` monomorphizes is emitted
into the
binary's own codegen units — the last unit on the critical path,
overlapping
with nothing.
`-Ztime-passes` on the binary unit (91.1s total):
```
LLVM_thinlto 39.8s
LLVM_passes 37.5s
codegen_to_LLVM_IR 5.3s
link_binary 5.3s (run_linker 3.9s)
type_check_crate 0.4s
```
The `openobserve` library does not depend on `openobserve-api-http`, so
its unit
compiles in parallel with the API crates — in the cold build it finished
23.9s
before the binary needed it. Moving the setup into `src/telemetry.rs`
(563
lines) puts that work in the slack instead of at the tail.
Two tests moved with the code and were dropped rather than relocated:
`test_setup_logs` and `test_enable_tracing_error_handling` both install
a global
tracing subscriber, and `setup_logs()` calls `.init()` unguarded, so
they can
only ever pass depending on ordering. They used to race just with each
other
inside the bin test binary; keeping them would have moved that race into
the lib
test binary next to the `cli` and `migration` tests. Neither asserted
anything —
the second wraps the call in `catch_unwind` and discards the result.
`main.rs` 1304 → 712 lines. **Binary unit 88.8s → 80.7s**; the library
unit grows
30.9s → 62.3s and is fully absorbed by the existing slack.
## 3. Unrelated: a dead-code warning in `openobserve-core`
Not a build-time change, just a warning fix noticed while checking the
above.
`COMPARE_ROLLUP_LOOKBACK_MICROS` was gated on `any(feature =
"enterprise",
test)`, but its only two readers (`fetch_arm_aggregate`,
`fetch_arm_failures`)
are `enterprise`-only and the unit tests below it exercise only the pure
aggregation helpers. So in an OSS build `cargo check -p openobserve-core
--all-targets` warned `constant COMPARE_ROLLUP_LOOKBACK_MICROS is never
used`.
Gate narrowed to `enterprise`, and the doc comment — which claimed the
opposite
— corrected. Happy to split this out if you'd rather it not ride along.
## 4. Review follow-ups
- `enable_tracing` became a public library API in §2 without a doc
comment,
unlike `setup_logs` beside it. Documented: which span processors attach
under
which env vars, why the returned provider must be held rather than
dropped,
and that it and `setup_logs` are mutually exclusive since both call
`.init()`.
- Added the `<base href>` test described under Verification below.
Not taken: the suggestion to re-add `test_setup_logs` /
`test_enable_tracing_error_handling` to `src/telemetry.rs`. Both install
a
global subscriber against `.init()`, so they cannot be made
order-independent,
and neither asserted anything — see §2.
`test_resource_key_value_creation` was
deleted for the same reason; it asserted that
`opentelemetry::KeyValue::new`
stores the key it was given.
## Measured and rejected
Recording these so nobody repeats them.
**Moving route registration into the crates that own the handlers.** 317
of the
371 routes were registered in one 622-line function in
`openobserve-api-http`.
Splitting them out to api-management (234), api-search (46),
api-pipelines (23)
and api-ingest (9) worked as designed — api-http went 145.0s → 63.0s —
but the
cold build only improved 654.4s → 637.7s. Splitting across a crate
boundary
**copies** work rather than relocating it: without LTO each crate
instantiates
the axum/tower machinery for itself, so the four receiving crates grew
222.1s
combined and total CPU rose 190.8s. It also just relocated the
bottleneck —
api-management became the new tail at 135.1s, with api-search finishing
only
6.9s behind it, so the tail is a plateau rather than a peak. Not carried
in this
branch.
| Tried | Result |
|---|---|
| sccache | 97.4% hit rate, **0** wall-clock gain; first fill +54% |
| Remove aws-lc-sys from the critical path | **0** — its `dur=72.0s` in
`--timings` is contention, it needs 15.6s alone |
| `codegen-units = 16` on the glue crates | **31.4s slower**, CPU
+297.8s — `lto` defaults to `false`, so thin-local LTO is on and its
cost scales with CGU count |
| Trim DataFusion features | `datafusion-core` depends on
`datafusion-functions` without `default-features = false`, so the
expression features cannot be turned off from the facade at all; what
remains is worth 13.8s of CPU (0.25%) |
A caveat on reading `cargo --timings`: a unit's `duration` is **not**
its
intrinsic cost — it is inflated by CPU contention when the machine is
saturated.
The aws-lc entry above is the clearest example.
## Verification
At this revision:
- `cargo check --all-targets` clean on OSS — 0 errors, 0 warnings — and
each
commit checks clean standalone, so `git bisect` works
- `cargo test -p web` passes
Carried over from the revision these changes were first written against
(the
edits since are doc comments, the `ui_routes` parameter rename, two
deleted
tests, a `Cargo.lock` revert, and the `cfg` narrowing in §3 — which is a
no-op
when `enterprise` is on):
- `cargo check` clean on the enterprise feature combination
- Cold release build succeeds
- Smoke-tested against a running binary: `/healthz` 200, UI and API
routes
resolve, unknown paths 404
The `<base href>` rewrite — the one runtime behaviour §1 touches — is
now
pinned by `base_href_is_rewritten_to_the_mount_point`, and checked end
to end
against the real embedded `index.html`: `ui_routes("/abc/web/")` serves
`<base href="/abc/web/" />`, matching what the old per-request
`format!("{base_uri}/web/")` produced for `ZO_BASE_URI=/abc`. The
committed test
drives a synthetic HTML response instead, so it still asserts something
in
builds where `web/dist` is absent. Verified by mutation: breaking the
rewrite
fails it.
|
||
|
Hengfei Yang
|
2bf19b2b44 |
feat(mcp): support MCP spec revision 2026-07-28 (stateless, dual-era server) (#13527)
Design at: #13526 ## Summary Implements MCP specification revision **2026-07-28** (stateless protocol) alongside the existing 2025-11-25 behavior on the same endpoint, following the spec's dual-era server model: requests carrying modern per-request `_meta` are served per 2026-07-28, while `initialize` selects legacy semantics. Legacy clients are unaffected. ## Changes - **Version negotiation** (`src/api/http/.../mcp/mod.rs`, `src/mcp/src/protocol.rs`): the `MCP-Protocol-Version` header is now validated against a supported-version set (`2026-07-28`, `2025-11-25`) instead of strict equality — the old check would have rejected every client that upgrades to the new revision. Unsupported versions (header or body `_meta`) get `400` + JSON-RPC `-32022` (`UnsupportedProtocolVersionError`) listing the supported versions so clients can retry. - **`server/discover`** (`src/mcp/src/protocol.rs`): new RPC required by 2026-07-28, returning `supportedVersions`, `capabilities`, server identity in `_meta`, `instructions`, and `ttlMs`/`cacheScope` cache fields. - **Modern result decoration** (`src/mcp/src/protocol.rs`): requests declaring `io.modelcontextprotocol/protocolVersion: 2026-07-28` in `_meta` get `resultType: "complete"` and `_meta` server identity on results, plus `ttlMs`/`cacheScope` on `tools/list`. Legacy responses carry none of these fields — byte-for-byte unchanged. - **Request metadata headers** (`src/api/http/.../mcp/mod.rs`): modern requests must mirror body fields into `Mcp-Method` / `Mcp-Name` headers (including the Base64 sentinel encoding for `Mcp-Name`); missing or mismatched headers are rejected with `400` + JSON-RPC `-32020` (`HeaderMismatch`), per the transport's server-validation rules. - **HTTP status mapping**: unknown methods on modern requests return `404` + `-32601` so dual-era clients can distinguish a modern server from a legacy one; version errors return `400`. Legacy requests keep errors in-band with `200` as before. - **Legacy paths untouched**: `initialize`, `ping`, the GET SSE keepalive channel, DELETE no-op, and notification → `202` handling are unchanged. Tool ordering was already deterministic (sorted at init), satisfying the new caching guidance. ## Testing - `cargo test -p openobserve-mcp` — 94 tests pass (new coverage: `server/discover` shape, unsupported-version error payload, modern decoration, legacy non-decoration, sentinel decoding, version set). - `cargo test -p openobserve-api-http --lib handler::http::request::mcp` — 13 tests pass (new coverage: modern happy path, missing/mismatched `Mcp-Method`/`Mcp-Name`, header↔body version mismatch, missing version header on modern requests, modern unknown method → 404, legacy header accepted). - `cargo clippy` clean on both crates. |
||
|
|
7d212f4621 |
feat: workflows incident connection (#13472)
This connects incidents to workflows, so any lifecycle event of incident will trigger workflow. design at: https://github.com/openobserve/designs/blob/main/infrastructure/workflows/design.md --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> Co-authored-by: Shrinath Rao <shnath@openobserve.ai> |
||
 Bhargav
|
e9596a10a3 |
feat: Mobile rum sdk v2 (#13503)
Design at: design/mobile-rum-sdk-evaluation/design-docs Backend changes exist to make the OpenObserve RUM intake accept the mobile SDK wire format alongside the existing browser format. src/api/src/handler/http/request/rum/ingest.rs 200 → 202 Accepted on /rum, /logs, /replay | ---> OpenObserve-family SDKs (which the mobile forks derive from) treat any non-202 as an unexpected error and drop the batch. #[serde(default)] on Event ---> Mobile replay omits creation_reason / index_in_view that the browser format sends. Multi-segment replay handling ---> Browser sends 1 segment + 1 event object; mobile sends N segment parts + one event JSON array. Handler now collects all segments, parses event as array-or-object, pairs them, emits NDJSON → one _sessionreplay row per segment. MAX_SEGMENTS=200, 16 MB decompressed cap via .take() ---> Decompression-bomb guard, newly needed now that N segments are accepted. |
||
|
udevandsai nikhil kethe
|
24938430db |
fix(evals): ensure o2_ingest_ts in defined schema (#13506)
commit-id:e9e87fba --- **Stack**: - #13506 ⬅ - #13491 - #13481 - #13453 - #13428 --- ⚠️ *Part of a stack created by [spr](https://github.com/ejoffe/spr). Do not merge manually using the UI - doing so may have unexpected results.* --------- Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> |
||
|
Harsh Mahajan
|
542e3420d5 | chore: incident small fix (#13493) |