mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-05 00:09:35 +00:00
6ca0eec1075053d4e97b35dfbf8ededcdcd1791a
3355
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Hengfei Yang
|
6ca0eec107 |
fix: MCP tool calls return 404 when ZO_BASE_URI is set (#13844)
Fixes #13845 ## What changed The MCP shared HTTP client (used to execute tool calls against the local REST API) now includes `ZO_BASE_URI` in its base URL: `http://localhost:{port}{base_uri}` instead of `http://localhost:{port}`. ## Why When `ZO_BASE_URI` is set (e.g. `/abc`), all HTTP routes are mounted under that prefix. The MCP endpoint itself was reachable (`POST /abc/api/default/mcp` → 200), but every tool execution failed: the internal client sent requests to the unprefixed path and got a 404. ``` INFO rmcp_openapi::http_client: Final URL: http://localhost:5080/api/default/streams INFO config::axum::middlewares::access_log: 127.0.0.1 "GET /api/default/streams HTTP/1.1" 404 ... ``` ## Notes for reviewers - `cfg.common.base_uri` is already normalized at config load to have no trailing slash (empty when unset), so the empty-prefix behavior is unchanged. - The join semantics are safe for path-prefixed base URLs: rmcp-openapi's `with_base_url` appends a trailing slash and `build_url` strips the tool path's leading slash before `Url::join`, covered by the crate's `test_build_url_with_base_url_containing_path`. So `/abc` + `/api/{org}/streams` → `/abc/api/{org}/streams`. - The other `localhost:5080` occurrence in `tools.rs` is a test-only mock client; the OAuth protected-resource metadata handler already includes `base_uri`. - `cargo check -p openobserve-mcp` and `cargo test -p openobserve-mcp` pass. |
||
|
Hengfei Yang
|
9d76505882 |
feat(traces): resolve session time ranges through the trace time index (#13837)
Design at: [https://github.com/openobserve/designs/blob/main/infrastructure/file-list/session-time-index.md](https://github.com/openobserve/designs/blob/main/infrastructure/file-list/session-time-index.md) ## Summary - carry an optional `session_id` attribute on trace time index records, resolved from span attributes by the existing `SESSION_ID_COLUMNS` precedence; the first non-empty value in the request wins for the whole trace, stamped onto all of its hour-bucket records - keep the aggregation key `(hour, trace_id)` unchanged; hourly compaction picks the attribute up with `MAX(session_id)`, which ignores NULLs - reuse the locate/expand timer for sessions with a 7-day expand window (traces keep 24h); a UUID v7 session id doubles as a probe anchor and tightens the scan's left bound to the session's lifetime neighborhood - aggregate session lookups per trace, so a lookup resolves the session's range plus every member trace's real range - reshape the time-range endpoint to query parameters so one endpoint serves both lookup kinds - make the session-details caller range optional, union it with the indexed range (never narrowed), take trace ids from the index, and keep the legacy source-stream scan for pre-coverage data - add `session_id` to the default bloom filter fields (gated by the existing `ZO_FEATURE_DEFAULT_INDEX_FIELDS_ENABLED`, folds into the secondary-index field set like `trace_id`); streams without the column build nothing ## Why Session detail lookup depends on a caller-provided time range: the sessions list passes its own picker window into the details request, so an LLM session that started before the window silently loses its earlier turns, and a deep link without a range is a hard 400. The trace time index already turns a `trace_id` into a time range at bounded cost; a session is the same problem one level up — `session_id` → range plus member traces. ## Environment variables No new variables. The existing `ZO_TRACE_TIME_INDEX_ENABLED` (default `true`) gates the feature, session column included. The existing coverage marker is reused; there is no historical backfill. This change must ship in the same release as #13827: reusing the coverage marker is valid only while no release separates the two. ## API changes All timestamp parameters and returned ranges use microseconds. ### Time-range endpoint reshaped ```http GET /api/{org_id}/{stream_name}/traces/time_range?trace_id=xx&hint_ts=... GET /api/{org_id}/{stream_name}/traces/time_range?session_id=xx&hint_ts=... ``` Exactly one of `trace_id` / `session_id` is required; zero or both is `400`. This replaces `GET /{stream_name}/traces/{trace_id}/time_range`; the old path form is removed without a compatibility alias since #13827 has not shipped in a release. ### Session details: range becomes optional `GET /api/{org_id}/{stream_name}/traces/session/details` now accepts `start_time` / `end_time` as an optional pair plus an optional `hint_ts` (defaulting to the caller range's midpoint). The effective range is the union of the caller range and the indexed session range, so it never narrows what the caller asked for; `400` only when neither side yields a range. When the caller range lies entirely inside index coverage the source-stream trace-id scan is skipped; otherwise it still runs over the caller range and its results are merged, so behavior is never worse than before. No frontend changes are required — the frontend keeps sending its list window and the union completes the session. ## Validation - `cargo test -p openobserve-core traces::time_index --lib` — 13 passed - `cargo test -p search datafusion::merge --lib` — 5 passed, including NULL-mixed session aggregation and pre-column file compatibility - `cargo test -p openobserve-api-search --lib` — 149 passed - `cargo test -p openobserve-api-http --lib` — 53 passed, including OpenAPI/MCP initialization - `cargo test -p config router:: --lib` — 8 passed - `cargo test -p infra db::trace_time_index --lib` — passed - `cargo clippy` on all touched crates (`--all-targets`) and `cargo check --workspace` — clean - the enterprise repository has no references to any changed API and no enterprise-gated code was modified |
||
 Julián ValdésandHengfei Yang
|
06c6b515fb |
fix(stats): only read file_list_dump_stats when dumping is enabled (#13830)
Fixes #13829. ## Problem `file_list_dump_stats` is created **conditionally** — `handle_partitioned_tables` only adds it when `ZO_COMPACT_FILE_LIST_DUMP_ENABLED` is set, and that flag defaults to `false`. But `update_stats_from_file_list_for_stream` reads it **unconditionally**. On a PostgreSQL metastore where the table was never created: ``` [STATS] run update stream stats from file list error: SqlxError# error returned from database: relation "file_list_dump_stats" does not exist ``` **The failure is wider than the dump.** The `?` propagates before `set_stream_stats`, so the entire stats update is aborted — every stream reports `doc_num: 0` while ingestion, search and compaction keep working. It reads as a UI problem rather than a failing job, which is what makes it expensive to track down. SQLite creates the table unconditionally, so this only surfaces on deployments with an external PostgreSQL metastore. ## Change Gate the read on the same flag that gates the creation. One call site — `query_dump_stats_by_date_range` has no other caller. When dumping is off there is nothing to merge, so the stats are unchanged: `merge` over an empty result was already a no-op. The difference is that the surrounding update now completes instead of aborting. ## Alternative considered Always creating the table would also fix it, at the cost of an empty partitioned table on every deployment. Gating the read seemed closer to the existing intent — the table is deliberately optional — but I am happy to flip it if you prefer the schema to be unconditional. ## Testing Reproduced on v0.92.1 with `ZO_META_STORE=postgres` and the flag at its default: stats fail every cycle and all counters sit at zero. Creating the table by hand with the DDL from `create_partitioned_dump_stats_table` makes the error and the zeros go away, which confirms the missing table is the whole cause. I have not been able to build the project locally to run the test suite — no Rust toolchain on the machine I reproduced this on. The change is a single conditional and the import follows the one already used in `src/compaction/src/merge.rs` (`use config::{get_config, ...}`), but it deserves a CI run before merging. Signed-off-by: Julián Valdés <julianvaldes24@gmail.com> Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com> |
||
 udev
|
35860aff9d |
fix: route annotation queue score lookups to queriers (#13801)
## Summary - route Discovery and annotation queue endpoints to queriers so score lookups, item hydration, and review reads do not fall through to ingesters - make Discovery source_stream required at the API response boundary - cover Discovery, queue item detail, and review routing with regression tests The QueueItem remains reference-only. Workbench and enqueue validation derive source_stream from timestamp-bounded _llm_scores queries in the enterprise implementation. Depends on https://github.com/openobserve/o2-enterprise/pull/2423 ## Testing - 5 querier routing tests - Discovery response serialization test with the enterprise feature - 16 enterprise annotation queue tests - 14 enterprise Discovery tests |
||
|
udev
|
d770d0245a |
fix: preserve online evaluation provider API keys (#13739)
## Summary - preserve the stored provider API key when an edit submits a blank value - replace the stored key when a nonblank value is supplied - show the create-time required marker only for providers that require authentication - retain keyless creation for self-hosted providers ## Testing - ProviderFormPage Vitest suite passed: 8 tests - Targeted ESLint passed - cargo fmt check passed - git diff check passed - Focused Rust test compilation is blocked by unrelated existing alert-state model errors Related to #13655. Companion enterprise PR: https://github.com/openobserve/o2-enterprise/pull/2390 |
||
  Omkar Kesarkhaneandgithub-actions[bot]
|
ddb67f3efb |
fix(synthetics): record from a specific step, and stop silently downgrading recorded actions (#13799)
Supersedes #13777, which carried the record-from-step commits alone. Closing that in favour of this, because the two changes land in the same three files (`types/synthetics.ts`, `useSyntheticsRecorder.ts`, `JourneySteps.vue`) and reviewing them apart means resolving the same conflict twice. ## Problem **A recorded hover was stored as a click.** Playwright 1.56 added `hover` to the recorder's action model, reachable from its "Choose action" picker. The extension's action mapper seeds every step with `action: 'click'` as a placeholder for its switch to overwrite, and returned that placeholder from `default:` — so any action name the switch did not know became a click the author never performed. On the usual hover target — a nav trigger, a dropdown opener — clicking it can navigate or submit. The run still reports green. There are three separate guards meant to stop exactly this, and none of them fired, because the relabel happens upstream of all of them: | Guard | Why it missed | |---|---| | `UNSUPPORTED_REPLAY_ACTIONS` (extension) | step already said `click` | | `RETIRED_ACTIONS` → `isStorableAction` | step already said `click` | | `V2_STEP_ACTIONS` (Rust validation) | `click` is valid | The same picker offers **Right click** and **Double click** as explicit choices. Both were flattened too: `click_count` was dropped by the mapper and never existed downstream, and the probe's executor called `locator.click({ timeout })` while reading neither `button` nor `click_count`. A deliberately recorded right click replayed as a left click; a double click as a single one. **Recording could only start from scratch.** Fixing step 12 of a 15-step journey meant re-recording all 15. ## Change **Record from a specific step.** Each step gets a record action that replays the prefix before it, then flips the recorder into insert mode at that point. A restore that cannot reach the requested step reports where it stopped (`prefixFailed`) rather than silently recording from the wrong place. **Every action Playwright can record is now stored and replayed as itself.** - `hover` is a first-class v2 action — mapped, stored, validated, and executed by `Locator.hover` in the probe and `Frame.hover` in the extension player. - `button` and `click_count` travel to storage, so right click and double click replay as recorded. - An action name the mapper does not know now **throws** instead of returning the `click` placeholder. This is the part that is not hover-specific: upstream added one action in nine minors and will add more, and the next one must be a deliberate decision rather than a silent downgrade. **The extension handshake reports capabilities, not just presence.** `getStatus` now returns `extVersion` and a `capabilities` array; O2 gates affordances on `hasCapability()` with a defined pre-handshake fallback, and warns when the installed build is older than the minimum. Previously the protocol carried no version at all — Chrome updates the extension independently of when O2 deploys, so the two could disagree about the wire with no way to say so. **Two dead message declarations, opposite problems.** `setSources` no longer exists in playwright-core (sources arrive alongside the actions), so its declared variant could never fire — while its `generatedCode` field is exactly what an eject-to-code feature would reach for. Deleted. `elementPicked` is the reverse: it fires on every "Pick locator" and every click in inspecting mode, and was being discarded, which made inspect mode a dead end. Now kept. ### Compatibility `button` and `click_count` are optional with defined absent-behaviour — absent means a single left click. Every journey stored today stays valid, means the same thing, and serialises byte-identically. **No schema version bump and no migration.** A test pins that rather than leaving it as an intention. `hover` is a new vocabulary entry with no default, so the probe must be able to execute one before O2 is allowed to store one — the probe change ships first. ## Implementation | File | | |---|---| | `src/config/src/meta/synthetics.rs` | `hover` joins `V2_STEP_ACTIONS`/`V2_ELEMENT_ACTIONS`; `button` and `click_count` added to `BrowserStepV2` with validation (`left\|middle\|right`, `1..=3`) | | `web/src/types/synthetics.ts` | `button`/`clickCount` on both step types; `extVersion` + `capabilities` on `RecorderStatus`; `setSources` variant deleted and `elementPicked` typed; `prefixFailed` and insert-mode fields on `recordingStarted` | | `web/src/utils/synthetics/mapRecordedStep.ts` | promotes `button`/`clickCount` from the wire, like every other stored field | | `web/src/utils/synthetics/buildV2Steps.ts` | `hover` in `ACTION_TO_V2`; crosses camelCase → snake_case for the two click fields | | `web/src/constants/synthetics.ts` | `hover` out of `RETIRED_ACTIONS`; `MIN_EXTENSION_VERSION` | | `web/src/composables/useSyntheticsRecorder.ts` | `startRecordingFrom`, `hasCapability`, `pickedSelector`, `extensionOutdated` | | `web/src/components/synthetics/journey/{BrowserJourney,JourneySteps}.vue` | per-step record action and its wiring | `click_count` is capped at 3: the recorder only ever emits 1 or 2, and an unbounded count is a way to hold a page's event loop open. The two click fields cross from camelCase to snake_case in `buildV2Step` and nowhere else — reading `step.wire` directly inside it would have worked and invented a second path into storage. ### Depends on a recorder extension build `hover`, the click fields and the capability handshake all need the matching extension build. O2 degrades honestly without it: `hasCapability()` falls back to the pre-handshake set, and an older build raises the outdated-extension warning instead of failing on a message shape it does not recognise. ## Tests - **Rust** — hover accepted as an element action; `button`/`click_count` validated at both bounds; a step carrying neither still validates (the compatibility guarantee). Two existing tests encoded "hover is rejected" and now say the opposite. 101 pass. - **Probe** — hover hovers rather than clicks; a right click sends the recorded button; a double click sends two clicks; a journey stored before these fields replays unchanged. The action union is asserted against the Rust vocabulary, so the next divergence fails a test instead of reaching an executor with no case for it. Also pinned: a selector carrying a frame hop resolves into the iframe, which had no coverage anywhere. - **Web** — 1213 across 60 files. Nine existing tests encoded "hover is retired" across constants, `buildV2Steps`, `buildPayload` and the save-gate schema; each now states the opposite deliberately. - **Extension** — the unmapped-action throw, hover round-tripping both directions, and a double click surviving the round trip as two clicks. `vue-tsc`, `cargo clippy -D warnings`, and `prettier --check` are clean. The last commit is formatting only: the three record-from-step files arrived failing `format:check`, which is a gate of its own and would not have been caught by lint. --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Hengfei Yang
|
41ff3cd9b6 |
feat: replace distinct_values_* streams with tantivy TopK for the _values API (#13834)
Design at: #13822 ## Summary `distinct_values_*` metadata streams existed to serve one reader: the `_values` API. They are a redundant — and lossier — materialization of what the tantivy secondary index already holds, and the index has the collectors needed to replace them outright (`SimpleTopN` / `SimpleDistinct`). This PR removes the distinct streams from the write and read paths and lets the `_values` API be served by the tantivy index optimizer. ### 1. Treat distinct value fields as secondary index fields (read-time union) - `get_stream_setting_index_fields()` — the single accessor used by both the index build side (ingester parquet/pack jobs, compactor merge) and the query side (leader/follower index optimizers) — now unions `distinct_value_fields` into the effective index field set, gated on `enable_distinct_fields`. Fields that cannot be secondary-indexed (full text search keys, partition keys, reserved columns) are skipped; the values API falls back to a normal scan for those. - The distinct list is deliberately **not merged into `settings.index_fields`**: the two stay independent, so removing a distinct value field (or disabling `enable_distinct_fields`) cleanly stops its indexing with no residue, and the user-managed `index_fields` list is never mutated behind their back. - Correctness gate: a distinct-only field is considered indexed since its `added_ts` (`get_stream_setting_index_updated_at_for_fields`), so parquet files older than that are served by DataFusion scan instead of the index (`split_file_list_by_time_range`). `added_ts` is set on add and reset when `enable_distinct_fields` is re-enabled (existing behavior). For streams configured before this release the fields only start being indexed at upgrade time, so `added_ts` is additionally clamped by a release-wide cutoff `/tantivy/distinct_fields/updated_at` — a lazy marker initialized to the first access after upgrade, following the existing `ttv_secondary_index_updated_at` pattern. No data migration and no `DB_SCHEMA_VERSION` bump needed. ### 2. `_values` API always queries the original stream - Removed `can_use_distinct_stream()` and the `distinct_values_*` / `SUM(count)` / `StreamType::Metadata` rewrite from `values_inner` (GET `/_values`, formerly `values_v1`) and `build_search_request_per_field` (POST `/_values_stream`). - The generated SQL shapes are exactly what the index optimizer detects: - `SELECT f AS zo_sql_key, COUNT(*) AS zo_sql_num FROM s GROUP BY zo_sql_key ORDER BY zo_sql_num DESC, zo_sql_key ASC` + limit → `SimpleTopN` - `no_count`: `SELECT f AS zo_sql_key FROM s GROUP BY zo_sql_key ORDER BY zo_sql_key ASC` + limit → `SimpleDistinct` - The `keyword` filter (`str_match_ignore_case`) is a valid index condition, so keyword-filtered values queries stay on the index path too. ### 3. Remove distinct values generation - Dropped the per-record distinct value collection in logs and traces ingestion and the whole `core::metadata` module (channel, mem-table, flush job, shutdown hook). - Removed the now-unused `DISTINCT_FIELDS` static and the `ZO_FEATURE_DISTINCT_EXTRA_FIELDS` / `ZO_DISTINCT_VALUES_INTERVAL` / `ZO_DISTINCT_VALUES_HOURLY` config knobs, including the `SUM(count)` merge rollup the last one gated — legacy `distinct_values_*` streams now compact as ordinary metadata streams. The default fields `DISTINCT_FIELDS` carried (`service_name`, `operation_name`) are already default secondary index fields via `SQL_SECONDARY_INDEX_SEARCH_FIELDS`, so traces `_values` queries on them remain index-served. - Renamed `values_v1` to `values_inner`: the `v2` variant was removed long ago (#5087) and the distinct stream switch inside `v1` is gone now too, so the version suffix was pure history. ### 4. Default fields after this change - `service_name`, `operation_name`: default secondary index fields (unchanged, via `SQL_SECONDARY_INDEX_SEARCH_FIELDS`). - `trace_id`: now also a built-in default **bloom filter** field (`BLOOM_FILTER_DEFAULT_FIELDS`, gated on `ZO_FEATURE_DEFAULT_INDEX_FIELDS_ENABLED`), in addition to being a default index field — bloom sits on top of the secondary index and prunes files for high-cardinality equality lookups. - There is no "default distinct fields" concept anymore. ### 5. Fix surfaced by e2e: SimpleDistinct returned null values as `""` The tantivy index encodes null/missing field values as empty strings (`index_builder` adds `""` for null docs), so the `SimpleDistinct` term-dictionary walk returned `""` as a distinct value — sorted first in ascending order, it surfaced as a blank first option in dashboard variable dropdowns, which the DataFusion `GROUP BY` path never returns (its null group sorts last). This was exposed once distinct fields became index-served; `handle_simple_distinct` now skips empty terms (with a unit test). ### 6. Drive-by: fix the failing `api_fuzz_tests` CI job `CreateAlertRequestSchema` was an untagged documentation union with two identical variants, producing `oneOf: [CreateAlertRequestBody, CreateAlertRequestBody]` in the OpenAPI spec. Under `oneOf` semantics (exactly one branch must match) no instance can ever validate, so schemathesis filtered out every generated example and `api_fuzz_tests` failed on `POST /api/v2/{org_id}/alerts` on every CI run since that schema landed. The handler deserializes `CreateAlertRequestBody` directly, so the annotation now documents exactly that type; runtime behavior is unchanged. ## What stays (intentionally) - `distinct_value_fields` / `enable_distinct_fields` in stream settings: still the user-facing way to declare values-API fields; the list now drives the read-time index-field union described above. - The `distinct_value_fields` DB table that tracks dashboard/report usage of these fields (still gates removal in the settings API). - `DISTINCT_STREAM_PREFIX` helpers, retention sync and stream-deletion cleanup for existing `distinct_values_*` streams — legacy stream data keeps compacting (as ordinary metadata streams) and ages out via retention; no new records are written to them. ## Testing - `cargo check --workspace --all-targets` and `cargo clippy --workspace --all-targets` clean. - Enterprise build verified with `Cargo.toml.openobserve` (`cargo check --all-targets`). - New unit tests for the index-field union (skip rules for FTS/partition/reserved fields, `enable_distinct_fields` gating) and the `added_ts` / release-cutoff indexed-since resolution; schema, infra, config and api-search unit tests pass. A few pre-existing failures on a clean tree (config `ssrf_guard`, one flaky sqlite-backed destinations test, one `.env`-dependent index-fields test) are local-environment issues and unrelated. Closes #13822 |
||
 Harsh Mahajan
|
7f74925605 | feat: banner as notification (#13667) | ||
 Prabhat Sharmaandgithub-actions[bot]
|
0dba67d638 |
feat: composite alerts (#13797)
Design at : alerts/composite_alerts/composite_alerts.md ## Summary - add composite alert graph persistence, evaluation, scheduling, lifecycle APIs, and deletion safeguards - add composite alert UI forms, preview, references, list/detail integration, and localization - add Rust, API, Vue, and Playwright contracts ## Verification - cargo fmt --all -- --check - cargo check -p openobserve-core - cargo check -p openobserve-api-management - npm run type-check:composite-alerts - npm run type-check:app - npm run lint:ci -- --quiet - design/token/style checks - targeted Rust/Vitest suites - browser QA against local alerts UI ## Follow-ups (intentionally deferred, documented in the spec) - `RunOutcome::Error`/`NotifyFailed` outcome records on composite delivery failure (§11.5) - Incident detail live-definition resolver for composites + "definition unavailable" retention (§9.6) - `alert_composite_*` observability metrics (§16) - SLO/stream delete-conflict 409 bodies with readable-vs-hidden parent split (§10.3) - Super-cluster startup preflight (§18) <img width="1650" height="979" alt="image" src="https://github.com/user-attachments/assets/3def5246-7bf7-48fe-9a42-c0ac08271ea9" /> --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
 TessaIO
|
30b938a1ed |
feat: introduce compactor ZO_COMPACT_DELETE_FILES_DELAY_MINUTES (#13828)
Resolves #13819 Signed-off-by: TessaIO <ahmedgrati1999@gmail.com> |
||
|
Hengfei Yang
|
3fda083f51 |
feat(traces): add trace time index (#13827)
Design at: [https://github.com/openobserve/designs/blob/main/infrastructure/file-list/trace-time-index.md](https://github.com/openobserve/designs/blob/main/infrastructure/file-list/trace-time-index.md) ## Summary - add an hourly Metadata time index per trace stream to resolve a trace ID to its full time range - add external time-range and trace-details APIs, including bounded locate/expand behavior and standard search responses - integrate index generation into trace ingestion and preserve query visibility through the synchronous Metadata write path - synchronize retention and deletion for associated Metadata streams - reuse the shared trace stream permission check across trace APIs ## Why Trace detail lookup currently depends on a caller-provided time range, which can omit spans outside that window. The time index provides a bounded, best-effort range lookup for `stream + trace_id` without scanning the full trace stream. Closes #13823 ## Environment variables | Variable | Default | Effect | | --- | --- | --- | | `ZO_TRACE_TIME_INDEX_ENABLED` | `true` | Enables per-trace-stream Metadata indexes, index writes during trace ingestion, and indexed time-range lookup. Only traces ingested after the feature is enabled are indexed; there is no historical backfill. | | `ZO_QUERY_TIMEOUT` | 600 seconds | Existing setting reused as the cumulative Trace Timer locate/expand deadline and as the default timeout for the trace-details search. | When enabled, each traces stream can create a related Metadata stream named `trace_time_index_{stream}`. Its retention settings follow the source stream, and it is removed with the source stream. ## API changes All timestamp parameters and returned time ranges use microseconds. ### Get a trace time range ```http GET /api/{org_id}/{stream_name}/traces/{trace_id}/time_range ``` Query parameters: - `hint_ts` (optional): timestamp hint used to start the index lookup Returns the source stream, trace ID, and the resolved `start_time` / `end_time`. The `range` field is `null` when the index cannot resolve the trace. ### Get trace details ```http GET /api/{org_id}/{stream_name}/traces/{trace_id}/details ``` Query parameters: - `start_time` and `end_time` (optional, but must be supplied together): caller-provided range - `hint_ts` (optional): timestamp hint for the time-index lookup - `timeout` (optional): trace-details search timeout in seconds The endpoint unions the caller range with the indexed range, so it never narrows a caller-provided range. If neither the caller range nor the index can provide a range, it returns `400`. The response uses the standard search response and includes the effective range in `new_start_time` and `new_end_time`. ## Validation - `cargo test -p openobserve-api-search --lib` — 145 passed - `cargo test -p openobserve-core traces::time_index::tests --lib` — 7 passed - `cargo test -p infra db::trace_time_index::tests::test_coverage_start_cache -- --exact --nocapture` — passed - `cargo check -p openobserve-api-search` - `cd web && npm run type-check:app` - end-to-end ingestion and query validation with the data generator ## Validation notes - targeted ESLint could not start because the local web dependencies are missing `@eslint/css` - enterprise-only compilation could not be completed from this OSS checkout because the local enterprise overlay is unavailable |
||
  
|
dbee572722 |
fix: slo bugs (#13784)
closes https://github.com/openobserve/o2-enterprise/issues/2397 --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Harsh Mahajan
|
0ed5336bf1 | chore: adding HA mode for o2-ai agent (#13525) | ||
 Huaijin
|
3901c548a2 |
perf(promql): use Utf8View for metric labels (#13796)
## Summary
- read PromQL metric label columns as Arrow `Utf8View` when the existing
`ZO_UTF8_VIEW_ENABLED` setting is enabled
- preserve `__hash__` and exemplars in their original representation
while retaining schema metadata
- make PromQL equality, negative, and regex matchers type-aware for
`Utf8View`; cast regex operands to `Utf8` to avoid the DataFusion 54
mixed-type rewrite issue
- support `StringViewArray` in series-label loading and keep
compatibility with the process-wide series-label cache
No PromQL-specific environment variable is introduced, and metrics
continue to use the existing global `ZO_BATCH_SIZE`.
## Performance
For the filtered histogram query below, a controlled local A/B using the
same release binary and identical settings except Utf8View measured P50
latency improving from 0.964s to 0.677s, about 29.8%:
```promql
histogram_quantile(0.9, sum by(le, path) (rate(codelab_api_request_duration_seconds_bucket{path="/api/service-1"}[5m])))
```
Test range was one hour with `step=15` and `use_cache=false`.
## Testing
- `cargo fmt --all -- --check`
- `cargo test -p promql --lib utils::tests` (10 passed)
- `cargo test -p promql --lib series_labels::tests` (11 passed)
- `cargo test -p search --lib
datafusion::exec::tests::test_metrics_query_schema` (2 passed)
- `cargo check -p promql-service -p promql`
|
||
|
Hengfei Yang
|
6a2dfbb2b4 |
fix(compact): stop retention from deleting one extra day (#13804)
Fixes #13800 ## Problem With a 7 day retention the stream only kept ~6 days of data. From the issue: retention is 7 days, but the stream data range starts at `2026-08-07T00:00 UTC` while now is `2026-08-13T04:52 UTC` — the whole `2026-08-06` partition is gone. Retention deletes data at day granularity (`delete_by_date` wipes a full `[day, day+1)` partition), but the job generator stepped through the days with: ```rust while start < time_range.end { ... } // time_range.end == lifecycle_end ``` `lifecycle_end = now - retention_days` is a point in the middle of a day, so the day it falls into also matched the condition and got a delete job. Deleting that partition also removes everything between `lifecycle_end` and the end of that day, which is still inside the retention period. Effective retention became `(N-1, N]` days instead of `[N, N+1)`. This is a regression from #8058, which switched retention from one job per range to one job per day. Before that change the range end was formatted as `%Y-%m-%d`, so the boundary day was never included. The day-aligned fast path in `generate_jobs()` already assumes the boundary day is retained — it skips the stream when `doc_time_min >= retention_day_start` — so the two paths disagreed today. ## Fix Only generate delete jobs for days that lie fully before the retention boundary. Extracted the day splitting into `generate_deletion_dates()` so it can be unit tested without a database. Retained window goes back to `[N, N+1)` days: with a 7 day retention the stream now holds 7–8 days of data depending on the time of day, which is the expected behaviour for day-granular deletion. Day-aligned range ends (the ranges produced by extended retention splitting) are unaffected — `start + 1day <= end` and `start < end` agree there. ## Tests Three new unit tests for `generate_deletion_dates`: a mid-day boundary (the regression case), a day-aligned end, and ranges shorter than a day. ``` test retention::tests::test_generate_deletion_dates_keeps_the_day_of_the_retention_boundary ... ok test retention::tests::test_generate_deletion_dates_with_day_aligned_end ... ok test retention::tests::test_generate_deletion_dates_less_than_a_day ... ok ``` `cargo clippy -p compaction --tests` and `cargo fmt --check` are clean. `test_generate_retention_job` / `test_delete_all` fail locally on `main` as well (sqlite data dir missing in my checkout), unrelated to this change. ## Note on the `time_range_start >= time_range_end` guard The removed loop had a `continue` when the two formatted dates compared equal, but it could never fire: both dates come from the same `start` variable, formatted before and after `start += day_micros(1)`, and in UTC adding 24h always advances the calendar date. Simulating the old loop with the numbers from the issue (`lifecycle_end = 2026-08-06T04:52 UTC`) produces jobs for `08-04`, `08-05` **and `08-06`** — the guard never triggers. It was meaningful before #8058, where the two dates came from the two ends of the range (`format(range.start)` / `format(range.end)`) and could land on the same day — and where `format(lifecycle_end)` as the exclusive end is exactly what used to keep the boundary day alive. Second commit removes it as dead code. |
||
|
Hengfei Yang
|
3d0085c43e |
refactor(traces): remove the trace_list_index feature (#13809)
## What Removes the `trace_list_index` metadata stream and everything that fed it. ## Why On every trace ingestion, `write_traces` built one `TraceListItem` per span (`_timestamp`, `stream_name`, `service_name`, `trace_id`) and wrote the batch into a per-org `trace_list_index` metadata stream — creating the stream, its schema, a bloom filter on `trace_id`, and (on enterprise) an OpenFGA ownership tuple on first write. Nothing ever read it back. A repo-wide search for the stream name and every associated symbol turns up only the writer itself: no query path, no API handler, no UI reference, and no reference in the enterprise tree. The feature was pure write amplification on the trace ingestion hot path, plus a stream per org that users had to see and pay retention on. ## Changes - Delete `src/core/src/metadata/trace_list_index.rs` (the `TraceListIndex` writer, its schema/settings bootstrap, and its tests). - Drop `MetadataItem::TraceListIndexer` and `MetadataType::TraceListIndexer`, and their dispatch arms in `metadata::write` / `MetadataManager::close`. - Remove the per-record index build and the trailing write call in `write_traces`. The `service_name` lookup there had no other consumer, so it goes too. - Remove the now-unused `ZO_TRACES_LIST_INDEX_ENABLED` config option. - `MetadataItem` is now single-variant, so the refutable `if let` in `distinct_values::write` becomes a plain destructuring `let` (an `if let` there would warn as irrefutable). ## Notes - **Existing data is untouched.** Already-ingested `trace_list_index` streams stay on disk and remain visible under the Metadata stream type; after this change nothing writes to them. They can be dropped through normal stream deletion. Say the word if you'd rather this ship with a cleanup migration. - `ZO_TRACES_LIST_INDEX_ENABLED` is removed rather than deprecated. Deployments that set it will simply see it ignored — `env_config` does not error on unknown `ZO_*` vars. ## Testing - `cargo check --workspace --all-targets` — clean, no warnings. - `cargo test -p openobserve-core --lib metadata::` — 22 passed. - `cargo test -p openobserve-core --lib traces::` — 378 passed. - `cargo test -p config --lib` — 3148 passed, 2 failed: `utils::ssrf_guard::test_async_validator_rejects_literal_private` and `..._resolves_hostname_to_private`. Both are DNS-dependent and fail identically on `main` at `d1f8941512`, so they are unrelated to this change. - The `enterprise` feature could not be compiled locally (the `o2_enterprise` submodule is a stub here, and `openobserve-cipher` fails to build against it independently of this change). The only enterprise-gated code in scope lived inside the deleted file, and the enterprise tree has no references to the feature. |
||
 
|
7f647109fc |
fix: preserve projection schema when rewriting match_all (#13808)
## Summary Forward-port of #13798 to `main`. - rebuild `ProjectionExec` expressions against the rewritten child schema before DataFusion validates the projection - carry full-text search fields through intermediate projections and preserve the original `FilterExec` output schema - reject a full-text search field that a projection already exposes as an alias for a different expression, instead of silently searching the wrong data - add a regression for a projection sitting between the filter and the datasource ## Root cause `RewriteMatchPhysical` expanded the `NewEmptyExec` projection with full-text search fields. When an optimizer rule (CSE, for example) had inserted an intermediate `ProjectionExec`, adding the FST field changed the child column positions while the projection still referenced its columns by their old indices, so DataFusion rejected the plan. The old `filter_projection` was also computed as positions inside `parquet_projection`, i.e. in the datasource output space. That is only valid when the filter reads the datasource directly. It is now mapped by name against the actual filter input schema, and the field order of `filter.schema()` is preserved so the operators above the filter keep reading the same columns. Two latent issues are fixed along the way: - a scan with `projection == None` (all columns) was narrowed down to only the FST fields - `schema.index_of(...).unwrap()` in the datasource rewrite panicked instead of returning a plan error ## Differences from #13798 `main` is on DataFusion 54 and builds the filter through `FilterExecBuilder`, so step 5 uses `apply_projection` and keeps `with_fetch`. The regression test also differs. The CTE query used in #13798 no longer produces an intermediate projection on DataFusion 54 — CSE does not extract `btrim(log)` there — so the test would be vacuous. `test_projection_between_filter_and_scan` builds the offending plan shape directly instead, which does not depend on optimizer internals. ## Verification - `cargo test -p search --no-default-features --lib datafusion::optimizer::physical_optimizer::rewrite_match` — 7 passed, 0 failed - `test_projection_between_filter_and_scan` fails on `main` without the fix (`SchemaError: Unable to get field named "__common_expr_1"`) --------- Co-authored-by: Neha P <neha@openobserve.ai> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Hengfei Yang
|
83d562325c |
fix(alerts): type the aggregation HAVING clause from the aggregate, not the column (#13790)
Fixes #13786 ## Why Aggregation alerts that worked before 0.92.0 now fail at query time: ``` SELECT host, COUNT("host") AS alert_agg_value, ... GROUP BY host HAVING "alert_agg_value" < '1.0' Optimizer rule 'simplify_expressions' failed caused by Arrow error: Cast error: Cannot cast string '1.0' to value of Int64 type ``` ## Root cause Two defects in `build_sql`, both introduced by #13547 (first shipped in 0.92.0), compound: 1. `having_filter_value` returns an `f64`, so a threshold of `1` becomes `json!(1.0_f64)`. The workspace enables serde_json's `arbitrary_precision`, so the number is stored as the literal text `"1.0"` — never `"1"`. 2. `build_expr` was handed the data type of the **source** column (`agg.having.column`, i.e. `host`: Utf8) rather than the type of the aggregate. `COUNT("host")` is Int64, but the Utf8 branch quoted the literal as `'1.0'`, which DataFusion then failed to cast. Before #13547 the un-widened `agg.having` value was passed straight through, so the clause read `< '1'` — a string DataFusion casts to Int64 without complaint. Hence the reporter's "it worked before 0.92.0". ### The same change also silently zeroed thresholds on numeric columns `build_expr`'s integer branch called `as_i64().unwrap_or_default()`, and `as_i64()` returns `None` for a float-serialized number (`N::Float(_) => None`; under `arbitrary_precision`, `"100.0".parse::<i64>()` fails too). So an alert on `AVG(response_time) > 100` quietly became: ```sql HAVING "alert_agg_value" > 0 ``` No error, no log line — just a threshold replaced by zero, firing on every evaluation (or never, for `<`). This affects more alerts than the reported cast error, and does so invisibly. ## Fix - **`having_data_type()`** resolves the comparison type from the aggregate's OUTPUT rather than the column: `COUNT` → Int64, `AVG`/`MEDIAN`/`P50`–`P99` → Float64, `MIN`/`MAX`/`SUM` → the column type (they do return their input type, so string min/max keeps its string comparison). - **`build_having_expr()`** extracts the widening out of `build_sql` so it is unit-testable without an `infra::schema` fetch, and normalizes an integral f64 threshold back to an integer `Number`. That keeps the Utf8 min/max path emitting `'1'`, byte-identical to pre-0.92. - **`build_expr`'s integer branch** falls back to `as_f64()` instead of defaulting to 0. This also covers plain WHERE conditions, which had the same silent-zero hazard. A fractional value stays fractional rather than being truncated; a non-numeric value errors instead of defaulting. The multi-alert path was never affected — `severity_order_sql` formats its f64 as a bare numeric literal. The reporter's alert now generates `HAVING "alert_agg_value" < 1`. ## Tests Nine unit tests, including the exact #13786 shape (COUNT over a Utf8 column), the silent-zero regression, the widened warning threshold, and the full `having_data_type` mapping. `cargo clippy` and `cargo fmt` clean. > [!NOTE] > One pre-existing failure in the same suite, `alerts::destinations::tests::test_alert_destination_template_is_optional`, fails identically on unmodified `main` (SSRF guard rejects `127.0.0.1`) and is unrelated to this change. ## Note for the reporter The alert in #13786 will now execute without error, but it still won't fire: `GROUP BY host HAVING COUNT("host") < 1` cannot match, because any group keyed by a non-null `host` has a count of at least 1. Detecting "this host stopped logging" needs a different alert shape, not an aggregation threshold. --- ## Second commit: comment cleanup (no behaviour change) `alerts_2.md` does not exist anywhere in the tree, so every "see §4.4, option B" pointer in this file sent the reader looking for a file they cannot open. Same for the bare requirement tags — `M-6`, `M-9`, `D8`, `T-9` — which name nothing a reader of this repo can resolve. Each reference is replaced with the reasoning it was standing in for, in plain terms: | was | now | | --- | --- | | `alerts_2.md §4.4, option B` | what widening the HAVING clause does, and why the critical threshold alone would lose the warning band | | `the M-6 cap` | `the max-groups cap` | | `only resolve via M-7's timeout` | `resolve later via the reaper timeout` | | `T-9: label the worst SERIES` | just the behaviour being described | Kept as a separate commit from the fix so the fix stays reviewable on its own. > [!NOTE] > 81 references to `alerts_2.md` remain in other files under `src/` (mostly `src/config/src/meta/alerts/`). Happy to clean those up in a follow-up. |
||
 Loakesh Indiran
|
4bb0908679 |
fix(synthetics): let the deployment ceilings survive a config reload (#13785)
Companion to o2-enterprise PR [#2375](https://github.com/openobserve/o2-enterprise/pull/2375). **Merge this first** — that branch does not compile without `set_limits`. ## The bug `job_lease_secs`, `max_check_budget_secs` and `max_net_timeout_ms` are declared in `o2_enterprise`'s `SyntheticsConfig` and pushed in here, because this crate cannot depend on `o2_enterprise`. The holder was a `OnceLock`, and `OnceLock::set` is a no-op after the first write. Boot wrote it once. A config reload is the second write, so it was silently discarded: an operator could edit `O2_SYNTHETICS_MAX_CHECK_BUDGET_SECS`, hit `/config/reload`, see no error, and still have every check validated against the boot-time ceiling until the process restarted. ## The change The holder is an `ArcSwap` and `set_limits()` overwrites. `ArcSwap` rather than a lock because `limits()` is read on every check-config validation — i.e. on request threads — while a reload writes it, and a reader-preferring lock would let those readers starve the writer. `init_limits` is unchanged for callers and now delegates to `set_limits`. ## One behaviour change worth reviewing **Rejection now keeps the last-good value rather than reverting to `DEFAULT_*`.** A deployment running a deliberately tight 540s/600s pair that reloads with a typo must not have its ceiling widened back to 840s/900s by the failure — that would silently accept checks it was configured to refuse, the opposite of what the operator asked for. At boot there is no last-good value, so the defaults stand and behaviour is unchanged from before. The `init_limits` doc comment is updated to say so. ## Tests 12 new tests in their own integration binary. That placement is deliberate: the holder is process-global and ~90 unit tests in this module validate check configs against `limits()`, so sharing a binary would make them fail intermittently whenever an overwrite test had a different ceiling installed. A separate `tests/*.rs` is a separate process. Two of them assert across `Synthetic::validate` rather than just reading `limits()` back — reading the holder proves the swap happened, not that validation actually moved. Hoisting `limits()` into a `LazyLock` would keep the holder tests green while freezing validation forever. The concurrency test asserts readers observe at least two distinct installed sets, so it cannot pass against a `set_limits` that silently drops writes. `cargo test -p config`: 12 new pass, 98 existing in `meta::synthetics` still pass. `cargo fmt --check` and `cargo clippy -p config --all-targets -- -D warnings` clean. --- *Replaces #13771 — identical content. Renamed the branch to `fix/lease-claimed-by-2270` to match the enterprise branch, because o2-enterprise's `unit-tests.yml` resolves its OSS checkout by looking for a branch of the same name and otherwise falls back to `main` (which lacks `set_limits`, so enterprise CI could not compile).* |
||
|
Hengfei Yang
|
c0d49810b3 |
perf: re-add process-wide promql series label cache (#13748)
## Summary Restores the process-wide PromQL series label cache that was added in #13109 and removed in #13148, ported onto the partitioned label-loading pipeline introduced by #13148. Labels are immutable per series hash, so a warm cache lets repeated queries skip the second (label-column) scan entirely — the biggest wins are on `by()`-grouped aggregations, where the label phase is a fixed cost that the cache removes. ## Changes - **`src/promql/src/label_cache.rs`** (ported from #13109): sharded LRU keyed by `(context fingerprint, series hash)`, bounded by **memory size** — `ZO_METRICS_LABEL_CACHE_MAX_SIZE` in MB, `0` (default) = auto: 5% of total memory clamped to [100 MB, 1 GiB]. - **Reads are always served; admission gates writes only.** A lookup cannot grow the cache, so even a query too big to admit benefits from entries other queries warmed. Admission is sized by the query's actual **misses** (the only entries it could insert) and bypasses **writes** when they exceed 50% of the budget, so one oversized query cannot evict the shared cache. Hit/miss metrics therefore reflect real lookup outcomes. - **`load_series_labels`** attaches cached labels partition-parallel (rayon) before the scan and collects per-partition missing hashes: - when every series hits, the DataFusion label scan is **skipped entirely** - the miss scan is narrowed to missing series: a hash in-list filter for up to 8192 misses, plus a per-missing-series max-timestamp filter (falls back to the sample scan's timestamp set when everything misses) - Partition tasks only attach/cache series in their missing set; extracted label sets are stored back into the cache, sharing the label interner's `Arc<Label>` values (hot label values are deduped inside the cache too). - Cached entries exclude the synthetic `__hash__` label; it is prepended per query, so raw-data and regular queries share entries. - Restores the `query_metrics_label_cache_hit_count` / `query_metrics_label_cache_miss_count` metrics (per organization). ## Performance Re-validated on the current partitioned pipeline: warm queries are **~20% faster than main** on the high-cardinality benchmark dataset below. The detailed tables are from #13109 (same cache design and dataset, measured on the pre-#13148 pipeline): single node (M-series Mac, 32GB), 1.34B rows / 3.17M series of a histogram bucket stream, result cache disabled, warm = median of 3 runs after first. **What the cache itself contributes** (cache on vs off on the same binary, `ZO_METRICS_LABEL_CACHE_MAX_SIZE=1` as the off-switch): | Query | cache on | cache off | cache contribution | |---|---|---|---| | `sum by (region) (rate(m[5m]))` | **3.62s** | 5.40s | **1.49x** | | `histogram_quantile(0.9, sum by (le) (rate(m[5m])))` | **3.53s** | 4.93s | **1.40x** | | `count by (path) (rate(m[5m]))` | **3.73s** | 5.56s | **1.49x** | | no-modifier aggregations (skip path), `topk` (write-bypassed), small filters | — | — | none (by design) | The cache is precisely a **`by()`-grouped-query optimization**: 1.4–1.5x there, zero effect (and zero overhead) everywhere else. Cold runs with the cache are on par with cache-off (`sum by (region)` cold: 5.33s cached vs 5.30s uncached). **Admission control** (`topk(5, rate(m[5m]))` needs full labels for all 3.17M series ≈ 3GB working set, past the 1 GiB auto budget): | Configuration | time | |---|---| | cache disabled (baseline behavior) | 10.4s | | cache without admission, 1 GiB budget (thrash) | 12.0s ⚠️ | | **cache with admission, 1 GiB budget (bypasses)** | **10.6s — parity restored** | | cache with `ZO_METRICS_LABEL_CACHE_MAX_SIZE=8192` (fits, admitted) | **4.5s — 2.3x faster** | The thrash row is what admission prevents — and it was caused by insert + eviction churn, which is why the current revision gates only writes and keeps reads always on. Notes on reproducing: this dataset is deliberately extreme, so its `by()` working sets (~0.6–0.9GB) exceed the default budget's 50% single-query share — set `ZO_METRICS_LABEL_CACHE_MAX_SIZE=2048` or higher to admit them. At typical production cardinality (≤1M series), `by()` working sets are admitted at defaults. Within a single query, selectors share the cache (the second operand of a binary op loads labels in 315ms vs the 2.57s the first paid). ## Operational notes Re-introduces `ZO_METRICS_LABEL_CACHE_MAX_SIZE` and the two cache metrics removed in #13148. Sizing guide: a typical series (~10 labels) costs ~1KB, so the 1 GiB auto cap holds roughly 1M series; column-pruned entries from `by()` queries are far smaller (~100B), so grouped queries fit ~10M series. A tiny budget (e.g. `1`) acts as a de-facto off switch for writes. ## Testing - 4 label-cache unit tests (get/put, byte-bounded eviction, admission control, fingerprint) ported from #13109 - New tests: partial-miss scan attaches only missing series and populates the cache; cached entries exclude `__hash__`; cached-attach hit/miss partitioning - All 374 `promql` crate tests pass; full workspace `cargo check` and clippy clean - Output verified numerically identical (#13109 methodology, max relative diff 1e-13 from float summation order) |
||
|
Huaijin
|
fd2306ffaa |
fix(search): preserve indexed top-n aggregation results (#13779)
- close: https://github.com/openobserve/openobserve/issues/13363 ## Summary - Prevent `AggregateTopkExec` from truncating file-local indexed aggregation rows before the final aggregate merges duplicate group keys. - Keep the Tantivy `SimpleTopN` optimization enabled while removing only the conflicting downstream top-k node. - Add a regression test that constructs the conflicting physical plan, verifies `AggregateTopkExec` is present before leader index optimization, and verifies it is removed afterward while `SimpleTopN` remains selected. ## Root cause `SimpleTopN` emits partial count rows per `(file, group key)`. `AggregateTopkExec` previously ranked and truncated those individual rows before the final aggregate could combine identical keys across files. With enough files, this silently undercounted results and removed entire groups while `is_partial` remained false. The rewrite removes the conflicting node bottom-up. This is important because `AggregateTopkExec` is inside the follower plan already serialized by `RemoteScanExec`; rebuilding the parent updates those serialized plan bytes before attaching `SimpleTopN`. ## Verification Using 121 Parquet files and 121 Tantivy index files containing 93,601 records across 12 groups: | Result | Before | After | |---|---:|---:| | Groups returned | 9 | 12 | | Sum of grouped counts | 83,201 | 93,601 | | Baseline `COUNT(*)` | 93,601 | 93,601 | | `is_partial` | false | false | Tests run: - `cargo fmt --all -- --check` - `cargo test -p search test_leader_rule_topn_removes_conflicting_aggregate_topk -- --nocapture` - `cargo test -p search datafusion::optimizer::physical_optimizer::index_optimizer::tests -- --nocapture` - `cargo test -p search aggregate_topk -- --nocapture` - `cargo check -p search` |
||
|
Prabhat Sharmaandgithub-actions[bot]
|
360b989272 |
feat(slos): PromQL support for count and time-sliced SLOs, plus SLO/alert regression fixes (#13761)
Supersedes #13738, whose commit is included here as the first of four. Adds PromQL support to count and time-sliced SLOs, lets metrics streams choose SQL or PromQL, and fixes three shipped bugs found along the way. ## Shipped bug: every SQL time-slice SLO recorded wrong history `fetch_rows`' `SliQueryPlan::Single` arm normalized every hit with `with_good_column: true`, but that arm serves **two** shapes and only one projects `zo_slo_good`: | shape | projects | result | |---|---|---| | count single-query | `zo_slo_good` + `zo_slo_value` | correct | | time slice | `zo_slo_value` only | `good` defaulted to `0.0` | So every SQL time-slice slice was classified from `0.0` instead of its aggregate. With a positive threshold, `<`/`<=` (e.g. `p95 < 300ms`) recorded every slice **good** — the SLO read ~100%, the budget never burned, and no alert on it could fire. `>`/`>=` recorded every slice **bad** and fired continuously. Shipped in v0.92.0 (`4772df1723`); `to_row` had no time-slice test, which is how it got through. Fixed by deriving the flag from the SLI type in a new `single_query_rows`. Deliberately **not** fixed inside `to_row` by falling back to `total` — that would read a count bucket whose `SUM` came back NULL as *fully good*, inventing uptime. A test forbids it. **Stored history is not rewritten.** It ages out within one window; an operator can force a rebuild sooner by making a real edit (a no-op re-save does nothing — `update` bumps only on `definition_changed`). No auto-migration: bumping every time-slice SLO at boot would freeze every attached alert org-wide until backfill cleared the coverage floor, and double each org's row budget for 97 days — a self-inflicted monitoring gap to repair a reporting error. ## PromQL support **Time slice** carried a `query_language` field the planner ignored, so a PromQL time-slice SLO saved cleanly and measured nothing, forever. - `SliQueryPlan::PromQlValue` — one expression, not a good/total pair, planned at slice ends from the same helper the count arm uses. - The expression is passed through **verbatim**. Grouping comes from series labels; wrapping it in `sum by (...)` the way the SQL arm injects GROUP BY would destroy the aggregate (the sum of four pods' p95 is not a p95). - `promql_value_rows` **rejects** two series colliding on one `(slice_start, group_key)` via `RejectReason::AmbiguousSeries` instead of summing. Summing is right for counters (what `promql_rows` does) and meaningless here. Detection is unconditional, so the ungrouped case — an unaggregated `histogram_quantile` returning one series per instance — is covered. Rejects are plumbed through `fetch_rows` into `PassResult.rejected` on both the live and backfill paths. **Count** already had a working `CountSource::PromQl` backend that no UI could reach. ## Metrics streams choose SQL or PromQL `language_suits_stream` refused SQL whenever `stream_type == "metrics"`, and the form derived the language from the stream type — so metrics had no SQL option. A metrics stream is an ordinary stream with a `value` column, and `SELECT histogram(_timestamp, '5 minute'), avg(value) FROM cpu_usage` is a perfectly good SLI. SQL is now valid for any stream type; PromQL stays metrics-only. The form asks via a toggle (defaulting to PromQL) instead of deriving. ## Validation - `scope` refused for PromQL — a SQL `WHERE` fragment with nowhere to go in a PromQL plan. - Empty `query` refused for **both** languages (the SQL aggregate was previously never validated — same silent-freeze shape). - Expressions parsed with `promql-parser` at save, in `core` (`config` has no parser). - **Parse-only by design.** A grouping check requiring the root to be an aggregation whose `by(...)` matches `group_by` wrongly rejects `histogram_quantile(0.95, sum by (le, region)(...))` (function-call root; `le` is a necessary label superset) and binary-op roots like `sum by (region)(increase(a[5m])) - sum by (region)(increase(b[5m]))`. A "warn instead" variant needs a warnings channel the save API doesn't have. ## Also fixed - **Time-slice comparator 422.** `SliConfig::TimeSlice.comparator` has no serde default, so a dropdown the user never opened was omitted and the save died with a bare 422 — blocking creation of *any* time-slice SLO from the UI. Seeded to `<` only when blank, so a stored comparator survives an edit. - **SLO/alert/service-stream regressions** (the commit from #13738): fractional-threshold serde round-trip, `SloAlertCondition` UI, alerts table, service streams. ## Verification - 487 config · 206 core · 45 infra · 418 frontend — all passing; clippy `-D warnings`, `cargo fmt`, prettier, eslint, `vue-tsc` all clean. - Driven through a real browser against a running server (isolated instance, own data dir). All four SLI/language combinations create and store correctly: count·PromQL, count·**SQL over metrics**, time-slice·PromQL, time-slice·**SQL over metrics**. Also verified: the toggle swaps field shapes both ways, scope shows for SQL and hides for PromQL, PromQL-only hints appear only in PromQL mode, and editing a stored SQL-over-metrics SLO reopens as SQL with its predicate intact. - Validation rejections confirmed live with actionable messages: PromQL-with-scope, unparseable PromQL, and an empty SQL aggregate. Known limitation, pre-existing: `threshold` likewise has no serde default and is not seeded, so leaving it blank still 422s. That is genuine user error rather than an untouched-control trap; adding form validation was out of scope. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Hengfei Yang
|
9f76ff6388 | refactor(search): enhance filtering logic for FTS fields in search function (#13767) | ||
 Ashish Kolhe
|
de0d37e23d |
fix(alerts): stop losing Slack alerts on undeliverable link URLs (#13742) (#13753)
Fixes #13742 Fixes openobserve/o2-enterprise#2394 ## The bug A content template link with a non-http scheme (`javascript:alert(1)`) made Slack reject the **entire** notification with `400 invalid_attachments`. The alert was lost outright and `triggers` recorded `notify_failed`. ## Root cause `safe_url` neutralises a hostile scheme by substituting `BLOCKED_URL` (`"#"`). That is **safe** but not **deliverable** — Slack, Teams, Discord and PagerDuty validate action URLs server-side and reject the whole payload when one is not an absolute `http(s)` URL. A security substitution was turning into total message loss. The two properties are different, and links need both. So the fix splits by medium: | renderer | link handling | |---|---| | Slack buttons, Teams actions, Discord `embed.url`, PagerDuty `href` | `dispatchable_url` → **drop** | | HTML email, email plaintext, SNS plaintext, Discord description | `safe_url` → substitute `#` | | Webhook envelope | unfiltered — versioned machine contract, documented | Dropping beats rewriting: the alert body is worth more than the button, and a button pointing somewhere misleading is worse than no button. `dispatchable_url` requires a **non-empty authority**, not just an allowlisted scheme — `http:javascript:alert(1)` and `https://` are valid *opaque* RFC 3986 http(s) URLs that Slack still rejects. It also rejects raw spaces and control characters, which variable substitution can introduce. ## Related issues fixed in the same area - **Empty `ZO_WEB_URL`** (not in the report): `alert_url` becomes a root-relative `/web/logs?…`, which `safe_url` passes through by design, and it lands in the always-appended "View in OpenObserve" button. Same `invalid_attachments`, on deployments that never authored a hostile template. - **PagerDuty** validates `links[].href` server-side too — same lost-alert class, converted. - **Save-time rejection** (`ContentSpec::validate`): a clear 400 naming the offending link, instead of a template whose link silently never works. Also wired into **preview** and **test-send**, which previously accepted a draft that couldn't be saved and echoed the raw hostile URL back to the UI. - **Slack image-fallback misdiagnosis**: removing the image proves the image was at fault, but only an image pointing at *our* `web_url` justifies tripping the process-wide chart suppression (1h, across every alert) and telling the operator to fix `ZO_WEB_URL`. A custom template's third-party image no longer does either. - **a11y**: `OInput` links its error message to the control via `aria-describedby`; the textarea branch also gained `aria-invalid`. ## o2-enterprise#2394 (fixed here, not in the enterprise repo) Clearing the Monaco body and saving toasted "Please fill required fields" with no indication of *where*. The schema already produced an error on the `body` path — a bare Monaco bridged in via `setFieldValue` just had no `OFormInput` error slot to render it. Now a red border + message driven by the same field state `OFormInput` reads. The aria wiring goes on the **shell**, not on `<QueryEditor>`: `CodeQueryEditor` sets `inheritAttrs: false` and binds `$attrs` to a non-focusable wrapper, so attributes passed to the component would be inert — false assurance for a screen-reader user. > The issue names the title field too. It's an `OFormInput` and already highlights correctly; I added a regression test pinning that rather than changing it. This fix is entirely in OSS (`web/`) since the template form isn't enterprise-gated, so there's no companion enterprise PR. ## Layering note Save-time validation is a **UX gate, not the security boundary**. Link URLs are templates — a scheme that only materialises after `{variable}` substitution can't be judged at save time. The render layer is what actually stops delivery, and it also covers every template saved before this validation existed. Both layers are needed; neither is sufficient. ## Testing - Every fix written test-first (verified RED) and **mutation-tested** — each new test fails when its fix is reverted in substance. - Golden files unchanged: legitimate URLs render byte-identically, so no over-blocking. - Rust: `alerts::notifications` 159 · `config` 3130 · `api-management` 424 — all passing. `cargo clippy --all-targets` and `cargo fmt --check` clean. - Frontend: full suite **41,597 passing** (run because `OInput` is shared library code). `vue-tsc`, `eslint`, and `npm run format:check` clean. - Two independent cold-start review passes; findings from both are folded in, including a P0 where the first version of `dispatchable_url` still let the original bug through. Pre-existing and unrelated: `alerts::destinations::tests::test_alert_destination_template_is_optional` fails locally on a DB connection reset — verified failing on unmodified `main` too. |
||
|
HuaijinandHengfei Yang
|
4a837ee4b9 |
chore(config): remove unused environment variables (#13690)
## Summary - remove 10 EnvConfig entries that have no runtime consumers - stop advertising ineffective knobs across SLO, common, runtime, query, metadata, NATS, S3, and pipeline configuration - preserve active replacements such as ZO_FAKE_ES_VERSION and ZO_PIPELINE_BATCH_SIZE_BYTES These variables were parsed but never read, so this does not change runtime behavior. ## Testing - cargo fmt --all - cargo check -p config - cargo test -p config config::tests --lib (64 passed) - cargo clippy -p config --lib -- -D warnings --------- Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com> |
||
|
Huaijin
|
d5fa24711c |
perf(metrics): choose label timestamp filter automatically (#13756)
## Summary - replace the `ZO_METRICS_INLIST_FILTER_ENABLED` switch with automatic timestamp filter selection - use an exact `IN` filter for up to 1024 timestamps and a bounded `BETWEEN` predicate above that limit - sort exact timestamp literals for stable query plans - add boundary coverage for empty, threshold, and over-threshold timestamp sets ## Rationale Large timestamp `IN` lists add planning and evaluation overhead, while always using `BETWEEN` can admit many unrelated label rows for sparse timestamp sets. A bounded exact-list strategy keeps the selective path for manageable sets without allowing query plans to grow indefinitely. Release benchmarks used 4.2M rows, five label columns, four hash partitions, and repeated median measurements: - with a representative sparse range returning about 262K rows through `BETWEEN`, 1024 timestamps favored `IN` (9.1 ms vs 10.1 ms), while 2048 favored `BETWEEN` (11.7 ms vs 10.2 ms) - when unique maximum timestamps equaled the series count and both predicates returned exactly one row per series, `BETWEEN` consistently won and the gap grew with list size, confirming the need for an upper bound ## Testing - `cargo test -p promql series_labels::tests` - `cargo clippy -p promql --tests --no-deps -- -D warnings` - `git diff origin/main...HEAD --check` ## Operational note `ZO_METRICS_INLIST_FILTER_ENABLED` is removed and no longer needs to be configured. |
||
|
Hengfei Yang
|
238b0d5262 |
fix: support zstd-compressed HTTP requests (#13728)
## Summary - enable tower-http zstd request decompression - add a router middleware regression test using a real zstd-compressed JSON payload - update decompression documentation to list zstd support ## Root cause The Axum router uses `RequestDecompressionLayer`, but the workspace did not enable tower-http's `decompression-zstd` feature. Requests from clients such as Vector configured with `compression: "zstd"` were therefore rejected with `415 Unsupported Media Type` before reaching the ingestion handler. ## User impact Vector HTTP sinks can continue using zstd compression when sending batches to OpenObserve ingestion endpoints. ## Validation - `cargo fmt --all -- --check` - `cargo test -p openobserve-api-http handler::http::router::decompression::tests` - `git diff --check` Fixes #13726 |
||
|
udevandsai nikhil kethe
|
bbd8601c64 |
feat(llm): add annotation queues and datasets (#13719)
## Summary - add annotation queue and dataset storage, management APIs, and review workflows - add discovery, dataset, queue, and review user interfaces - add RBAC synchronization, super-cluster support, tags, version history, and review progress ## Validation - CI should pair this branch with the identically named Enterprise branch for cumulative compilation testing ## Migration note This is an additional cumulative PR created from the committed ud/ebb3 tip. The existing SPR pull requests and branches are intentionally left unchanged. --------- Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> |
||
|
Hengfei Yang
|
ca75da8b78 |
fix(db): scope postgres catalog lookups to the active schema (#13723)
Fixes #11342. ## Problem Upgrading fails on any deployment using a non-`public` postgres schema: ``` [POSTGRES] Fresh install: creating partitioned table file_list db init failed: SqlxError# relation "file_list" already exists ``` The DDL and DML we emit are **unqualified** and resolve through `search_path`, but the catalog / `information_schema` lookups that decide *what* to emit were pinned to a literal `public`. The two disagree: `get_table_relkind` reports "table missing" for a table that plainly exists, we take the fresh-install branch, and `CREATE TABLE` collides with the table already sitting in the user's schema. **The panic is not the worst case.** When a stale same-named table exists in `public` (old deployment, another tenant), the probe returns `'r'` and we take the *regular-to-partitioned migration* branch against the wrong schema's table — renaming its indexes. That matches the index-name conflicts the reporter described. The reporter found 2 sites; there were 14 in total. ## Approach Replace the literal `'public'` with `current_schema()` inline in each catalog query. It evaluates on the very connection that runs the query — the first existing entry of that connection's `search_path`, i.e. exactly where its unqualified DDL/DML lands — so a probe can never disagree with the writes again. No Rust-side state. Two details: - `pg_class` lookups join `pg_namespace` on `nspname = current_schema()` instead of casting to `regnamespace`: the cast parses its input as an identifier and fails on schema names with special characters (e.g. `My-Schema.X`). - `verify_schema_consistency()` at startup checks the read-write, read-only and DDL pools resolve to the same schema. They come from three separately configured DSNs, and `ZO_META_DDL_DSN` is documented as a different **role**, which can carry its own `search_path` via `ALTER ROLE ... SET search_path` — e.g. `cache_indices` reads on `CLIENT` while `create_index` writes on `CLIENT_DDL`, so a divergence would silently relocate the same bug class. Fail loudly instead. ## Sites fixed | File | Functions | |---|---| | `file_list/postgres.rs` | `get_table_relkind`, `check_partition_exists`, `drop_empty_partitions`, `get_varchar_column_width`, `apply_autovacuum_tuning`, `align_identity_sequence` | | `db/postgres.rs` | `cache_indices` (had **no** schema filter at all), `add_column`, `drop_column`, meta `start_dt` probe | | `migration/adapter/postgres.rs` | `list_tables`, `get_columns`, `get_foreign_keys`, `sync_sequences` | | 2 sea-orm migrations | postgres branch had no schema filter, while the mysql branch correctly used `DATABASE()` | The migration-adapter ones matter independently: under a non-public schema `list_tables()` returned **zero** rows, so the cross-database migration tool reported success having migrated nothing. ## Verification Against a real PostgreSQL 18.4 instance with the app configured with `search_path=o2`: - **Fresh install** into schema `o2`: boots clean, all 121 tables land in `o2`. - **Forced upgrade path** (`db_schema_version` lowered) **with a stale regular `public.file_list` planted**: logs `Table file_list already partitioned, ensuring partitions`, upgrade completes, and the stale `public` table is untouched. On main this exact setup either panics (`relation "file_list" already exists`) or migrates the wrong schema's table. - Every rewritten query shape additionally verified in psql under schema `o2test` and under a schema named `My-Schema.X` (stale-table isolation, partition listing, varchar width, identity-sequence alignment, index cache, adapter `list_tables`). `cargo check --bin openobserve`, `cargo fmt --check`, `cargo clippy -p infra --all-targets` all clean; 108 infra postgres unit tests pass. ## Not covered here 1. **Recovery for already-broken databases.** This prevents the failure but does not unstick installs that are mid-migration with renamed indexes. Worth a follow-up (idempotent cleanup, or a documented recovery procedure). 2. **CI coverage.** There is no job exercising a non-`public` `search_path`, which is why these 14 sites accumulated across different changes. A postgres job with `search_path=o2test` would stop this recurring. |
||
|
Ashish Kolhe
|
b3e6a77270 |
fix(correlation): service-streams P2 behavioral fixes + discovered services flat table (#13710)
Correlation P2 round: behavioral/security fixes across the service-streams backend and correlation FE, plus the Discovered Services UX rework and the correlation e2e suites. **Companion enterprise PR (must merge together):** openobserve/o2-enterprise#2381 — the enterprise side calls the new 3-arg `put(org, record, max_streams_per_type)` introduced here. ## Commit 1 — behavioral fixes (F2–F38) **Backend** - F25/F26: RBAC on `list_services` (`service_streams` GET — the one previously-unenforced endpoint in the module) + validate `distinguish_by` ids on identity-config save - F2: validate trace filter param before WHERE-clause interpolation - F6: `_reset` invalidates caches on all nodes via reload event; org deletion clears service-streams caches - F8: shared coverage-based identity-set ranking for ingest and read - F10: purge rows of removed/changed identity sets on config save - F19: `put` returns deleted orphan keys for cache eviction - F28: discriminate no-match from failure; stop aliasing 404 to no-match - F32: cap DB-side stream union at `max_streams_per_type` - `all_dimensions` unions on put so late-tracked dimensions surface - F14: `StreamInfo.dropped_dimensions` contract field; allow empty `tracked_alias_ids` when identity sets exist **Frontend** - F35/F36/F31: per-stream filter-edit resolution via semantic groups; dimension edits in both key spaces; subject overrides never guess fields - F27/F30: echo `source_stream`/`source_type`; stop guessing another stream's filters; chip dedup - F2/F38: SQL field/value escaping via shared `quoteSqlLiteral` helper ## Commit 2 — Discovered Services UX + e2e - Flat table (one row per instance), same-name services merge their name cell (OTable `pivotRowColumns`); optional per-service collapse with aggregated summary row; search/filters override collapse - Fixes an OTable pivot-merge bug (value-keyed merge map blanked single-column runs) + regression test - Semantic Field Groups: category tabs with counts + cross-category search with match highlighting - Correlation Phase-1 API + Phase-2 UI Playwright suites ## ⚠️ Release note (enterprise + RBAC deployments) Viewing Discovered Services now requires the **Service Streams** permission (previously unenforced). Root/admin users are unaffected; custom roles may need a one-time grant via the role editor (resource already exists in the OFGA model). ## Checks - `cargo fmt --all --check` ✅ · `cargo clippy --all-targets` ✅ enterprise + OSS feature sets (0 warnings) - prettier ✅ · eslint 0 errors · 207 unit tests on touched specs ✅ - Compiles with paired enterprise branch ✅ |
||
|
Prabhat Sharma
|
087e1498ab |
feat(slos): add SLO burn-rate alerts (#13704)
design at /slo ## What Adds **SLO burn-rate alerts** — an alert family that fires off an SLO's error budget rather than a raw query — and finishes the SLO surface around it: navigation entry, detail-page fixes, and the flag removal now that SLOs have shipped. ## Changes **SLO burn-rate alerts (backend)** - New burn-rate evaluation (`config/src/meta/slo/burn.rs`, `condition.rs`) with multi-window burn conditions, and `core/src/slo/evaluate.rs` extended to evaluate them on the SLO job. - Alerts carry an `slo_id`; `infra/src/table/alerts` gains the column, filtering, and query support so an SLO's alerts can be listed and deleted with it. - Alert levels (`meta/alerts/level.rs`) and tags extended for the SLO family; scheduler handlers route SLO alerts through the same notification path. **SLO alert UI** - `SloAlertForm` + `AlertDetailSlo`: create, edit and view burn-rate alerts from the SLO detail page and the alert list. - Payload/routing helpers (`utils/alerts/sloAlertPayload.ts`, `sloAlertRouting.ts`, `alertPayload.ts`) with unit tests. - `AlertDestinationsField` extracted as the complete Destination form field — side label, tooltip, combined destinations/workflows picker, inline error, refresh, and create-in-new-tab. It replaces two verbatim copies in the generic alert form and is reused by the SLO form and synthetics `CheckAlerts` (which sets `supportsWorkflows: false`, since a check has no workflow routing). Fixes the SLO form's previously unwired refresh and Add Destination buttons. **SLO detail page fixes** - Burndown and burn-rate charts drew as bare axes when coverage was sparse: `showSymbol: false` means an isolated measured bucket between gaps has no segment and paints nothing. Symbols are now sized per point, and a sparse-coverage note states how many buckets actually recorded events. - Nothing scrolled below ~700px of viewport height — `OPageLayout`'s fixed flex body plus `OTabPanels` `scroll="none"` clipped the panels. Panels now grow and scroll; the stat strip and tab bar stay pinned. Verified at 1024x500, 1280x620, 1512x945. **Other** - `ZO_SLO_ENABLED` removed — SLOs have shipped, so the flag and its status/route gating are gone. - SLOs surfaced in the main nav. - Dashboard chart panel header gets its own tint via new `--color-panel-bar-bg` / `--color-panel-bar-border` tokens, and the bar is extracted into `PanelBar.vue` (it existed as five hand-copied class strings that had already drifted). - `tsconfig.vitest.tsbuildinfo` untracked; third-party vendor name removed from comments and test names. ## Testing - New Rust tests across `meta/slo/burn.rs`, `condition.rs`, `core/src/alerts/alert.rs` and `infra/src/table/alerts`. - New frontend suites: `sloAlertPayload.spec.ts`, `sloAlertRouting.spec.ts`, `alertPayload.spec.ts`, `AlertDetailSlo.spec.ts`, `SloDetailAlerts.spec.ts`, `SloListDelete.spec.ts`, `SloTimeSlicePreview.spec.ts`. - `cargo clippy --all-targets` and `web` `lint:ci` clean. |
||
 Recoordinate
|
93ee50ca17 |
docs: fix occured typo in alert and folder doc comments (#13705)
`occured` should be `occurred`. Four occurrences, all in rustdoc comments on error variants: - `src/core/src/alerts/alert.rs:235,240` - `src/db/src/folders.rs:83,88` No identifiers, error messages or code paths change. The remaining two hits in the tree are in `src/config/src/utils/async_walkdir/`, which is a vendored copy of the `async_walkdir` crate and carries its upstream copyright headers, so I left those untouched. |
||
 Pinguladora
|
575e8ea4d5 |
feat: add zstd compression on OTLP gRPC ingest (#13703)
Design at: #13702 / inline in PR description Add the `zstd` feature to tonic and enable `accept_compressed(Zstd)` on the OTLP logs, metrics and trace gRPC services in both `run_common` and run_router`. Closes #13702 ## Out of scope - Response compression on `send_compressed(Zstd)` - OTLP/HTTP path (tower-http zstd features) - snappy would be the better choice under CPU pressure, but tonic has no snappy support. |
||
|
Hengfei Yang
|
8303be0b4c |
refactor: rename AlertManager node role to Scheduler (#13696)
## Summary - rename the `AlertManager` node role to `Scheduler` across Rust services, jobs, protobuf bindings, UI labels, and tests - emit `scheduler` as the canonical role name - continue accepting legacy `alertmanager` and `alert_manager` configuration values - keep protobuf enum value `5` unchanged for wire compatibility ## Why The role schedules and executes alerts, reports, pipelines, synthetics, and other background work. `Scheduler` describes that broader responsibility more accurately. ## Compatibility - existing role configuration values remain accepted through parser and serde aliases - the protobuf numeric value remains stable - coordinated changes are included in the Enterprise, Helm chart, and documentation repositories ## Verification - `cargo test -p config test_role` - `cargo test -p openobserve-node` - `cargo check -p openobserve-jobs` - targeted web tests: 307 passed, 13 skipped - Rust formatting and targeted Prettier checks |
||
|
Yashodhan JoshiandYashodhan Joshi
|
24083c018e |
fix: cache invites (#13613)
This PR introduces cache layer for user invites, as well as removes the ratelimit from org list api --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> |
||
|
Hengfei Yang
|
dacc6d5e40 |
fix: prevent NATS KV watch consumer accumulation on reconnect storms (#13676)
## Incident A ~100 node production cluster accumulated **587 consumers on a single KV stream** (`KV_<prefix>_compact`). 580 of them were created inside one ~10 minute fleet-wide reconnect window (262 in a single second), and none were reclaimed in the 10 days since. Each kept a 5s idle heartbeat alive (~117 msg/s of dead traffic). Diagnosis confirmed the server side is at fault for the non-reclamation: `nats consumer info` reports `Active Interest: Active` while a `subsz --filter-subject <deliver subject>` sweep across the entire supercluster returns zero matching subscriptions — stale interest state left behind by the connection storm keeps the 30s `inactive_threshold` from ever firing (nats-server 2.11.4, same family as [nats-io/nats-server#5352](https://github.com/nats-io/nats-server/issues/5352)). The client-side contribution: every watcher error tore down and recreated the whole `watch_all()` chain in a 1s loop, and each recreation leaves a brand-new anonymous ephemeral consumer behind — during connection flapping this multiplies consumers across the fleet, handing the server bug hundreds of victims. ## Fix (deliberately minimal) Keeps the official `watch_all()` API; no custom consumer management. `kv_watch` now: - applies **exponential backoff (1s..30s)** to watcher recreation, reset once a healthy entry arrives, so a nats disruption no longer turns into a consumer creation storm - **tolerates transient stream errors** (up to 3 consecutive, e.g. missed heartbeats while the connection re-establishes — the ordered consumer recovers on its own) instead of recreating the watcher, and its consumer, on the first error - **resumes from the last processed revision** (`watch_all_from_revision`) instead of dropping events published while the watcher was down, with a guard for the bucket being recreated - decodes keys with `try_key_decode` so a malformed key off the wire cannot panic the watcher task ## Notes for operators - The existing stale consumers must be cleaned manually (`nats consumer rm`, or rolling-restart the NATS servers — they are memory-storage R1); this PR reduces the recurrence surface but the actual reaping failure is a server bug: upgrade beyond 2.11.4 (2.11.9 / 2.12.4 contain interest-tracking fixes) and it is worth reporting upstream with the subsz evidence. ## Testing - `cargo clippy -p infra` clean, `cargo test -p infra db::nats` 28 passed (new test for panic-free key decoding). Supersedes #13594. |
||
 Dhruv PatelandHengfei Yang
|
0dfb28b771 |
fix: evaluate every metrics record against realtime alerts (#13581)
Fixes #13458 Reduced per review — this now changes only the three metrics ingestion paths. ## The bug `stream_trigger_map` was written after the first record of a stream regardless of whether anything matched: ```rust let need_trigger = !stream_trigger_map.contains_key(&stream_name); if need_trigger && !stream_alerts_map.is_empty() { ... stream_trigger_map.insert(stream_name.clone(), Some(trigger_alerts)); // runs even with zero matches } ``` `contains_key` is then true for the rest of the request, so every later record of that stream is skipped. With the payload from the report — matching row second — record 1 is evaluated, doesn't match, writes the key, and record 2 is never looked at. Swapping the rows makes it fire. `kube_node_status_condition` arrives as one row per (node × condition), so the matching row is rarely first, which is why it presented as firing a couple of times a day rather than continuously. ## The fix Follows the existing logs behaviour (`logs/mod.rs`): an `evaluated_alerts` set keyed on the alert, each record checked against the alerts that have not matched yet, and an alert marked evaluated only after it matches. ```rust if let Some(alerts) = cur_stream_alerts && triggers.len() < alerts.len() { let end_time = now_micros(); for alert in alerts { let key = format!("{}/{}/{}/{}", org_id, StreamType::Metrics, alert.stream_name, alert.get_unique_key()); if evaluated_alerts.contains(&key) { continue; } match alert.evaluate(Some(&record), (None, end_time), None).await { Ok(trigger_results) if trigger_results.data.is_some() => { triggers.push((alert.clone(), trigger_results.data.unwrap())); evaluated_alerts.insert(key); } Ok(_) => {} Err(e) => log::error!("[METRICS] Error while evaluating realtime alert: {e}"), } } } ``` The insert moves after the record loop and only happens when there is something to send: ```rust if !triggers.is_empty() { stream_trigger_map.insert(stream_name.clone(), Some(triggers)); } ``` One trigger per alert per request is unchanged. `Option<TriggerAlertData>` and all three drain loops are untouched. `json.rs` needs one structural change: it fetched its alerts *inside* the record loop, which cannot coexist with a borrow of `stream_alerts_map` held across that loop (E0502). That block moves above the loop, otherwise unmodified. `prom.rs` and `otlp.rs` already fetch before the loop and needed nothing. ## Testing ### Unit ``` cargo test -p openobserve-core --lib test result: ok. 1227 passed; 0 failed; 10 ignored; 0 measured ``` `rustfmt --check` clean on the three changed files; `cargo clippy -p openobserve-core --lib` reports no warnings or errors. (`users::tests::test_create_new_user` flakes independently of this change — it fails on a clean checkout too, roughly 2 runs in 6, with `unable to open database file`. It touches the users/DB path, not metrics.) ### Manual verification A 40-row batch (10 nodes x 4 conditions) with the single matching row placed at three positions, run against a build with this change and a build without it, using a webhook destination to observe delivery: | Path | match at 0 | match at 5 | match at 39 (last) | |---|---|---|---| | `_json` | fires / fires | **silent** / fires | **silent** / fires | | OTLP | fires / fires | **silent** / fires | **silent** / fires | | Prometheus remote-write | fires | **silent** / fires | **silent** / fires | (before / after.) The matching row's position is the only variable; before the change, anything past the first record was never evaluated. Also checked: a batch with 20 matching rows still produces exactly one notification carrying one row, so the one-trigger-per-alert-per-request behaviour is unchanged. Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com> |
||
|
Yashodhan JoshiandYashodhan Joshi
|
f5285ad0c2 |
fix: add env to completely skip ssrf checks (#13682)
This gives an way out to self hosted installations to completely skip ssrf checks. false by default. Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> |
||
  Abhay Padamaniandktx-akshay
|
739054c7bd |
feat: metric chart improvement (#13546)
Design At: designs/dashboards/metric-chart-override-support/ 1. For metric chart conditional text color, background color, value override support. 2. sparkline for metric chart. --------- Co-authored-by: ktx-akshay <akshay.patil@kiara.tech> |
||
|
Huaijin
|
ee71df7ce7 |
chore: support per-stream file formats (#13677)
## Summary - support a default ZO_FILE_FORMAT with logs, metrics, and traces overrides - resolve the configured format when producing merged files while preserving Parquet output for metrics on ingesters - reject invalid or duplicate overrides and preserve string serialization compatibility ## Examples - ZO_FILE_FORMAT=parquet - ZO_FILE_FORMAT=vortex - ZO_FILE_FORMAT=parquet,metrics=vortex - ZO_FILE_FORMAT=vortex,logs=parquet,traces=parquet ## Testing - cargo test -p config test_file_format --lib - cargo test -p config test_common_config_preserves_file_format_overrides --lib - cargo test -p search test_merge_output_file_format_uses_parquet_for_ingester_metrics --lib - cargo clippy -p config -p search --lib -- -D warnings |
||
|
21dfaf36cf |
fix: handle auth for invite reject correctly (#13566)
API call for rejecting an invite was not handled properly and would log out an user with unauth. This PR fixes that. --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Subhra264 <chakrabortysubhradeep556@gmail.com> |
||
|
Huaijin
|
aaed33543c |
refactor(search): group DataFusion and Tantivy feature configs (#13664)
## Summary - add ZO_FEATURE_METRICS_PUSHDOWN_FILTER_ENABLED as an opt-in filter pushdown switch for metrics queries - group DataFusion and Tantivy feature configuration under the search config struct - move inverted-index result cache sizing, footer caches, skip threshold, TopN group limit, and bloom footer cache settings from limit to search - preserve the moved cache settings' environment variable names and defaults - apply the metrics-specific setting to object storage, WAL, and Flight query contexts ## Testing - cargo fmt --all -- --check - cargo check for config, infra, search, tantivy_utils, search_service, and promql-service - cargo test -p config every_env_config_default_parses --lib - cargo test -p config inverted_index_config --lib - cargo test -p search test_create_session_config --lib - cargo clippy for all affected crates with -D warnings |
||
|
Ashish Kolhe
|
b5c2740f85 |
fix: canonicalize model names in service graph to prevent node splitting (#13561)
## Summary - The Agent Graph model-edge query previously used the raw `gen_ai_request_model`/`gen_ai_response_model` string verbatim as the node key. Different instrumentation frameworks/vendors report the same model under different raw strings (e.g. `anthropic/claude-sonnet-4-6` vs. `claude-sonnet-4-6`), which split what should be one model into two separate graph nodes. - Adds `canonical_model_pattern`/`model_patterns` to `otel::pricing`, reusing the existing `MODEL_PRICING` regex table (already used for cost calculation) so cost lookup and node identity stay consistent. - Wraps the model-edge node-key expression in a SQL `CASE` in `service_graph::processor::compute_stream_edges`, applied at query time (no ingest-time mutation, no backfill needed — raw `gen_ai_*` columns are untouched for audit/debugging, and historical data self-heals on the next hourly aggregation pass). ## Test plan - [x] `cargo test -p openobserve-core --lib traces::otel::pricing::` — 25/25 pass, including 5 new tests covering vendor-prefix merge, dated/undated suffix merge, older-snapshot distinct-bucket behavior, unknown-model fallback, and pattern-list sanity — using real-world sample strings gathered from both a Google-ADK/gateway-style agent and CrewAI/LiteLLM-style instrumentation. - [x] `cargo build -p openobserve-core` (non-enterprise, default) — green, confirms zero OSS-build impact (new code is fully `#[cfg(feature = "enterprise")]`-gated). - [x] `cargo fmt --all -- --check` clean for both changed files. - [ ] Full `--features enterprise` build/test of `processor.rs` itself — not completed in this session (local worktree lacked a fully-wired sibling `o2-enterprise` checkout); recommend CI or a follow-up local verification pass before merge. - [ ] Live verification against a running instance's `_o2_service_graph` data — not done, would need a deployed instance. See `docs/ak_review/pr-fix-model-node-canonicalization-review.md` for the full review-pr writeup (0 P0/P1 findings; 2 P2 notes on PB-scale timing and minor duplication, 1 P3 doc note). |
||
|
Ashish Kolhe
|
bafc2be4f0 |
fix(correlation): P0/P1 fixes from service-streams correlation review (#13668)
## Summary OSS half of the correlation P0/P1 fix set from the line-by-line review (`docs/ak_review/correlation-line-by-line-review-2026-08-06.md`). Companion enterprise PR: same branch name in o2-enterprise (required for CI manifest pairing). ### Fixes (by review finding) | Finding | Fix | |---|---| | **F14** | `StreamInfo.dropped_dimensions` contract field (`#[serde(default, skip_serializing_if)]`, backward-compatible wire format) + TS type | | **F8** | Shared coverage-based identity-set ranking (`resolve_best_set_by_coverage`) in `config::meta::correlation` — full coverage > ratio > count, declaration-order ties; ingest and read now rank identically | | **F10** | `save_identity_config` purges rows of removed/changed sets (`obsolete_set_ids`) + cache clear + org reload event | | **F6** | `_reset` routes through enterprise `ServiceStorage::delete_all` (cache clear + super-cluster replication) and emits an org-scope reload; super-cluster Delete consumer emits org reload instead of per-service delete | | **F19** | `infra::table::service_streams::put`/`delete_subset_orphans` return deleted orphan disambiguations so callers can evict cache keys | | **F2** | `config::utils::sql::validate_where_fragment` (sqlparser) applied to the `filter` param at both `traces/latest*` interpolation sites — statement smuggling and comment-truncation payloads → 400 | | **F2/F38** | FE: shared `quoteSqlIdentifier`/`quoteSqlLiteral`/`buildSqlCondition` adopted by all six SQL builders (two trace paths were fully unescaped) | | **F35** | `applyFilterOverlay`: filter edits resolve per-stream via semantic groups instead of literal key identity; dead `removeFilter`/`resetFilters` removed; `semanticGroups` threaded to `CorrelatedLogsTable` parents | | **F36** | `applyDimensionEditsToFilters`: dimension-bar edits work in both key spaces (raw-field and semantic-ID), case-insensitive shared field→group map | | **F31** | `mergeSubjectOverrides`: subject overrides replace the same-group key (never union two aliases into contradictory WHERE), schema-less field guessing removed | | F14 (FE) | Warning banner shipped, then removed by reviewer decision (`dropped_dimensions` stays in the API) | ### Verification - `cargo test -p config` correlation/service_streams/sql: 121 tests green; enterprise-feature and OSS-only cfg builds both compile - FE: 263 vitest tests green across the correlation suites; vue-tsc clean - **Live-verified** against a local build (fresh DB): 19/19 API checks (raw-value emission, set filing, obsolete-set purge, reset cache invalidation, orphan eviction, filter validation) plus browser checks (escaped SQL on the wire: `"environment" = 'Prod-EU'`) Deliberately out of scope (per review decision): F15, F17/F37, F9. |
||
|
Omkar KesarkhaneandClaude
|
d171d98873 |
fix(synthetics): recorder extension install link, record/replay setup gate, configurable store URL (#13658)
## Summary The OpenObserve Recorder Chrome extension is now published on the Chrome Web Store. This PR wires the browser-test UI to it: - **Install link + setup checklist** — the CreateBrowserTest gate links to the Web Store listing and walks through install → Allow-in-Incognito → connect, with the connect step verified live via the extension probe. The checklist is extracted into a shared `ExtensionSetupChecklist` component. - **Record/replay setup dialog** — `BrowserJourney.vue` now gates both Record and Replay behind an `ExtensionSetupDialog` when the extension is not detected (previously Replay was ungated and would hang in "running" until the 15-min watchdog). - **Loader on Record journey** — the gate's Record button shows a spinner while the extension probe awaits its ~4.5s worst-case timeout. - **Configurable store URL** — the Web Store URL is served by `/config` as `synthetics_recorder_extension_url`, sourced from the new env var `O2_SYNTHETICS_RECORDER_EXTENSION_URL`. The frontend falls back to the built-in constant when the field is absent or empty, so the UI works against older backends and OSS builds. ## Test plan - `ExtensionSetupChecklist.spec.ts` (8), `ExtensionSetupDialog.spec.ts` (11), `BrowserJourney.spec.ts` dialog gating tests — all passing - `npm run lint` / `npm run type-check` clean (3 pre-existing `eslint.config.js` duplicate-key errors untouched) - Manual: record + replay gating verified against the published extension --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Subhra264
|
6cbb62ffa8 | fix: metric stream name rbac migration (#13521) | ||
|
Ashish Kolhe
|
58c7dafa55 |
fix: notification templates v2 — rich content templates + alert chart images (#13640)
## Summary
Two coupled alerting features:
**1. Notification Templates v2 ("content" templates)**
- New `kind: content` template type: a structured spec (title, body
markdown, fields, links, rows, per-channel title overrides,
severity-filtered visibility) rendered natively per channel — existing
`custom` (raw body) templates are untouched and byte-identical on the
send path.
- Per-channel renderers: Slack Block Kit, Teams (Adaptive Card +
MessageCard), Discord embeds, PagerDuty Events v2, Opsgenie, ServiceNow,
SNS, HTML/text email, and a stable machine `webhook` envelope.
- Escaping is renderer-owned (mrkdwn entities, HTML encoding, markdown
stripping) with a shared URL-scheme allowlist; golden-fixture tests per
channel.
- Template editor UI: guided content form (fields/links row tables,
compact rows config, channel title overrides), live preview panel with
per-channel preview + payload tab, test-send (rate-limited,
`[TEST]`-marked).
- Migration `m20260802_000001_add_template_kind`; prebuilt default
template + org backfill.
**2. Alert chart images (stateless, storage-free)**
- Alerts can attach a metric-history chart PNG rendered from the
triggers self-reporting stream (`actual_value` per evaluation), with
warn/crit threshold lines and the firing point appended.
- Zero storage: the chart URL *is* the data — deflate+base64url payload
signed with truncated HMAC-SHA256 (org-bound, expiry embedded), rendered
on fetch by the HTTP-terminating node (`GET
/api/v2/{org}/alerts/charts/render`). Signature verified before inflate
(decompression-bomb guard), uniform 404s, per-node render semaphore +
per-minute query budget + small LRU.
- Renderer: plotters + embedded DejaVu Sans (Bitstream Vera license
included), gap-broken line segments so sparse evaluation history doesn't
draw false plateaus.
- Channel delivery: Slack image block, email inline CID (no remote
fetch), Discord multipart attachment, Teams/PagerDuty image URLs;
per-destination opt-in via template chart flag.
- Slack layout live-A/B'd on a real channel: content + buttons/chart
split across two same-colored attachments so the severity stripe spans
everything while staying under Slack's "Show more" fold; clickable link
title; "View in OpenObserve" keeps a reserved primary-button slot.
- Slack image-delivery fallback: Slack validates image URLs server-side
and rejects the ENTIRE message with `400 invalid_attachments` when its
proxy can't fetch the chart (VPN-only/private `ZO_WEB_URL`) —
live-verified. On that rejection the send strips the image blocks and
resends once (alert always delivers), sets a process-wide flag so later
sends pre-strip (reachability is deployment-wide, not per-destination),
and re-probes hourly. Zero config; observable via `slack_image_rejected`
/ `slack_image_pre_stripped` metric events.
## Testing
- 147 notification unit tests (resolve/render/preview/test-send incl.
per-channel goldens and hostile-input/injection coverage), 8 chart
payload tests, renderer PNG tests.
- Live-verified end to end: Slack (Slackbot chart fetches ~100ms,
cache-immutable semantics), SES email inline image, multi-severity
firing scenario (warning→critical) with correct threshold lines.
- FE: AddTemplate/ContentTemplateForm/preview specs; UX audit pass with
fixes.
- `cargo clippy -D warnings` clean in both OSS and enterprise
arrangements.
## Enterprise pairing
Must merge together with openobserve/o2-enterprise#2353 (workspace
twin-manifest mirror for the new OSS deps).
|
||
|
Hengfei Yang
|
d5d0b23666 |
fix: accept org ingestion tokens for OTLP gRPC (#13627)
## Summary - accept organization ingestion tokens (`o2oi_`) for OTLP gRPC logs, metrics, and traces - keep internal cluster, search, node, and other non-OTLP gRPC services on the existing authentication path - add coverage for accepted, missing, and internal-RPC token cases ## Root cause The HTTP ingestion authentication path validates organization ingestion tokens, but the global gRPC interceptor only recognized internal gRPC tokens and user credentials. As a result, the same organization token succeeded over HTTP OTLP and failed over gRPC OTLP with `Unauthenticated: No valid auth token[5]`. This change assigns the organization-token-aware interceptor only to external OTLP services. Internal RPCs continue using the original interceptor, preserving the existing security boundary. ## Testing - `cargo test --release -p openobserve-api-grpc handler::grpc::auth::tests --lib` (6 passed) - `cargo build --release --bin openobserve` - live local E2E on latest `main`: created a real organization ingestion token, exported an OTLP log over gRPC, then searched the stream and verified the expected record (`total=1`) Fixes #12540 |
||
|
|
6f0b2dcb07 |
fix: workflow history (#13588)
This enhances the workflow history , stores all runs instead of just failed ones and also some UX improvements on FE --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> |
||
|
Huaijin
|
12dce4c274 |
ci: lint all Rust workspace targets (#13615)
## Summary - run Clippy against every workspace member and target in the unit-test CI - fix the existing test-target Clippy findings, including the non-snake-case test name - keep warnings as hard failures so `unit_tests_summary` can be used as the required check ## Validation - `cargo fmt --all -- --check` - `cargo clippy --workspace --all-targets -- -D warnings` - `cargo test -p search --lib catalog_functions_documents_what_only_the_server_can_supply` - `cargo test -p config --lib` — 2967 passed, 2 ignored - `cargo test -p infra --lib` — 1243 passed, 35 ignored; the full parallel run also exposed two pre-existing local-only failures: an environment-sensitive default-index-field assertion and a shared mock database initialization race. The ratelimit test passes in isolation. |