mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-04 16:09:05 +00:00
c49d156e7ea77e2bf64e2d62ba173b4b4479c2d4
6716
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Shrinath Rao
|
c49d156e7e |
test(alpha1): multi-org sharding for the alpha1 e2e run (#13530)
Companion OSS change to enterprise 2278 (alpha1 workflow). Same branch slug on both repos. ## What Supports running the Alpha1 (cloud) e2e matrix with its shards **divided across multiple orgs** for data isolation, on top of the shared-auth login (one Dex login, reused by all shards). Both changes are in \`global-setup-alpha1.js\`. ## Changes 1. **Ingest into the shard's own org (\`ORGNAME\`)** instead of the org baked into \`cloud-config.json\`. The shared-auth barrier logs into one org; without this, every shard ingested into that one org even when \`ORGNAME\` differed. 2. **Re-fetch the shard's own org passcode in the shared-auth path.** Each org has its own \`o2oi_\` default ingestion token (see \`core/organization.rs\` \`get_passcode\`). The barrier captured only its org's token, so shards on other orgs got \`401 Unauthorized Access\` on ingest. \`verifySharedAuth\` now calls \`fetchCloudConfig(page)\` to overwrite \`cloud-config.json\` with the current shard's passcode (fetched via the session, which is a member of every org). Keeps the single shared login — no login storm — while giving each shard the correct per-org credential. ## Context Part of the alpha1 parallelization effort: - Shared-auth artifact removes the 13-way concurrent Dex login storm (enterprise 2278). - Shards divided across 6 orgs (enterprise 2278 matrix) for data isolation. - These two OSS fixes make per-org ingestion actually work. ## Testing Validated by dispatching the enterprise \`playwright_alpha1.yml\` with \`o2_opensource_repo=test/alpha1-parallel-shards\`. Verifying each shard now ingests \`e2e_automate\` into its own org with \`status:200\` (previously \`401\` for non-barrier orgs). |
||
 Hengfei Yang
|
4231ca5833 |
perf(schema): share stream settings via Arc and centralize cache mutation (#13535)
Design at: #13465 PR 2 of the plan in #13465 (finding 1: the settings cache O(S) deep clone per mutation). ## Background The stream settings cache is a locked map (`STREAM_SETTINGS`) plus an ArcSwap read snapshot (`STREAM_SETTINGS_ATOMIC`) introduced in `405794db08` for lock-free ingestion reads. That architecture is kept — the read/write ratio justifies it. The problem was the constant factor: - every mutation republished the snapshot by **deep-cloning every cached `StreamSettings`** (vectors, UDS field lists, maps). The snapshot is republished on every schema change event cluster-wide (any new stream or field, on every node) plus every cache-miss backfill — at 100k+ streams this is a full deep copy of the map at minute-level frequency; - every point read (`get_settings`, called on each ingestion request) deep-cloned the returned `StreamSettings`, including UDS lists that can hold hundreds of strings; - the two maps each held a complete deep copy of all settings. ## Changes **Arc-ified values.** Both maps now store `Arc<StreamSettings>`. Snapshot republish is a shallow clone (String keys + refcount bumps), point reads return an `Arc` clone, and the maps share one set of values, roughly halving the cache's memory. **Centralized mutation.** The map statics are private; all writes go through `put_stream_settings` / `put_stream_settings_batch` / `remove_stream_settings`, which republish the snapshot under the write lock so snapshots cannot be published out of order. Startup load collects settings and publishes once via the batch API. Callers can no longer accidentally rebuild the snapshot — the pattern that caused the original quadratic startup. Lock-free reads stay available via `get_stream_settings_atomic`; the status API uses a new `get_stream_settings_stats()` accessor. **API ripple, kept minimal.** `get_settings()` now returns `Option<Arc<StreamSettings>>`. Field access and `unwrap_or_default()` call sites compile unchanged; the few paths that edit a local copy before persisting (`update_stream_settings`, LLM/gen-ai settings, dashboard distinct fields) do an explicit `(*s).clone()`. The `get_stream_setting_*` helpers are generic over `Borrow<StreamSettings>`, so both `&Option<StreamSettings>` and `&Option<Arc<StreamSettings>>` callers work — including the downstream enterprise call sites, which were checked and need no changes. **`SchemaCache::fields_map` Arc-ified** — cloning a cached schema entry no longer deep-copies the field-name index, which holds thousands of entries for wide streams. ## Measurements (criterion bench used during development, not kept in-tree) | operation | result | |---|---| | bulk load 10k / 50k / 100k streams | 0.80ms / 4.4ms / 8.7ms — linear, ~87ns per stream | | point write against 100k-entry cache | 3.5ms (shallow snapshot republish) | | point read against 100k-entry cache | 7.9ns (lock-free) | For comparison, the field report in #13465 measured ~27ms per stream insert at 58k cached streams under the old per-mutation deep clone; reads previously deep-cloned the settings value on every call. ## Testing - new unit test for the mutation API (put/batch/remove, snapshot sharing via `Arc::ptr_eq`, republish keeps other keys) - full test suites pass for all touched crates (infra, db, schema, stream, core, api-search, api-management, search_service, compaction, enrichment-data); the only failures are pre-existing local-environment ones also failing on main (sqlite-path retention tests, one env-dependent index test, one enterprise doctest) - `cargo clippy --workspace --all-targets`: no findings in touched files Finding 2 (grouped stream enumeration) landed in #13536; this branch has been rebased onto main (the #13531 overlap is resolved). |
||
 Huaijin
|
3d96c9670c |
chore: regenerate Rust CycloneDX SBOMs (#13534)
## Summary - regenerate Rust CycloneDX SBOMs with cargo-cyclonedx cyclonedx - update 9 existing SBOMs and add 32 workspace crate SBOMs - exclude all generated files under src/enterprise ## Validation - cargo-cyclonedx version: 0.5.9 - all 41 retained XML files pass xmllint --noout - malformed vcs_url PURL pattern from 0.5.7 is absent - git diff --check passes - src/enterprise has no changes |
||
|
Hengfei Yang
|
d702518446 |
perf(schema): enumerate streams with one grouped cache scan per job cycle (#13536)
Design at: #13465 PR 3 of the plan in #13465 (finding 2: full schema-cache rescans per (org, stream type) pair). ## Background `list_streams_from_cache(org, type)` scans every key in `STREAM_SCHEMAS_LATEST` and allocates a `Vec` per key to parse it. The periodic jobs called it inside `for org { for stream_type { ... } }` loops, so one cycle cost O(orgs × types × streams) key visits. With 100k+ streams, hundreds of orgs and 8 stream types, a single compactor generate cycle performs billions of allocating key parses — and several of these jobs run every cycle (`run_generate_job`, retention, stats) or every 60s (metadata metrics). ## Changes **New API:** `list_all_streams_grouped() → HashMap<org, HashMap<StreamType, Vec<name>>>` — one pass over the cache, allocation-free key parsing (iterator-based split instead of `split().collect::<Vec<_>>()`), same name-truncation and dedup semantics as the per-pair function. Covered by a unit test that also asserts it matches `list_streams_from_cache` output. **Callers rewritten to one scan per cycle:** - compaction `run_generate_job` and (enterprise) `run_generate_downsampling_job` - compaction retention `generate_jobs` - compaction stats `update_stats_from_file_list` - flatten compactor `run_generate` - metadata metrics job (also derives the org list and org-count metric from the same grouped result, dropping the separate `list_organizations_from_cache` full scan) - trace service-graph stream discovery (orgs still come from the org table; the per-org cache scan is gone) - CLI `reset_index_updated_at` and `gc-file-list` Per-type skip conditions (EnrichmentTables/Filelist in retention and gc, Index/Filelist in stats, Logs-only in flatten, Logs/Metrics/Traces in metrics) and the org blocklist checks are unchanged. Iterating the grouped result is equivalent to iterating `ALL_STREAM_TYPES` since that constant covers all `StreamType` variants and absent groups were empty scans before. `list_streams_from_cache` stays for point callers (per-stream lookups, `get_stream_count_cached`'s TTL cache is untouched). ## Testing - new unit test `test_list_all_streams_grouped` (grouping, per-type contents, equivalence with the per-pair enumeration) - test suites pass for db, compaction, jobs, core; the only failures are the two pre-existing sqlite-path retention tests that also fail on main - clippy: no findings in touched files Note: overlaps #13531 (metrics.rs/stats.rs hunks) — whichever merges second rebases trivially. Independent of #13535. |
||
|
Hengfei Yang
|
93b3fec5cf |
perf(cache): remove per-org full scans and hot-path shrink_to_fit calls (#13531)
Design at: #13465 PR 1 of the plan in #13465 (the mechanical fixes: findings 3, 4, 5). ## Changes **Metadata metrics job (finding 3)** — `update_metadata_metrics` counted users by scanning all of `ORG_USERS` once per organization, with a `format!` allocation inside the inner loop: O(orgs × org_users) iterations and allocations every 60 seconds on the compactor. Replaced with a single pass that aggregates counts per org (`count_org_users()`); per-org metric output is unchanged, including 0 for organizations with no users. Covered by a unit test. **Incremental stats (finding 4)** — `update_stats_from_file_list` checked every enumerated stream against `updated_streams: Vec<String>` with a linear `contains`, O(streams × updated) and effectively quadratic when most streams changed. The list is now collected into a `HashSet` once; the empty-set-means-update-all semantics are preserved. **Per-operation `shrink_to_fit` (finding 5)** — removed from: - the schema watcher delete path, which shrank four global maps (`STREAM_SCHEMAS`, `STREAM_SCHEMAS_LATEST`, `STREAM_RECORD_ID_GENERATOR`, `STREAM_SETTINGS`) after every single-stream deletion — bulk deletes rehashed a large table repeatedly while holding write locks; - the WAL search-lock release paths (`release_files`, `release_request`), which shrank `SEARCHING_FILES`/`SEARCHING_REQUESTS` on every request completion. `shrink_to_fit` rehashes the entire table but only reclaims the bucket array — the entry payloads are already freed by `remove()` — so these calls cost O(map size) under a write lock for a few MB upper-bound of savings. The shrink inside the rare full-clear `clean()` path is kept. **Test flakiness fix (pre-existing)** — the WAL locking tests mutate the shared global map and run in parallel; `clean_lock_files()` from one test could clear locks another test had just asserted on. Verified flaky on main (2 of 3 runs failed). The tests that assert on lock presence now serialize on a test-local mutex; 5 consecutive full runs pass. ## Testing - `cargo test -p common infra::wal` — 6 passed, stable across 5 runs - `cargo test -p compaction stats` — 10 passed - `cargo test -p openobserve-jobs job::metrics` — new `test_count_org_users` passes - `cargo clippy --all-targets` clean on the four touched crates Findings 1 (settings cache Arc-ification + mutation centralization) and 2 (grouped stream enumeration) follow in separate PRs per the plan. |
||
|
Huaijin
|
c723c4322f |
chore: add Vortex native compression toggle (#13533)
## Summary - add ZO_VORTEX_USE_NATIVE_COMPRESSION with a default of false - keep the custom UTF8/Zstd compressor as the default and allow opting into the native Vortex strategy - share the configured strategy across regular merge and downsampling writes - move Vortex writing into the merge call site and keep the custom compressor private ## Testing - cargo check -p search - cargo test -p search datafusion::vortex::tests --lib - cargo fmt --all -- --check - git diff --check |
||
|
Huaijin
|
17ef03b8d6 |
refactor: cut release build time in the web and telemetry paths (#13528)
Two changes that cut build time, both measured on the same machine (M2
Max,
12 cores) with `cargo build --features mimalloc --profile release`.
## 1. The `web` crate no longer depends on `config`
`src/web` is a leaf crate so the RustEmbed expansion over `web/dist`
only
re-runs when the assets change — that expansion is 149.8s, of which
144.1s is
macro expansion (the crate already sets `#[br = false]` / `#[gzip =
false]`, so
this is not compression).
It also depended on `config`, for exactly one value:
`cfg.common.base_uri`. That
put a 50MB embed downstream of a 724-crate subgraph, so **every `config`
edit
paid the 148.5s embed again** — which is precisely what the crate's own
doc
comment says it exists to avoid.
`ui_routes()` now takes the `<base href>` value from the caller and
`base_href_middleware` carries it as middleware state.
`create_app_router()`
already computed that exact string (`"{base_uri}/web/"`) for its own
redirect,
so it is now resolved once and passed in rather than recomputed per
request
from a second `get_config()` call. `base_uri` is env-derived, normalised
once at
config init and never touched by `refresh_config`, so hoisting it out of
the
request path cannot drift.
The web crate's dependency tree goes **724 → 92 crates**, and `cargo
tree -i
config` no longer lists it.
## 2. Logging and tracing setup moved out of `main.rs`
`setup_logs`, `enable_tracing` and their span-processor / exporter types
compose
very large generic trees. Everything `main.rs` monomorphizes is emitted
into the
binary's own codegen units — the last unit on the critical path,
overlapping
with nothing.
`-Ztime-passes` on the binary unit (91.1s total):
```
LLVM_thinlto 39.8s
LLVM_passes 37.5s
codegen_to_LLVM_IR 5.3s
link_binary 5.3s (run_linker 3.9s)
type_check_crate 0.4s
```
The `openobserve` library does not depend on `openobserve-api-http`, so
its unit
compiles in parallel with the API crates — in the cold build it finished
23.9s
before the binary needed it. Moving the setup into `src/telemetry.rs`
(563
lines) puts that work in the slack instead of at the tail.
Two tests moved with the code and were dropped rather than relocated:
`test_setup_logs` and `test_enable_tracing_error_handling` both install
a global
tracing subscriber, and `setup_logs()` calls `.init()` unguarded, so
they can
only ever pass depending on ordering. They used to race just with each
other
inside the bin test binary; keeping them would have moved that race into
the lib
test binary next to the `cli` and `migration` tests. Neither asserted
anything —
the second wraps the call in `catch_unwind` and discards the result.
`main.rs` 1304 → 712 lines. **Binary unit 88.8s → 80.7s**; the library
unit grows
30.9s → 62.3s and is fully absorbed by the existing slack.
## 3. Unrelated: a dead-code warning in `openobserve-core`
Not a build-time change, just a warning fix noticed while checking the
above.
`COMPARE_ROLLUP_LOOKBACK_MICROS` was gated on `any(feature =
"enterprise",
test)`, but its only two readers (`fetch_arm_aggregate`,
`fetch_arm_failures`)
are `enterprise`-only and the unit tests below it exercise only the pure
aggregation helpers. So in an OSS build `cargo check -p openobserve-core
--all-targets` warned `constant COMPARE_ROLLUP_LOOKBACK_MICROS is never
used`.
Gate narrowed to `enterprise`, and the doc comment — which claimed the
opposite
— corrected. Happy to split this out if you'd rather it not ride along.
## 4. Review follow-ups
- `enable_tracing` became a public library API in §2 without a doc
comment,
unlike `setup_logs` beside it. Documented: which span processors attach
under
which env vars, why the returned provider must be held rather than
dropped,
and that it and `setup_logs` are mutually exclusive since both call
`.init()`.
- Added the `<base href>` test described under Verification below.
Not taken: the suggestion to re-add `test_setup_logs` /
`test_enable_tracing_error_handling` to `src/telemetry.rs`. Both install
a
global subscriber against `.init()`, so they cannot be made
order-independent,
and neither asserted anything — see §2.
`test_resource_key_value_creation` was
deleted for the same reason; it asserted that
`opentelemetry::KeyValue::new`
stores the key it was given.
## Measured and rejected
Recording these so nobody repeats them.
**Moving route registration into the crates that own the handlers.** 317
of the
371 routes were registered in one 622-line function in
`openobserve-api-http`.
Splitting them out to api-management (234), api-search (46),
api-pipelines (23)
and api-ingest (9) worked as designed — api-http went 145.0s → 63.0s —
but the
cold build only improved 654.4s → 637.7s. Splitting across a crate
boundary
**copies** work rather than relocating it: without LTO each crate
instantiates
the axum/tower machinery for itself, so the four receiving crates grew
222.1s
combined and total CPU rose 190.8s. It also just relocated the
bottleneck —
api-management became the new tail at 135.1s, with api-search finishing
only
6.9s behind it, so the tail is a plateau rather than a peak. Not carried
in this
branch.
| Tried | Result |
|---|---|
| sccache | 97.4% hit rate, **0** wall-clock gain; first fill +54% |
| Remove aws-lc-sys from the critical path | **0** — its `dur=72.0s` in
`--timings` is contention, it needs 15.6s alone |
| `codegen-units = 16` on the glue crates | **31.4s slower**, CPU
+297.8s — `lto` defaults to `false`, so thin-local LTO is on and its
cost scales with CGU count |
| Trim DataFusion features | `datafusion-core` depends on
`datafusion-functions` without `default-features = false`, so the
expression features cannot be turned off from the facade at all; what
remains is worth 13.8s of CPU (0.25%) |
A caveat on reading `cargo --timings`: a unit's `duration` is **not**
its
intrinsic cost — it is inflated by CPU contention when the machine is
saturated.
The aws-lc entry above is the clearest example.
## Verification
At this revision:
- `cargo check --all-targets` clean on OSS — 0 errors, 0 warnings — and
each
commit checks clean standalone, so `git bisect` works
- `cargo test -p web` passes
Carried over from the revision these changes were first written against
(the
edits since are doc comments, the `ui_routes` parameter rename, two
deleted
tests, a `Cargo.lock` revert, and the `cfg` narrowing in §3 — which is a
no-op
when `enterprise` is on):
- `cargo check` clean on the enterprise feature combination
- Cold release build succeeds
- Smoke-tested against a running binary: `/healthz` 200, UI and API
routes
resolve, unknown paths 404
The `<base href>` rewrite — the one runtime behaviour §1 touches — is
now
pinned by `base_href_is_rewritten_to_the_mount_point`, and checked end
to end
against the real embedded `index.html`: `ui_routes("/abc/web/")` serves
`<base href="/abc/web/" />`, matching what the old per-request
`format!("{base_uri}/web/")` produced for `ZO_BASE_URI=/abc`. The
committed test
drives a synthetic HTML response instead, so it still asserts something
in
builds where `web/dist` is absent. Verified by mutation: breaking the
rewrite
fails it.
|
||
|
Shrinath Rao
|
184268ff1b |
test(ci): loud stdout banner when Playwright retries a test (#13529)
## What Adds a small **log-only** Playwright reporter that prints a LOUD, unmissable banner to stdout **the moment any test starts a retry attempt**, plus an end-of-shard summary listing every test that was retried. ## Why In CI the reporter is `blob`, which writes nothing to stdout. So when a failed test is retried (`retries: process.env.CI ? 3 : 0` in `playwright.config.js`) there is **no signal in the GitHub Actions logs that a retry is even happening** — a test silently re-runs up to 3 more times and the log just shows the next `Test Started` line. This makes flaky-vs-real failures hard to spot when scanning CI output. The retry is internal to `npx playwright test`, so the workflow YAML can't see it — it has to be emitted from the Playwright reporter layer. This is that layer. ## What it looks like ``` 🔁🔁🔁══════════════════════════════════════════════════════════🔁🔁🔁 🔁 R E T R Y # 1 · attempt 2 of 4 (test previously failed — re-running) 🔁 ▶ chromium › logsqueries.cte.spec.js › should query with simple TotalCols CTE 🔁 ▶ Logs/logsqueries.cte.spec.js 🔁🔁🔁══════════════════════════════════════════════════════════🔁🔁🔁 ``` …and once per shard at the end: ``` 🔁 RETRY SUMMARY — N test(s) were retried in this shard: 🔁 • [2 retries] chromium › … › should query … (Logs/logsqueries.cte.spec.js) ``` ## Scope / safety - Added to the **CI** reporter list only, alongside `blob` (multiple reporters are fine). It only `console.log`s — never sends anything, never fails the run. - **ENT gets this for free**: the ENT Playwright workflow checks out and runs the OSS `tests/ui-testing` tree, so this one OSS change makes retries loud in both OSS and ENT CI. There is no ENT test tree to change. - Verified locally with a forced-failure probe (`CI=true`, `retries: 2`): the per-retry banner fires on attempt 2 and 3, and the summary lists the retried test. ## Files - `tests/ui-testing/playwright-tests/utils/retry-banner-reporter.js` — new reporter - `tests/ui-testing/playwright.config.js` — wire it into the CI reporter array |
||
 Sai Nikhil
|
d0fca58714 |
fix: should restrict patterns count api on logs page (#13515)
should restrict the patterns count call on logs page , search or visualize mode ent #2292 |
||
 ktx-akshay
|
8ef7988e5d | fix: extended retention date picker rendering blank for streams with unlimited retention (#13510) | ||
|
Hengfei Yang
|
2bf19b2b44 |
feat(mcp): support MCP spec revision 2026-07-28 (stateless, dual-era server) (#13527)
Design at: #13526 ## Summary Implements MCP specification revision **2026-07-28** (stateless protocol) alongside the existing 2025-11-25 behavior on the same endpoint, following the spec's dual-era server model: requests carrying modern per-request `_meta` are served per 2026-07-28, while `initialize` selects legacy semantics. Legacy clients are unaffected. ## Changes - **Version negotiation** (`src/api/http/.../mcp/mod.rs`, `src/mcp/src/protocol.rs`): the `MCP-Protocol-Version` header is now validated against a supported-version set (`2026-07-28`, `2025-11-25`) instead of strict equality — the old check would have rejected every client that upgrades to the new revision. Unsupported versions (header or body `_meta`) get `400` + JSON-RPC `-32022` (`UnsupportedProtocolVersionError`) listing the supported versions so clients can retry. - **`server/discover`** (`src/mcp/src/protocol.rs`): new RPC required by 2026-07-28, returning `supportedVersions`, `capabilities`, server identity in `_meta`, `instructions`, and `ttlMs`/`cacheScope` cache fields. - **Modern result decoration** (`src/mcp/src/protocol.rs`): requests declaring `io.modelcontextprotocol/protocolVersion: 2026-07-28` in `_meta` get `resultType: "complete"` and `_meta` server identity on results, plus `ttlMs`/`cacheScope` on `tools/list`. Legacy responses carry none of these fields — byte-for-byte unchanged. - **Request metadata headers** (`src/api/http/.../mcp/mod.rs`): modern requests must mirror body fields into `Mcp-Method` / `Mcp-Name` headers (including the Base64 sentinel encoding for `Mcp-Name`); missing or mismatched headers are rejected with `400` + JSON-RPC `-32020` (`HeaderMismatch`), per the transport's server-validation rules. - **HTTP status mapping**: unknown methods on modern requests return `404` + `-32601` so dual-era clients can distinguish a modern server from a legacy one; version errors return `400`. Legacy requests keep errors in-band with `200` as before. - **Legacy paths untouched**: `initialize`, `ping`, the GET SSE keepalive channel, DELETE no-op, and notification → `202` handling are unchanged. Tool ordering was already deterministic (sorted at init), satisfying the new caching guidance. ## Testing - `cargo test -p openobserve-mcp` — 94 tests pass (new coverage: `server/discover` shape, unsupported-version error payload, modern decoration, legacy non-decoration, sentinel decoding, version set). - `cargo test -p openobserve-api-http --lib handler::http::request::mcp` — 13 tests pass (new coverage: modern happy path, missing/mismatched `Mcp-Method`/`Mcp-Name`, header↔body version mismatch, missing version header on modern requests, modern unknown method → 404, legacy header accepted). - `cargo clippy` clean on both crates. |
||
 Loakesh Indiran
|
cc155fca6c |
fix(synthetics): report a folder that still holds checks, and lift run-detail contrast (#13524)
Two unrelated synthetics fixes, both small. ## 1. Deleting a folder that still contains checks Deleting a synthetics folder with checks in it showed the user a raw database error: ``` DbError# SeaORMError# Execution Error: error returned from database: (code: 787) FOREIGN KEY constraint failed ``` Alerts handles the same situation properly — *"Folder contains alerts, please move/delete alerts from folder"*. ### Cause The guard already existed. `FolderError::DeleteWithSynthetics` is defined, and `folder_error_response` already maps it to *"Folder contains synthetics, please move/delete synthetics from folder"*. It simply never fired, because the two sides of the comparison are different identifiers: ``` folders: id = 3H8CpCLKyBqPokByb5pKetSlgnw folder_id = 7487841328038936576 name = test synthetics: folder_id = 3H8CpCLKyBqPokByb5pKetSlgnw <- stores folders.id ``` The delete endpoint receives the user-facing `folders.folder_id` (`7487841328038936576`), but `ListSyntheticsParams.folder_id` filters `synthetics.folder_id`, which holds the folder's **primary key**. The count returned 0 for every folder, the guard passed, and the database rejected the delete on the foreign key. Dashboards, alerts and reports were unaffected: their list queries reach the folder through a join and filter on `folders.folder_id`, so they compare like with like. ### Fix Translate the id with the existing `folders::get_pk_by_name` before counting. A folder that doesn't exist falls through to the `exists` check below, which already reports `NotFound`. ## 2. Run-detail contrast Every field label in the protocol run detail — Status, Response time, Error, Location, Runtime, Probe ID, Trigger, plus the not-evaluated assertion rows — used `text-text-muted`. In light mode that token resolves to `grey-300`, and its own definition documents it as *"disabled, placeholder, de-emphasized"*. These are captions, not disabled controls, so they now use `text-text-secondary` (`grey-500`). ## Testing - `cargo check -p db --features enterprise` — clean. - `cargo test -p db --features enterprise folders` — 14 pass. - `ProtocolRunSummary.spec.ts` — 22 pass. The id mismatch was confirmed against a live local SQLite metadata DB (the row values quoted above are real), and the failing delete reproduced exactly the 500 in the screenshot. Not covered by an automated test: `delete_folder` has no DB-backed test harness today, and the existing tests in that module only assert on error-variant construction. Worth adding one if this area gets touched again. |
||
 IluckySiandHengfei Yang
|
c278682cca |
feat(web): add categraf option in custom metric ingestion (#13466)
This PR adds categraf as a explicit custom metric ingestion. The configuration as follows： > [[writers]] > url = "http://localhost:5080/api/default/prometheus/api/v1/write" > basic_auth_user = "root@example.com" > basic_auth_pass = "o2oi_dMJ0RsB50MzVt2CVPCripths0WaGE832" <img width="3064" height="1162" alt="image" src="https://github.com/user-attachments/assets/189c3c96-441c-4476-8c31-943acaa788b0" /> <img width="3840" height="1862" alt="image" src="https://github.com/user-attachments/assets/0408704b-b1d1-4546-8d8b-86dd5895d8bc" /> Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com> |
||
|
Hengfei Yang
|
94bddd7a7b |
feat: pack ingester wal persist into large files to avoid massive small files (#13497)
Design at: https://github.com/openobserve/openobserve/issues/13509 Closes #13509 ## Problem On clusters with 100k+ metrics streams, every memtable rotation persists **one parquet file per stream × hour**, producing thousands of tiny files per second on the ingester. The legacy file mover cannot keep up: it re-scans the whole `wal/files` tree every `file_push_interval` (capped at `file_push_limit` = 10000 files per cycle ≈ a hard ceiling of ~1000 files/s), repeatedly stats not-yet-expired files, and pays a full DataFusion plan + S3 PUT per stream×hour for a few KB of data. Generation outpaces consumption and wal files pile up unboundedly. ## Design Behind `ZO_FEATURE_WAL_PACK_ENABLED` (bool, default **off**), applying to **metrics streams only** — the stream-cardinality problem is metrics-specific; other stream types keep the legacy per-stream persist path. With the flag on, the "split by stream" step moves from persist time (disk) to upload time (memory): - **Persist**: all streams of a memtable are appended into a few large **pack** files (`wal/pack/{idx}/{memtable_id}.{seq}.pack`, rolled at `ZO_MAX_FILE_SIZE_ON_DISK`, default 512MB). Each segment inside a pack is a **complete, self-contained parquet file**; the pack footer (JSON + gxhash trailer, versioned) indexes segments with stream, partition key, offset/length and FileMeta. One rotation now writes O(1) sequential files instead of O(streams) tiny ones. - **Query**: an in-memory segment index (rebuilt from footers on startup) resolves stream → byte ranges; segments are materialized into record batches and reuse the memtable search table-building path. Dedup against in-memory data uses the existing memtable_ids mechanism. - **Mover**: pending streams are snapshotted from the segment index (no readdir/stat storm, no `file_push_limit` cap). A stream is flushed when its segments are big enough or the oldest exceeds `max_file_retention_time`. A **single segment is uploaded as-is (zero decode/re-encode)**; multiple segments are merged in memory via DataFusion MemTable + the existing `merge_parquet_files`. Object keys, file_list, incremental compaction trigger, tantivy index, cache_latest_files and enterprise service-streams all follow the legacy mover. **S3 layout, file_list schema, compactor and querier are completely unchanged.** - **Crash recovery**: packs follow the same `.lock` five-step flow as legacy files (`.pack.tmp` → `.lock` → delete wal → rename → unlock), keeping exactly-once semantics. Lock recovery stays synchronous on startup (it must run before the pack index rebuild), while the orphan `.par` sweep over the potentially millions-of-files `wal/files` tree moved to a background task so startup no longer blocks. - **Pack recycling**: a registry tracks per-pack `{total, consumed, readers}`; a pack file is deleted once all segments are consumed and no in-flight query read holds it (read guards replace the legacy `lock_files`/pending-delete mechanism for packs). Also included: shared-memtable hash changed from `{thread_id}_{org_id}` to `{org_id}_{stream_name}` so one stream maps to exactly one bucket, removing the per-request round-robin fan-out that multiplied file/segment counts. ## Testing - Unit tests: pack format roundtrip, size-based rollover, corruption/truncation rejection, real-parquet segment read through the index with time-range/memtable-id filtering, consume lifecycle (pack deleted when fully consumed), reader-guard deferred deletion, flush conditions. - End-to-end with the real binary (4 phases): 1. wal binary format is byte-identical to main (`git diff main... -- src/wal src/ingester/src/entry.rs` is empty); 2. old-format wal written with the flag off **replays into packs** with the flag on — no migration needed; 3. mixed-source search returns exact counts with legacy wal parquet + pack segments + memtable coexisting (300/400/500 across three batches, filters and row fetches correct); 4. pack mover merges multiple segments into one storage file, generates the tantivy index, deletes consumed packs; final storage-side queries and `match_all` are exact. ## Unrelated fixes included Two startup paths shared the same "treat any db error as absence" bug: - `init_db` used to treat ANY error from `get_db_schema_version` as version 0 ("assuming default of 0 and trying upgrade") and kicked off the full db upgrade — so a merely unreachable or overloaded database got migration DDL piled on top of its existing load. Now `get_db_schema_version` returns 0 only when the version is genuinely absent (missing key, or the meta table itself not created yet on a fresh install); transient errors are retried 5 times with backoff and then abort the startup so the orchestrator restarts the node once the database recovers. - bootstrap did the same with the **instance id**: `instance::get()` reported a missing key as an error, so "not stored yet" and "db unreachable" landed in one arm that generated a fresh instance id — and could overwrite the stored one once the db recovered. The instance id must stay stable for the lifetime of a deployment, so a new one is now generated only when it is genuinely absent; read errors are retried with backoff and then abort the startup, and a failed write of a newly generated id is no longer ignored. ## Notes for reviewers - Flag is default **off**; with the flag off, behavior is unchanged except the startup fixes and the shared-memtable hash change. The flag only gates the persist path — pack search always runs (no-op on an empty index), so packs already on disk stay searchable after the flag is turned off while the mover drains them. - Verified with the real binary that with the flag on, a metrics stream is packed while a logs stream keeps the legacy per-stream parquet, both queryable (SQL + PromQL). - Pairs with o2-enterprise branch `feat/ingester-wal-pack-drain` (drain completion check also scans `wal/pack`); that change must land before/with enabling the flag in production, otherwise a draining node could shut down with pack segments not yet uploaded. - Known follow-ups (not in this PR): pack-specific metrics, extending pack to other stream types, large-scale load test for segment-index memory sizing, `wal-reader`-style debugging tool for packs. |
||
 ktx-vaidehi
|
c9fc8d3b55 | fix: dashboard builder redesign (#13419) | ||
 Abhay Padamani
|
5a0b10e7af |
fix: _timestamp alias migration (#13489)
## How `rewriteQueryTimestampAlias` works
Example query:
```sql
SELECT a AS _timestamp FROM t GROUP BY _timestamp
```
### Step 1 — walk the string, one token at a time
Move left to right, grouping characters into tokens (words, quotes,
punctuation). Keep a tiny bit of state: **"which clause am I in"** and
**"did I just see AS"**.
### Step 2 — update state on keywords, decide on `_timestamp`
| Token | State update | If it's `_timestamp`, decision |
|---|---|---|
| `SELECT` | clause = SELECT | — |
| `a` | — | — |
| `AS` | expectAlias = true | — |
| `_timestamp` (chars **12–21**) | — | in SELECT + just saw AS → **alias
definition** → record an edit |
| `FROM` | clause = FROM | — |
| `t` | — | — |
| `GROUP` | clause = GROUP | — |
| `BY` | (stay in GROUP) | — |
| `_timestamp` (chars **39–48**) | — | in GROUP + we saw an alias →
**reference** → record an edit |
### Step 3 — the edits we recorded (just notes, nothing changed yet)
```
edit 1: replace chars 12–22 with "ts"
edit 2: replace chars 39–49 with "ts"
```
That's it — the scan's only job is to find the **character positions**
of alias `_timestamp`s. It figures out positions itself (by tracking
clause state), instead of asking a parser for them.
### Step 4 — rebuild by copying the original + splicing
```
out = sql.slice(0, 12) → "SELECT a AS "
+ "ts"
+ sql.slice(22, 39) → " FROM t GROUP BY "
+ "ts"
+ sql.slice(49) → ""
```
Result: `SELECT a AS ts FROM t GROUP BY ts`
### Why this guarantees "only the alias changed"
Everything in the output comes from `sql.slice(...)` — literal copies of
the original — **except** the spliced `"ts"`. There's no re-generation
step (no parse/stringify), so nothing else *can* change.
|
||
 Shrinath RaoandClaude Opus 4.8
|
6af766f651 |
test: fold Kinora upload into Merge Reports job (no re-merge) (#13513)
## What Moves the Kinora upload **into the `merge_reports` job** instead of a separate `report_kinora` job, and renames it to **`Merge Reports & Report to Kinora`** so it's obvious the upload happens there. ## Why The standalone `report_kinora` job re-downloaded every `blob-report-*` artifact and re-ran `playwright merge-reports` before uploading — repeating exactly what `merge_reports` already did. On large runs that step was **hitting the 20-min timeout** (the merge + sequential trace uploads). Now the Kinora upload + sticky PR comment run as the **last two steps of `merge_reports`**, reusing the `report.json` + extracted `trace.zip`s already on that runner. **No second download, no second merge** — the upload just piggybacks on the merge that already happened. ## Details - Renamed job `Merge Reports` → **`Merge Reports & Report to Kinora`**; removed the separate `report_kinora` job. - Added `pull-requests: write` to the job (for the sticky comment). - Still **non-fatal** (`continue-on-error`) — can't fail the build; still gated by `UPLOAD_TO_KINORA`. - Reuses `playwright-results/report.json` (traces resolve because they were extracted on this same runner). Companion enterprise PR uses the same branch name (`test/kinora-in-merge-job`). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
 
|
7d212f4621 |
feat: workflows incident connection (#13472)
This connects incidents to workflows, so any lifecycle event of incident will trigger workflow. design at: https://github.com/openobserve/designs/blob/main/infrastructure/workflows/design.md --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> Co-authored-by: Shrinath Rao <shnath@openobserve.ai> |
||
 Harsh Mahajan
|
bf901df035 |
fix(incidents): anchor Created at the start of a reconstructed timeline (#13511)
## What Fixes event ordering in the reconstructed incident timeline: `Created` now always opens the timeline, instead of being sorted below an earlier alert. ## Why `build_reconstructed_events` sorted all events by timestamp alone. But the alert that triggers an incident is recorded in `alert_incident_alerts` *before* the incident row is written, so `alert_fired_at` can legitimately precede `created_at`. On a real local incident the gap was ~464ms (`..419417` vs `..883003`). The result was a rebuilt timeline that opened with an `Alert` and only then showed `Created` — an incident whose first alert appears to arrive before the incident exists. **Before** Alert, Created, Alert, Alert, Alert, Resolved **After** Created, Alert, Alert, Alert, Alert, Resolved ## How `Created` is prepended after the sort rather than sorted in with everything else. An incident cannot precede its own creation, so it is pinned to the front; every other event keeps its true chronological position. No event is reordered relative to any other, and no timestamp is altered. Only affects the reconstruction path for timelines that were emptied by the pre-fix read/write cycle. Intact timelines are untouched. |
||
 Bhargav
|
e9596a10a3 |
feat: Mobile rum sdk v2 (#13503)
Design at: design/mobile-rum-sdk-evaluation/design-docs Backend changes exist to make the OpenObserve RUM intake accept the mobile SDK wire format alongside the existing browser format. src/api/src/handler/http/request/rum/ingest.rs 200 → 202 Accepted on /rum, /logs, /replay | ---> OpenObserve-family SDKs (which the mobile forks derive from) treat any non-202 as an unexpected error and drop the batch. #[serde(default)] on Event ---> Mobile replay omits creation_reason / index_in_view that the browser format sends. Multi-segment replay handling ---> Browser sends 1 segment + 1 event object; mobile sends N segment parts + one event JSON array. Handler now collects all segments, parses event as array-or-object, pairs them, emits NDJSON → one _sessionreplay row per segment. MAX_SEGMENTS=200, 16 MB decompressed cap via .take() ---> Decompression-bomb guard, newly needed now that N segments are accepted. |
||
 udevandsai nikhil kethe
|
24938430db |
fix(evals): ensure o2_ingest_ts in defined schema (#13506)
commit-id:e9e87fba --- **Stack**: - #13506 ⬅ - #13491 - #13481 - #13453 - #13428 --- ⚠️ *Part of a stack created by [spr](https://github.com/ejoffe/spr). Do not merge manually using the UI - doing so may have unexpected results.* --------- Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> |
||
|
ktx-akshay
|
6e0a7374e5 |
test: add DashboardFavorites page object and associated tests for fav… (#13362)
## Summary Adds a Playwright E2E test suite covering the dashboard Favorites feature — favoriting/unfavoriting a dashboard, the `Favorites` pseudo-folder view, and its interaction with single and bulk delete flows. ## What's covered New `dashboard-favorites.spec.js` (6 tests) + `DashboardFavorites` page object: - Add a dashboard to favorites and verify it appears in the Favorites view - Remove a dashboard from favorites — confirms it drops from the Favorites view while the dashboard itself remains untouched - Favorite state persists across a page reload - Bulk-delete a favorited dashboard from the Favorites view completes successfully with no error - Deleting a dashboard directly also removes it from Favorites automatically - Bulk-deleting a dashboard from the Favorites view doesn't leave a stale copy behind in its original folder ## CI `dashboard-favorites.spec.js` runs in its own dedicated `Dashboards-Isolated` shard with a fresh backend instance, keeping it independent from the rest of the dashboard test suite. |
||
|
Loakesh Indiran
|
967433fe9e |
fix(synthetics): stop the HTTP check card vanishing on non-GET methods (#13505)
## Problem
On a new HTTP / API check, picking **POST** (or any method other than
GET/HEAD) makes the entire **HTTP Request** card disappear — method,
timeout, follow-redirects, headers, assertions, all of it. The console
shows only:
```
SyntaxError: 2
at ... nextToken ... parse ...
```
## Cause
The request-body field is the only thing that renders for non-GET/HEAD
methods, and its placeholder message was the literal text `{"key":
"value"}`:
```json
"bodyPlaceholder": "{\"key\": \"value\"}",
```
To vue-i18n `{...}` is an interpolation slot, so its parser hits `"`
where an identifier must be:
```
SyntaxError: Invalid token in placeholder: '"key":'
```
Messages compile lazily (`__INTLIFY_JIT_COMPILATION__: true` in
`vite.config.ts`), so nothing happens until the message is first
rendered. The throw then lands inside render and tears down the whole
`CheckHttpConfig` subtree — which is why the card vanishes, and why it
only vanishes for the methods that show a body.
## Fix
Escape the braces as `{'{'}` / `{'}'}`. Rendered output is
byte-identical — verified through `createI18n().t()`:
```
raw -> "{\"key\": \"value\"}" (logs a compilation error, throws under the app's config)
escaped -> "{\"key\": \"value\"}" (compiles clean)
```
Applied across all 15 locale files, since the crash is
locale-independent.
Two other **en-US** messages carried the same latent crash and are
escaped the same way:
- `pipeline.streamSelectionNameInvalid` — a raw `@`, parsed as
linked-message syntax
- `aiToolset.genericDataPlaceholder` — `{}`, an empty placeholder
## Guard
New `enUsMessages.spec.ts` compiles every en-US message the way vue-i18n
does (`baseCompile` with a rethrowing `onError` — the default collects
errors and returns a best-effort AST, hiding exactly this). The next
such message fails CI instead of silently deleting a form section.
The other locales still have ~250 broken messages, overwhelmingly
placeholder *names* that were translated (`{方向}`, `{秒}`). Those break
interpolation semantics as well as compilation and need per-locale
review — out of scope here, so the guard is en-US only for now.
## Testing
**Before the fix**, driven in a real browser against a local build:
```
card on load: 1 | aria-expanded: true | options: 7
select POST -> card still there: 0 | HTTP Request heading: 0
[console] SyntaxError: 2
```
**After the fix**, every method exercised through the real select:
| method | selected | card | body field | assertions | new console
errors |
|---|---|---|---|---|---|
| POST | POST | 1 | 1 | 1 | 0 |
| PUT | PUT | 1 | 1 | 1 | 0 |
| DELETE | DELETE | 1 | 1 | 1 | 0 |
| PATCH | PATCH | 1 | 1 | 1 | 0 |
| HEAD | HEAD | 1 | 0 | 1 | 0 |
| OPTIONS | OPTIONS | 1 | 1 | 1 | 0 |
| GET | GET | 1 | 0 | 1 | 0 |
Body field visibility is correct in every case (hidden for GET/HEAD
only), zero console errors throughout.
**Full round-trip**, also in the browser — creating a non-GET check was
impossible before this fix:
- Create with POST + body `{"ping": "pong"}` → `POST
/api/default/synthetics` → **200**, redirect to the list, 0 console
errors. Wire payload carries `"method":"POST"` and the body verbatim.
- Reopen for edit → card present, method `POST`, body field populated
with `{"ping": "pong"}`, 0 console errors.
**Unit tests:** 598 passed, 0 failed across `locales`,
`components/synthetics`, `views/synthetics`, `utils/synthetics`,
`composables/synthetics`, including the new `enUsMessages.spec.ts`.
One suite, `views/synthetics/MonitorResults.spec.ts`, fails to load with
`Cannot read properties of undefined (reading 'value')`. That is
pre-existing and unrelated — `useLocalTimezone` gets no localStorage
under Node 26's vitest environment, and it fails identically on a clean
tree.
|
||
|
ktx-akshay
|
3af2a67727 |
fix: independent scrolling for dimensions list and charts in Insights drawer (#13504)
## Summary Fixes shared scrolling in the Insights drawer (Logs → Analyze) where the Dimensions sidebar and the charts panel scrolled together as one unit. With many dimensions (e.g. 40+ Kubernetes fields), users had to scroll through the entire Dimensions list just to reach the charts below. ## Root cause The dashboard content relied on `flex-1` / `min-h-0` to constrain its height, but its parent container wasn't a flex container, so those styles had no effect. The content simply grew to fit everything, and the drawer's outer scrollbar ended up scrolling both panels together instead of letting each one scroll independently. |
||
|
Harsh Mahajan
|
542e3420d5 | chore: incident small fix (#13493) | ||
 Ashish Kolhe
|
60b1942bf2 |
fix: agent env version filters (#13488)
Makes **environment** and **version** first-class identity dimensions for GenAI agents, and adds an A/B **version-compare** experience to LLM Insights. ### What's in it - **Schema / identity (OSS, ungated):** migration adds `env` + `agent_version` to `gen_ai_agents`; entity/table ops and agent-mapping config carry them through; env/version surfaced across all AI screens via a shared Env→Agent→Version cascade. - **Version compare (LLM Insights):** pick two versions of an agent, align their windows (**Since each rollout** / **Same wall-clock** / **Manual**), and read latency / cost / error-rate deltas with confidence intervals plus a tri-state failure-class diff (introduced / fixed / shared). - **Handlers + service wiring:** `#[cfg(feature = "enterprise")]` call sites for the compare/agent-signals endpoints (enterprise impl + OSS 403 stub); the sketch-merge compute lives in the companion enterprise PR. ~120 files; primarily `web/src` (AI Observability + traces) and `src/infra`/`src/core`/`src/config` for the schema and thin service layer. |
||
|
Loakesh Indiran
|
72be79e829 |
fix(synthetics): send a readable timestamp in run notifications (#13500)
## Problem Synthetics Slack alerts render the run time as a raw unix epoch: ``` 🔴 EU Cloud Login Check is failing Monitor: EU Cloud Login Check (browser) Target: https://eu1.openobserve.ai/ Locations checked: 2 Time: 1785202020 <-- unreadable ``` The line used Slack's date markup with the epoch as its **own fallback**: ```rust format!("*Time:* <!date^{checked_secs}^{{date_time_secs}}|{checked_secs}>") ``` Wherever the markup isn't interpreted — which is what happens on these webhook posts — the reader is left with the bare number. Email and SNS were worse: neither carried a timestamp at all. ## Change Format the stamp server-side (`checked_at_utc`) and use it as the fallback, so the markup still localises the time for anyone whose client honours it and degrades to something readable for everyone else: ``` Time: 2026-07-28 01:24:51 UTC ``` Added the same stamp to the email card (new Time row) and the SNS plain-text body, so every destination answers "when did this fail?" without a converter. ## Testing Three unit tests on the formatter: normal micros → UTC string, sub-second remainder truncated rather than rounded, and out-of-range input falling back to the raw value rather than dropping the field. `cargo test -p openobserve-core --features enterprise synthetics::tests` — 3 pass. |
||
 Subhra264andClaude Opus 5
|
47daba4d44 |
fix: prevent deadlock when batch upserting pipeline errors (#13480) (#13487)
## Problem
Postgres is killing pipeline-error flushes with `deadlock detected`:
```
ERROR: deadlock detected
DETAIL: Process 3407043 waits for ExclusiveLock on tuple (82,8) of relation 2739499; blocked by process 3407136.
Process 3407136 waits for ShareLock on transaction 2384324529; blocked by process 3406875.
Process 3406875 waits for ExclusiveLock on tuple (81,4) ...
Process 3406910 waits for ShareLock on transaction 2384321712 ...
Process 3406931 waits for ExclusiveLock on tuple (82,8) ... <-- cycle closes
```
All five participants run the identical `UPDATE "pipeline_last_errors"
SET ...` statement. Same rows, different orders — lock-order inversion,
not contention or a slow query.
Two things combine to cause it:
1. **`batch_upsert` wrote rows in arbitrary order.** The batch is built
straight from the queue with no sorting, and every row is updated inside
one transaction, which holds each row lock until commit. Node A may
update pipeline X then Y while node B does Y then X.
2. **All nodes flush on the same tick.** Every log line in the incident
is within the same second, so the transactions overlap almost perfectly.
## Fix
Deduplicate the batch by `pipeline_id` (keeping the newest error per
pipeline) and sort by `pipeline_id` before the write loop.
- **Sorting** is the actual deadlock fix: with a global lock order, a
wait cycle cannot form. Nodes may still block on each other briefly, but
Postgres will never find a cycle to kill.
- **Dedup** shrinks each transaction. A batch usually carries many
errors from the same pipeline, which previously became repeated updates
of the very same row inside one transaction — more rows touched, longer
lock-hold window, more chances to interleave.
Everything else is unchanged: same transaction, same up-front `SELECT`,
same "skip the write if the error content hasn't changed" check, same
per-row update/insert loop. The diff is additions only.
## Testing
- New unit test `test_dedup_and_sort` covers both the ordering and the
newest-wins behavior.
- `cargo clippy --lib --tests` and `cargo fmt --check` clean.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Loakesh Indiran
|
cca95a9ec6 |
fix(synthetics): stop reporting unevaluated HTTP assertions as passed (#13498)
## Problem A failing HTTP check showed **Failed** at the top of the run detail and **Assertions: Passed** below it — a green tick against `status_code eq 200` on a run that never got a status code. The probe is right to omit `assertions_passed` when the request never returned a response (connection refused, EOF, timeout): it returns before evaluating anything, because there is nothing to assert against. The view had two branches for three states: ```vue :variant="run.assertionsPassed === false ? 'error' : 'success'" ``` `null` is not `false`, so "never ran" landed in the success branch. | `assertions_passed` | means | UI showed | |---|---|---| | `true` | ran, all passed | Passed ✅ | | `false` | ran, one failed | Failed ✅ | | *absent* | never ran | Passed ❌ | ## Changes 1. **Three states, not two.** Absent renders as a neutral **Not evaluated** badge with neutral rows and a line explaining why. 2. **Render the probe's per-assertion results** when the record carries them (see synthetic-o2-agent#4), so a run with several failing assertions shows each one with its own comparison string instead of a single inferred red row. Records written before the probe echoed that array keep the old inference path. 3. **Fix `response_time_ms` on the legacy path.** The probe's detail leads with `response_time`, so `startsWith` never matched the configured field name and a failing latency assertion left every row green. The two renamed fields now go through a lookup map rather than one special case. 4. Reported rows are self-describing, so the assertions card no longer depends on the check-config fetch succeeding (403 on community builds, deleted check, …). ## Testing `ProtocolRunSummary.spec.ts` — 7 new tests covering the not-evaluated state, the still-passes case, the reported-array path, config-fetch failure, the legacy single-failure record, and `response_time_ms`. Full synthetics sweep: 499 passed, 0 failed. Verified end to end against a local deployment with five checks covering each case: | Check | Assertions | Record | UI | |---|---|---|---| | AT-A | `status_code eq 200` | `assertions_passed: true` | Passed, row green | | AT-B | `status_code eq 999` | `false` + array | Failed, row red | | AT-C | `status_code eq 999` + `body contains NOPE` | `false` + array of 2 | Failed, **both** rows red | | AT-D | `response_time_ms lt 1` | `false` + array | Failed, row red | | AT-E | unreachable target | `assertions_passed` absent | **Not evaluated** | Before: AT-C showed one red row and one green, AT-D showed no red row, AT-E showed Passed. |
||
|
Loakesh Indiran
|
ee3ce2296a |
feat(synthetics): pick net or browser agent in the private-agent setup drawer (#13496)
## What
The private-agent setup drawer composed a single install command, so the
only way to get a browser-capable private location was for the caller to
pin `agentType` — which the org-level **Private Locations** page never
does. A location that could run browser journeys was unreachable from
the page dedicated to private locations.
- **Agent-type tabs** (`Net` / `Browser`) drive `--type` on the composed
command. The caller's `agentType` is now the opening default rather than
a lock, so either kind can be installed into the same location from
anywhere the drawer opens.
- **Linux/Windows hidden under Browser** — `install.sh`'s linux path
downloads a Go release binary and Windows runs `install.ps1`; neither
can run Chromium. Switching to Browser from one of those tabs snaps the
platform back to Docker, so the command never composes an install that
cannot work.
- **Locations card empty state** — the check form now offers the "Set up
private agent" CTA when there are no locations at all, instead of a dead
"No locations available" box. That is exactly the state a fresh
self-hosted org starts in, and the one where the CTA matters most.
Applies to both the browser and HTTP/net check forms (shared
`CheckLocations.vue`).
- **`open` watcher gains `immediate`** — a drawer mounted already-open
never fired the transition and composed from defaults instead of its own
props.
## Also
`vite.config.ts`: the monaco plugin is gated behind `!isTesting`. Its
`rmdirSync(..., { recursive: true })` was removed in Node 26, which
stopped vitest from booting at all. Tests already swap in
`monacoEditorTestResolver`.
## Testing
- New `AgentSetupDrawer.spec.ts` — 7 tests covering the type tabs, the
`--type=browser` flag, and the platform fallback.
- `CheckLocations.spec.ts` — 2 tests for the empty + `allowPrivate`
path.
- Full synthetics sweep: 597 passed, 0 failed.
Verified end to end against a local deployment: net and browser agents
both registered into one `lcl-dev` private location, which then
advertises `BROWSER HTTP SSH TCP TLS`.
|
||
|
Harsh Mahajan
|
b7ae83d2ad | chore: UX revamp for Incidents RCA analysis (#13388) | ||
|
Huaijin
|
ef3617f157 |
refactor: split HTTP APIs by domain (#13479)
## Summary
- organize API crates under
`src/api/{common,grpc,http,ingest,management,pipelines,search}`
- keep ingest, management, pipelines, and search independent from one
another; compose them only in the HTTP frontend
- move pattern APIs into search and simplify `main.rs` wiring
- update workspace manifests, lockfile, integration tests, and
cargo-deny entries
- keep `web/` unchanged
## Validation
- `cargo fmt --all -- --check`
- `cargo clippy --all-targets -- -D warnings`
- `cargo deny check licenses`
- enterprise overlay: `cargo build --no-default-features --features
enterprise`
- enterprise overlay: `cargo build --no-default-features --features
cloud`
- enterprise overlay: `cargo clippy --all-targets -- -D warnings`
The matching `o2-enterprise` overlay PR will be linked after creation.
|
||
 Omkar Kesarkhane
|
c047dfe4cf |
fix: updated browser check action buttons (#13482)
Editing a browser check offered Cancel / Continue / Update check on the Journey step, where neither save-bearing label matched its behaviour: Continue advanced to Configure without saving, and Update check saved but deliberately stayed on the page. There was no obvious save-and-leave action. Edit mode now shows Cancel / Save & Continue / Save & Exit on the Journey step, and the Configure step's primary button becomes Save & Exit. Create mode is unchanged — a new check has no name, URL or locations yet, so saving from the Journey step would always fail validation. |
||
|
Omkar Kesarkhane
|
237fbca3af |
fix: Synthetic check runs table sorting issue (#13477)
## What changed This PR makes the Synthetics runs table sortable (with a default sort by last-run, newest first) on all columns, using raw underlying values (e.g. `durationMs`, `statusRank`, `locationName`) instead of the formatted display strings the cells render. It also removes a `tableFilterKey`-based force re-render hack and an inline error-badge cell/`filterByError` helper that duplicated existing `filterByErrorPattern` filtering. Separately, it fixes a loading-state flash in `RunDetail.vue`/`ProtocolRunSummary.vue`: `MonitorResults.vue` now passes a resolved `overrideMonitorType` down to `RunDetail`, `RunDetail` seeds `monitorType` from that override in drawer mode to skip a redundant fetch, and `loadRun()` now sets `synthetics.loading.value = true` synchronously so the skeleton shows instead of an empty/wrong-branch UI while monitor type resolves. A new `ProtocolRunSummarySkeleton.vue` component replaces a generic `OSkeleton` block to mirror the actual result-card layout. The runs query limit in `useSyntheticResults.ts` was bumped from 500 to 1000, and the runs page size default is kept at 10. ## Why Reviewing `MonitorRuns.vue`'s pagination, default sort, and column sorting surfaced that columns weren't sortable and had no default sort, and that sorting on formatted values would produce incorrect ordering (e.g. "900ms" sorting after "1.2s", or status ordering depending on locale). Separately, the session's testing turned up a UI flash where `RunDetail`/`ProtocolRunSummary` briefly rendered the wrong branch (empty state or "browser" default) before `fetchCheck()`/monitor-type resolution completed. |
||
|
ktx-akshay
|
6a366c4d48 | fix(dashboards): remove misleading "New dashboard" action and fix icon copy on empty Favorites state (#13454) | ||
|
Omkar Kesarkhane
|
640e0647c6 |
fix: group Synthetics and RUM under an Experience menu (#13474)
Collapse the two top-level RUM and Synthetics rail items into a single Experience group tile, mirroring how Data groups Streams/Pipelines and Dashboards groups Reports. Driven entirely by a new NAV_GROUPS entry in navGroups.ts; when Synthetics is disabled the group auto-collapses and RUM falls back to a plain link. |
||
|
ktx-vaidehi
|
3200c202c7 | fix: emptystate animation migration (#13440) | ||
|
Hengfei Yang
|
1b56b3da05 |
fix: skip account column widen when already VARCHAR(128) (#13471)
Design at: #10521 Port of #13469 to main. ## Summary The `ALTER TABLE {table} ALTER COLUMN account TYPE VARCHAR(128)` migration in `create_table` ran the ALTER unconditionally on every startup for `file_list`, `file_list_history`, and `file_list_deleted`. This ALTER takes an `ACCESS EXCLUSIVE` lock on each table (and cascades to all partitions for partitioned tables), which can block or fail startup when the tables are busy — even though the column is already `VARCHAR(128)` on any instance that has started once since the migration was introduced. This applies the same treatment as the `file` column widening in `apply_column_width_compat` (#10521), without changing any control flow: - New `widen_varchar_column` helper extracted from `apply_column_width_compat`: checks `information_schema.columns` first and skips the ALTER when the column is already at (or above) the target width, uses `ALTER TABLE IF EXISTS`, and logs a warning instead of failing startup if the ALTER errors (info log on success). - The account loop in `create_table` now goes through this helper (still unconditional, same tables), so the ALTER only executes when the column is actually narrower than 128. - `apply_column_width_compat` (file → VARCHAR(1024)) also uses the helper and stays under the existing `meta_partition_mode == "auto"` gate, unchanged. ## Testing - `cargo check -p infra` passes. |
||
|
442ca09688 |
test(functions): un-skip JS-in-all-orgs tests + add per-event runtime-error coverage (#13341)
Replaces the `skip` band-aids that PR #13156 ("workflows v1") left on the **Functions** tests with real, running coverage of the current behaviour — on the correct edition — plus new tests for behaviour #13156 introduced but never covered. Test files only, no feature-code churn. Clean branch off `main`. ## Background — what #13156 actually changed #13156 bundled a Functions backend change unrelated to Workflows, and the old tests still asserted the *pre*-#13156 rules, so they were silenced with `pytest.skip(...)` / `test.skip(...)`. Two rules changed: 1. **JS is no longer `_meta`-only at the backend.** `save_function` / `test_run_function` have **no org gate on any edition** — JS (`transType:1`) is accepted in every org, including `default`, on **both OSS and Enterprise**. The `_meta`-only rule now lives **only in the frontend** (`AddFunction.vue` gates the JS radio on `isEnterprise || isCloud || org === "_meta"`, baked at build time by `VITE_OPENOBSERVE_ENTERPRISE`). 2. **The compile-check reports only *syntax* errors, not *runtime* errors.** A syntax error still returns **400**; code that compiles but throws at runtime now returns **200** with the error surfaced **per-event** in `results[].message` — for **both VRL and JS**. Because the API tests hit the backend directly, there is **no edition difference to assert for them** — the earlier "OSS asserts 400 / ENT asserts 200" framing was incorrect. Only the **e2e** (frontend JS-radio) is genuinely edition-dependent. ## API tests (pytest · `tests/api-testing`) **`test_functions.py`** - Keep the compile-error → 400 test (`test_test_endpoint_with_invalid_vrl_returns_400`) — it was never actually failing; `test`/`=====` are genuine VRL *compile* errors (`undefined variable` / syntax) and still 400. - Add `test_test_endpoint_with_js_runtime_error_returns_200` and `test_test_endpoint_with_vrl_runtime_error_returns_200` — runtime errors now return 200 with a per-event `results[].message`. **`test_functions_meta_org_restriction.py`** - Flip the 3 previously-skipped "blocked in default" tests → assert **200** (JS allowed in every org): `..._blocked` → `..._allowed`. - Repurpose `TestErrorMessages` (its old restriction-message assertions no longer exist) into per-event runtime-error coverage: JS `ReferenceError` content, VRL + JS surfacing consistency, partial-failure (only the failing event carries a message), and a syntax-error → 400 regression guard. - Update the class/module docstrings to the real contract. **`test_js_enrichment.py`** - Flip `test_js_function_in_default_org_returns_400` → `test_js_function_in_default_org_allowed` (200, transform runs). ## e2e (Playwright · `tests/ui-testing/Functions/js-transform-type.spec.js`) The JS-radio rule *is* edition-dependent. Instead of a skip (or an env-var hack), the two tests now detect edition at runtime via `EditionFeaturesPage.detectEdition()` — which reads the header edition button, driven by the **same** build flag that gates the JS radio, so it's the exact source of truth and needs **no CI change**: - **OSS build** → assert JS radio **hidden** in a non-meta org. - **Enterprise / cloud build** → assert JS radio **visible** in every org. The edition-agnostic steps that were trapped inside the skipped tests now **always run**: VRL-function create in a non-meta org (a P0 smoke path), and JS-visible in `_meta`. Added `P1: VRL runtime error surfaces in error-details` (mirrors the existing JS `ReferenceError` case; covers the VRL half of the per-event-error change). ## Skips removed (before → after) | Old (skipped) | New (runs & passes) | |---|---| | `test_create_js_function_in_default_org_blocked` | `..._default_org_allowed` | | `test_test_js_function_in_default_org_blocked` | `..._default_org_allowed` | | `test_update_vrl_to_js_in_default_org_blocked` | `..._default_org_allowed` | | `test_error_message_content` / `_consistency` | per-event runtime-error tests | | `test_js_function_in_default_org_returns_400` (enrichment) | `..._default_org_allowed` | | e2e `P0: Non-meta org - JS hidden, VRL works` | `P0: Non-meta org - JS radio edition-gated, VRL works` | | e2e `P1: Radio visibility across orgs` (skipped) | same name, edition-aware, no `test.skip` | `grep -rn "JS fns are now allowed" tests/` and `grep -rn "JS is allowed in all orgs on enterprise" tests/` → **both empty**. (The 7 remaining pytest skips are the unrelated `JS_HELPER_UNAVAILABLE` enrichment-helper cases — a separate pre-existing limitation, not part of this change.) ## Verification (local, against current `main` source) Built the current-`main` OSS binary and ran everything live: - **API:** `32 passed`, `0` target skips (7 unrelated `JS_HELPER_UNAVAILABLE` skips remain). - **e2e OSS build:** `7/7` — `detectEdition=opensource`, JS hidden in non-meta. - **e2e Enterprise-frontend build:** `3/3` edition tests — `detectEdition=enterprise`, JS visible everywhere, VRL runtime error renders in error-details. All backend behaviour was confirmed by probing the live `/functions` + `/functions/test` endpoints, not assumed. ## What CI proves, and where - **This PR (openobserve):** `api-testing.yml` runs the API tests; `playwright.yml` (`Functions` shard) runs the e2e on the **OSS** edition (JS-hidden branch). - **Enterprise edition:** dispatch `o2-enterprise` → `playwright.yml` (workflow_dispatch) with `o2_opensource_repo = test/functions-skipped-tests-13318` to run the same specs on the **enterprise** build (JS-visible branch). Pairs with the ENT Functions work in #13318 / o2-enterprise#2217. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com> |
||
|
Shrinath Rao
|
30c80cd2ce |
fix(synthetics): make probe-token, location & agent migrations idempotent (#13468)
## What Makes three synthetics DB migrations idempotent so they don't panic `db init` on startup when re-run against a database that already has the tables. ## Why While upgrading an environment (schema 48 → 53), startup crashed repeatedly: ``` db init failed: Execution Error: error returned from database: relation "idx_synthetics_probe_tokens_org_id" already exists ``` …then the same for `idx_synthetics_locations_org_id`, then the agent indexes. Root cause: the `create_index` calls in the probe_tokens, locations and agents migrations omitted `.if_not_exists()`. Their sibling migrations from the very same feature (monitors / runs / jobs, PR 12849) **do** use `.if_not_exists()`. When a migration re-runs against a DB where the table + index already exist but the migration row is not recorded in `seaql_migrations`, sea-orm re-issues `CREATE INDEX`, the DB rejects it as already existing, and startup panics. The siblings survive that exact condition (they log `already exists, skipping`); these three didn't because the flag was missing. So this was a genuine miss, not house style. ## Changes - Add `.if_not_exists()` to the four index creates (probe_tokens ×1, locations ×1, agents ×2), matching the sibling synthetics migrations. - Guard the probe_tokens backfill to skip orgs that already have a token, so a re-run can't seed a duplicate `system` token per org. This mirrors the idempotency check already used in `m20260331_000001_create_sys_rca_agent_service_accounts`. ## Testing - `cargo check -p infra` passes. - Verified against the affected environment: after clearing the leftover objects the migrations apply cleanly and record; with this patch the `CREATE INDEX` re-run is skipped instead of crashing. |
||
|
Hengfei Yang
|
ec6091543e |
fix(db): avoid O(n^2) stream settings clone during schema cache load (#13462)
Design at: #13461 Fixes #13461 ## Problem Startup schema cache loading degrades quadratically: with ~122k streams it starts at ~0.1s per 1000 streams and slows to 14s+ per 1000, taking 40+ minutes in total. ## Root cause `set_stream_settings_atomic(w.clone())` inside the per-stream loop of `db::schema::cache()` clones the entire `STREAM_SETTINGS` map for every stream and swaps it into the ArcSwap atomic cache. The map grows by one entry per iteration, so total work is O(n^2). ## Fix Only insert into `STREAM_SETTINGS` inside the loop; sync the atomic cache once after the loop completes. `cache()` runs during startup init before the node serves traffic, so deferring the atomic sync to the end has no behavioral impact. The per-event single updates in the watch handler keep the in-place `w.clone()` sync — those are single-key updates on a warm cache, not a loop. ## Verification - `cargo check -p db` passes - Same fix already validated on the `hotfix/v92-schema-cache` branch built from the production commit |
||
|
Shrinath Rao
|
7ade62610a |
test: heal logs scroll-retention selector after histogram-pinning change (#13460)
## Problem The scheduled Playwright Regression runs failed on `Logs-Regression` in **both repos** on the same 3 tests in `logs-9044-7354.spec.js` (issue 9044 scroll-retention): - OSS run [30150748275](https://github.com/openobserve/openobserve/actions/runs/30150748275) - ENT run [30150717648](https://github.com/openobserve/o2-enterprise/actions/runs/30150717648) All three failed identically — `expect(scroll).toBeGreaterThan(10)` received `0`, i.e. scrolling the results container no longer moved it. ## Root cause `web/src/plugins/logs/SearchResult.vue` now **pins the histogram** and hands the scrollbar to the results table (`TenstackTable` → `[data-test="logs-search-result-logs-table"]`). Its scroll parent is the `ref="parentRef"` div (`.o2-scroll-container`, `overflow: auto`). The outer `scrollContainerRef` pane is now `overflow: hidden` in the logs view. The page-object helpers `getScrollContainerPosition()` / `scrollToResultsBottom()` walked *down* from the pagination header looking for a scrollable sibling, which latched onto a non-overflowing element — so `scrollTop` stayed `0`. (This is the same area healed once before in #13316; the histogram-pinning change moved the scroller again.) ## Fix Discover the scroll container from the results table itself: walk **up** from `[data-test="logs-search-result-logs-table"]` to the nearest ancestor with a scrollable computed `overflow-y` (the `.o2-scroll-container`). Matched by computed style so class renames don't break it. The old pagination sibling-walk is kept as a fallback for the patterns view (which still scrolls as one pane). Only the shared OSS page object `logsPage.js` changes, so this heals both OSS and ENT (ENT merges the OSS test folder). ## Verification Ran against a fresh local build (localhost:5080): ``` 3 passed (46.8s) Scrolled to: 358 → Scroll correctly reset to top on page change ``` Confirmed via a DOM probe that `.o2-scroll-container` is the real scroller (scrollHeight 1112 > clientHeight 754; scrollTop → 358 after scrolling to bottom). |
||
|
udev
|
8ed0f95ea0 |
fix: resolve AI session schema and AnyValue handling (#13459)
## Summary - remove legacy `tool_name` and `tool_args` references from the session span SQL projection - use the canonical `gen_ai_tool_name` field for session tool calls - handle `gen_ai.input.messages` and `gen_ai.output.messages` as OTEL `AnyValue` payloads in backend session summaries and frontend turn rendering - support structured values, JSON-encoded span values, plain strings, scalar values, and unstructured object fallbacks without dropping message content - add regression coverage for the SQL schema and message normalization paths ## Testing - frontend unit tests: 145 passed - backend session extraction tests: 10 passed - frontend application type-check passed - targeted ESLint passed - Rust formatting and diff checks passed ## Reference - https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-events.md |
||
|
Loakesh Indiran
|
3f49343426 |
feat(synthetics): private-agent token lifecycle, browser private agents & P0 lease fixes (#13432)
Synthetics work spanning private-agent tokens, browser private agents, and job-lease correctness. (Enterprise backend: openobserve/o2-enterprise#2260.) ## Private-agent probe tokens - Named, org-level, rotatable `o2syn_` probe tokens — **create / list / rotate / disable**. Rotate = create-new + keep-old for zero-downtime overlap; disable = revoke. Fixes the gap where a private-agent token was an unrevocable standing credential. - IAM **Synthetics Tokens** page (create / rotate / disable) with an **Agents** column — how many agents currently use each token, so a token is only revoked once it's safe. - `POST /synthetics/agent-tokens` to mint a named token; agent `token_id` tracking; per-agent **Restart config** action. ## Browser checks on private locations - Allow browser checks on private, **browser-capable** locations (previously public/Lambda only). - Agent-setup drawer emits `--type=browser` so the install command pulls the browser probe image. ## Private locations as agent pools - A private location is treated as a **pool of interchangeable agents** — list shows live/total agent count + C/min (deliberate naming instead of auto-generated `private-location-XXXX`). ## Lease correctness (P0) - **Atomic job lease** — no duplicate runs when a location has multiple agents (guarded `status = 0` claim + winner-scoped return). - **Capability-filter the lease by agent type** — a net agent and a browser agent can now share one pool; the net agent no longer wins (and fails with `unsupported check type: browser`) a browser job. ## Scheduling & RBAC - Expose `next_run_at` on DueMonitor for drift-free scheduling. - CREATE-folder query is authoritative (RBAC); require PUT permission to move checks between folders; pass the folder query param on per-check API calls. |
||
|
Omkar Kesarkhane
|
23675360fe | fix: added beta badge to all synthetics page headers (#13456) | ||
|
Huaijin
|
d165299bcd |
refactor: extract TLS and node services from core (#13449)
## Summary - extract TLS certificate loading and client configuration from openobserve-core into openobserve-tls - extract node discovery, cluster information, ingester gRPC channel selection, and Tokio runtime metrics into openobserve-node - update Core, API, API Management, Jobs, and the root binary to depend on the new crates directly - preserve one-way dependencies: openobserve-core depends on openobserve-node, while openobserve-node does not depend on Core - register both crates in the enterprise workspace overlay ## Validation - cargo test -p openobserve-node --lib - CARGO_INCREMENTAL=0 cargo clippy -p openobserve-node -p openobserve-core -p openobserve-jobs -p openobserve-api-management -p openobserve-api -p openobserve -- -D warnings - RUSTFLAGS="--cfg tokio_unstable" cargo check -p openobserve-node - enterprise overlay: CARGO_INCREMENTAL=0 cargo clippy -p openobserve --no-default-features --features enterprise -- -D warnings - cargo deny check licenses --hide-inclusion-graph - cargo fmt --all -- --check - git diff --check Companion enterprise PR: openobserve/o2-enterprise#2267 |
||
|
Abhay Padamani
|
49b7c3ce5a | fix: metrics page improvements (#13452) | ||
|
Bhargav
|
1d99426df9 | fix: browser sdk package version upgraded to 0.4.1 (#13455) | ||
|
Abhay Padamani
|
f13e28a46e |
fix: dashboard and metrics page label overlap issues (#13450)
<img width="1172" height="482" alt="image" src="https://github.com/user-attachments/assets/3a49a47a-15c8-4eda-8044-ecca31d6eaf7" /> <img width="721" height="128" alt="image" src="https://github.com/user-attachments/assets/6cdf5b58-c71b-47a5-b92a-ae76f76867d7" /> <img width="167" height="225" alt="image" src="https://github.com/user-attachments/assets/dd0fa886-0e22-45ca-89d0-a8254a67e70d" /> <img width="215" height="234" alt="image" src="https://github.com/user-attachments/assets/19109ee0-0f02-47e3-b16f-e8c28b76bb8d" /> <img width="215" height="230" alt="image" src="https://github.com/user-attachments/assets/ed1777b4-53be-4279-aab3-0ca1910b1050" /> |
||
|
Hengfei Yang
|
f5be876931 | fix(evaluation): add queue pull_consume API and return 500 on task publish failure (#13448) |