mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-04 16:09:05 +00:00
fac24d59bb15fd3932b6bc1fd19b54b0e8a9f22e
6958
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Omkar Kesarkhaneandgithub-actions[bot]
|
fac24d59bb |
fix(traces): trace-log correlation fixes and Trace Graph height (#13901)
## Problem
Four defects around traces and the trace↔log correlation flows:
1. **Trace Graph is clipped.** The service map renders inside a 200px
band, vertically centred in an otherwise empty pane, instead of using
the available height.
2. **A span row's View Logs goes nowhere.** In OSS builds it navigates
to `/logs?stream=&from=NaN&to=NaN`; the logs page drops the filter and
falls back to its default window, so the user gets an unfiltered result
set instead of that span's logs.
3. **The log detail drawer has no View Trace button.** Opening the trace
for a log record only works from the inline expanded row.
4. **View Trace lived inside the JSON tab**, so it disappeared when
switching to Table.
## Root causes
**Trace Graph.** The ancestor chain was already correct — the sibling
Waterfall tab fills the same box. Only the leaf was clamped: the chart
carried `h-50! min-h-50!` (12.5rem, `!important`), which beat its own
root `h-full w-full`. That clamp dates to when the map lived in a small
collapsible timeline strip, where 200px was the intended design. When it
was promoted to a full-height tab, the scoped rule `.trace-details
.trace-chart-height { height: 12.5rem }` was left behind and its
two-class specificity silently defeated the `h-full!` added at the time.
The later design-token migration deleted the rule but preserved the
rendered result by translating it into utilities, which made the clamp
look deliberate.
**Span View Logs.** `buildQueryDetails()` reads the raw API shape
(`start_time`/`end_time`, nanoseconds), but the waterfall passes
formatted nodes, which `getFormattedSpan()` rebuilds as a fresh object
literal — camelCase, times as `startTimeUs`/`endTimeUs` in microseconds,
snake_case originals dropped. Reading `start_time` off one yields
`undefined`, hence `NaN`. The helper already had a `spanId || span_id`
fallback for the id, which is why the query string was correct while the
window was not.
**Drawer View Trace.** `JsonPreview` decides whether to offer the action
once, in `onBeforeMount`. `DetailTable` fills `rowData` in an `async
created()` — the `await` defers assignment past the child's mount even
though the work is synchronous — so the preview mounted with an empty
record, the check failed, the traces streams were never fetched, and
nothing recomputed it.
## Changes
- Trace Graph chart uses `h-full`; the orphaned `trace-chart-height`
class is dropped (its rule no longer exists) and two nested wrappers
collapse into one. Matches the full-bleed pattern in `ServiceGraph.vue`.
- The span row resolves the raw span from `spanMap` before calling the
helper — the same lookup this component already does for span data, HTTP
status and events — rather than teaching the shared helper to accept two
shapes. Guarded, since a span missing `span_id` gets a synthetic id
absent from the map.
- The span row also warns instead of navigating when no logs stream is
selected. The header's own View Logs button is disabled in that state
but the row-level one is not, so it was a silent dead end.
- `JsonPreview` re-evaluates the View Trace gate when the previewed
record changes, so the flag tracks its input rather than a single
snapshot.
- The action moves to the drawer's header row beside the wrap toggle,
reachable from every tab, with `hide-view-trace` on `JsonPreview`
(mirroring the existing `hide-view-related`). The gate, stream fetch and
filter move into `useViewTraceAction` rather than being copied into
`DetailTable` — `service_streams_enabled` is already hand-rolled in
three files and drifted. `JsonPreview` keeps exposing the same names, so
its specs are untouched.
Two latent issues surfaced while moving the action:
- `hiddenMenus` is seeded as an array and only becomes a `Set` once
`MainLayout` has read `custom_hide_menus`, so `.has()` is not always
available. Both shapes are now accepted.
- `OSelect`'s root sets `w-full`, which won a plain `w-40` and pushed
the button outside the drawer — the same class-collision shape as defect
1. Pinned with `w-40!`.
## Tests
No behaviour is asserted by new specs; existing suites cover the touched
paths and pass unmodified:
- `JsonPreview.spec.ts` — 91 passed (unchanged, including the assertions
on `tracesStreams`)
- `DetailTable.spec.ts` — 70 passed, 3 skipped
- `TraceTree.spec.ts`, `useTraces.spec.ts`,
`TraceDetailsSidebar.spec.ts` — 297 passed
- `TraceDetails` suites — 234 passed
- `SearchResult.viewTrace.spec.ts`,
`CorrelatedLogsTable.viewTrace.spec.ts` — passed
Prettier and ESLint are clean on every touched file.
## Verification
Checked against live trace and log data sharing a trace id:
- Trace Graph fills its pane (measured 549px where it previously
measured 200px), survives a tab round-trip, and reflows on window
resize.
- A span row's View Logs lands on `/logs` with the span+trace filter and
a ±60s window whose bounds match the span's real timestamps exactly;
with no stream selected it warns and does not navigate.
- The drawer's header shows the picker and button on both the JSON and
Table tabs with no overflow, and opens the trace with the correct id and
a ±15 min window.
- Console clean across every navigation.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
|
||
 Hengfei Yang
|
e28de6c6c3 |
fix(infra): serialize every remaining sqlite write path (#13905)
## Problem In local mode with the default sqlite meta store, concurrent metadata writes fail with: ``` Execution Error: error returned from database: (code: 517) database is locked ``` 517 is `SQLITE_BUSY_SNAPSHOT`: a deferred sqlite transaction pins its read snapshot at the first SELECT, and when it later performs its first write the upgrade fails immediately if any other connection committed in between — `busy_timeout` never retries it. The codebase convention is to serialize sqlite writes with `infra::table::get_lock()`, but a full audit found ~100 write functions that never take it, concentrated in the newer subsystems. ## What changed (31 files) - **Table layer, previously zero locks**: `alert_states`, `alert_incidents`, `incident_events`, `dashboards`, `folders`, `ratelimit`, `sessions`, `system_settings`, `alert_composites`, `slo` / `slos` / `slo_budget` / `slo_backfill_jobs`, `synthetics_jobs` / `synthetics_runs`. - **Partially covered modules**: the specific unlocked writers in `users`, `org_users`, `organizations`, `org_ingestion_tokens`, `synthetics_agents`, `synthetics_checks`, `compactor_manual_jobs`, `trial_quota_usage`, `alert_eval_intervals`, `search_job/search_jobs`. - **Service layer**: `db/pipeline_errors` (self-reporting flush), alert deduplication state, the composite-alert scheduler transaction in `core/alerts/scheduler/handlers.rs`, raw anomaly-detection config updates, and two direct writes in `core/alerts/composite/service.rs`. - **Bonus fixes**: `search_job_partitions::submit_partitions` ran its `FOR UPDATE` probe on the pool instead of its open transaction (on Postgres that row lock was not tied to the transaction at all); a ratelimit unit test raced the global `ORM_CLIENT` OnceCell and failed depending on suite ordering. ## Lock-placement rules (why some functions deliberately do NOT lock) The mutex is not reentrant, so the diff enforces **exactly one acquisition per call chain**: - The function that owns the transaction locks before `begin()`; helpers that receive an open transaction (`persist_update_in_transaction`, `apply_status_in_txn`, `record_evaluation_with`, the `alert_states` public wrappers, …) never lock. - Guards are dropped before coordinator emits, `db::scheduler` calls, and helpers that lock internally (e.g. `dashboards::delete_all` locks only after `distinct_values::batch_remove`, which takes the lock itself). ## Verification - Full workspace `cargo check` + clippy clean; 928 infra table tests pass (run twice for ordering stability). - Combined build with the enterprise crate passes; **companion enterprise PR uses the same branch name** `fix/sqlite-lock-remaining` per paired-CI convention. - Live smoke on a sqlite local-mode instance: 40 concurrent folder creations (previously an unlocked path) plus mixed ingestion/metadata load — all requests succeed, zero locked errors in the log. ## Note for reviewers This is the per-call-site fix consistent with the existing convention. A structural alternative (single-connection sqlite RW pool, which makes 517 impossible and would let the ~330 manual lock sites retire) was prototyped separately and can follow as its own change. |
||
 Huaijin
|
08d31371cc |
refactor(metrics): prepare indexed metrics groundwork (#13907)
## Summary - rename the existing TSID-major config, modules, types, merge modes, and persisted file prefixes to indexed-metrics terminology - add the promql-service dependency declarations needed by the follow-up metrics-index work - carry the exact Cargo.lock content from #13899, including h2 0.4.16 and the promql-service dependency entries ## Scope This is no-new-feature groundwork split from #13899. It retains the existing metrics layout mechanics under the new names and does not add .midx sidecar writing, index lookup/query pruning, or index lifecycle handling. The old environment variable and file prefixes are intentionally not kept as compatibility aliases because this experimental path has not been used in a deployment. ## Validation - cargo fmt --all -- --check - cargo metadata --locked --no-deps --format-version 1 - cargo check -p promql-service - cargo test -p config --lib -- layout (5 passed) - cargo test -p search --lib -- datafusion::merge (8 passed) - cargo test -p compaction --lib -- merge (32 passed) - cargo check -p openobserve-jobs |
||
|
Huaijin
|
8ef1fd4cbb |
refactor(compaction): split merge/mod.rs by function into job/stream/file modules (#13900)
## Summary Pure code-move refactor: the 1795-line `src/compaction/src/merge/mod.rs` is split by responsibility into three modules under `merge/`: - **`job.rs`** — job generation: `generate_job_by_stream`, `generate_old_data_job_by_stream`, `generate_downsampling_job_by_stream_and_rule`, plus the shared `job_range_end` helper (now `pub(super)`) and its test - **`stream.rs`** — per-stream merge orchestration: `merge_by_stream` with its private helpers `write_file_list` and `sort_by_time_range`, plus all `sort_by_time_range` tests - **`file.rs`** — file-level merge execution (called by the worker): `merge_files` with its private helpers `cache_remote_files` and `generate_inverted_index` - **`mod.rs`** — now only module declarations and `pub use` re-exports; `tsid_major.rs` untouched No logic changes — only `use` blocks and relative paths were adjusted (`super::is_past_hour` → `crate::is_past_hour`, etc.). The public API paths are unchanged, so the call sites in `worker.rs` and `lib.rs` need no edits. ## Testing - `cargo check -p compaction` and `cargo clippy -p compaction` clean, zero warnings - All 30 merge-module tests pass (`cargo test -p compaction merge::`) |
||
|
Huaijin
|
1dbf32baa2 |
fix(metrics): preserve numeric JSON series hash (#13893)
## Summary - exclude the generated `__hash__` field from JSON metric label stringification - normalize metrics `__hash__` schemas inferred as `Int64` or `Float64` to the canonical `UInt64` type - keep the existing hash calculation order and the rest of the JSON ingestion flow unchanged ## Why The JSON metrics path inserted the numeric series hash and then stringified every label, including `__hash__`. This left JSON-ingested streams with an `Utf8` hash and prevented the UInt64-only TSID-major compaction path from being selected. For hashes above `i64::MAX`, Arrow text JSON schema inference reports the field as `Float64`. The source `serde_json::Number` still retains the exact `u64`, so normalizing only the metrics system hash schema back to `UInt64` preserves the full hash range without changing ordinary floating-point fields. ## Validation - `cargo fmt --all -- --check` - `cargo test -p config --lib utils::schema::tests -- --nocapture` (19 passed) - `cargo test -p openobserve-core --lib metrics::json::tests -- --nocapture` (41 passed) - `cargo clippy -p config -p openobserve-core --lib --tests -- -D warnings` ### E2E Built the PR with `cargo build --features mimalloc` and started OpenObserve with an isolated local data directory. A JSON gauge was ingested through `/api/default/ingest/metrics/_json`, then verified through the stream schema and SQL search APIs: - schema: `__hash__ = UInt64` - schema: `value = Float64` - queried value: `1.25` - queried hash: `14531083140834660755` (greater than `i64::MAX`, returned exactly) - repository metrics API suite: `tests/ingest/test_metrics.py` — 4 passed |
||
 ktx-akshay
|
aa00c5528e |
test: improve Alpha automation test stability (#13841)
…color-by-series features |
||
|
Huaijin
|
789eee463f |
feat(metrics): TSID-major Parquet layout behind ZO_METRICS_TSID_MAJOR_ENABLED (#13882)
The storage-layout half of #13839, without the `.sidx` series index and its PromQL query path (those stay in #13839, which will rebase on this). ## What Behind `ZO_METRICS_TSID_MAJOR_ENABLED` (default off), Parquet metrics files of a stream whose `__hash__` column is `UInt64` are written ordered by `(__hash__, _timestamp)` so every series is contiguous: - **`config::meta::promql::tsid_layout`** — `metrics_tsid_major_enabled(stream_type)` (flag + configured Parquet), `metrics_tsid_major_stream(stream_type, schema)` (also `__hash__: UInt64`; JSON-ingested metrics store `__hash__` as a string and keep the classic layout), and `MetricsFileLayout { Legacy, HashSorted, TsidMajor }` encoded in the file-name prefix (`tsid-sorted-*`, `tsid-major-v3-*`) so readers and later merges know the physical order without opening the file. - **`MergeMode::HashSorted`** (open hour): the ingester movers and incremental compactor merges write one hash-ordered file named `tsid-sorted-*`. In pack mode the single-segment upload-as-is fast path is skipped for such streams (segments are in arrival order). - **`MergeMode::TsidMajor`** (closed hour): `merges_whole_batch()` — every file of the hour, regardless of size — into size-split `tsid-major-v3-*` files written at RecordBatch boundaries using `ZO_COMPACT_MAX_FILE_SIZE` as logical original size (`search/datafusion/merge/tsid_major.rs`). `MergeMode::for_compactor / for_ingester` take the schema and pick the mode via `metrics_tsid_major_stream`; downsampling keeps precedence. The compactor skips a closed hour that is already all `tsid-major-v3-*`. - **`MergedFile.layout`** decides the output file name (`layout.file_name` / `mark_file_key`) in the compactor and the ingester movers. - **Merging pre-sorted inputs**: `MergeMode::input_sort_order(&files)` declares `FileSortOrder::HashTimestampAsc` when every input is hash-ordered (`tsid-sorted-*` / `tsid-major-v3-*`). Overlapping hash ranges make DataFusion drop the order for its size-based groups; the listing adapter regroups one file per partition (single-file groups are trivially ordered) and keeps the declared `output_ordering`, so `EnforceSorting` turns the `SortExec` into a `SortPreservingMergeExec` over N pre-sorted streams (memory N × batch instead of the whole hour). A batch that mixes layouts declares no order and falls back to the full sort. Two file states only: `tsid-sorted-*` (ordered, not finalized) and `tsid-major-v3-*` (finalized). Nothing changes for streams without the flag, non-Parquet metrics, or a non-UInt64 `__hash__`. ## Known limitation The SQL query path still derives `sorted_by_time` from the SQL shape only, so with the flag on a `SELECT … FROM <metrics> ORDER BY _timestamp DESC LIMIT n` may assume a timestamp order that hash-sorted files do not have. PromQL is unaffected (no ordering declared). To be addressed with the query-side work. ## Validation - `cargo fmt --check`; `cargo check` / `cargo clippy` on config, search, compaction, jobs, promql-service, search_service (`--tests`) clean - `cargo test -p config --lib -- tsid_layout` 2 passed; `cargo test -p search --lib -- merge:: mode:: tsid_major listing_adapter` 9 passed (mode selection, per-layout input order, hash `ORDER BY`, size-split rotation, SPM plan shape); `cargo test -p compaction --lib -- merge` 31 passed - End-to-end on #13839 (debug build, local disk, OTLP gauge 300 series): ingester uploads `tsid-sorted-*` verified `(__hash__, _timestamp)` ordered; hour-end `merge [tsid_major] input sort order: hash_timestamp_asc` → plan `SortPreservingMergeExec` ← `DataSourceExec{N single-file groups}` (no `SortExec`) → `tsid-major-v3-*`; incremental `merge [hash_sorted]` 2 → 1 `tsid-sorted-*`; JSON-ingested stream (Utf8 `__hash__`) keeps the classic layout with no compactor errors. ## TODO (follow-up, not in this PR) **Sort TSID-major streams by `(__hash__, _timestamp)` already at WAL dump time.** `Partition::persist` full-sorts every dumped chunk anyway (today by `_timestamp DESC` in `merge_record_batches`); switching the sort key for TSID-major streams is near-free at dump time and removes the biggest ingester-side cost of this layout — the full `SortExec` over the merge input: 1. **persist**: lexsort the chunk by `(__hash__ ASC, _timestamp ASC)` (same gate as the writers, `metrics_tsid_major_stream`) and stamp a `sort_order: hash_timestamp_asc` key-value into the Parquet footer. A file-name marker is not an option for WAL files — `parse_time_range_from_filename` splits on `.` and a prefix would break WAL time pruning. 2. **ingester mover**: it already reads every input's footer for the schema; declare `HashTimestampAsc` on the table only when *all* inputs carry the footer marker (else `FileSortOrder::None`, full sort — handles mixed files during rolling upgrades / flag flips, same role `MetricsFileLayout::of` plays for the compactor). The merge then becomes a `SortPreservingMergeExec` over pre-sorted files instead of a full sort. 3. **pack, single segment**: the as-is fast path (skipped for TSID-major streams in this PR because segments were time-sorted) can come back — the segment is already hash-sorted, upload it unchanged and mark it `tsid-sorted-*`. 4. **pack, multiple segments**: one `MemTable` partition per segment with a declared sort order, so the in-memory merge is also an SPM instead of a full sort. Query-side is already safe for this change: metrics SQL never declares `sorted_by_time` and PromQL declares no order, so hash-sorted WAL files cannot mislead a reader. The cost is slightly coarser time clustering inside WAL files, negligible at WAL file sizes. Steps 1–2 carry most of the win; 3–4 are incremental. |
||
  Vamsi Karthik RandShrinath Rao
|
6930a9b724 |
test(mobile): RUM SDK end-to-end suite across 4 platforms + self-contained CI (#13628)
## What this PR adds end-to-end test suite for the **Mobile RUM SDK** under `tests/mobile-testing/`, plus a **self-contained CI workflow** that runs it on real emulators/simulators against an OpenObserve **built from this PR's source**. Test / fixture / CI only — no product-code/existing test changes ## How each test works (3 layers) Every scenario is a real end-to-end journey, so a failure points at the actual broken layer: 1. **Drive** — a Maestro flow taps through the app like a user (`maestro/<track>/*.yaml`). 2. **API** — query the `_rumdata` stream and assert the telemetry is correct (`utils/ooClient.js`, poll-and-retry for async ingestion). 3. **UI** — open the RUM dashboard and assert the sessions ## Coverage (42 tests across 4 platforms) The **native platforms (android-native, ios-native) are now at parity with React Native** — each new native test (masking, network, bg/fg, user-identity, no-phone-home) was built + validated on a real emulator/simulator before landing. | Capability | Layer | RN-Android | Android-native | RN-iOS | iOS-native | |---|---|:-:|:-:|:-:|:-:| | Core telemetry (views/actions/handled-error/crash → `_rumdata`) | API | ✅ | ✅ | ✅ | ✅ | | Crash session viewable in dashboard | UI | ✅ | ✅ | ✅ | ✅ | | Attributes: env / service / version tagging | API | ✅ | ✅ | ✅ | ✅ | | User identity (`setUser`) | API | ✅ | ✅ | ✅ | ✅ | | Network: fetch → resource + 4xx/5xx | API | ✅ | ✅ | ✅ | ✅ | | Session-replay PII masking | UI | ✅ | ✅ | ✅ | ✅ | | Background / foreground view transitions | API | ✅ | ✅ | — | ✅ | | No-phone-home (no egress to Datadog hosts) | security | ✅ | ✅ | — | ✅ | - Shared per-capability factories keep the specs DRY (`utils/rumChecks.js`, `utils/coreRumSpec.js`). - Negatives are covered: 4xx capture, PII-must-not-leak (masking), no phone-home, handled-error ≠ crash. - `o2-enterprise#2289` dashboard bugs are kept as inline skips that flip green once fixed. - Full capability × platform detail + on-device status: `docs/COVERAGE-MATRIX.md`. ## Fixture apps + build-time config - Four vendored apps under `apps/`: `o2-rum-tester` (React Native — drives both RN platforms), `o2-native-android` (Kotlin), `o2-native-ios` (Swift / xcodegen). - Each reads its RUM target (endpoint / org / token) and SDK version from a **build-time-overridable** config. The committed default is the current pin, so local dev is unchanged; CI regenerates it to point at the runner's OpenObserve and can pin the latest SDK on release. ## CI — `.github/workflows/mobile-sdk-e2e.yml` (self-contained, no secrets) - **Build once per OS from source.** `build_oo_linux` / `build_oo_macos` compile OpenObserve (binary + embedded web UI) from this PR's code and cache it. Device jobs download and run it **natively**, and the app posts real RUM to it (Android reaches the host at `10.0.2.2`, the iOS simulator via `localhost`). No Docker, no shared cluster, no secrets. - **Device jobs run in parallel.** `android` (Linux + KVM emulator → rn-android + android-native) and `ios` (macOS simulator, pinned Xcode 16.4 → rn-ios + ios-native). - **`merge_reports`** combines both platforms into one browsable HTML report — even when a device job fails. - **`decide` picks which platform runs, to keep scarce macOS runners in check:** - PR touching Android or shared mobile-RUM code → **Android auto-runs**; iOS is gated behind the `mobile-e2e` label. - `mobile-e2e` label → both platforms (full pre-merge gate). - `repository_dispatch(mobile-sdk-release)` from an SDK repo → runs that SDK's platform (companion PRs wire this up: openobserve-react-native-rum#3, openobserve-sdk-android#1, openobserve-sdk-ios#1). - `workflow_dispatch` → manual, choose platforms. *Separate workflow :* it needs macOS **and** Android emulators, builds 4 phone apps, and is dispatch-driven — matching the repo's existing per-suite split ## Status Green end-to-end on GitHub runners (Android + iOS device jobs + merge) : https://github.com/openobserve/openobserve/actions/runs/31733717758/job/94565520891 --------- Co-authored-by: Shrinath Rao <shnath@openobserve.ai> |
||
|
ktx-akshay
|
6fe56575e2 |
test(datetime-picker): E2E cover for copy/paste in the date-time picker (#13894)
Adds E2E coverage for the date-time picker copy/paste feature from #13374, which shipped with no E2E tests. **14 tests — 3 P0, 9 P1, 2 P2** Logs picker (auto-apply, 12): controls render with aria-labels; copy resolves a *relative* range to absolute epoch µs; full copy→change→paste round-trip; paste of epoch µs / epoch seconds / ISO-with-Z / slash-absolute / human-log formats; single-value proximity rule on both boundaries; unparseable text and empty clipboard rejected without side effects. Metrics picker (manual-apply, 2): paste stages until Apply commits; closing without Apply reverts the trigger label — guards the `triggerLabel` change that #13374's own review flagged as a behavioural warning. **Notes** - The paste grammar is already exhaustively unit-tested in `web/src/utils/dateTimeRangeParse.spec.ts` (497 lines), so this suite covers the UI wiring those can't reach plus one representative case per format family. - Assertions use a copy-back round-trip rather than wall-clock strings: `getConsumableDateTime()` resolves through `store.state.timezone`, so pasting epoch µs and re-reading via Copy is exact and timezone-independent. Wall-clock assertions are used only for the two timezone-naive formats. - `navigator.clipboard.readText()` requires a secure context. localhost and https qualify; a plain-HTTP host does not, and paste has no `execCommand` fallback (copy does). CI targets localhost, so this is satisfied there. - Registered in the `GeneralTests` CI shard. |
||
|
Omkar Kesarkhaneandgithub-actions[bot]
|
cb1b064a5d |
fix(synthetics): support to read a run's evidence without leaving the step (#13879)
## What Makes the evidence a synthetic browser check captures readable **where you already are**, instead of one tab away. Two related changes on the run-detail view: **1. An evidence row opens to its full record.** The table renders three fields the probe already captures and nothing displayed: the stack trace on a page error, the resource type, and the first-party flag. Plus the full URL — the row strips the host and then truncates, so a login URL died mid-query-string with no way to see the rest. **2. The step expansion's "Page activity" card now hosts that same table.** It used to show five severity-ranked rows and a `5 of 136` count — a card whose headline metric was its own incompleteness. It now renders the step's whole bucket with the same header, sorting, row expansion, filters, wrap and paging the Evidence tab has. Supporting work: the filter row is extracted into one component shared by both surfaces, so they cannot drift onto different controls; and the Evidence tab's step scope is a select you can change rather than a banner you can only dismiss. ## Why Every question past "was there anything?" cost a tab switch, and the correlation you arrived with — this step, this failure — had to be rebuilt by hand against a step filter. ## Notable fixes found along the way - **The Step column was sortable but sorted nothing.** It had no accessor, so the comparator read a field the row does not have, compared `undefined` to `undefined`, and left every row in place while the header showed a working arrow. - **The step expansion's content column had no width clamp.** As a flex item it kept its content's intrinsic width, so a long URL pushed it past the container and every card in it overflowed — and `truncate` and `overflow-x-auto` both went inert, since each needs a bounded parent. Measured at 1487px inside a 1000px container. - **Expand chevrons did not line up.** The row status rail's class also insets the first cell, so rows without a rail sat half a rem left. Benign rows now reserve the slot transparently. - **"Took" was never right-aligned** — the class sat on an inline span, where it aligns nothing. ## Testing `components/synthetics/`, `views/synthetics/`, `composables/synthetics/`, `utils/synthetics/` — **1398 passing**. `type-check`, `lint:design:strict` and `format:check` all clean. New behavioural coverage where it was thin: sorting is driven through the header and asserted on rendered row order rather than on column config, which is what let the non-functional Step sort ship. ## Reviewer notes - `EvidenceEvents.vue` is shared by both surfaces. The only differences left are the Step column and the page size — everything else is deliberately identical. - The step select is gated on being passed options, which is how it stays off the step card, where the step is already the scope. A test on each side pins that. - The per-step card's five-row cap is deliberately removed; pagination bounds the height instead, which bounds the card *and* keeps every event reachable in place. --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
 udev
|
a6af96289f |
fix(ofga): migrate annotation queue and dataset permissions (#13888)
## What Adds the OpenFGA migration for the `annotation_queues` and `datasets` resources, gated on model version `0.0.39`, following the pattern used for every other resource in `init()`: - a `need_annotation_queues_datasets_migration` flag - a `v0_0_39` version constant - an `existing_model_version < v0_0_39` gate - `get_ownership_all_org_tuple(org_name, "annotation_queues", …)` and `…"datasets"…` in the per-org loop Those two strings are the `OFGA_MODELS` map keys added in o2-enterprise, which resolve to the `annotation_queue` and `dataset` object types. ## Why `annotation_queue` and `dataset` were added to the OpenFGA model without a migration, so existing deployments never receive the `_all_<org>` ownership tuples for them. ## Depends on o2-enterprise#2477, which adds the two types to `model.json` and bumps `o2_version` to `0.0.39`. Without it this migration never fires, since the version gate is never crossed. ## Verification Verified on a local enterprise instance running both changes: ``` [OFGA:Local] model version changed: 0.0.38 -> 0.0.39 [OFGA:Local] annotation queues and datasets permissions migration needed [OFGA:Local] Data migrated to openfga ``` - `org:<org> owningOrg annotation_queue:_all_<org>` and `… dataset:_all_<org>` are written, with no errors in the batch. - Creating a role and saving `annotation_queue` / `dataset` permissions returns 200 and reads back correctly. - OpenFGA `check`, with the `org_context` contextual tuple the app sends, resolves correctly: `AllowAll` on `annotation_queue` permits GET/POST/DELETE; `AllowList` only on `dataset` permits LIST and denies DELETE. - `cargo check -p openobserve-core --features enterprise` is clean. No frontend change is needed — the roles edit page builds its resource list from the `/resources` endpoint, and the entity fetchers for both resources already shipped in #13719. |
||
 
|
0f4b606e8e |
feat(synthetics): super-cluster support, then move synthetics to OSS (#13795)
> ### ⚠️ This PR carries four changes — review alongside openobserve/o2-enterprise#2420 > > 1. **Super-cluster support (consumer half)** — `8ceb041c43 … 5885c57b3a`. > 2. **Synthetics OSS migration** — `3a380087a3 … 2596300dbf`, nine commits — one per build-order step, plus a ninth for fixes found by running the binary. Moves synthetics out of `o2_enterprise` into `src/synthetics` (`openobserve-synthetics`), leaving only the private-VPC-agent path enterprise: `job_api::lease`, `agent/*`, private locations and the agent-setup install flow. Paired with openobserve/o2-enterprise#2420. > > 3. **Last-check columns read from the results stream** — `6af411e808`, `e7a3f2c759`. Outside the region whose scheduler runs a check, the list showed a stale status and a blank Last check / response time: `last_check_status` replicates only on change, `last_triggered_at` is a runtime column that does not replicate, and `last_response_ms` was never populated anywhere. The `synthetics_results` stream already carries all three and is federated, so the list API reads it instead of replicating the columns — no queue traffic, no schema change. Best-effort: any miss keeps the stored DB values, so search being unavailable cannot take the list down. OSS-only; no enterprise counterpart. > > 4. **Check secrets are readable in every region** — `d47a6b2345` (+ o2-enterprise `da71d7ec`). The per-org DEK is minted by each region for itself, so a check created outside the scheduler region replicated fine and then silently never ran — `resolve` could not decrypt its credentials (o2-enterprise#2451). Secrets now cross the queue **in clear** and the applier re-encrypts them under its own DEK on arrival, fixing the class at the replication boundary with no key to distribute or keep in sync. > > ⚠️ **This puts customer secrets in plaintext on the inter-region bus** — a deliberate, recorded trade-off (`o2oi_` ingest tokens and probe tokens already travel it in clear). Securing the bus itself (TLS on the gateway links, short retention on secret-carrying subjects) is the real fix and is **not** in this PR. > > ⚠️ **Deploy consumers before producers.** The payload's bytes did not change, only the meaning of the secret strings, so an un-upgraded consumer handed the new payload would write customer passwords to disk in clear. The plaintext-carrying messages therefore use new variant names (old ones retired as `CreateEncrypted`/`UpdateEncrypted`), so an old region fails serde and redelivers until upgraded instead of silently persisting plaintext. > > An OSS build can now create, schedule, dispatch to Lambda, take `resolve`/`ack` and report a check. 7 393 lines moved into the new crate; 679 lines stay enterprise. > > **🛑 Deployment prerequisite (2 of 2).** `ZO_SYNTHETICS_ENCRYPTION_KEY` must be set to the **same** base64 64-byte value in every region of a super cluster. Without it synthetics falls back to the per-org DEK and checks carrying secrets silently fail to run outside the scheduler region. It is read once at startup and pinned — changing it later requires a restart, and changing it at all orphans every secret already encrypted under the old value. > > **🛑 Deployment prerequisite (1 of 2).** 14 `O2_SYNTHETICS_*` env vars are hard-renamed to `ZO_SYNTHETICS_*` with **no fallback read**, and `ZO_SYNTHETICS_ENABLED` defaults to `false`. A deployment still setting the old names does not lose tuning — **synthetics stops running entirely, and the first symptom is checks silently not firing.** Update env vars *before* rolling the binary. > > #### Things a reviewer should look at first > > - **`super_cluster_queue` does not depend on `openobserve-synthetics`** (`cargo tree` → 0), which is what keeps the applier's loop-prevention guarantee a property of the crate graph rather than a convention. `a7d5fdd969` adds the assertion that fires if that ever changes. > - **`job_upload` was skipping authorization in OSS builds** (`4c3a0b8f0d`). Unreachable before, because no synthetics route was registered in OSS; registering it would have made it an unauthenticated artifact upload. `authorize_probe` is ungated and the bypass is gone. > - **The dependency edge had to reverse in a single commit** (`3247d4093f`). Cargo rejects an `o2_enterprise` ↔ OSS package cycle at the manifest level — `optional` and feature gating do not help — so the two edges could never coexist, and everything needed for the reversal lands together. This is why the spec's steps 4 and 5 are one commit and the config moved earlier than planned. > > #### Accepted OSS limitations, all documented in the commits > > - No notifications: `notify_check_result` ends in `alerts::alert::dispatch_notification`, which is enterprise. An OSS build runs the check and records the run; it has nowhere to send an alert, exactly as for alerts themselves. > - No per-resource RBAC on synthetics, and no OpenFGA ownership tuples — both are enterprise capabilities across the platform. > - Private locations are refused with a message rather than created. > - Agent-served public `net-*` pools are unavailable, so an OSS user needs a Lambda for every public pool (spec §1). --- Consumer half of synthetics super-cluster support. **Depends on openobserve/o2-enterprise#2420** (producer, message types, queue struct) — and **this side must deploy first**: `MessageType` bytes 83–85 hit a panicking `unreachable!()` in `From<u8>` on a consumer that does not know them, which takes down the subscriber task rather than failing one message. Based on `fix/lease-claimed-by-2270` (#13785), not `main`, so the diff is only this work. ## What this fixes Synthetics writes are not querier routes, so a create lands on ingesters in whichever region received the request and commits to that region's Postgres. There is no synthetics topic on the super-cluster queue, so a check created in a region that runs no scheduler is never seen by the region that does: the UI shows it enabled, the scheduler has never heard of it, and nothing reports the gap. ## Commits | commit | what | | --- | --- | | `8ceb041c43` | orphan-detection query + metrics | | `1c3e10da48` | apply replicated check config | | `79e6abc052` | apply replicated locations and probe tokens | | `16d4fa0fde` | render unknown live status instead of a false pending | | `5885c57b3a` | share the job-cluster predicate | ## Design notes worth reviewing **Config columns only, and the list is not restated.** `synthetics_checks::update` rebuilds its `ActiveModel` from the row already in this region, and `update_mutable_fields` writes only user-editable columns — so `next_run_at` and the alerting counters keep local values for free. Hand-listing them in the consumer would duplicate a list that already exists and let the two drift. A test asserts every runtime column comes back `Unchanged`. **`put()` is deliberately unused.** Its insert branch builds from `build_active_model`, which ends in `..Default::default()` and never assigns `owner`, `tz_offset` or `next_run_at` — a replicated create through it would drop the owner and fire a scheduled-later check immediately. **`create()` gains `use_given_id`**, matching `table::alerts::create`. It generated its own uuid, which would have made every region invent a different primary key for the same check. **Consumers apply through `infra::table` only.** The publish lives one layer up in the enterprise service, and `infra` cannot reach the enterprise crate — that layering is what stops a region re-broadcasting what it just applied. `source_cluster` does not cover this: it only stops a region re-consuming its *own* message. Each consumer greps its own source to keep this true. **Runtime tables are never replicated** — `synthetics_jobs`, `synthetics_runs`, `synthetics_agents`, and the locations `down_notified_at` claim. Asserted at both producer and consumer. ## Known gaps, named not hidden - If two regions mint a token (or location) with the same name for one org before either message crosses, the UNIQUE constraint rejects the arriving row. Local row wins, logged loudly. Erroring would redeliver to dead-letter, which is worse. - Pre-existing and untouched: `write_triggers_stream`/`write_results_stream` in the enterprise reaper post to `_meta` with the *org's* ingest token, which 401s, and neither checks response status. Orphan detection fixes this in its own path only. - Pre-existing: `create` never sets `tz_offset`, so a new check gets DB default 0 until its first edit. Replicates identically everywhere, so it causes no divergence. ## Spec corrections The spec (`docs/synthetics-lcl/2026-08-11-p1-3-super-cluster-spec.md`, local notes) is wrong in five places, all found by reading the tree: §1's invariant evidence (see the enterprise PR), §10's never-run guard, §4's column list (and it lists six runtime columns — there are seven, `alerting` is missing), §3's "private rows read-only when remote", and §7's claim that non-scheduler regions render OFFLINE (they render `pending`). ## Verification infra 42, super_cluster_queue 32, jobs 21, web 111/111 vitest + type-check + design lint, `cargo fmt --check` and clippy clean, full `--features enterprise` build. Pre-existing failures on this branch, unrelated and untouched: `super_cluster_queue::schemas::tests::test_parse_key_{valid,metrics}`, `infra::table::ratelimit::tests::test_add_batch_missing_rule_id`, and 8 web vitest failures in `buildPayload.spec.ts`/`payloadContract.spec.ts`. --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
ktx-akshay
|
78998b5fa3 |
test(logs): E2E cover for Build tab stream list with no stream selected (#13884)
Adds E2E regression cover for the fix in #13812. The Logs → Build toggle mounts PanelFieldList with pageKey "build" in edit mode, which deferred its stream-list fetch until a stream was already set. Since ZO_QUERY_ON_STREAM_SELECTION defaults to true, the logs page selects nothing, so Build mounted blank and its Stream dropdown was left enabled, openable and permanently empty on "No options found" — no way to pick a stream, no way out. ## Tests (Logs-SelectStar shard) - **P0** — with no stream selected on the logs page, the Build tab's Stream dropdown is populated and a stream can actually be picked: no empty state, options render, `e2e_automate` is offered, selecting it lands, and the field list loads that stream's schema. - **P1** — changing Stream Type with a blank stream refetches the list: logs → traces drops `e2e_automate` (a logs stream) and loads a real traces list, switching back restores it. Covers the stream_type watcher, which is gated on the same `initialStreamsLoaded` flag the fix arms. `beforeEach` asserts the logs page starts with no stream selected — without that precondition both tests would pass against the buggy build. ## Notes - Page-object locators are scoped to `[data-test="logs-build-query-page"]`: the Visualize tab mounts a second PanelEditor that stays in the DOM under `v-show`, so an unscoped `index-dropdown-stream` is a strict-mode violation. - Verified 6/6 green over 3 repeats locally. The shard runs the backend with ZO_QUICK_MODE_ENABLED=true, which the local instance does not — assertions avoid named fields so quick mode's field narrowing can't affect them. |
||
|
Omkar Kesarkhaneandgithub-actions[bot]
|
d6c726c222 |
fix(synthetics): show where recorded steps will land (#13860)
## What Shows where recorded steps will land in a browser journey — as a labelled divider above the target row, previewed on hover and held for the whole recording session. ## Why Each step row carries a Record control meaning "restore the journey up to here, then record new steps before this row". It was the most expensive click in the editor — it opens an incognito window and replays every preceding step — and nothing said where the result would land until after that had already started. The only signal was a "Recording here" badge rendered inline in the anchored row's details cell, between the action icon and the action label. Sitting inside the row, it read as *this row is recording* when it means *new steps go above this row*. ## What changed - The badge is replaced by a divider on the row's top edge: two rule segments with a centred label between them, straddling the boundary so it names a position **between** rows rather than a property of one. - Hovering or focusing a row's Record control draws the same marker in a lighter tone, so the destination is legible while the click is still free. - The preview is suppressed wherever the control is disabled — the first row, a locked table, or an extension too old to restore. A preview of an action you cannot take is worse than no preview. - Focus is bound alongside hover, so the affordance is not mouse-only. Two edge cases surfaced while testing and are fixed here: - A replay started from the toolbar locks the table while the pointer rests on a row control. Nothing moves, so no `mouseleave` arrives, and the preview sat on a locked table offering a click that was now refused. - Both render modes share the details cell slot, so an anchor id reaching results mode drew a marker on a finished run — over a cell with no positioning to hold it. ## Implementation notes No `OTable` change. The marker is absolutely positioned against the details `<td>`, which becomes a containing block via the column's `meta.cellClass`. Two consequences worth knowing: - **Nothing reflows.** Toggling the marker on hover repaints only. A rule that took layout would jitter the whole table as the pointer crossed the action column. - **The gap is real, not painted.** The label sits between two `flex-1` segments rather than over a continuous rule. That centres it without measuring anything, and means it needs no background to avoid being struck through. `getCellStyle` lifts the cell's `overflow: hidden` on the marked row only, so the label can overhang the boundary. Step-name truncation is unaffected — that ellipsis lives on the inner wrapper, not the `<td>`. ## Testing - `JourneySteps.spec.ts`: 43 tests (18 new), covering the committed marker, the hover and focus preview, suppression on every disabled path, tone precedence when a committed anchor and a hover coexist, anchor reassignment, the positioning contract, and results mode. - Full journey suite green: 336 passed, 1 skipped, `BrowserJourney.spec.ts` unmodified — which is what shows the marker rides the existing `anchorId` prop rather than duplicating state. - Guards were mutation-checked: deleting each one turns the suite red, so none of them pass vacuously. - `lint`, `type-check`, `format:check` and `check-design-consistency.mjs` all clean. ## Not covered jsdom has no layout, so positioning is pinned by class assertions only — this has not been verified visually. Worth an eye on the label at the dense row height, in both themes. Out of scope, and unchanged by this PR: - The marker does not survive into recording — `BrowserJourney` replaces the whole journey table with a capture-only table while recording, so there is no row to mark. - The toolbar Record button draws no marker below the last row. - Downstream steps keep stale pass ticks after a splice, and the passed banner recomputes its total from the new step count. --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
4b6491a0ea |
fix: session detail output message fixed (#13889)
## Summary - web/src/plugins/traces/SessionDetails.vue - web/src/plugins/traces/composables/useSessions.ts - web/src/plugins/traces/threadView.utils.ts (new) - web/src/plugins/traces/threadView.utils.spec.ts (new) ## Test plan - [x] `npx vitest run` on `threadView.utils.spec.ts` - 114 passed - [x] `vue-tsc --noEmit` (`type-check:app`) - clean --------- Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> Co-authored-by: Shrinath Rao <shnath@openobserve.ai> |
||
|
Vamsi Karthik R
|
c3056cc720 |
test(alerts): E2E suite for email destinations and distribution lists (#13876)
Automated coverage for **email alert destinations and distribution lists**, and unblocks the email tests that have been skipped since they were written. **32 cases** — 27 active, 5 skipped regressions tracking open defects (openobserve/o2-enterprise#2471). **The unblocking.** The two skipped tests said *"the test environment does not have email infrastructure set up"*. That over-stated it: creating, editing, validating and deleting an email destination needs SMTP switched **on**, and never reads a mailbox. Only delivery assertions do. The suite is tiered on that: | Tier | Needs | Cases | |---|---|---| | A | SMTP enabled | 24 | | B | a readable sink | 2 | | C | an alias that fans out | 1 | **Tests** - `Alerts/dl-email-destinations.spec.js` (32) — the DL half (alias accepted, addressed, fanned out) and the recipient-list half, where our code actually runs: splitting/trimming/lowercasing, the org-membership gate, edit round-trips, add/remove/delete, both form paradigms (prebuilt free-text vs the custom path's org-user picker), template auto-attach, duplicate names, cancel, type-switching, keyboard, dark mode at 420px. - `utils/mail-sink.js` — pluggable sink: Mailpit, a Mailinator inbox, or a clean skip with a stated reason. - `utils/o2-api.js` — storage is asserted from the API, never the form field: an input can echo what you typed while the stored value differs. - `scripts/dl-relay.js` — SMTP relay that expands one alias into per-member deliveries, so fan-out runs with no external provider. Same pattern as the in-test HTTP receiver in `alerts-content-templates.spec.js`. - `alertDestinationsPage.js` — +17 methods; the spec carries zero raw locators. **Replaces, without losing coverage.** The two skipped tests in `alerts-destinations-prebuilt.spec.js` are removed for a pointer mapping their intent to its new home: create (`D-01`), edit-add (`D-07`), edit-remove (`ML-08`), delete (`UI-12`), format validation (`UI-02`). Three of those did not exist before. That file now has no skips. **CI.** `ZO_SMTP_ENABLED: true` was already set in `playwright.yml` but had no host or port, so it fell back to `localhost:25` where nothing listens — destinations could be **saved** while every **send** failed silently. The workflow now starts Mailpit and a DL relay as static binaries before the app (no Docker-in-Docker on these runners) and points `ZO_SMTP_*` at them. All three tiers run with no secrets and no egress. Verified in the CI topology: **27 passed, 5 skipped**, `D-02` confirmed reaching all three member mailboxes. **Behaviour.** Self-cleaning via the `auto_dest_dl_*` prefix `cleanup.spec.js` already sweeps, plus an API `afterAll`. `mode: 'parallel'` — each test owns its fixtures. **Findings** are filed in openobserve/o2-enterprise#2471, not fixed here. Two P1s: the *"Show matching rows"* toggle is ignored so log rows are always embedded, and a user removed from the org keeps receiving alerts while the destination can no longer be saved. The regression tests for each are in this PR, skipped until the fix lands. **Paired ENT PR: openobserve/o2-enterprise#2472.** ENT's workflow sets `ZO_SMTP_ENABLED: false`, so without it the suite skips all 27 cases there. The shard matrix needs no ENT change — ENT builds it from this repo's `ci_matrix.json` plus its overlay, so the new spec is picked up automatically. Merge the two together: this one alone leaves enterprise with no email coverage. |
||
|
Hengfei Yang
|
46e28fe42e |
fix(auth): return 403 instead of panicking when user is not in the org (#13878)
## What
`check_stream_permissions` in `src/core/src/authz.rs` unwrapped the
result of the user lookup:
```rust
let user: User = crate::users::get_user(Some(org_id), user_id).await.unwrap();
```
When a non-root user hits an endpoint that checks permissions against an
org they are not a member of, `get_user` returns `None` and the handler
panics. The connection is aborted and the load balancer surfaces it as a
502.
The concrete trigger is `GET /api/{org}/search/profile` (search
inspector): it always checks against the `_meta` org's `default` traces
stream regardless of the org in the URL, so any SSO user who is not a
member of `_meta` got a 502 when clicking the Inspect button.
This change replaces the unwrap with a let-else that returns the same
`403 forbidden("Unauthorized Access")` response as the function's
existing deny path.
## Why
A user not being a member of the checked org is a normal deny scenario,
not an internal error. All 15 call sites of this function (search,
traces, alerts handlers) treat `Some(response)` as an opaque deny, so
they all benefit from the graceful path.
## Reviewer notes
- The change is entirely inside `cfg(feature = "enterprise")` code.
Verified with the enterprise Cargo.toml: `cargo check` and `cargo clippy
-D warnings` on `openobserve-core`, `openobserve-api-search`, and
`openobserve-api-management` all pass.
- Users who should be able to use Search Inspect still need search
permission on the `default` traces stream in `_meta` — this PR only
fixes the failure mode (502 → 403), not the permission model.
|
||
|
Shrinath Rao
|
8a46a836c1 |
fix(ci): make AI review process exit reliably after posting (#13886)
## Problem A **successful** AI review shows up as a **red 25-minute CI timeout**. Two plumbing bugs, both proven from run logs: **1. Trace-export hang** (OSS #13875) ``` 10:19:53 AI Code Review complete ← review posted in 147s 10:19:53 Exporting 14 trace spans to OpenObserve... 10:20:53 Trace export failed: Span processor did not complete within 60000 ms Error: The operation was canceled ← job killed 25 min later ``` After the comment is posted, `TRACE.flush()` exports OTel spans. If OpenObserve is slow/unreachable, `forceFlush()` times out and throws, so `shutdown()` never runs — the `BatchSpanProcessor` timer + exporter sockets stay open and keep Node alive until the job's 25-min timeout. **2. Update-404** (OSS #13871) ``` Updating existing review comment #5336607219 Fatal error: gh api ... PATCH ... gh: Not Found (HTTP 404) ``` If the captured review comment is deleted mid-run, the PATCH 404s and fatal-errors the whole run. ## Fix (plumbing only — review logic untouched) 1. Bound `forceFlush()` **and** `shutdown()` against a 15s deadline (deadline timer `.unref()`'d), always run `shutdown()` in a `finally`, and **`process.exit()` explicitly** after flush so lingering OTel handles can't keep the process alive. 2. On a 404 from the comment PATCH, **fall back to posting a fresh comment**. ## Verified - `node --check` passes. - Runtime repro with an **unreachable trace endpoint**: the process exits cleanly (code 1) instead of hanging to the 25-min job timeout. ## Review feedback (addressed) - 🟡 **Flush deadline no longer hardcoded to 15s.** It's now `Math.max(traceExportTimeoutMs() + 5_000, 15_000)`, so the outer race sits above the configured `forceFlushTimeoutMillis` — a healthy-but-slow OpenObserve gets its full `O2_TRACE_EXPORT_TIMEOUT_MS` window (default 60s) and the precise inner timeout fires first. **Behavior note:** an unreachable endpoint now exits in ~65s at the default config instead of 15s (still well under the 25-min cap); lower `O2_TRACE_EXPORT_TIMEOUT_MS` for a faster exit. - 🔵 **Accurate comment-span attribution.** `updateReviewComment`/`postReviewComment` now return the real `{action, id}`, so the `github.pr.review_comment` span records `create` + the new comment id on a 404 fallback instead of the stale `update` + deleted id. Engine logic in this change is identical across OSS/ENT. (Note: the ENT branch has since merged the OpenObserve Code Review branding from `main`, so the full file is no longer byte-identical to OSS — only the plumbing touched here is.) No change to what the reviewer finds or decides. |
||
|
Shrinath Rao
|
e2b35f43f5 |
ci: Auto Merge Queuer — re-add PRs dequeued by failed checks (up to 2x) (#13891)
## What
Adds `.github/workflows/merge-queue-auto-requeue.yml` — **Auto Merge
Queuer**.
When a PR is kicked **out of the merge queue because its required checks
failed** (`reason == CI_FAILURE`) or **timed out** (`reason ==
CI_TIMEOUT`), the workflow adds it straight back to the queue — **up to
twice**.
## Behavior
- **Trigger:** `pull_request: [dequeued]`, gated at the job `if` to
`CI_FAILURE` / `CI_TIMEOUT` only.
- **Manual removals** (`reason == MANUAL`) and every other reason
(`MERGE_CONFLICT`, `QUEUE_CLEARED`, `BRANCH_PROTECTIONS`, …) are
**ignored** — the job never runs for them.
- **Limit:** attempt count tracked via labels `auto-requeue-1` /
`auto-requeue-2` (max 2 re-adds).
- **Re-queue:** GraphQL `enqueuePullRequest` mutation, authenticated
with `secrets.ORG_ADMIN_TOKEN` (a `GITHUB_TOKEN` enqueue would not
re-trigger the `merge_group` check workflows).
- **One sticky comment, edited in place** (found by a hidden marker):
- 1st kick → "added back (attempt 1 of 2)"
- 2nd kick → same comment edited → "re-added a second time (2 of 2) —
last automatic retry"
- 3rd kick → same comment edited → "🛑 no-go for auto re-add" and it does
**not** re-queue.
All `gh` calls run under the PAT, so the comment, labels and enqueue are
authored by the token owner (not `github-actions[bot]`).
## Notes for reviewers
- Requires repo secret `ORG_ADMIN_TOKEN` with merge-queue write (already
used by 26 workflows in this repo).
- `github.event.reason` on the `dequeued` payload drives the gate; worth
confirming on the first real CI-caused dequeue. Safe failure mode if
absent: the job no-ops.
|
||
|
Shrinath Rao
|
503a7054fb |
feat(ci): rebrand automated PR review as "OpenObserve Code Reviewer" (#13871)
## What & why Repackages our existing automated PR reviewer under a branded identity — **OpenObserve Code Reviewer** — and gives its comment a colorful, attention-grabbing format. This is **purely aesthetic + repackaging**. > [!IMPORTANT] > The review **engine is frozen**. The 5 specialist agents, coordinator judgment/rubric, risk tiering, XML finding contract, re-review PATCH-in-place logic, total-outage guard, DeepSeek/opencode wiring, read-only security model, and OTel tracing are all **unchanged**. Only *who signs the comment* and *how it looks* change. ## The two things that move **1. Identity (fail-safe).** The workflow mints a GitHub App installation token via `actions/create-github-app-token`, **gated on the `OO_REVIEWER_APP_ID` variable**. If the app isn't configured, it skips and falls back to `github.token` (posts as `github-actions[bot]`, exactly as today). So this PR is **safe to merge before the app exists** and auto-activates once the secrets are provisioned. **2. Presentation.** The coordinator now emits: - Decision as a native GitHub **callout box** — 🟢 `[!TIP]` approved · 🔵 `[!NOTE]` approved-with-comments · 🟡 `[!WARNING]` minor · 🔴 `[!CAUTION]` significant. - Rebranded heading `## 🔎 OpenObserve Code Review`. - Per-category glyphs 🔒 ⚡ 🧩 📝 📦 on each finding line. `sanitizeReviewBody` normalizes the heading (tolerates a missing emoji / legacy heading, idempotent on re-review) and the deterministic fallback renderer gets the same cosmetic treatment. ## To activate the brand (one-time, org admin) Register a GitHub App "OpenObserve Code Reviewer", install on this repo, then set: - Variable `OO_REVIEWER_APP_ID` (numeric App ID — this is the gate) - Secret `OO_REVIEWER_APP_PRIVATE_KEY` (the app's `.pem`) Remove the variable to instantly roll back to `github-actions[bot]`. ## Verification - JS parses; both workflows valid YAML. - Engine files byte-identical with the o2-enterprise counterpart PR. - 6/6 sanitizer heading cases pass (fresh, no-emoji, legacy, preamble, idempotent, trailing-words). - This PR's own review comment will preview the new format. Paired with the identical o2-enterprise PR (same branch name). |
||
|
udev
|
84b666ce0b |
fix: annotation_queues item hydration intermittent 500s (#13881)
## Summary
RCA (2026-08-19, 08:20-08:22 UTC): `GET
/api/{org}/annotation_queues/{queue_id}/items/{item_id}` (and its
`/reviews` sibling) intermittently returned 500. Originally attributed
to search work-group admission capacity exhaustion; tracing the actual
code found a different root cause.
**Real mechanism**: `eval_jobs::search::execute()`'s registered
executors (in `src/jobs/src/job/mod.rs`) called
`search_service::search("", ...)` with a hardcoded empty `trace_id`.
With no propagated OTel span in that detached task,
`search_service::search()` falls back to OpenTelemetry's *invalid* trace
ID for every such call - the same `000...000` seen in every failing log
line. Every concurrent internal search therefore shared one
`QueryManager` map key: one query's cleanup (`SEARCH_SERVER.remove`)
deleted another's still in-flight registration, so the next
`insert_sender` failed with "trace_id not found" -> `SearchCancelQuery`
-> 500. `work_group_checking` never reached the real concurrency-wait
branch in this failure mode - "work-group capacity" was never the actual
mechanism.
## Changes (3 commits)
1. **`fix(annotation-queues): merge item detail and reviews into one
call`** - the Queue Workbench always fetches item detail + reviews
together (confirmed: no caller ever needs one without the other,
including after review submission); fold them into one backend call,
cutting the per-item load from 2 HTTP round trips to 1. See companion PR
openobserve/o2-enterprise#2473 for the response-shape change this
depends on.
2. **`fix(annotation-queues): map search admission cancellation to
429`** - `annotation_queue_error_response` unconditionally 500'd on
`AnnotationQueueError::Search`/`Hydration`, even when the underlying
cause already has a defined 429 `http_status()` (`SearchCancelQuery` /
`RatelimitExceeded`). Downcast and map correctly instead.
3. **`fix(search): stop colliding on trace_id in internal eval
searches`** - the actual root-cause fix: generate a real per-call
`trace_id` via `config::ider::generate_trace_id()` at all three internal
`search_service::search("", ...)` call sites instead of passing `""`.
## Dependency
This PR's enterprise build depends on openobserve/o2-enterprise#2473
(same branch name `ud/fix-annotation-queue-hydration-500s`) via the path
dependency in `Cargo.toml.openobserve` - that PR adds the `reviews`
field on `AnnotationQueueItemDetail` this PR's merge (commit 1)
consumes.
## Test plan
- [x] `cargo build --features enterprise`
- [x] `cargo test -p openobserve-api-management --lib --features
enterprise annotation_queue` - 11 passed (incl. 2 new tests for the 429
mapping), no regressions
- [x] `cargo test -p o2_enterprise --lib annotation_queues` (companion
repo) - 16 passed, no regressions
- [x] `cargo clippy --features enterprise -p openobserve-jobs -p
search_service -p openobserve-api-management --no-deps` - clean (no
warnings in touched files)
- [x] `cargo fmt --check` - clean
- [x] Frontend: `npx vitest run` on `llm-queues.service.spec.ts` +
`QueueWorkbenchPage.backTarget.spec.ts` - 9 passed
- [x] `vue-tsc --noEmit` (`type-check:app`) - clean
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
Shrinath Rao
|
6eedf5dd16 |
chore(ci): remove redundant Qodo PR-Agent workflow (#13887)
## What Removes the **Qodo PR-Agent** workflow (`qodo-ai/pr-agent@main`). ## Why - It auto-runs its code-suggestions tool on every PR via OpenAI; that call now fails, so it posts **"Failed to generate code suggestions for PR"** as noise on **every** PR (e.g. o2-enterprise#2475). - It's **superseded by the OpenObserve Code Reviewer** app, which now handles automated PR review. - Keeping it means a failing, redundant OpenAI dependency pinned to a **mutable `@main`** tag. Added Jun 2025 (OSS #7041 / ENT #577); effectively unused since the branded reviewer shipped. ## Scope Deletes only the PR-Agent workflow file. No other CI touched. Paired OSS/ENT change. |
||
|
Shrinath Rao
|
30ef3ec242 |
test(traces): heal service-graph/catalog rail-flyout nav for #13852 ?tab= reroute (#13875)
## What Heals the two Traces-Regression tests that broke after #13852 (\`fix(ui): clean up SLO detail and traces navigation\`). Failing scheduled runs: - ENT: https://github.com/openobserve/o2-enterprise/actions/runs/32213206401/job/95953026991 — \`traces-ent.spec.js\` Service Graph test - OSS: https://github.com/openobserve/openobserve/actions/runs/32213278081 — \`traces-regression.spec.js\` Service Catalog test Both failed all 4 retries with \`locator.click: Timeout 45000ms exceeded\` on the left-rail flyout item. ## Root cause #13852 rerouted the Traces rail-flyout children through the Traces page's in-page \`?tab=\` views. That changed **both**: | | before | after | |---|---|---| | flyout \`data-test\` | \`nav-group-item-serviceGraph\` | \`nav-group-item-traces-service-graph\` | | | \`nav-group-item-servicesCatalog\` | \`nav-group-item-traces-services-catalog\` | | URL | \`/traces/service-graph\` | \`/traces?tab=service-graph\` | | | \`/traces/services\` | \`/traces?tab=services-catalog\` | (\`childDataTest = nav-group-item-\${child.name}-\${child.tab}\`; legacy paths now only exist as redirects.) The stale \`data-test\` made \`.click()\` wait the full 45s action timeout. ## Fix \`serviceGraphPage.navigateToServiceGraph()\` and \`servicesCatalogPage.clickServiceCatalogTab()\`: - point at the new flyout \`data-test\` ids - \`waitFor\` the teleported flyout item to be **visible** before clicking — the flyout opens after ONavGroup's 120ms \`OPEN_DELAY\`, so the item isn't in the DOM the instant \`hover()\` resolves - match the new \`/traces?tab=\` URL Engine/other nav helpers untouched. ## Verification Run locally against a #13852 frontend (localhost:5080, enterprise build): - ✅ Live flyout click → \`/web/traces?org_identifier=default&tab=service-graph\` and \`...&tab=services-catalog\` - ✅ **OSS** \`traces-regression.spec.js\` "Clearing search filter should not blank out Service Catalog table" — **passes** - ✅ **ENT** \`traces-ent.spec.js\` Service Graph nav now works — test reaches the chart assertion (line 54); the only residual local failure is absent service-graph topology (0 nodes locally; CI seeds it via ingestion + the daemon) ## Follow-up (not in this PR / not in any CI matrix) \`Traces/traces-toolbar-tabs.spec.js\` + \`tracesPage.switchToServiceMaps()\` still assert the pre-#13852 "flyout → standalone route" dual-semantics that #13852 removed. Not run by any CI matrix, so left for a separate change aligned with the new single-\`?tab=\` semantics. |
||
|
Hengfei Yang
|
c572727863 |
fix: wait for WAL settle window before merging the previous hour (#13880)
## What The compactor's hour-merge gate (`is_past_hour`) now measures the 3 × `ZO_MAX_FILE_RETENTION_TIME` settle window from the **end** of the offset hour instead of its start. ## Why The original design delayed merging the previous hour until the current hour's top plus 3 × `ZO_MAX_FILE_RETENTION_TIME` (default 30 min), giving data still buffered in ingester WAL when the hour closed time to upload. Since the multi-worker compactor refactor (#3526), the gate compared `now - offset` (offset = start of the hour to merge) against the window — a condition that is always already satisfied the moment the hour passes (1 h > 30 min), so the check was dead code and the merge fired right at the top of the hour. Files uploaded afterwards for that hour arrived as small files that the normal path never re-merged; the old-data fallback only picks an hour up once it has ≥ 10 small files, so a handful of stragglers stayed unmerged forever. ## How - `is_past_hour` rounds `offset` up to its hour's end and requires `now - hour_end > 3 * max_file_retention_time`. One fix covers all three call sites: scheduled job generation, the incremental-mode decision in `merge_by_stream` (jobs inside the settle window now correctly run incrementally: seal full-size groups only, carry the remainder), and the file_list dump gate. - Catch-up of older hours is unaffected — their hour end is long past the window, so they pass immediately (the behavior #3526 wanted to preserve). ## Reviewer notes The behavior change operators will see: the previous hour's final merge now starts ~30 minutes (3 × retention) after the hour closes instead of immediately, matching the pre-#3526 design. Streams with the ingester incremental trigger enabled keep getting intra-hour merges during that window. |
||
|
Shrinath Rao
|
be67ed442a |
fix(deps): add missing @noble/hashes@2.3.0 to web lockfile (npm 10 npm ci) (#13872)
## Problem
Every frontend build is red on `npm ci` in `web/`:
```
npm error `npm ci` can only install packages when your package.json and
package-lock.json ... are in sync.
npm error Missing: @noble/hashes@2.3.0 from lock file
```
## Root cause (not a feature change)
- `web/package-lock.json` was last regenerated with **npm 11**, whose
resolver **omits** the explicit `@noble/hashes@2.3.0` entry that
`jsdom`'s transitive `@exodus/bytes@1.15.1` **peer-requires** (`^1.8.0
|| ^2.0.0`).
- The **Playwright build pins `node-version: 22` → npm 10**
(`.github/workflows/playwright.yml`). npm 10's `npm ci` recomputes the
ideal tree, needs that entry, finds it missing, and hard-fails.
- Reproduced locally: `npm@10 ci` **fails**, `npm@11 ci` **passes**, on
the exact committed lock. So it's a **toolchain mismatch**, independent
of any PR — it breaks `main` and every open PR building the frontend.
## Fix
Regenerated the lock with `npm@10 install --package-lock-only` → adds
the nested `node_modules/jsdom/node_modules/@noble/hashes@2.3.0` entry.
**+15 lines, lockfile only**, `lockfileVersion` unchanged (3).
Verified `npm ci` now passes under **both npm 10 and npm 11**.
## Note
The durable fix is aligning the Playwright build to **Node 24 / npm 11**
(matching the AI-review workflow and the Node 20/22 runner deprecation).
This PR is the minimal unblock; the Node bump can follow separately.
|
||
  Yashodhan JoshiandYashodhan Joshi
|
3f40d13f21 |
fix: always create file_list_dump_stats table (#13873)
Currently observed on v0.92.x , when file list dump is not enabled we do not create the dump stats table, but for some reason compactor tries to use the table and fails. Apart from this, even in the case when someone has not initially setup the dump enabled, and later enables it, the table will be missing as migration will not be run if db schema version matches. We should always create all the table schemas, as plain schema and indices will not take much db resources, but would be problematic to later debug and fix if missing. Thus this pr always created the file_list_dump_stats table irrespective of the env value. Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> |
||
|
Vamsi Karthik R
|
7b79fdcdbe |
ci(playwright): stop SLO-Measurement backfill from serializing 21 daily chunks (#13870)
## What this fixes The `SLO-Measurement` shard (`slo-timeslice.spec.js` + `slo-alerts.spec.js`) has been taking **~26 minutes** ## Root cause Both specs seed **8 days of history** and create SLOs with a **7-day rolling window**, then block in `beforeAll` until the SLO is actually measured (`waitForSloMeasured`). The server backfills history in `ZO_SLO_BACKFILL_CHUNK_SECS` chunks — default **86400s (1 day)** — and only **one chunk at a time** (`ZO_SCHEDULER_SLO_BACKFILL_CONCURRENCY=1`, by design, so bulk backfill never starves real-time alerts). The shard backfills **3 SLOs** across the two specs, each needing 7 daily chunks to cover its 7-day window: - `slo-timeslice.spec.js` `beforeAll`: 2 SLOs (`sloLt`, `sloGt`) × 7 chunks = 14 - `slo-alerts.spec.js` `beforeAll`: 1 SLO × 7 chunks = 7 **21 chunks total, strictly serial**, at roughly 40s/chunk ≈ **~14 minutes of pure backfill wait**, before any UI assertion runs. This is the server behaving correctly and exactly as designed for production — the test is just paying for it the slow way. ## Fix Neither spec asserts on per-chunk timing or partial coverage, only on the final measured state, so raising the chunk size to cover the whole window in one pass is free wall-clock, not a behavior change. - Threaded `slo_backfill_chunk_secs` through `build-ci-matrix.js`'s existing per-shard passthrough, same pattern already used for `ingest_allowed_upto` / `quick_mode_enabled`. - Set it to `604800` (7 days) for the `SLO-Measurement` shard only — the server-wide default (`86400`) is untouched everywhere else. - Documented the new knob in `tests/ui-testing/ci-matrix/README.md`. ## Validation **Synthetic (API-only), same seed shape as the real specs** (8 days, 1-min interval, deterministic 3h fault window), against a local build: | | Default (86400s chunk) | Fixed (604800s chunk) | |---|---|---| | Time to measured | 246.9s (~4.1 min) for **one** SLO, stepping 1/7 → 2/7 → ... every ~40s | **3.0s** | | Final SLI | 98.21428571428571% | 98.21428571428571% (**identical**) | Same math, same result — only the number of scheduler round-trips changes. **Real shard, end-to-end**, run exactly as CI invokes it (`npx playwright test slo-timeslice.spec.js slo-alerts.spec.js --workers=1`) against a server mirroring this shard's full env including the fix: - **25/25 tests passed**, 0 regressions - Total wall-clock (server start → global teardown): **4m Avg** Local timing won't map 1:1 onto the GitHub Actions runner (different disk/scheduler characteristics — even the "before" baseline was faster locally than the ~26min/~5min-per-spec seen in CI), so the authoritative number is whatever this PR's own CI run reports. But the structural win — 1 chunk instead of 7 per SLO — is verified correct and regression-free independent of machine. ## Companion PR The enterprise repo's `playwright.yml` independently consumes per-shard matrix fields (it explicitly mirrors OSS's `ingest_allowed_upto` handling for the same reason), so it needs the equivalent env passthrough or it'll keep paying the old cost even after this merges. Companion PR: openobserve/o2-enterprise#2460 |
||
|
Shrinath Rao
|
87f7960d48 |
test(traces): accept redirected ?tab=service-graph URL in Service Graph smoke (#13868)
## What The `@enterprise` Service Graph smoke test `P0: Navigate to service graph and verify chart renders` (`tests/ui-testing/playwright-tests/Traces/service-graph.spec.js:128`) hard-asserts: ```js await expect(page).toHaveURL(/\/traces\/service-graph/); ``` ## Why [#13852](https://github.com/openobserve/openobserve/pull/13852) converts the standalone `/traces/service-graph` route into a **redirect** to the canonical `/traces?tab=service-graph` tab. The chart still renders — every other Service Graph spec passes — but this literal URL assertion no longer matches, so the **Playwright E2E Crosscheck (Enterprise gate)** fails on #13852 on every attempt. ## Fix Accept both the legacy route and the `?tab=service-graph` form: ```js await expect(page).toHaveURL(/\/traces(\/service-graph|\?.*\btab=service-graph\b)/); ``` Backward-compatible and test-only: - Against **main** (standalone route) → `…/traces/service-graph?…` → matches ✅ - Against **#13852** (redirect) → `…/traces?…&tab=service-graph` → matches ✅ - Rejects plain `/traces` and `tab=services-catalog` ✅ (verified) ## Ordering note This assertion lives in the OSS repo (`@enterprise`-tagged, runs only in the ENT gate build). Once this merges to `main`, **#13852 must sync/rebase on main** for its crosscheck to pick up the tolerant assertion. |
||
 Chaitanya Sistlaandgithub-actions[bot]
|
351b356620 |
feat(alerts,slos): export as Terraform and OpenTofu (#13849)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
 Prabhat Sharmaandgithub-actions[bot]
|
ecc1d5dbff |
fix(ui): clean up SLO detail and traces navigation (#13852)
## Summary This PR combines the SLO detail cleanup and the Traces navigation fixes into one change set. ### SLO cleanup - remove the internal definition generation value from the SLO detail title - remove the unused translation key and update generated locale files - normalize SLO health icon identifiers - avoid rewriting unchanged locale files during translation updates ### Traces navigation - add Spans as the first item in the Traces flyout - route Spans, Traces, Service Graph, and Service Catalog through /traces?tab=... - redirect legacy Service Graph and Service Catalog paths while preserving query state - support same-route menu switching and retain stream, time-range, and query context - remove the deprecated search_mode parameter from generated and legacy URLs - keep Traces as the default for tabless URLs ## Validation - focused Traces Vitest suites: 436 passed, 2 skipped - app TypeScript check - full frontend ESLint check - Prettier checks - live browser verification of Traces and Spans menu switching --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Huaijin
|
0641e1e486 |
refactor(merge): decide the merge kind once (MergeMode) and share the DataFusion pipeline (#13866)
## Why The kind of merge — plain time-ordered file, trace time-index aggregation, file-list ordering, enterprise metrics downsampling — was re-derived at every layer, each from stream type / stream name / config / batch time range: - compactor scheduler (`merge_by_stream`): `skip_group_files` (downsampling rule lookup #1) → whole-hour batch vs size groups - `merge_files`: `is_match_downsampling_rule` (lookup #2) → early returns, size-limit bypass - `merge_parquet_files`: lookup #3 + SQL selection by stream type, with `is_ingester: bool` standing in for the writer's role `merge_parquet_files` and `merge_parquet_files_with_downsampling` also carried two copies of the DataFusion context / union table / plan / execute-stream / channel / read-task plumbing. Reading the code meant reconciling three derivations of the same fact to know what a batch would produce. ## What - **`search::datafusion::merge::mode::MergeMode { Classic, TraceTimeIndex, FileList, Downsampling(rule) }`** — decided once: `MergeMode::for_compactor(stream_type, stream_name, max_ts)` by the scheduler (carried in `MergeBatch.mode`), `MergeMode::for_ingester(stream_type, stream_name)` by the movers. Every layer only asks it questions: `merges_whole_batch()`, `input_sort_order()`, `sql(schema)`. - **`MergeOutput { file_format, parquet_compression }`** (`for_ingester` / `for_compactor`) replaces the `is_ingester` bool. - **`merge_parquet_files(schema, tables, bloom, meta, &mode, output)`**: one `run_merge_query()` pipeline for every mode, then the mode's writer. Downsampling is a `MergeMode` arm calling `downsampling::write_files` instead of an early-return second implementation (`merge_parquet_files_with_downsampling` removed, −81 lines). - Compactor: `skip_group_files` / `is_match_downsampling_rule` and their `cfg(enterprise)` twin declarations are gone; `compaction` no longer imports `get_largest_downsampling_rule`. Commit 1 (`672ab315a1`) is pure refactor, no behavior change. ## Behavior change (commit 2, `8030eefd04`): whole-hour merges only for a closed hour, over every file of the hour Metrics downsampling used to merge the whole batch whenever the rule matched the batch's newest sample — including incremental rounds on the still-open hour — and it only ever saw the files below the 95%-of-`ZO_COMPACT_MAX_FILE_SIZE` bound of `query_for_merge`: files already above the bound were never downsampled, and small outputs were re-aggregated round after round. - `MergeMode::for_compactor(stream_type, stream_name, range_end_ts, finalize)` is decided **once per hour, before the file_list query**, from the end of the job's date range; a whole-hour mode (downsampling) is only chosen when `finalize` (the hour is closed). Incremental merges of a matching hour use the plain per-stream mode and leave the aggregation to the hour-end pass. - `query_for_merge` takes the `original_size` bound from the caller (`infra::file_list::merge_max_original_size()` = the classic 95% rule); a whole-hour merge asks for **every** file of the hour (`i64::MAX`), the same input range the TSID-major hour-end merge in #13839 uses. - Rule matching uses the hour's end instead of the batch's max `max_ts` (≤ 1h difference; the hour is decided as a unit). ## Guards (commit 3, cherry-picked from #13839) - compactor: fail the merge instead of deleting the sources when `merge_parquet_files` produced no files - search storage `file_list::get_scan_selection` strips the multi-account `{account}/::/` prefix so selections registered by `set` are found again (+ round-trip test) - `deleted.rs`: `delete_from_storage` takes the file slice and a key closure (`Cow`) instead of a pre-collected `Vec` ## Not in this PR The write stage still dispatches to four writers (`write_parquet`, `write_vortex`, `write_downsampled_parquet`, `write_downsampled_vortex`) that share the open / write / rotate / close skeleton and differ only in the rotation policy and how `FileMeta` is derived. Unifying them into one rotating writer is a follow-up (needs a decision on deriving `min_ts/max_ts` from the output batches for the classic path too). ## Validation - `cargo fmt --check`; `cargo clippy -p search -p compaction -p openobserve-jobs --tests` clean - `cargo test -p search --lib -- merge::` 6 passed (new: `mode_by_stream`, `sql_by_mode`, `output_file_format_uses_parquet_for_ingester_metrics`; trace-time-index merge test now goes through `MergeMode::for_ingester` and asserts `TraceTimeIndex`) - `cargo test -p compaction --lib -- merge` 30 passed; `cargo test -p infra --lib -- file_list` 49 passed - `downsampling.rs` is enterprise-only and not compilable locally; `write_files` / `pub(super) generate_downsampling_sql` were edited by hand and need the enterprise CI. |
||
|
Hengfei Yang
|
7dd19d78ad |
fix: reduce eval match preview window to one hour (#13859)
## Summary - reduce the new evaluation job matched-target preview from a rolling 24-hour window to one hour - update span, trace, and session preview labels to show `last 1h` - add deterministic coverage for the one-hour query range ## Why Selecting a stream immediately runs a matched-target count query. The hard-coded 24-hour window can scan a large amount of trace data on high-volume streams even though this number is only a creation-time preview. ## Impact This only changes the matched-target preview query and its label. It does not change the evaluation job schedule or execution range. ## Root cause The preview composable and its labels were hard-coded to 24 hours. ## Validation - `npm run test:unit -- --run enterprise/components/onlineEvals/composables/useJobMatchedTargets.spec.ts enterprise/components/onlineEvals/forms/job/JobPreviewPanel.spec.ts` - `npm run type-check:app` - focused ESLint and Prettier checks - `npm run lint:design:strict` ## Companion PR - openobserve/o2-enterprise#2452 fixes the OpenAI token-limit parameter and raises the LLM judge defaults. |
||
|
Hengfei Yang
|
4be23092a7 |
fix: skip session_id in trace time index merge when files lack the column (#13863)
## Problem Ingester merge jobs fail repeatedly for trace time index streams: ``` merge_parquet_files error for stream: default/metadata/trace_time_index__evaluator, ... err: Schema error: No field named session_id. Valid fields are tbl._timestamp, tbl.max_ts, tbl.min_ts, tbl.trace_id. ``` The failing file can never merge, so the job retries it forever and the file stays stuck in the WAL, never uploaded to object storage. ## Root cause - The trace time index write path only inserts a `session_id` value when the trace actually carries one; the field is nullable in the schema. - When the ingester persists WAL files, `merge_record_batches` prunes columns that are entirely null — so a file whose records all lack a session is written **without the `session_id` column at all**. Since most traces have no session id, this is the common case, not an edge case. - The ingester merge job builds the DataFusion table schema from the files' own fields (intentionally, to avoid the wide latest schema), and `merge_parquet_files` hardcoded `MAX(session_id) AS session_id` in the merge SQL for trace time index streams — planning fails when the column is absent. ## Fix Only include `MAX(session_id) AS session_id` in the merge SQL when the resolved schema actually contains the field. Files without the column merge with the original 4-column shape (matching what search already handles via schema adaptation); when any file in the batch has the column, the union schema null-fills the rest and the full query works. This also covers files written before `session_id` was added to the index schema. ## Testing Existing `test_trace_time_index_merge_aggregates_by_trace_id` covers the with-session path through the conditional SQL; verified the module's tests pass. |
||
|
Huaijin
|
4e805038fe |
refactor: merge result / access-plan / deletion helpers groundwork for TSID-major metrics (#13861)
Behavior-neutral refactors split out of #13839 so that PR only carries the TSID-major feature. Nothing here changes what is written, read or deleted; every commit is independently reviewable. ## Commits - `cba75d244b` **promql**: `apply_matchers` split into `matcher_predicates(schema, matchers) -> Vec<Expr>` + `apply_matchers` (unchanged behavior; lets other inputs evaluate the same matcher semantics). - `b5ff6fed21` **parquet**: `new_parquet_writer` / `append_metadata` generic over `W: AsyncFileWriter` instead of `&mut Vec<u8>`; existing callers unchanged. - `b07d84216a` **merge**: `MergeParquetResult::{Single, Multiple}` → `MergeParquetResult { files: Vec<MergedFile { buf, meta }>, file_format }` + `into_single()`. The compactor's two copies of the upload / index / push sequence become one loop (`compressed_size` is set from the buffer for every file, as the `Multiple` arm already did); downsampling maps its `(bufs, metas)` into it; ingester movers use `into_single()` instead of panicking/erroring on `Multiple`. - `014b6c5721` **compactor**: `deleted.rs` `derived_files()` / `delete_from_storage()` replace three copies of collect / `storage::del` / ignore-not-found (data, `.ttv`, flattened). - `7163786362` **search**: `generate_parquet_access_plan_from_ranges` extracted from the bitmap path (`set_slices` → ranges); an out-of-range, empty or unsorted range is rejected with `assert!` (`56de386522`; a producer that violates the sorted/non-overlapping contract is a bug, not something to degrade around) instead of the previous `unreachable!()` on the bitmap path. Adds `FileSelection::RowRanges(Arc<Vec<Range<usize>>>)` as the matching selection kind (no producer yet; Parquet only, Vortex full scan like `RowGroups`). - `754014ccf3` **test**: `ListingTableAdapter` plan-shape test for `FileSortOrder::HashTimestampAsc` (from #13853): three overlapping hash-sorted files → `SortPreservingMergeExec` over three single-file groups, no `SortExec`, globally ordered rows. `tempfile` as `search` dev-dependency. ## Validation - `cargo fmt --check` - `cargo clippy -p config -p promql -p search -p compaction -p openobserve-jobs -p search_service -p promql-service --tests` clean - `cargo test -p promql --lib -- utils` 10, `-p search --lib -- merge:: helpers listing_adapter` 14, `-p compaction --lib -- merge deleted` 31, `-p config --lib -- test_file_key` 4 passed - `merge/downsampling.rs` is enterprise-only and was updated by hand; needs the enterprise CI compile. |
||
 ktx-vaidehi
|
a6c910f015 |
fix(web): logs scroll jump, dashboard tab-rename width, and banner usage (#13858)
Three unrelated UI fixes. ### `fix: logs page wrap scroll` The Logs view delegates its scroll element to a container that also holds the histogram. TanStack Virtual compensates `scrollTop` whenever a measured row differs from its estimate — correct for a self-contained list, wrong for a shared scroller. With wrap on, rows were remeasured on every query refresh and the deltas pushed the whole page downward. Adds `preserveScrollOffsetOnRowResize` to `useTableVirtualization`, which pins `shouldAdjustScrollPositionOnItemSizeChange` to `() => false`. `OTable` opts in only when it has both a delegated `scrollEl` and dynamic row height, so self-contained tables keep the existing behaviour. ### `fix: dashboard tab rename size issue` The rename input's auto-sizer grid track was `minmax(0, max-content)`. Once the tab strip overflowed, the `0` minimum collapsed the track to `0px` and the input vanished. Now a bare `max-content`. ### `fix: ingestion and webinar obanner usage` Replaces two hand-rolled banner `div`s (`Ingestion.vue`, `WebinarBanner.vue`) with `OBanner`. To make that possible without appearance overrides, `OBanner` gains: - a `promo` variant (the existing marketing gold, so the standalone webinar bar still matches); - `center`, which groups message and actions in the middle of a bar instead of stranding the action at the far edge; - a `role` override, for a bar that is page furniture rather than a notification; - `dense` now actually applies in `bar` mode (`py-1` vs `py-2`). ### Tests `TabList.spec.ts` and `OTable.wrapMeasure.spec.ts` updated alongside. |
||
|
ktx-vaidehiandgithub-actions[bot]
|
c53a16b262 |
fix: script i18n (#13840)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
|
71ac1071be |
fix: draft workflows (#13666)
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yashodhan@openobserve.ai> |
||
|
Huaijin
|
617303a307 |
refactor(search): drop FileSortOrder from the distributed plan wire format (#13856)
Follow-up to #13853. The `(__hash__, _timestamp)` order is only relevant to the compactor merge; distributed query plans never carry it, so the wire-level `FileSortOrder` is unnecessary. ## Changes - `plan.proto` / generated: remove `enum FileSortOrder` and `NewEmptyExecNode.sort_order = 8`; `NewEmptyExecNode` keeps only the legacy `bool sorted_by_time = 6`. - `codec/empty_exec.rs`: encode/decode with the bool only (`FileSortOrder::from_sorted_by_time`); drop the enum-first/legacy-fallback logic and its test. - `sort_order.rs`: drop `to_proto` / `from_proto` and the round-trip test; `HashTimestampAsc` is documented as compactor-merge only. - `empty_exec.rs` tests: cover `None` / `TimestampDesc` output ordering only. Rolling upgrade: #13853 nodes still write and read `sorted_by_time`, nodes with this change only use `sorted_by_time` — no incompatibility. ## Validation - `cargo check -p search -p search_service --tests`, `cargo fmt --check` - `cargo test -p search --lib -- sort_order empty_exec` 26 passed |
||
|
ktx-akshay
|
411be8968a |
test: fix flaky report folder move test (#13847)
## Problem `reportFolders.spec.js` failed intermittently in CI. `P2: Move button disabled when source equals destination` failed on all 3 retries with a freshly-created report each time — the signature of poisoned shared state, not timing flake. ## Root causes Four independent test-side defects, all rooted in specs sharing one org, one user, and one `default` folder under `fullyParallel: true, workers: 5`: 1. **Reports bound to a stranger's dashboard.** `createReportViaApi` ignored the dashboard the test had just created and attached to `dashboards[0]` — whichever dashboard sorted first alphabetically in the shared folder. Deleting a dashboard removes its reports, so a neighbouring spec's cleanup made the report vanish mid-test. 2. **Blanket cleanups.** `cleanupDashboards()` deletes *every* dashboard owned by the automation user — every other spec's included (16 in one observed run). `reportFolders` also deleted every report matching `test_report_`, which swept up `scheduledReportsDrawer`'s `test_report_sched_*`. 3. **Bulk select-all across the shared folder.** `reports-bulk-operations` did "select all in `default` → bulk move/delete". Under parallel load that captured other specs' reports. Confirmed on a live server: `reportFolders`' report was found sitting in `bulk_test_folder_*`, created 78ms apart. 4. **A retry that could never succeed.** `expectReportVisibleInTable` retried by re-typing the same value into the search box. The Vue model never changed, and the in-folder search is a client-side filter over already-fetched rows — it cannot pull in a report the page hasn't loaded. The poll burned its full 45s doing nothing. Separately, `clickMove()` swallowed its dialog-close wait, so a move that never landed passed silently and failed a *later* test instead. ## Changes |
||
|
Huaijin
|
516e225bde |
refactor(search): replace sorted_by_time bool with FileSortOrder enum (#13853)
## Summary Every table provider, `NewEmptyExec`, the memtable `SortExec`, the merge `ORDER BY` and the `ListingTable` `file_sort_order` shared a single `sorted_by_time: bool` that implicitly meant `_timestamp DESC`. TSID-major metrics files (see #13839) are sorted by `(__hash__ ASC, _timestamp ASC)` instead, so a bool can no longer describe the physical layout of the files behind a table. This PR introduces `search::datafusion::sort_order::FileSortOrder { None, TimestampDesc, HashTimestampAsc }` as the single place that knows how each order maps to DataFusion logical / physical sort expressions, and threads it through the whole pipeline: - `DataFusionContextBuilder::sort_order` → enables `split_file_groups_by_statistics` for any sorted order - `TableBuilder::sort_order` → `ListingOptions::with_file_sort_order(order.logical_sort_exprs())` - `ListingTableAdapter` → regroups files by statistics using the declared order (`order.physical_ordering(schema)`) instead of a hard-coded `_timestamp DESC`; the "reverse file groups" fallback is kept only for `TimestampDesc` - `NewMemTable` → `SortExec` on the declared order (no-op if a sort column is missing) - `NewEmptyTable` / `NewEmptyExec` → equivalence properties from the declared order; display shows `sort_order=timestamp_desc` - `search_service` (`flight` / `storage` / `wal` / `create_tables_from_files`) pass the enum through - merge `ORDER BY` string derived from `FileSortOrder::TimestampDesc.order_by_clause()` ### Wire format `NewEmptyExecNode` gains `FileSortOrder sort_order = 8`. The legacy `bool sorted_by_time = 6` is still **written** (`true` iff `TimestampDesc`) and **read** as a fallback when `sort_order` is `NONE`, so mixed-version clusters keep working during rolling upgrades. ### No behavior change Every previous `sorted_by_time(true)` maps to `FileSortOrder::TimestampDesc`; `Sql.sorted_by_time` stays a bool (it describes the query, not the files) and is converted at the boundary. `HashTimestampAsc` is not yet used by any caller — that lands with the TSID-major work. ## Validation - `cargo fmt --all -- --check` - `cargo check --workspace --tests` - `cargo clippy -p search -p search_service -p compaction -p openobserve-jobs --tests` clean - `cargo test -p search --lib`: all pass except 12 pre-existing `sql::histogram` failures caused by a local `.env` override (`ZO_HISTOGRAM_BREAKDOWN_FIELDS=""`), unrelated - `cargo test -p search_service --lib -- grpc::` 16 passed - New tests: `sort_order` (logical/physical exprs, ORDER BY clause, proto roundtrip, missing column), `NewEmptyExec` output ordering per variant, codec roundtrip for all variants + legacy-flag fallback decoding |
||
|
Vamsi Karthik R
|
99b1a23773 |
test(slo): E2E + API suite for SLOs (#13836)
Adds automated test coverage for **SLOs** (#13547, #13577, #13579, #13704, #13761, #13784), which shipped with none. **149 tests** — 77 Playwright + 72 pytest — all three SLI types, both query languages, happy / negative / edge paths. Clean branch off `main`. **UI (Playwright · `tests/ui-testing`)** - `pages/sloPages/` — 4 page objects (verified `data-test` selectors, hydration + Monaco-sync waits, OSelect commit-retry, virtualized-list filtering). - `playwright-tests/SLO/slo-crud.spec.js` (24) — create count/time-slice/grouped through the form, detail, edit, SLI-type change, tags, pause, delete, filters; negative: dismissed delete, empty search, cancel, unmeasured SLI is absent not zero. - `playwright-tests/SLO/slo-form-validation.spec.js` (20) — server errors surfacing in the form, conditional fields per SLI type, count + time-slice previews, **PromQL count and time-slice created end to end**; negative: no stream, invalid aggregate, over-long name, target boundaries. - `playwright-tests/SLO/slo-alert-sli.spec.js` (8) — **the `alert` SLI type**: source picker, create, config carries only the alert id, detail names the source; negative: ineligible source offered-but-disabled, grouped alert SLI, slice finer than source cadence. - `playwright-tests/SLO/slo-timeslice.spec.js` (10) — **#13761 P0 regression** (measured SLI, not the preview), PromQL, previews, gaps, language-flip clearing. - `playwright-tests/SLO/slo-alerts.spec.js` (15) — burn-rate + error-budget kinds, presets, edit, freeze door, burndown; negative: no destination, cancel, blank/invalid name. - `playwright-tests/utils/slo-seed.js` — backdated seeding, metrics seeding, eligible-source-alert fixture, ground-truth query, fail-fast env guard, transient-store retry, full fixture teardown (alerts → streams → destinations → templates). **API (pytest · `tests/api-testing`)** - `tests/slos/test_slo_validation.py` (72) — full REST surface (create/get/update/enable/delete/groups/slo-eligible), validation on **both create and update** (name, target, window, slice, config shapes, comparators, float-threshold round-trip), boundaries (target 0.001/99.999, name 256 vs 257, threshold 0/negative/1e12/NaN, unicode names), alert-source eligibility (silence-gated, too-infrequent, cron), PromQL config shapes, auth 401/404. **CI** - Two shards: `SLO` (`workers: 3`) and `SLO-Measurement` (`workers: 1`). Split because the backfill job runs with `ZO_SCHEDULER_SLO_BACKFILL_CONCURRENCY=1`, so measurement specs sharing an instance starve each other's waits — `fullyParallel` races spec *files*, which `mode:'serial'` cannot prevent. The 3-worker pin is SQLite: at 5, concurrent SLO/alert/destination writes trip `database is locked` on the single-node meta store. - New optional per-shard `ingest_allowed_upto` (`240`) alongside the existing `quick_mode_enabled`: the workflow default of 5 **hours** silently drops the 8 days of history an SLO window needs, while still answering `200`. Other shards keep the default. Same precedent as `api-testing.yml`'s `48`. **Feature code**: 9 additive `data-test`/`dataTest` attributes in `SloDetail.vue`, `SloList.vue`, `SloAlertCondition.vue` — no logic changes, `vue-tsc` clean. The stat tiles, tabs and burn-window inputs had no stable selectors; `OStatStrip`/`OTab` already forward the attribute. **Behaviour**: self-cleaning (`e2e_slo_*`, worker-scoped since `afterAll` runs per worker). **Verified in CI** on this branch: `e2e / SLO` **52 passed** (1.8m, workers pinned to 3), `e2e / SLO-Measurement` **25 passed** (11.3m, workers pinned to 1) — the measurement shard passing confirms the per-shard `ingest_allowed_upto` took effect, since those tests are impossible under the 5-hour default. pytest **72 passed** locally; in CI's 599-test run one setup delete hit a transient 500 (see findings) and now retries. **Findings for the feature owner** (reported, not fixed here): - A blank "good when" shows `Request failed with status code 422` — `pruned()` drops the empty key, so the API's real 400 message never reaches `save()`. - A rejected alert name shows no banner: `SloAlertForm.submit()` returns on `nameError` before setting `saveError`. - `avg(latency_ms); DROP TABLE users--` is accepted at create time (parsed only at measurement). - The name-length message says "less than 256 characters" but 256 is accepted — 257 is the first rejection. - Duplicate alert names on one SLO are accepted (duplicate SLO names are a 409), and the panel lists them by name, so they are indistinguishable. - SLO names accept characters that break the list's own `slos-slolist-edit-${name}` selectors; alert names correctly reject them. - All chart/preview root `data-test` ids are unreachable — a parent's fallthrough attribute overrides the child root's. - Both edit forms hydrate asynchronously, so a blank form can flash before the stored values appear. - `DELETE /slos/{id}` on a freshly-created SLO returned **500** once under CI load (not reproducible locally in 12 rapid iterations). Contention is arguably a retryable 503, and the body carried no actionable message. - **A scheduled alert whose source stream is deleted is retried forever** (`Stream "…" not found in schema`), and the SLO backfill lane shares the scheduler's worker budget — so accumulated dead alerts *starve SLO measurement*. Observed: 65 of them held coverage at 1 of 7 chunks indefinitely; deleting them restored 0.143 → 1.0 in 240s. Harmless in CI (fresh instance per shard) but it degrades any long-lived environment. The suite now cleans up after itself; the retry-forever behaviour itself may deserve a look. **Paired ENT PR: openobserve/o2-enterprise#2434.** ENT's matrix is this repo's manifest plus an overlay, merged by this repo's `build-ci-matrix.js`, so both SLO shards reach ENT automatically — verified: 31 merged shards, both SLO entries carrying `ingest_allowed_upto=240` and their worker pins, no other shard affected. What ENT was missing is *consuming* those two fields, which #2434 adds. Merge the two together: this one alone leaves ENT booting the SLO shards on the 5-hour ingest window, and #2434 alone is inert. *(An earlier review reported ENT as using an inline matrix with no SLO shard — that turned out not to be the case.)* |
||
 
|
69bd367d9b |
feat: notification dependency view — where-used for templates, destinations & alerts (#13820)
## What Adds a **"View dependencies"** affordance to the **Templates**, **Notification Destinations**, and **Alerts** lists. A small `graph-1` icon on each row opens a compact popover that shows everything that entity connects to along the **Template → Destination → Alert** chain, with search, pagination, and per-row **Open** / **Delete**. ## Why The linkage between alert templates, destinations, and alerts is invisible today. There is **no reverse-lookup API** — the only way to learn "which alerts use this destination" is to try deleting it and hit the backend's `409 "Destination is currently used by alert: X"`. That's a dead end, not a map. This surfaces the dependency proactively so you can see the blast-radius *before* you change or delete something, and act on it in place (open or delete a related entity, with the same `409` guard still protecting in-use items). ## Design The three entities are linked **by name** (there are no ID foreign keys): - `Alert.destinations: string[]` → `Destination.name` - `Destination.template` → `Template.name` (a destination's default) - `Alert.template` → `Template.name` (an alert-level override) ```mermaid flowchart LR T["Template"] -->|"Destination.template"| D["Destination"] D -->|"Alert.destinations[]"| A["Alert"] T -. "Alert.template (override)" .-> A ``` Because the alerts **list** DTO previously omitted `destinations`, the whole graph would have needed N per-alert detail fetches. Instead a small **opt-in additive field** on the list item makes the entire graph **three list calls**, cross-referenced client-side and cached per-org: ```mermaid flowchart TD Row["List row · graph-1 icon"] --> Pop["DependencyChainPopover<br/>(OPopover + ConfirmDialog)"] Pop --> Panel["DependencyUsagePanel<br/>search · collapsible sections · pager"] Panel --> Rows["DependencyUsageRow ×N<br/>Open · Delete"] Panel --> Load["useDependencyGraph.loadGraph()<br/>per-org cache"] Load -->|"alerts?include_dependencies=true<br/>+ destinations + templates (3 list calls)"| Build["buildGraph() → nodes + edges"] Panel --> Focus["buildFocusChain(graph, focus)<br/>directional: a shared template never<br/>drags in sibling destinations"] Rows -. "requestDelete(node)" .-> Pop Rows -. "open(node) — same-kind, in-place" .-> Pop Pop -. "ConfirmDialog → service.delete()<br/>(409 'used by' → toast)" .-> Done["emit deleted → list refreshes"] ``` Why the ConfirmDialog lives in the popover **wrapper**, not the panel: a modal can't render inside the popover without dismissing it, so the row/panel only *request* a delete and the wrapper owns the real confirmation. **Open** works the same way for the in-place editors: templates/destinations edit on their own list route (which only opens from the page's mount-time handler), so a same-page open is bubbled to the host page to invoke its native editor rather than a no-op `router.push` to the same route. ## UX / behaviour - **Where-used, grouped:** collapsible **Templates · Destinations · Alerts** sections, each a clickable row (kind-coloured icon, name, and for destinations an "N alerts" **total-usage** badge). - **Search-first:** one search box filters all groups — the key affordance when a destination feeds thousands of alerts. - **Paged:** the alerts group renders **50 at a time** with a pager pinned beside its (sticky) header, so 4,000-alert destinations stay responsive. - **Act in place:** every row has **Open** (templates/destinations → their editor; alerts → the alert's detail page) and **Delete** (the same `ConfirmDialog` the list pages use; the backend `409` still blocks in-use items). - Consistent trigger on all three lists; **purely additive** — no existing button, icon, or column was moved or replaced. ## Backend change `ListAlertsResponseBodyItem` (`src/api/management/src/models/alerts/responses.rs`) gains two additive fields — the linkage the dependency view cross-references: - `destinations: Vec<String>` - `template: Option<String>` (the alert-level override) They serialize **only when the caller passes `?include_dependencies=true`** (which just the dependency popover does); the default paginated alerts-list path is unchanged, so ordinary consumers pay no extra bytes and receive no module-scoped destination/template names. Both are also `skip_serializing_if`-empty. Backward-compatible, OSS, no new endpoint. ## Testing - **Unit (vitest) — 17 tests:** - `useDependencyGraph.spec.ts` — `buildGraph` (linkage, unused/orphan, dangling refs, override edges, shared destinations) and directional `buildFocusChain` per focus kind (incl. the alert-focus total-usage count). - `DependencyUsagePanel.spec.ts` — sections render every kind, search + 50/page paging, collapse, `requestDelete`, the same-kind `open` bubble, and the `include_dependencies` opt-in. - `DependencyChainPopover.spec.ts` — trigger + lazy panel mount, `requestDelete` → ConfirmDialog → API + `deleted`, and the `open` bubble + close. - **Backend:** serialization test asserts `destinations`/`template` are omitted when empty. - `type-check`, `lint:design:strict`, and `prettier` all clean. - A popover e2e (Playwright) is a follow-up in a separate PR — the earlier spec targeted an interim dedicated page that this PR does not ship, so it was removed. ## Notes - Net change is popover-only: an earlier iteration added a dedicated Reliability page + route + rail entry; those were dropped in favour of the on-the-fly popover, so `router.ts` / `navGroups.ts` are untouched vs `main`. - Icon `graph-1` was added to the icon registry (additive) for the popover trigger. - **Deferred (see design doc Non-goals):** a server-side where-used / `?destinations=` endpoint so the popover fetches only the neighbourhood instead of the full alert list; and enterprise RBAC name-filtering on the gated fields. Design at: https://github.com/openobserve/designs/blob/main/alerts/notification-dependencies/design.md (designs#8: `alerts/notification-dependencies/design.md`) --------- Co-authored-by: Abhay Padamani <141122205+ktx-abhay@users.noreply.github.com> Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech> |
||
|
Hengfei Yang
|
42d4e2a203 |
fix: return empty when usage is not there instead of report error (#13842)
- [x] Should start check job after node start, otherwise it will report gRPC can't connect. - [x] Should skip usage check when there is no usage stream, otherwise it will always report error. |
||
  
|
1c849a9c71 |
refactor(web): migrate templates + shadows to tokens, delete the debt baseline (#13769)
## What
Deletes `web/scripts/design-debt-baseline.json` — **not by relaxing the
rules, by fixing the debt**. Every design-token category is now
zero-tolerance, SCSS is gone from the app, and 144 hand-written shadow
values collapse into one scale.
## The guard was under-reporting (three separate holes)
1. **Comment contamination.** `check-design-consistency.mjs` fed comment
text into its exemption checks, so a `keep()` comment that merely
*named* `:deep(` / `::after` / `@keyframes` silenced the rule after it —
and a `;` or `{` inside a comment corrupted the offsets, so stripping
comments afterwards couldn't help. Comments are now blanked
(offset-preserving) before the scan. **Uncovered 9 occurrences in 6
files.** Real debt was 110, not 101.
2. **`arbShadow` only scanned the template.** `box-shadow:` in a
`<style>` block and `boxShadow:` in a JS/`:style` object were invisible
— 40 literals sat there while the guard reported zero. It now scans all
three syntaxes across `.vue` and `.ts`.
3. **`rawVarInTemplate` missed the fallback form.**
`border-(--color-x,rgba(…))` and `bg-[var(--color-x,#fff)]` slipped
through. The token is always defined, so the fallback can never fire —
it only hid the read.
## stylelint had never run on 14 files
`postcss-scss` isn't installed, so `postcss-html` **silently skipped**
every `<style lang="scss">` block. The CI step named *"Lint styles
(stylelint — bans legacy --o2-* tokens)"* was a no-op for those files —
including `color-no-hex` and the `.body--dark` selector ban. All 14 are
plain CSS now and `vue/block-lang` errors on `scss`/`sass`/`less`, so it
cannot silently recur.
## What moved
| | before | after |
|---|---|---|
| `rawVarInComponent` (in `<style>`) | 110 | **0** |
| template `x-[var(--color-*)]`, utility exists | 225 | **0** |
| template reads, no utility | 27 | **0** |
| `style` / `:style` colour literals | 29 | **0** |
| `color-mix(token N%, transparent)` | 160 | **0** |
| hand-written shadows | 233 (144 distinct) | **0** |
| SFCs using SCSS | 14 | **0** |
Guard categories: 22 → 24. `rawVarInTemplate` and `arbShadow` are new;
`arbPx` was deleted because main's `local/no-hardcoded-px` ESLint rule
(#13541) now owns px across every file type.
## Shadows are a system now
Stripped of colour, the 144 values were the same handful of geometries
rewritten in px vs rem, with and without an explicit `0` spread.
```
--shadow-xs / sm / md / lg elevation
--shadow-sticky-left / -right directional (sticky columns)
--shadow-rail inset status rail
--shadow-glow-sm / -md / -lg coloured, 3 intensities
--shadow-scroll-top / -bottom / -both scroll affordance
--shadow-ring-hairline 1px ring where a utility can't reach
--shadow-ai-focus AI composer focus glow (flips in dark)
```
Colour is a **separate axis**. Tailwind rewrites a `--shadow-*` token as
`<geometry> var(--tw-shadow-color, <default>)` — verified by compiling a
probe, and it holds for multi-layer tokens — so a template writes
`shadow-md shadow-ai-accent/35`. A plain stylesheet can't add a utility,
so the role and glow tokens read `--glow-color`; that keeps the pipeline
canvas colour-coding nodes by type from CSS and lets the status rail be
painted per row from JS.
Also reclassified: `0 0 0 2px <colour>` is a **ring**, not a shadow, and
`0 1px 0 0 <colour>` is a **border**.
## color-mix, analysed by intent
All 331 occurrences were classified rather than treated as one problem:
- **160** mixed a token with `transparent` — only an alpha change, so
it's Tailwind's `/N` modifier written the long way (verified: `/N`
compiles to `color-mix(in oklab, <token> N%, transparent)`, and mixing
with `transparent` is premultiplied, so the interpolation space doesn't
matter)
- **81** blended two colours — a derived colour, so it's a token now
(~50 minted)
- **75** were shadow/ring geometry, handled by the scale
- **8 gradients + 1 `calc()`-driven percentage** genuinely have no
utility form and remain; the guard exempts gradient/mask/filter
contexts, matching what the style-block counter already exempts
## Merge with main
Main was running the same migration in parallel (**#13541 "px to rem"**
removed all SCSS, added `no-hardcoded-px`, and built its own shadow
scale), so the merge collided in 68 files / 155 hunks. Resolved by
taking main's source files and re-applying this branch on top, with the
guards driving convergence (429 → 0) rather than hand-resolving hunks.
On shadows, **main's structure wins** — geometry in `semantic.css`
`@theme`, colours as `--shadow-*-c*` sub-tokens, wrapped in its px
disable; its `--shadow-sm/md/lg`, `--shadow-scroll-*` and
`--shadow-glow` keep their definitions. This branch adds what main
didn't have: `xs`, the directional/inset roles, and the
elevation-matched coloured glows.
## New tokens
No new *colours* — each names a value a component already hardcoded,
because a registered palette utility (`text-orange-500`) is banned by
`rawPalette`, so "just register the ramp" was never available.
`--color-splitter-hover`, `--color-status-flaky` (+`-text`),
`--color-chat-bubble-user-border`, `--color-thread-avatar-bg`,
`--color-thread-accent-strong`, `--color-menu-badge-ring`,
`--color-progress-track-fill`, `--color-ingest-accent-*` (5), ~50
derived blend tokens, plus `--color-border-strong` /
`--color-border-subtle` / `--color-warning-300` registered for utility
use.
## Visual deltas to review
Everything else is value-preserving. These are not:
- **4 hover accents** move from a fixed `primary-600` to `accent` — 600
→ 400 **in dark only** (2 chevrons, an empty-state icon fill, one link)
- **sticky table header** in `TraceDetailsSidebar` was a fixed light
grey that never flipped; now `border-default`, so it themes
- **shadow canonicalization** snaps near-identical geometries together
and moves ~40 hardcoded `rgba(0,0,0,x)` onto the scale's token colour
Highest-surface single file: `AlertSettingsHelpDrawer.vue` — a whole
style block deleted with ~44 class names moved into template utilities.
## Verification
`lint:design` · `lint:tokens` · `lint:token-purity` · `lint:styles` ·
`format:check` · `lint:ci` · `vue-tsc` — all clean.
**94 spec files covering every file this branch touches: 4395 passing, 0
failing.**
Not run: the full 1236-file unit suite, Playwright/E2E, and any browser
visual check.
---------
Co-authored-by: ktx-vaidehi <vaidehi.akhani@kiara.tech>
Co-authored-by: Abhay Padamani <141122205+ktx-abhay@users.noreply.github.com>
Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech>
Co-authored-by: ktx-vaidehi <134508096+ktx-vaidehi@users.noreply.github.com>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
|
||
|
Hengfei Yang
|
6ca0eec107 |
fix: MCP tool calls return 404 when ZO_BASE_URI is set (#13844)
Fixes #13845 ## What changed The MCP shared HTTP client (used to execute tool calls against the local REST API) now includes `ZO_BASE_URI` in its base URL: `http://localhost:{port}{base_uri}` instead of `http://localhost:{port}`. ## Why When `ZO_BASE_URI` is set (e.g. `/abc`), all HTTP routes are mounted under that prefix. The MCP endpoint itself was reachable (`POST /abc/api/default/mcp` → 200), but every tool execution failed: the internal client sent requests to the unprefixed path and got a 404. ``` INFO rmcp_openapi::http_client: Final URL: http://localhost:5080/api/default/streams INFO config::axum::middlewares::access_log: 127.0.0.1 "GET /api/default/streams HTTP/1.1" 404 ... ``` ## Notes for reviewers - `cfg.common.base_uri` is already normalized at config load to have no trailing slash (empty when unset), so the empty-prefix behavior is unchanged. - The join semantics are safe for path-prefixed base URLs: rmcp-openapi's `with_base_url` appends a trailing slash and `build_url` strips the tool path's leading slash before `Url::join`, covered by the crate's `test_build_url_with_base_url_containing_path`. So `/abc` + `/api/{org}/streams` → `/abc/api/{org}/streams`. - The other `localhost:5080` occurrence in `tools.rs` is a test-only mock client; the OAuth protected-resource metadata handler already includes `base_uri`. - `cargo check -p openobserve-mcp` and `cargo test -p openobserve-mcp` pass. |
||
|
Hengfei Yang
|
9d76505882 |
feat(traces): resolve session time ranges through the trace time index (#13837)
Design at: [https://github.com/openobserve/designs/blob/main/infrastructure/file-list/session-time-index.md](https://github.com/openobserve/designs/blob/main/infrastructure/file-list/session-time-index.md) ## Summary - carry an optional `session_id` attribute on trace time index records, resolved from span attributes by the existing `SESSION_ID_COLUMNS` precedence; the first non-empty value in the request wins for the whole trace, stamped onto all of its hour-bucket records - keep the aggregation key `(hour, trace_id)` unchanged; hourly compaction picks the attribute up with `MAX(session_id)`, which ignores NULLs - reuse the locate/expand timer for sessions with a 7-day expand window (traces keep 24h); a UUID v7 session id doubles as a probe anchor and tightens the scan's left bound to the session's lifetime neighborhood - aggregate session lookups per trace, so a lookup resolves the session's range plus every member trace's real range - reshape the time-range endpoint to query parameters so one endpoint serves both lookup kinds - make the session-details caller range optional, union it with the indexed range (never narrowed), take trace ids from the index, and keep the legacy source-stream scan for pre-coverage data - add `session_id` to the default bloom filter fields (gated by the existing `ZO_FEATURE_DEFAULT_INDEX_FIELDS_ENABLED`, folds into the secondary-index field set like `trace_id`); streams without the column build nothing ## Why Session detail lookup depends on a caller-provided time range: the sessions list passes its own picker window into the details request, so an LLM session that started before the window silently loses its earlier turns, and a deep link without a range is a hard 400. The trace time index already turns a `trace_id` into a time range at bounded cost; a session is the same problem one level up — `session_id` → range plus member traces. ## Environment variables No new variables. The existing `ZO_TRACE_TIME_INDEX_ENABLED` (default `true`) gates the feature, session column included. The existing coverage marker is reused; there is no historical backfill. This change must ship in the same release as #13827: reusing the coverage marker is valid only while no release separates the two. ## API changes All timestamp parameters and returned ranges use microseconds. ### Time-range endpoint reshaped ```http GET /api/{org_id}/{stream_name}/traces/time_range?trace_id=xx&hint_ts=... GET /api/{org_id}/{stream_name}/traces/time_range?session_id=xx&hint_ts=... ``` Exactly one of `trace_id` / `session_id` is required; zero or both is `400`. This replaces `GET /{stream_name}/traces/{trace_id}/time_range`; the old path form is removed without a compatibility alias since #13827 has not shipped in a release. ### Session details: range becomes optional `GET /api/{org_id}/{stream_name}/traces/session/details` now accepts `start_time` / `end_time` as an optional pair plus an optional `hint_ts` (defaulting to the caller range's midpoint). The effective range is the union of the caller range and the indexed session range, so it never narrows what the caller asked for; `400` only when neither side yields a range. When the caller range lies entirely inside index coverage the source-stream trace-id scan is skipped; otherwise it still runs over the caller range and its results are merged, so behavior is never worse than before. No frontend changes are required — the frontend keeps sending its list window and the union completes the session. ## Validation - `cargo test -p openobserve-core traces::time_index --lib` — 13 passed - `cargo test -p search datafusion::merge --lib` — 5 passed, including NULL-mixed session aggregation and pre-column file compatibility - `cargo test -p openobserve-api-search --lib` — 149 passed - `cargo test -p openobserve-api-http --lib` — 53 passed, including OpenAPI/MCP initialization - `cargo test -p config router:: --lib` — 8 passed - `cargo test -p infra db::trace_time_index --lib` — passed - `cargo clippy` on all touched crates (`--all-targets`) and `cargo check --workspace` — clean - the enterprise repository has no references to any changed API and no enterprise-gated code was modified |
||
|
Abhay Padamaniandgithub-actions[bot]
|
f112ac01a5 |
fix: dashboard layout improvements (#13815)
<h2 style="unicode-bidi: plaintext; color: rgb(204, 204, 204);
font-family: -apple-system, "system-ui", "Segoe UI",
Roboto, sans-serif; font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none; widows: 2;
word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal;
background-color: rgb(24, 24, 24); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color:
initial;">Dashboard layout & UX improvements</h2><p
style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em;
unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family:
-apple-system, "system-ui", "Segoe UI", Roboto,
sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures:
normal; font-variant-caps: normal; font-weight: 400; letter-spacing:
normal; orphans: 2; text-align: start; text-indent: 0px; text-transform:
none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(24, 24, 24); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color: initial;">Polish
pass on the dashboard experience: settings layout, drag/resize grid
feedback, empty states, and per-tab panel counts.</p><h3
style="unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family:
-apple-system, "system-ui", "Segoe UI", Roboto,
sans-serif; font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none; widows: 2;
word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal;
background-color: rgb(24, 24, 24); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color: initial;">✨
Highlights</h3><p style="white-space: pre-wrap; margin-top: 0.1em;
margin-bottom: 0.2em; unicode-bidi: plaintext; color: rgb(204, 204,
204); font-family: -apple-system, "system-ui", "Segoe
UI", Roboto, sans-serif; font-size: 13px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal; font-weight:
400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(24, 24, 24);
text-decoration-thickness: initial; text-decoration-style: initial;
text-decoration-color: initial;"><strong>Grid feedback while
dragging/resizing panels</strong> (<a
href="vscode-webview://0vkpvjgb486eanj04qjfqcnqmnatevjacbbbqdp3ecrs4efo5krm/web/src/views/Dashboards/RenderDashboardCharts.vue"
target="_blank" rel="noopener noreferrer" style="color: rgb(77, 170,
252); text-decoration: rgb(77, 170,
252);">RenderDashboardCharts.vue</a>)</p><ul
style="padding-inline-start: 2em; color: rgb(204, 204, 204);
font-family: -apple-system, "system-ui", "Segoe UI",
Roboto, sans-serif; font-size: 13px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal; font-weight:
400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; white-space: normal; background-color:
rgb(24, 24, 24); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color: initial;"><li
style="unicode-bidi: plaintext;">Added a live alignment grid that
appears only during a drag or resize, drawn from GridStack's real cell
metrics (192 columns × row height).</li><li style="unicode-bidi:
plaintext;">The grid is<span> </span><strong>operation-aware</strong>:
lines snap to the top/left edges
while<span> </span><em>moving</em><span> </span>and the bottom/right
edges while<span> </span><em>resizing</em>, so the edge you're dragging
always lands on a line and every line is reachable (no skipped
positions).</li><li style="unicode-bidi: plaintext;">Panels are made
opaque during interaction (<code style="font-family: monospace; color:
rgb(208, 208, 208); background-color: rgb(60, 60, 60); padding: 2px 4px;
border-radius: 3px; word-break: break-word; font-size:
0.9em;">--color-surface-base</code>) so the grid only shows in the
gutters/empty cells, not through chart or table panels.</li></ul><p
style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em;
unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family:
-apple-system, "system-ui", "Segoe UI", Roboto,
sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures:
normal; font-variant-caps: normal; font-weight: 400; letter-spacing:
normal; orphans: 2; text-align: start; text-indent: 0px; text-transform:
none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(24, 24, 24); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color:
initial;"><strong>Empty states</strong> (<a
href="vscode-webview://0vkpvjgb486eanj04qjfqcnqmnatevjacbbbqdp3ecrs4efo5krm/web/src/lib/core/EmptyState/presets.ts"
target="_blank" rel="noopener noreferrer" style="color: rgb(77, 170,
252); text-decoration: rgb(77, 170, 252);">presets.ts</a>, <a
href="vscode-webview://0vkpvjgb486eanj04qjfqcnqmnatevjacbbbqdp3ecrs4efo5krm/web/src/components/dashboards/settings/VariableSettings.vue"
target="_blank" rel="noopener noreferrer" style="color: rgb(77, 170,
252); text-decoration: rgb(77, 170, 252);">VariableSettings.vue</a>, <a
href="vscode-webview://0vkpvjgb486eanj04qjfqcnqmnatevjacbbbqdp3ecrs4efo5krm/web/src/views/Dashboards/ScheduledDashboards.vue"
target="_blank" rel="noopener noreferrer" style="color: rgb(77, 170,
252); text-decoration: rgb(77, 170,
252);">ScheduledDashboards.vue</a>)</p><ul style="padding-inline-start:
2em; color: rgb(204, 204, 204); font-family: -apple-system,
"system-ui", "Segoe UI", Roboto, sans-serif;
font-size: 13px; font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; font-weight: 400; letter-spacing: normal;
orphans: 2; text-align: start; text-indent: 0px; text-transform: none;
widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px;
white-space: normal; background-color: rgb(24, 24, 24);
text-decoration-thickness: initial; text-decoration-style: initial;
text-decoration-color: initial;"><li style="unicode-bidi:
plaintext;">Replaced the generic<span> </span><code style="font-family:
monospace; color: rgb(208, 208, 208); background-color: rgb(60, 60, 60);
padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size:
0.9em;">NoData</code><span> </span>placeholder with<span> </span><code
style="font-family: monospace; color: rgb(208, 208, 208);
background-color: rgb(60, 60, 60); padding: 2px 4px; border-radius: 3px;
word-break: break-word; font-size:
0.9em;">OEmptyState</code><span> </span>for the Variables settings table
and the Scheduled Dashboards list.</li><li style="unicode-bidi:
plaintext;">Added a new<span> </span><code style="font-family:
monospace; color: rgb(208, 208, 208); background-color: rgb(60, 60, 60);
padding: 2px 4px; border-radius: 3px; word-break: break-word; font-size:
0.9em;">no-variables</code><span> </span>preset (illustration + copy +
"Add variable" action); reused the existing<span> </span><code
style="font-family: monospace; color: rgb(208, 208, 208);
background-color: rgb(60, 60, 60); padding: 2px 4px; border-radius: 3px;
word-break: break-word; font-size:
0.9em;">no-reports</code><span> </span>preset for scheduled
dashboards.</li><li style="unicode-bidi: plaintext;">Fixed the Variables
table so a long
list<span> </span><strong>scrolls</strong><span> </span>instead of
clipping, and made the Scheduled Dashboards table fill the drawer height
so the empty state centers.</li></ul><p style="white-space: pre-wrap;
margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi: plaintext; color:
rgb(204, 204, 204); font-family: -apple-system, "system-ui",
"Segoe UI", Roboto, sans-serif; font-size: 13px; font-style:
normal; font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start;
text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(24, 24, 24);
text-decoration-thickness: initial; text-decoration-style: initial;
text-decoration-color: initial;"><strong>Dashboard settings
layout</strong> (<a
href="vscode-webview://0vkpvjgb486eanj04qjfqcnqmnatevjacbbbqdp3ecrs4efo5krm/web/src/components/dashboards/settings/GeneralSettings.vue"
target="_blank" rel="noopener noreferrer" style="color: rgb(77, 170,
252); text-decoration: rgb(77, 170,
252);">GeneralSettings.vue</a>)</p><ul style="padding-inline-start: 2em;
color: rgb(204, 204, 204); font-family: -apple-system,
"system-ui", "Segoe UI", Roboto, sans-serif;
font-size: 13px; font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; font-weight: 400; letter-spacing: normal;
orphans: 2; text-align: start; text-indent: 0px; text-transform: none;
widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px;
white-space: normal; background-color: rgb(24, 24, 24);
text-decoration-thickness: initial; text-decoration-style: initial;
text-decoration-color: initial;"><li style="unicode-bidi:
plaintext;">Reworked the General settings layout.</li></ul><p
style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em;
unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family:
-apple-system, "system-ui", "Segoe UI", Roboto,
sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures:
normal; font-variant-caps: normal; font-weight: 400; letter-spacing:
normal; orphans: 2; text-align: start; text-indent: 0px; text-transform:
none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(24, 24, 24); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color:
initial;"><strong>Tab panel-count badge</strong> (<a
href="vscode-webview://0vkpvjgb486eanj04qjfqcnqmnatevjacbbbqdp3ecrs4efo5krm/web/src/components/dashboards/tabs/TabList.vue"
target="_blank" rel="noopener noreferrer" style="color: rgb(77, 170,
252); text-decoration: rgb(77, 170, 252);">TabList.vue</a>)</p><ul
style="padding-inline-start: 2em; color: rgb(204, 204, 204);
font-family: -apple-system, "system-ui", "Segoe UI",
Roboto, sans-serif; font-size: 13px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal; font-weight:
400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; white-space: normal; background-color:
rgb(24, 24, 24); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color: initial;"><li
style="unicode-bidi: plaintext;">Each dashboard tab now shows a badge
with its panel count (section headers excluded), so tab density is
visible without opening each tab.</li></ul><p style="white-space:
pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em; unicode-bidi:
plaintext; color: rgb(204, 204, 204); font-family: -apple-system,
"system-ui", "Segoe UI", Roboto, sans-serif;
font-size: 13px; font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; font-weight: 400; letter-spacing: normal;
orphans: 2; text-align: start; text-indent: 0px; text-transform: none;
widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(24, 24, 24); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color:
initial;"><strong>Variable dependency indicator</strong> (<a
href="vscode-webview://0vkpvjgb486eanj04qjfqcnqmnatevjacbbbqdp3ecrs4efo5krm/web/src/components/dashboards/settings/VariableSettings.vue"
target="_blank" rel="noopener noreferrer" style="color: rgb(77, 170,
252); text-decoration: rgb(77, 170,
252);">VariableSettings.vue</a>)</p><ul style="padding-inline-start:
2em; color: rgb(204, 204, 204); font-family: -apple-system,
"system-ui", "Segoe UI", Roboto, sans-serif;
font-size: 13px; font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; font-weight: 400; letter-spacing: normal;
orphans: 2; text-align: start; text-indent: 0px; text-transform: none;
widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px;
white-space: normal; background-color: rgb(24, 24, 24);
text-decoration-thickness: initial; text-decoration-style: initial;
text-decoration-color: initial;"><li style="unicode-bidi:
plaintext;">Variables that depend on others show a "Depends on: …" chip
(tree icon + count) in the settings table.</li></ul><p
style="white-space: pre-wrap; margin-top: 0.1em; margin-bottom: 0.2em;
unicode-bidi: plaintext; color: rgb(204, 204, 204); font-family:
-apple-system, "system-ui", "Segoe UI", Roboto,
sans-serif; font-size: 13px; font-style: normal; font-variant-ligatures:
normal; font-variant-caps: normal; font-weight: 400; letter-spacing:
normal; orphans: 2; text-align: start; text-indent: 0px; text-transform:
none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(24, 24, 24); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color:
initial;"><strong>Panel header token</strong> (<a
href="vscode-webview://0vkpvjgb486eanj04qjfqcnqmnatevjacbbbqdp3ecrs4efo5krm/web/src/lib/styles/tokens/component.css"
target="_blank" rel="noopener noreferrer" style="color: rgb(77, 170,
252); text-decoration: rgb(77, 170, 252);">component.css</a>)</p><ul
style="padding-inline-start: 2em; color: rgb(204, 204, 204);
font-family: -apple-system, "system-ui", "Segoe UI",
Roboto, sans-serif; font-size: 13px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal; font-weight:
400; letter-spacing: normal; orphans: 2; text-align: start; text-indent:
0px; text-transform: none; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; white-space: normal; background-color:
rgb(24, 24, 24); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color: initial;"><li
style="unicode-bidi: plaintext;">Lightened<span> </span><code
style="font-family: monospace; color: rgb(208, 208, 208);
background-color: rgb(60, 60, 60); padding: 2px 4px; border-radius: 3px;
word-break: break-word; font-size:
0.9em;">--color-panel-bar-bg</code><span> </span>(grey-200 → grey-100)
for a softer chart-header band; the grey-300 divider still carries the
header/body edge.</li></ul><h3 style="unicode-bidi: plaintext; color:
rgb(204, 204, 204); font-family: -apple-system, "system-ui",
"Segoe UI", Roboto, sans-serif; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px;
text-transform: none; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; white-space: normal; background-color:
rgb(24, 24, 24); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color: initial;">
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
|
||
|
ktx-akshay
|
33c6e9e0e1 |
test(dashboards): cover CTE/subquery match_all queries through the panel search path (#13838)
## What First test coverage for the dashboard **panel** endpoints and the **dashboard search path**, running CTE/subquery `match_all` queries as real dashboard panels. Nothing in the repo previously touched either: `grep -rn "/panels\|search_type=dashboards" tests/api-testing/` returned zero hits. ## Why Panels query `_search?type=logs&search_type=dashboards&dashboard_id=…` — a different route into the engine than the Logs page (`search_type=ui`). The `match_all` + FST projection rewrite fixed in #13808 runs inside it, so a query that passes via Logs can still regress on a dashboard. The rewrite has to preserve the projection of the block the predicate sits in. When that block is the whole statement, the projection is obvious. When it's a CTE body feeding a JOIN, a window, or a four-branch UNION ALL, it has to preserve the *CTE's* output schema while the outer query reads different columns from it — which is where it went wrong. ## Coverage **71 distinct queries across 29 combination families**, all with `match_all` inside a `WITH …` body or a derived table: | Axis | Variants | |---|---| | Nesting depth | 1, 2, 3, and 5-level CTE chains | | Nesting kind | CTEs and derived tables (and one query using both) | | UNION ALL | 2, 3, and 4 separate `match_all` branches | | JOIN | CTE⋈CTE, derived⋈derived, cross-stream | | Window | `ROW_NUMBER`, `RANK`, `DENSE_RANK`, partitioned aggregates | | Negation | `NOT match_all`, and mixed with positive `match_all` | | Combined with | `COUNT(DISTINCT)`, `CASE`, `HAVING`, `histogram()` | Each case creates a real panel, runs it via the dashboard search path, then asserts row count against the DuckDB oracle and that no expected column was dropped from the projection. A further 30 cases run the same SQL through **both** Logs and dashboard and diff the results — that's what catches a rewrite firing on only one path. Queries are reused from the existing query-agent corpus rather than duplicated, so the two suites can't drift: query-agent verifies the values via the Logs path, this verifies the same SQL survives the dashboard path. Cases are **discovered by classifying the SQL**, not from a hand-written ID list, so new combinations added to the corpus are picked up automatically. Test ids name the shape (`Q1022-cte1_union_cdist_4ma`), so a failure identifies the combination without opening the corpus. Plus 14 tests for the panel endpoints themselves — AddPanel / UpdatePanel / DeletePanel, contract + CRUD lifecycle + 7 error paths derived from `DashboardError` in `src/core/src/http.rs`. ## Changes | File | | |---|---| | `tests/dashboards/test_dashboard_cte_panels.py` | new — 103 tests | | `tests/dashboards/test_dashboard_panels.py` | new — 14 tests | | `tests/dashboards/conftest.py` | new — shared fixtures | | `support/panel_queries.py` | new — discovery + panel helpers | | `support/endpoints/dashboards.py` | +78, additive | | `support/endpoints/search.py` | +36, additive | | `support/factories.py` | +73, additive | The three shared-file edits are **187 added lines, 0 modified** — nothing existing can break. ## Results 168 tests in tests/api-testing/tests/dashboards/ (117 new + 51 pre-existing) 166 passed, 1 skipped, 1 xfailed # full nested-query corpus present 74 passed, 2 skipped, 1 xfailed # current corpus on main Repeated 2× and 4× respectively, no flakes. Discovery scales with the corpus: on main it finds 8 nested `match_all` cases, and 71 once the larger sentinel set is present. Cases that aren't in the corpus skip with an explanatory message instead of failing, so this is safe to merge in either order. |
||
 Julián ValdésandHengfei Yang
|
06c6b515fb |
fix(stats): only read file_list_dump_stats when dumping is enabled (#13830)
Fixes #13829. ## Problem `file_list_dump_stats` is created **conditionally** — `handle_partitioned_tables` only adds it when `ZO_COMPACT_FILE_LIST_DUMP_ENABLED` is set, and that flag defaults to `false`. But `update_stats_from_file_list_for_stream` reads it **unconditionally**. On a PostgreSQL metastore where the table was never created: ``` [STATS] run update stream stats from file list error: SqlxError# error returned from database: relation "file_list_dump_stats" does not exist ``` **The failure is wider than the dump.** The `?` propagates before `set_stream_stats`, so the entire stats update is aborted — every stream reports `doc_num: 0` while ingestion, search and compaction keep working. It reads as a UI problem rather than a failing job, which is what makes it expensive to track down. SQLite creates the table unconditionally, so this only surfaces on deployments with an external PostgreSQL metastore. ## Change Gate the read on the same flag that gates the creation. One call site — `query_dump_stats_by_date_range` has no other caller. When dumping is off there is nothing to merge, so the stats are unchanged: `merge` over an empty result was already a no-op. The difference is that the surrounding update now completes instead of aborting. ## Alternative considered Always creating the table would also fix it, at the cost of an empty partitioned table on every deployment. Gating the read seemed closer to the existing intent — the table is deliberately optional — but I am happy to flip it if you prefer the schema to be unconditional. ## Testing Reproduced on v0.92.1 with `ZO_META_STORE=postgres` and the flag at its default: stats fail every cycle and all counters sit at zero. Creating the table by hand with the DDL from `create_partitioned_dump_stats_table` makes the error and the zeros go away, which confirms the missing table is the whole cause. I have not been able to build the project locally to run the test suite — no Rust toolchain on the machine I reproduced this on. The change is a single conditional and the import follows the one already used in `src/compaction/src/merge.rs` (`use config::{get_config, ...}`), but it deserves a CI run before merging. Signed-off-by: Julián Valdés <julianvaldes24@gmail.com> Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com> |
||
 Neha PandClaude Opus 4.7
|
8a4f7ea23b |
test(query-agent): add 70 match_all sentinel queries for FST rewrite path coverage (#13824)
## Summary Adds Q1003–Q1072 (70 queries) to `tests/test-data/query-agent/queries/full_text_search.json`. All exercise the `match_all` + FST projection rewrite path fixed in #13808, under diverse plan shapes. Structural patterns mirror real production customer queries (fan-out UNION ALL, multi-CTE, CSE-triggering scalar wrappers on the FTS field) but use **only test-fixture fields and values** — no customer identifiers. ## Shape coverage | Family | Range | Count | What it exercises | |---|---|---|---| | A. Subquery + `match_all` + GROUP BY | Q1003–Q1012 | 10 | Simple inline-view aggregations | | B. CTE with fan-out UNION ALL, `match_all` per branch | Q1013–Q1027 | 15 | Dominant customer pattern (multi-branch UNION + per-branch FTS + LIKE/regex + outer aggregate) | | C. CTE + scalar wrapper on `log` (trim/upper/split_part/regexp_replace/substring/concat/length) + `match_all` | Q1028–Q1037 | 10 | **CSE-trigger sentinels** — the wrappers cause DataFusion CSE to insert the intermediate `ProjectionExec` that #13808 fixes | | D. Deep multi-CTE + `match_all` + regex/LIKE/CASE combos | Q1038–Q1047 | 10 | Deep plan trees, many projection candidates | | E. Edge cases (DISTINCT, HAVING, CASE, IN-list) | Q1048–Q1052 | 5 | Corner combinations | | F. `match_all` inside JOINs (self-join, LEFT JOIN, cross-stream) | Q1053–Q1058 | 6 | JOIN builders inject their own projections | | G. `match_all` + window functions (ROW_NUMBER, RANK, DENSE_RANK, SUM/COUNT OVER) | Q1059–Q1063 | 5 | Window planning inserts projections above filter | | H. Negated `match_all` (NOT, AND NOT, UNION of pos/neg) | Q1064–Q1066 | 3 | Rewriter's negation code path | | I. Multiple `match_all` in same query (AND, OR, nested) | Q1067–Q1069 | 3 | Two FST fields injected simultaneously | | J. `histogram` + `match_all` time-series | Q1070–Q1072 | 3 | Common customer pattern; histogram optimizer + FST rewriter together | ## Verification - All 70 queries have DuckDB oracle results auto-populated via `compute_counts.py`. - Two initial cross-stream JOIN queries (Q1055, Q1058) were rewritten to disambiguate the FTS field between streams — now they scope `match_all` inside a per-stream CTE before joining. - Audit for customer-token leakage (pharma, tomcat, o2_test, usgm, storenumber, tagname, cash, credit_card, etc.) returns **zero hits** across Q1003–Q1072. ## Implementation notes - Added generator script [tests/test-data/query-agent/gen_q1003_q1052.py](tests/test-data/query-agent/gen_q1003_q1052.py) alongside existing `gen_q201_q400.py` / `gen_q511_q556.py` convention. Deterministic regeneration. - Wide JSON churn across the other 12 category files is `compute_counts.py` re-baselining expected results against a fresh `BASE_TS` (`base_ts_override.json` is untracked); tuple-order shuffles are harmless under `results_mode: rowsort`. - Coverage is most meaningful under the `remove_filter_with_index_false` CI leg (added in #13810), where `match_all` actually reaches `RewriteMatchPhysical`. Under the default flag (`true`), the index optimizer removes `match_all` before the rewriter runs, masking the bug path. ## Test plan - [ ] `api_query_agent / single_node_opt_true` — green - [ ] `api_query_agent / single_node_opt_false` — green - [ ] `api_query_agent / remove_filter_with_index_false` — green (this is the leg that actually validates the rewriter under all 70 shapes) - [ ] Q1003–Q1072 all pass on both memtable and parquet phases 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |