mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-05 08:09:22 +00:00
ab7bdd5e3127a4d805fa1b89beb8a2666ea9aecf
904
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Hengfei Yang
|
9c82beeeed |
fix: OTLP/JSON metrics silently dropping histograms and int values (#12615)
Close #12345 ## Problem OTLP histogram metrics sent over HTTP/JSON (e.g. otel-collector with `encoding: json`) created streams but stored 0 events, with no `_count`/`_sum`/`_bucket` streams. Int-valued counters/gauges were silently recorded as 0. The protobuf path was unaffected, which is why this only surfaced for collectors explicitly configured with JSON encoding. ## Root cause `otlp_json` deserializes the body directly into opentelemetry-proto's serde-derived types, which only partially implement the proto3 JSON mapping (64-bit integers encoded as strings, enums by name, zero/empty fields omitted). `Metric.data` is a flattened `Option` oneof, and serde converts any inner deserialization error of a flattened `Option` into `None` instead of failing — so the whole metric silently became `data: None`, while the stream itself had already been created by the schema setup. Same mechanism on `NumberDataPoint.value` turned string `asInt` values into 0. ## Changes - Upgrade opentelemetry stack 0.31 → 0.32, which fixes string-encoded u64 handling for plain histograms and summaries upstream. - Add `otlp_json_compat` normalizer for the gaps still present in 0.32 (each verified empirically, with unit tests): - string `asInt` values on data points and exemplars - `aggregationTemporality` as enum name - `Exemplar`'s value oneof is not flattened in 0.32 - missing `serde(default)` on `ExponentialHistogramDataPoint`/`Buckets`/`ValueAtQuantile`/`Exemplar`, which rejected omitted zero/empty fields - Count and log metrics whose data oneof failed to decode (`partial_success.rejected_data_points`) instead of silently skipping them, and stop overwriting specific partial-success error messages with the generic retention message. - Adapt to OTLP 0.32 API additions (`KeyValue.key_strindex`, `AnyValue::StringValueStrindex`). - Upgrade workspace reqwest 0.12 → 0.13, required because opentelemetry-http 0.32 only implements `HttpClient` for reqwest 0.13 (`rustls` feature replaces `rustls-tls-native-roots` via the platform verifier; `json`/`query`/`form` are now explicit features). The `segment` crate is pinned to reqwest 0.12, so `TELEMETRY_CLIENT` moved into `telemetry.rs` as a plain `reqwest::Client` that posts the Track message itself, replicating `segment::HttpClient::send`. - Remove the unused pyroscope profiling integration (optional feature, dependencies, agent setup, `ZO_PROF_PYROSCOPE_*` config). ## Verification - 10 unit tests in `otlp_json_compat` covering the full payload matrix (collector-style string integers, enum names, omitted zero fields, exemplars with and without trace context, already-normalized payloads, malformed input). - End-to-end: posted the issue's collector-style JSON payload to `/api/default/v1/metrics` — `_bucket` (3 events, correct cumulative counts with exemplars attached), `_count`, `_sum` populated for histogram and exponential histogram, summary quantiles stored, `asInt` counter stored as 42 instead of 0. - `cargo clippy -- -D warnings` clean; lib test failures are pre-existing local-environment issues that reproduce identically on main. |
||
  Bhargavandgithub-actions[bot]
|
957d7714a0 |
fix: incident page status filter added (#12613)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
|
Bhargavandgithub-actions[bot]
|
5d39efa572 |
fix: introduced empty state design for home, logs, traces. rum (#12610)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
 ktx-vaidehi
|
715695b58c | fix: ux revamp pipeline and dashboard issue (#12598) | ||
 
|
3e89667b13 |
test: add edition-features documentation validation spec (#12573)
## Summary - Adds a single GeneralTests spec (`edition-features.spec.js`) that validates the header edition button, the `EnterpriseUpgradeDialog`, every enterprise feature card, and the `o2.ws/*` documentation links each card opens. - Two tests live in the spec; exactly one runs per environment. The active edition is detected from the rendered header button label (most direct signal of what the user sees, tolerates frontend/backend build-flag mismatches like an OSS frontend served by an ENT backend). - Pairs with the same-named branch on `openobserve/o2-enterprise` so ENT CI tree-merges this spec in and runs the ENT variant. ## What the spec validates - Header button label per edition (`Get OpenObserve Enterprise (Free)` vs `Edition: Enterprise`) - Hero panel (title, badge, primary button, features title, Learn More button) - 20 enterprise feature cards (presence + external-link icon + BETA/HA badges) - Cloud-only `Bring Your Own Bucket` card stays hidden outside Cloud - Every feature card click opens a popup that lands on the expected `openobserve.ai` doc page - Hero external link buttons (`Download Now`/`Learn More` on OSS, `Learn More` on ENT) — `Get Free License` intentionally not clicked because it switches the active org to the meta org ## Files - `tests/ui-testing/playwright-tests/GeneralTests/edition-features.spec.js` (new) - `tests/ui-testing/pages/generalPages/editionFeaturesPage.js` (new — page object + truth table of slugs / expected doc paths) - `tests/ui-testing/pages/page-manager.js` (register `pm.editionFeaturesPage`) ## Test plan - [x] OSS local run: 1 passed, 1 skipped (~2.0m) - [x] ENT local run: 1 passed, 1 skipped (~2.0m) - [x] Sentinel audit passed (POM compliance, no raw selectors in spec, no hardcoded secrets, no console.log) - [ ] OSS CI green on this PR - [ ] ENT CI green on companion PR 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
  
|
5c56d05703 |
fix: layout ux3 (#12560)
Co-authored-by: Kirtan Thakkar <kirtan@kiara.tech> Co-authored-by: ktx-mihir <mihir.patel@kiara.tech> Co-authored-by: ktx-riya <riya.kukadiya@kiara.tech> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
 Omkar Kesarkhaneandgithub-actions[bot]
|
96f6ea0d70 |
fix: updated time buffer for RUM queries in traces and updated RUM config in datasources (#12555)
## What changed
- Extracted parser-limitation guards into `isParserLimitation()` and
refactored `useSearchQuery.ts` to **not block the search** when the PEG
parser flags a false-positive error — the squiggle still shows but the
query proceeds to the backend
- Added a 4th parser-limitation guard for `found="("` inside window
function clauses (covers `SUM(COUNT(*)) OVER (...)`, `NTILE(N) OVER
(PARTITION BY COALESCE(...)...)`)
- Unified all four `fetchRumEventsForTrace` queries to a consistent
±60,000,000 µs buffer around the trace window
- Added `allowedTracingUrls` config + `propagatorTypes` to the RUM SDK
config snippet in `FrontendRumConfig.vue`
- Added an "Explain" query dropdown menu item in the logs SearchBar
- Fixed various traces/RUM UI padding and border issues
(PlayerEventsSidebar, TraceDetails, TracesMetricsDashboard,
SessionViewer)
- Updated i18n keys across all 12 language files
## Why
Two valid SQL queries produced false-positive validation errors
(`SUM(COUNT(*)) OVER (...)`, `NTILE(...) OVER (PARTITION BY
COALESCE(...)...)`), and even after suppressing those, the query was
still **blocked from executing** by `useSearchQuery.ts`. The PEG parser
has known gaps that the DataFusion backend handles correctly — there was
no reason to block valid queries. The RUM event queries also had
inconsistent time buffers across different calls (some used ±10s, others
±60s), causing missed RUM events in trace correlation.
## Approach
- Extracted the 4 parser-limitation guards from `validateSql()` into a
pure sync `isParserLimitation()` function exported from
`sqlDiagnostics.ts`, then reused it in `useSearchQuery.ts` to let query
execution continue when only parser limitations are detected
- Replaced inline constants with a shared `RUM_TIME_BUFFER_US =
60_000_000` in `useRumSpanBuilder.ts` so all four
`fetchRumEventsForTrace` queries use the same ±60s window
- Added `{ timeout: 30000 }` to the aggregate mutation test that was
timing out on ~5,500 broken queries
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
|
||
 ktx-akshay
|
b62f1e41f0 |
test: fix failing dashboard E2E test cases (#12556)
…ease timeout for network idle state |
||
|
Shrinath RaoandClaude Sonnet 4.6
|
0ff45001d3 |
test: fix search-across-folders toggle selector (-btn suffix) (#12551)
## Summary - **Root cause**: `alertManagement.js` used `alert-list-search-across-folders-toggle-btn` but `AlertList.vue` renders a plain `<button>` with `data-test="alert-list-search-across-folders-toggle"` — no `-btn` suffix. - Fixed the same stale `-btn` selector in `dashboardPage.js` (dashboard toggle is also a plain `<button>`). ## Failing tests fixed - `alerts-regression.spec.js:128` — Bug #9967: PromQL alert creation - `alerts-stream-switching-regression.spec.js:338` — Bug #11577: Alert preview chart y-axis Both were timing out at 45s trying to click the nonexistent `*-toggle-btn` selector. ## Files changed - `tests/ui-testing/pages/alertsPages/alertManagement.js` — 2 selector fixes (lines 214, 481) - `tests/ui-testing/pages/dashboardPages/dashboardPage.js` — 1 selector fix + removed stale OSwitch comment ## Test plan - [ ] Run `npx playwright test playwright-tests/RegressionSet/alerts-regression.spec.js` locally - [ ] Run `npx playwright test playwright-tests/RegressionSet/alerts-stream-switching-regression.spec.js` locally - [ ] Verify CI Alerts-Regression job passes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
 Neha PandClaude Opus 4.7
|
1484f920a9 |
test: expand query agent suite to 570 queries with production SQL patterns (#12534)
## Summary - Added **46 new queries** (Q511-Q556) to the query agent regression suite, bringing the total from 524 to **570 unique queries** - Mined from real production SQL patterns (june7.sql) and adapted to use existing test data fields - Fixed **15 pre-existing duplicate IDs** (Q226-Q240 existed in both aggregation.json and basic_select.json — renumbered to Q557-Q571) ## New query distribution | Category | New | Description | |---|---|---| | string_functions | +10 | spath() + str_match_ignore_case, split_part() + chr(), multi-COALESCE chains, CAST AS VARCHAR | | histogram | +7 | 30-second/min intervals, COUNT FILTER, SUM(CASE WHEN), NULL breakdowns | | date_time | +6 | arrow_cast(), date_trunc(), to_timestamp_micros(), timestamp arithmetic, LAG for gaps | | combined | +6 | count(distinct) in CASE WHEN, re_match + re_not_match combo, FILTER in aggregation, HAVING percentages | | full_text_search | +5 | re_match + re_not_match, regexp_like, match_all(), str_match_ignore_case with COALESCE | | cte_subquery | +4 | CTE with spath, CTE with COALESCE fallbacks, correlated subquery | | aggregation | +2 | IS NOT NULL exploration patterns, multi-column GROUP BY | | window | +2 | NTILE with COALESCE PARTITION, LAG with delta calculation | | basic_select | +2 | Complex COALESCE + CAST chains, NULLS LAST ordering | | cross_stream | +1 | LEFT JOIN on dual keys across streams | | union | +1 | UNION ALL with different metric types | ## New OO-specific functions covered Four functions previously had **zero test coverage**: - `chr()` — 1 query (string splitting by character code) - `arrow_cast()` — 1 query (DataFusion timestamp arithmetic) - `date_trunc()` — 1 query (minute-level timestamp bucketing) - `to_timestamp_micros()` — 1 query (microsecond-to-timestamp conversion) Plus expanded coverage of rare functions: `spath()` (+3), `split_part()` (+3). ## Test modes - **38/46** new queries have full sqllogictest comparison (DuckDB computed expected results) - **8/46** use legacy assertions for OO-specific functions that DuckDB cannot replicate (spath, arrow_cast, date_trunc, to_timestamp_micros) - Overall suite: **533** sqllogictest, **37** legacy ## Changes | File | Change | |---|---| | `data_gen.py` | `NUM_QUERIES` 520 → 575 | | `gen_q511_q556.py` | New: query generation script | | 13 category JSON files | Updated with new queries + dedup fix | ## Verification - `compute_counts.py` ran successfully on all 570 queries - 0 duplicate IDs across all 13 category files - Only gap: Q025 (pre-existing) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Shrinath RaoandClaude Sonnet 4.6
|
3f63fd4d2d |
test: strip identical retries from TestDino upload to reduce quota usage (#12539)
## Summary - TestDino quota (~75k/month) was being consumed at ~4x rate because each failing test uploads one result entry per retry attempt (original + 3 retries = 4 entries per test) - This PR adds deduplication: if all retry results share the same `status` and error first-line, only the final attempt is uploaded - Retries with differing status or error message are preserved in full — they carry real diagnostic signal (e.g. a test that times out then fails with a different error) ## How it works ``` getErrorFingerprint(result) → first line of error.message (stable across retries) allIdentical = every retry has same status AND same error fingerprint → true: upload only results[last] (strips ~3 entries per test) → false: upload all results as before (keeps diagnostic data) ``` ## Expected quota impact Based on last month's data: ~65k attempts → ~16k (estimated ~75% reduction for OSS Portal alone) ## Test plan - [ ] Trigger a CI run that has failures with retries — confirm the workflow log shows `stripped N identical retry results` - [ ] Trigger a run with a genuinely flaky/environment-variable failure — confirm all retries are preserved in TestDino 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
|
4ca7a04864 |
test: model pricing E2E tests (13 journey flows) + Build tab timing fix (#12525)
## Summary - **13 Playwright E2E journey tests** for the Model Pricing feature (\`/settings/model_pricing\`), consolidating all P0–P2 scenarios into journey-based flows - New page object \`ModelPricingPage\` under \`pages/generalPages/\` - API cleanup integrated into the centralized \`APICleanup\` class and called from \`cleanup.spec.js\` - CI matrix updated: \`ZO_MODEL_PRICING_ENABLED=true\` env var added; \`model-pricing.spec.js\` added to the \`GeneralTests\` job - **Bug fix**: \`waitForBuildTabLoaded\` now polls the Monaco editor for non-empty content (Phase 3), fixing a timing race where Build tab SQL tests read an empty query string on slower CI machines ## Test coverage | Journey | Scenarios covered | |---------|------------------| | List, tabs & search | Controls, tab switching, search, built-in tab | | Create & validation | All form validation cases + successful save | | Edit, cancel & delete | Full lifecycle | | Duplicate & clone | Org model duplicate + built-in clone | | Bulk actions | Select, export JSON, bulk delete | | Toggle | Disable/re-enable with correct visual state | | Test match dialog | Empty state, no-result, clear, match result | | Import | Valid JSON, correction flow, cancel | | Editor features | Multi-tier, pricing template, pattern examples | | Pricing drawer | Overflow link → drawer with all price keys | | Empty state | CTA when org has no custom models | | Cost applied | LLM span matched → cost stored correctly | | No matching model | Unknown model name → zero cost stored | ## Bug fix: Build tab query editor timing \`waitForBuildTabLoaded\`'s Phase 2 (chart-type-selected signal) fires as soon as \`initializeFromQuery\` sets the chart type — before \`makeAutoSQLQuery\` has run. On slower CI machines this left \`queries[0].query\` empty when tests read the editor, causing the "SQL ON + Auto mode" and "Auto → Custom" tests to receive \`""\` instead of generated SQL. Phase 3 polls the Monaco editor (\`[data-test="logs-search-bar-query-editor"]\`) for non-empty content, confirming \`onBuildQueryGenerated\` has been called and the query is ready. ## Test plan - [x] All 13 model pricing tests pass locally - [x] \`logsQueryBuilder-editor.spec.js\` (23 tests) — all pass with Phase 3 fix applied - [x] Sentinel code audit passed (no blocking issues) - [x] API cleanup added to \`cleanup.spec.js\` via \`apiCleanup.cleanupModelPricingModels()\` — 404-safe for instances where the feature is disabled 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
  
|
44d8e56f0f |
fix: Fix/layout exp1 (#12521)
Co-authored-by: Kirtan Thakkar <kirtan@kiara.tech> Co-authored-by: ktx-riya <riya.kukadiya@kiara.tech> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: ktx-vaidehi <134508096+ktx-vaidehi@users.noreply.github.com> Co-authored-by: Subhra264 <chakrabortysubhradeep556@gmail.com> Co-authored-by: ktx-mihir <mihir.patel@kiara.tech> Co-authored-by: ktx-vaidehi <vaidehi.akhani@kiara.tech> Co-authored-by: Ashish Kolhe <ashish.j.kolhe@gmail.com> |
||
 Abhay Padamani
|
03dfc88bfb |
feat: default fields for add Panel and metrics page (#12509)
Design at: dashboards/default-panel-fields |
||
  Sai Nikhilandudev
|
57e57a56c9 |
fix: import and export functionality for score and score configs in llm evals (#12517)
Co-authored-by: udev <uddhav@openobserve.ai> |
||
|
Neha PandClaude Sonnet 4.6
|
146d5e7612 |
feat(tests): vortex file format test suite (scenarios 1–83) (#12472)
## Summary - Adds **239 new tests** covering the vortex columnar file format across 78 scenarios from the test plan - **`tests/api-testing/tests/vortex/`** — 9 test modules (83 tests): format conversion, schema evolution, data integrity, FTS, downsampling, API format awareness, server config, and performance/stress - **`tests/api-testing/tests/query_agent/test_3_vortex.py`** — 156 tests: the full 155-query SQL correctness suite + scenario 39 (self-join) run post-flush against vortex files - **`api-testing.yml`** — adds `--ignore=tests/vortex` to the OSS default-set run; vortex tests require the enterprise binary with `ZO_FILE_FORMAT=vortex` and are gated to enterprise CI only ## Test plan coverage | Module | Scenarios | |--------|-----------| | `test_format_conversion.py` | 1–13, 7, 8, 9, 10 | | `test_schema_evolution.py` | 14–25 | | `test_data_integrity.py` | 55–62 | | `test_fts.py` | 51–54 | | `test_downsampling.py` | 47–50 | | `test_api_format.py` | 71–73 | | `test_config.py` | 66–67 | | `test_performance.py` | 74–78 | | `test_3_vortex.py` | 26–46 (155 queries + JOIN) | ## Enterprise CI Two dedicated jobs added in the companion `openobserve/o2-enterprise` PR on the same branch: - **`api_query_agent_vortex`** — runs `test_3_vortex.py` with `ZO_FILE_FORMAT=vortex` - **`api_vortex_format_tests`** — runs the full `tests/vortex/` suite with 4-phase server lifecycle (parquet→vortex→parquet→vortex+compaction) OSS CI is unchanged except for the ignore rule — no vortex tests run against the OSS binary. ## Test checklist - [ ] `tests/vortex/` collects cleanly: `83 tests collected` - [ ] `tests/query_agent/test_3_vortex.py` collects cleanly: `156 tests collected` - [ ] Enterprise CI `api_query_agent_vortex` job passes - [ ] Enterprise CI `api_vortex_format_tests` job passes 🤖 Generated with [Claude Code](https://claude.ai/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Abhay Padamani
|
8c23dd000e |
feat: multi-query support (#12374)
Design At: https://github.com/openobserve/designs/tree/main/dashboards/multi-sql-query-support Issue list : https://github.com/openobserve/openobserve/issues/11278 |
||
|
Shrinath RaoandClaude Sonnet 4.6
|
a6e833bb99 |
test: fix isQueryEditorExpanded selector and enableHistogram state detection (#12502)
## Summary
- **Streams-Regression** — `isQueryEditorExpanded()` used
`container.locator('.editor-fullscreen')` which searches *descendants*,
but Vue adds `editor-fullscreen` directly to the container element
itself. Fixed by using combined CSS selector
`.query-editor-container.editor-fullscreen`.
- **Logs-Regression** — `enableHistogram()` checked canvas visibility to
detect histogram state. Canvas is absent before the first query runs, so
it always assumed histogram was OFF and toggled it OFF when it was
already ON. This removed `[data-test="logs-search-result-bar-chart"]`
from the DOM (it is `v-if`-gated), breaking
`expectBarChartHasContent()`. Fixed by using
`logsQueryPage.isHistogramOn()` (reads button `data-state` attribute),
consistent with how `ensureHistogramState()` already works.
## Failing tests fixed
- `should allow query expand icon to work after navigating from stream
explorer (#9337)` — Streams-Regression
- `should not show blank histogram after cancelling query @bug-4315` —
Logs-Regression
- `collapsing/expanding index list should redraw histogram chart` —
Logs-Regression
## Test plan
- [ ] Ran `streams-regression.spec.js --grep "9337"` locally — passes
- [ ] Ran `logs-regression-bugs.spec.js --grep "blank histogram after
cancelling"` locally — passes
- [ ] Ran `logs-regression-bugs-2.spec.js --grep "collapsing/expanding
index list"` locally — passes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
|
||
|
Neha PandClaude Opus 4.7
|
008398163b |
test: add more queries to query regression (#12487)
## Summary - Adds **100 new structural-pattern queries** (Q411-Q510) across all 13 categories, bringing the suite from 425 to **525 queries** - Adds **cross-stream JOIN/UNION support**: `stream_offset` in data_gen.py, `STREAM2` in conftest, dual-table DuckDB oracle - 15 cross-stream queries (Q401-Q415) covering INNER/LEFT/RIGHT JOIN, EXISTS/NOT EXISTS, UNION ALL across streams - Bumps data_gen.py from 510 to 520 query capacity ## Changes - **Infrastructure**: [conftest.py](tests/api-testing/tests/query_agent/conftest.py), [compute_counts.py](tests/test-data/query-agent/compute_counts.py), [data_gen.py](tests/test-data/query-agent/data_gen.py) - **New file**: [cross_stream.json](tests/test-data/query-agent/queries/cross_stream.json) — 15 queries - **12 categories**: 85 queries across aggregation, basic_select, combined, cte_subquery, date_time, full_text_search, histogram, math_functions, pagination, string_functions, union, window ## Test plan - [x] `compute_counts.py` generates expected results for all 525 queries - [ ] Run `pytest tests/query_agent/ -v` against target environment (565 tests expected) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Shrinath RaoandClaude Sonnet 4.6
|
af7d0e6696 |
test: harden rerun check step against GitHub API failures (#12486)
## Summary
- Adds `|| echo '{"jobs":[]}'` fallback on the `curl` call so a network
error or GitHub API rate-limit response never produces an empty/invalid
variable
- Adds `2>/dev/null || echo "0"` on both `jq` calls so a malformed API
response (non-JSON error body) falls back to `"0"` instead of
propagating a non-zero exit under `-eo pipefail`
- Changes the arithmetic comparison to `[ "${CURRENT_ATTEMPT_JOBS:-0}"
-eq "${TOTAL_MATRIX_JOBS:-0}" ]` with `2>/dev/null` so a `null` jq
output can't trigger "integer expression expected" and kill the step
**Failure mode fixed:** On re-runs, multiple matrix jobs query the same
GitHub API endpoint concurrently with the shared `GITHUB_TOKEN`. A
rate-limit `403` or transient network hiccup returns non-JSON, `jq`
exits non-zero, and with `set -eo pipefail` the whole "Check if this is
a rerun" step fails — blocking the job before any tests run.
**Degraded behaviour:** When the API is unavailable, both counts fall to
`0`, `0 -eq 0` → `rerun_type=all` → step succeeds → all tests run (same
as a normal rerun-all). The TestDino optimisation is skipped but nothing
is blocked.
Companion PR: openobserve/o2-enterprise#<ENT_PR>
## Test plan
- [ ] Trigger a re-run of a workflow run — "Check if this is a rerun"
step should pass even if GitHub API is slow/rate-limited
- [ ] Verify `rerun_type` is still correctly set to `failed` when the
API responds normally
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
|
||
|
Neha PandClaude Opus 4.7
|
0334dda9a3 |
test: add data variety and NULL stress to query agent data generation (#12485)
## Summary Two high-value improvements to the query agent regression test data: 1. **Data variety** — Per-field rotation offsets (`qi*13 + i*7`) replace the old uniform `idx % len(pool)` approach. Previously all fields picked from the same pool position, creating only 5 fixed row shapes per query. Now field-value combinations vary independently per field. 2. **NULL stress** — 8 fields (`bot_flag`, `info_tag`, `upstream_error_code`, `page_url`, `endpoint_path`, `resource_path`, `variant_tag`, `threat_flag`) get `None` for 40% of records per query. This exercises `IS NULL` / `IS NOT NULL` / `COALESCE` paths that were previously untested. ## Changes - `data_gen.py`: Rewrote `make_record()` with per-field rotation and NULL injection - 24 queries: Adjusted WHERE/HAVING filters to match new data distribution - All 12 query JSONs: Regenerated expected results via `compute_counts.py` ## Impact - 400 queries all have results (18 pre-existing legacy/skip queries unchanged) - No change to record count (still 5/query, 2000 total) — test time unaffected (~7 min) ## Test plan - [x] `compute_counts.py` regenerates all expected results - [ ] Run `pytest tests/query_agent/ -v --tb=short` against dev environment 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
|
61aebff552 |
fix: refactor!: remove SQL mode toggle in favor of auto-detection (#12483)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Shrinath Rao <shnath@openobserve.ai> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Neha PandClaude Opus 4.7
|
5eb63395f1 |
test: broaden coverage with structural query templates (#12482)
## Summary - Adds 200 new SQL queries (Q201-Q400) across all 12 categories, extracted as structural templates from production `june6.sql` query patterns - All queries use the existing synthetic `FIELD_POOL` schema — zero real stream/field names - Bumps `data_gen.py` from 200 → 400 queries (4000 records total, 10 per query) - Introduces new structural patterns previously missing from the suite ## New patterns - **CONCAT + COALESCE chains**: Multi-field string construction for display messages - **CASE WHEN inside histogram COUNT**: Platform mapping + conditional aggregation - **Multi-field LIKE / NOT LIKE OR chains**: Text search across 4-8 fields - **NOT IN + IS NOT NULL**: Exclusion filters with null safety - **Long IN lists**: 5+ values in WHERE clauses - **Percentage calculations**: `100.0 * COUNT(CASE WHEN ...) / COUNT(*)` patterns - **approx_percentile_cont in subqueries**: P95/P99 in nested SELECTs - **Histogram with breakdown**: `GROUP BY _time, breakdown` pattern - **char_length CASE fallbacks**: `CASE WHEN char_length(f) > 0 THEN f ELSE fallback END` - **CTE with multiple subquery references**: Self-join and multi-CTE patterns ## Distribution | Category | New | Range | |----------|-----|-------| | aggregation | 25 | Q201-Q225 | | basic_select | 25 | Q226-Q250 | | combined | 20 | Q251-Q270 | | cte_subquery | 25 | Q271-Q295 | | date_time | 10 | Q296-Q305 | | full_text_search | 20 | Q306-Q325 | | histogram | 25 | Q326-Q350 | | math_functions | 8 | Q351-Q358 | | pagination | 7 | Q359-Q365 | | string_functions | 15 | Q366-Q380 | | union | 12 | Q381-Q392 | | window | 8 | Q393-Q400 | ## Test plan - [x] `compute_counts.py` generates DuckDB expected results for all 400 queries - [ ] CI: 800 tests (400 × 2 phases) pass 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Neha PandClaude Opus 4.7
|
4d48bbb21b |
test: expand query agent regression suite (#12479)
## Summary - Expands the SQL regression catalog from **155 to 200 queries** across all 12 categories - Adds **7 queries** extracted from abc.sql (Q156-Q162) — synthesized from real-world OpenObserve SQL patterns with all data/fields/streams generated from scratch - Adds **230 auto-generated queries** (Q163-Q392) from templates to cover query format diversity - Both test phases pass: **200 memtable + 200 parquet = 400 total** ## Key changes - **data_gen.py**: 43 new fields (record_type, site_name, latency_ms, payload_size, alert_tags, phase_data, db_info, etc.), generates 3920 deterministic records - **compute_counts.py**: new translations for regexp_like, str_match_ignore_case, approx_distinct, json_get_str/json_get_json chaining; empty-aggregate result detection (OO returns 0 rows vs DuckDB's 1 NULL row) - **conftest.py**: PID-based stream name suffix for cross-invocation isolation (prevents data accumulation) ## Test plan - [x] test_1_memtable.py — 392 passed - [x] test_2_parquet.py — 392 passed - [x] All 7 abc.sql queries pass (Q156-Q162) - [x] No real stream names, field names, or data copied from external sources 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
|
0aab1ae261 |
feat: online evaluations (#12352)
- **feat(online-eval): add PipelineKind::Evaluation and dual realtime pipeline support** - **feat(online-eval): add provider, score_config, scorer, and online_eval_jobs DB entities** - **feat(online-eval): add config CRUD services and API handlers** - **feat(online-eval): reconcile jobs into hidden evaluation pipelines** - **feat(online-eval): add scorer preparation and execution backends** - **feat(online-eval): write scores and evaluator traces to system streams** - **refactor(online-eval): remove legacy eval templates backend** - **test(online-eval): add API integration coverage for eval resources** --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> |
||
|
Neha PandClaude Sonnet 4.6
|
73ce75c883 |
fix(tests): bump query_agent ingest timeout 120s → 300s (#12444)
## Summary - ENT CI runners are slower than local/OSS runners — the `ingest_query_agent_data` fixture was hitting the 120s timeout before 1550 records became searchable, causing all 310 query_agent tests to fail at setup - Bumped `wait_until` timeout to 300s ## Test plan - [ ] OSS `api_query_agent_tests` passes as before - [ ] ENT `api_query_agent_tests` no longer hits WaitTimeout on ingest 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
ktx-mihir
|
004e320b6d |
fix: ux revamp add condition layout and crosslink combobox (#12443)
- Add condition layout fix - drilldown close button alignment and size sm - OCombobox item align center - Crosslink combobox and input based on condition - Actions timezone alignment - Ai toogle expand shortcut enhance |
||
|
|
aa81dc3f13 |
fix: OTable resize and add/remove column feature (#12433)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Shrinath Rao <shnath@openobserve.ai> |
||
|
Neha PandClaude Opus 4.7
|
1196045636 |
test: add org-level ingestion token API and Playwright tests (#12439)
PR #11691 introduces org-level ingestion tokens with o2oi_ prefix (37 chars). This commit adds comprehensive automated test coverage: ## API Tests (35 tests in test_ingestion_tokens.py) ### Token CRUD (9 tests) - Create token returns o2oi_ prefix and 37-char length - Create token persists name, created_by, enabled=true, is_default=false - Create token with name only (no description) — description is empty - Create token with hyphens and underscores allowed - Duplicate name returns 400 with "already exists" message - Empty name returns 400 with "cannot be empty" message - Whitespace-only name returns 400 with "cannot be empty" message - Name too long (>256 chars) returns 400 with non-empty error message - Parametrized: special chars (spaces, @, ., /) rejected with "alphanumeric" message ### List Tokens (3 tests) - GET returns data array - Created token appears in list - Token values are unmasked in list response ### Enable/Disable (5 tests) - Disable token via PATCH {enabled: false} - Enable disabled token via PATCH {enabled: true} - Double disable is a no-op (idempotent) - Toggle nonexistent token returns 404 - DELETE returns 405 (not supported) ### Ingestion Auth — Org Token (6 tests) - Ingest with org token (org_id:o2oi_token) succeeds - Ingest with disabled org token fails (401) - Ingest with nonexistent o2oi_ token fails (401) - Ingest to wrong org with valid token fails (401) - Org token blocked on search endpoint (401) - Org token blocked on dashboards endpoint (401) ### Ingestion Auth — Backward Compat (6 tests) - User password ingestion still works (pre-PR behavior) - Non-o2oi_ service account token ingestion works (falls through to user lookup) - User password on search succeeds (200) — mirrors org token blocking tests - User password on dashboards succeeds (200) — mirrors org token blocking tests - Non-o2oi_ SA token NOT blocked by o2oi_ prefix check on search (≠401) - Non-o2oi_ SA token NOT blocked by o2oi_ prefix check on dashboards (≠401) ### Multi-token and Security (6 tests) - Create 5 tokens, all visible in list - Disable one token, others remain enabled - SQL injection in token name rejected (special chars → 400) - Empty name, whitespace name, too-long name (with message assertions) - Special chars parametrized: spaces, @, ., / ## Playwright UI Tests (7 tests in ingestionTokens.spec.js) - Ingestion Tokens tab renders with table and columns (P0) - Create token via UI dialog with name and description (P0) - Create token with empty name — button disabled, enabled on fill, disabled on clear - Revealed token dialog — copy token with clipboard verification - Toggle token enable/disable from table actions (P0) - Manage Tokens button on ingestion page navigates to IAM - Ingestion page token selector dropdown shows available tokens ## Page Object (ingestionTokensPage.js) - data-test selectors for IAM nav, token list, create dialog, toggle, toast, revealed dialog - Registered in centralized PageManager --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
ktx-vaidehiandktx-mihir
|
2fcd670aac |
fix: ux revamp drawer to dialog for less fields (#12424)
Co-authored-by: ktx-mihir <mihir.patel@kiara.tech> |
||
|
Shrinath Rao
|
a647e6aee9 |
test: add retry_on_db_lock helper to support/wait.py (#12434)
## Summary
- Adds `retry_on_db_lock()` to `tests/api-testing/support/wait.py`
- Companion to the enterprise-side fix (openobserve/o2-enterprise#TBD)
that applies the helper across all RBAC entity base tests
## Root cause
The `ent_rbac` CI shard runs with `-n 2` (two pytest-xdist workers).
SQLite's single-writer model means parallel workers racing on write
operations produce transient `500 {"message": "database is locked"}`
responses (SQLite error codes 5 / 517). These surface as flaky assertion
failures across many PRs.
## Fix
`retry_on_db_lock(fn)` wraps an HTTP call and retries up to 3 times with
exponential backoff (0.5 → 1 → 2 s) only when the response is `500` with
`"database is locked"` in the body. Parallelism is preserved; the brief
retry window absorbs the time the other writer needs to commit and
release the lock.
## Test plan
- [ ] OSS CI passes with this branch
- [ ] Enterprise PR with the same branch name picks up the new utility
and applies it in all RBAC entity base tests
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
Neha PandClaude Opus 4.7
|
eb7cce9b96 |
test: query agent SQL regression suite — 155 queries, sqllogictest-style, two-phase (#12384)
## Summary
155-query SQL regression test suite covering 12 categories. Adding a new
query requires zero test code changes — drop a JSON entry into the right
category file and run `compute_counts.py`.
**Sqllogictest-style comparison**: DuckDB pre-computes full expected
result sets (not just row counts). The test runner extracts actual
values from OO hits, sorts both sides, and compares cell-by-cell with 5%
relative float tolerance. This catches value-level regressions that
row-count-only checks miss.
Two-phase execution: **memtable** (pre-flush) then **parquet**
(post-flush + Tantivy). **310/310 pass**.
## Architecture
```
tests/
├── api-testing/tests/query_agent/
│ ├── conftest.py # Shared fixtures, run_query(), tolerance comparison
│ ├── test_1_memtable.py # Phase 1: all queries pre-flush (FTS skipped)
│ └── test_2_parquet.py # Phase 2: post-flush (module-scoped fixture triggers flush)
└── test-data/query-agent/
├── data_gen.py # Deterministic data generator (single source of truth)
├── compute_counts.py # DuckDB oracle: computes expected result sets
├── base_ts_override.json # Locks BASE_TS so compute and tests share same timestamps
└── queries/ # 12 JSON files, 155 queries with expected results
```
### How it works
1. **Deterministic data** — `data_gen.py` generates 10 records per query
(1550 total) with a warehouse/supply-chain IoT schema. Each query Q{N}
gets isolated timestamps. `BASE_TS` is locked via
`base_ts_override.json` so compute and tests share identical timestamps
(prevents histogram bucket drift on minute boundaries).
2. **Unique stream per session** — `STREAM =
f"query_agent_test_{int(time.time())}"` prevents accumulated-data bleed
from prior runs.
3. **DuckDB oracle** — `compute_counts.py` loads the same dataset into
DuckDB, creates a pre-filtered VIEW per query (so time filters work for
CTEs, subqueries, UNIONs — no fragile SQL injection), translates
OO-specific functions, and writes `expected.results` blocks into the
query JSONs.
4. **Histogram translation** — `histogram(_timestamp, 'N unit')` is
translated to integer arithmetic matching OO's `date_bin` with origin at
2001-01-01T00:00:00 UTC. Verified against DataFusion CLI — 100/100
records match across all interval types (15s, 30s, 1m, 2m, 3m, 4m, 5m).
ISO timestamp post-processing converts epoch-µs bucket values to
`YYYY-MM-DDTHH:MM:SS` strings matching OO's serialization format.
5. **Sqllogictest comparison** — `run_query()` checks for
`expected.results`. When present, it extracts column values from hits as
strings, sorts both actual and expected (`results_mode: "rowsort"`), and
compares cell-by-cell with 5% relative float tolerance (handles
cross-engine floating-point drift in AVG/STDDEV/etc.).
6. **Two-phase testing** — `test_1_memtable.py` runs all queries
pre-flush (FTS sqllogictest skipped — Tantivy indexes don't exist yet).
`test_2_parquet.py` has a module-scoped `post_flush` fixture that
triggers `/node/flush`, re-polls until data is searchable, then runs all
queries with full assertions.
### Comparison modes
| Mode | Count | Mechanism |
|------|-------|-----------|
| **Sqllogictest** | 140 | Cell-by-cell comparison with 5% float
tolerance |
| **Skip sqllogictest** | 9 | DuckDB results for documentation, fall
back to legacy assertions |
| **Legacy** | 6 | 5 auto-interval histograms + Q122 (STRING_AGG
DISTINCT ORDER BY) |
**Skip sqllogictest queries** — known OO-DuckDB divergence:
| Query | Reason |
|-------|--------|
| Q037 | ROW_NUMBER tie-breaking differs between engines |
| Q058 | Self-join pairings non-deterministic across engines |
| Q072 | FULL OUTER JOIN histogram — NULL handling differs |
| Q080 | FTS `match_all` + histogram — Tantivy vs SQL LIKE |
| Q111 | ROW_NUMBER ties in pagination |
| Q117 | STRING_AGG ordering differs between engines |
| Q121 | STRING_AGG concatenation order |
| Q127 | UNION across 5 overlapping groups — boundary records compete
for LIMIT spots |
| Q152 | PERCENT_RANK tie-breaking differs between engines |
### Key design decisions
| Decision | Why |
|----------|-----|
| Full result-set comparison | Catches value-level regressions (e.g.,
ROUND producing wrong decimal) |
| `rowsort` mode | DuckDB and OO order ties differently — sorting both
sides eliminates spurious failures |
| 5% float tolerance | Cross-engine AVG/STDDEV/PERCENTILE differ at
least-significant digits |
| `''` = `'None'` normalization | OO serializes NULL as empty string;
DuckDB uses `"None"` |
| DuckDB VIEW for time filtering | WHERE injection into CTEs/subqueries
is fragile — VIEW filters at scan level |
| Dynamic columns excluded | `_timestamp` and `_time` differ between
compute time and test time |
| OO-dropped LAG columns excluded | `prev_tput`, `hours_2_back`,
`delta_from_prev`, `temp_diff` |
| Histogram → DuckDB integer math | DuckDB lacks `date_bin`; integer
formula verified via DataFusion CLI |
| BASE_TS override file | Prevents histogram timestamp drift when
compute and test run at different minutes |
| 1s time window margins | Minimizes cross-query data bleed while
capturing all 10 records |
| `pytest.fail()` on fixture failure | Stops all tests instead of
running 310 against an empty stream |
### Categories covered
| Category | Queries | Description |
|---|---|---|
| aggregation | Q001-Q020 | GROUP BY, COUNT, AVG, SUM, MIN, MAX, FILTER,
HAVING |
| basic_select | Q021-Q035 | WHERE, IN, BETWEEN, LIKE, ORDER BY,
DISTINCT, CASE |
| combined | Q036-Q045 | JOINs, multi-condition filters, combined
aggregations |
| cte_subquery | Q046-Q062 | CTEs, subqueries, derived tables, EXISTS |
| date_time | Q063-Q073 | Date parsing, formatting, extraction, interval
arithmetic, histogram |
| full_text_search | Q074-Q082 | match_all, match against fields |
| histogram | Q083-Q093 | histogram() bucketing, aggregation over
buckets |
| math_functions | Q094-Q102 | ROUND, SQRT, ABS, POWER, arithmetic
expressions |
| pagination | Q103-Q112 | LIMIT, OFFSET, combination with filters and
sorts |
| string_functions | Q113-Q122 | SUBSTR, CONCAT, UPPER, LOWER, TRIM,
LIKE patterns |
| union | Q123-Q134 | UNION ALL, UNION (dedup), CTE + UNION combinations
|
| window | Q135-Q155 | ROW_NUMBER, RANK, DENSE_RANK, LAG, LEAD, SUM/AVG
OVER |
## Test plan
- [x] All 155 queries pass in both phases (310/310)
- [x] 140 queries with sqllogictest cell-by-cell comparison
- [x] 17 histogram queries converted from legacy to sqllogictest
- [x] 9 queries with skip_sqllogictest for known OO-DuckDB divergence
- [x] 6 queries in legacy mode (auto-interval histograms + Q122)
- [x] BASE_TS locked via override file — no timestamp drift
- [x] Dedicated CI job (`api_query_agent_tests`) runs in parallel
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
Hengfei YangandClaude Opus 4.8
|
ad9fef2242 |
fix: fuzzy tests (#12405)
- [x] Bulk ingestion panic - [x] Create function panic - [x] Dashboards/Alerts/Incidents panic when list data - [x] Match http response error code - [x] Skip some endpoint that fuzz can't generate data --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Shrinath RaoandClaude Sonnet 4.6
|
b6093cb996 |
test: repair two failing dashboard E2E tests (#12395)
## Summary - **camelCase zero-value line chart** (`dashboard.spec.js:826`): SQL referenced `kubernetes_namespace_name` / `kubernetes_container_name` / `kubernetes_pod_id` with underscores, but `e2e_automate` stores those fields with dot notation (`kubernetes.namespace_name`). DataFusion returned "column not found", `errorDetail.message` was set, `ChartRenderer` was unmounted (behind a `v-if`), and the test timed out waiting for `[data-test="chart-renderer"]`. Fixed by switching to `level` / `method` / `stream` — fields guaranteed to exist — with impossible conditions so all series values are zero. - **Sankey no-data test** (`dashboard-sankey.spec.js:400`): After `waitForChartToRender()` there is an async gap between the apply button re-enabling and Vue committing the "No Data" text to the DOM. The old `waitForFunction` used `offsetParent !== null`, which fires immediately for the empty zero-height no-data div (`position: absolute`, no content → 0px height). `isVisible()` then returned `false`, producing `Expected true, Received false`. Fixed by replacing the custom logic with `expect.poll()` that reads `textContent` — retrying until the text is actually committed to the DOM. ## Test plan - [x] Both tests verified passing locally against a hosted enterprise OO instance (localhost:5080) - [ ] CI run on `o2-enterprise` PR to confirm `Dashboards-Core` and `Dashboards-Charts` jobs go green --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Shrinath RaoandClaude Opus 4.7
|
b0118090e1 |
test: api-testing follow-ups — pytest-xdist dep + standardize test user display name (#12393)
## Summary
Companion PR to o2-enterprise PR 1841 (api-testing follow-ups). All
ENT-side test/CI work lands there; this OSS PR carries the two changes
that must live on the OSS side because they touch shared code.
Both PRs live on the same-named branch (`test/api-test-followups`) so
ENT CI's cp-merge picks up this branch and sees both changes together.
## What's in this PR
### 1 — Add `pytest-xdist` dependency
Adds `pytest-xdist>=3.0` to `tests/api-testing/pyproject.toml`. Needed
by the ENT api-testing CI's `api_integration / ent_rbac` shard to run
RBAC tests in parallel (`-n 4`).
CI-verified speedup: **~70 s → 21.7 s (~69% reduction)** on the ENT RBAC
shard's 280+ tests, with zero failures.
Foundation only — no test code in this repo uses xdist yet. Available
for any future shard that wants it.
### 2 — Standardize automated-test user display name to "Automation
TestUser"
Two helper files in `tests/api-testing/helpers/workflow/pages/` had a
personal name (`"Shyam P"` / `"Shyam Panjiyar"`) hardcoded in their user
/ service-account create payloads. Those helpers are consumed by the
workflow chain (`helpers/workflow/create_objects.py` +
`edit_objects.py`), which doesn't reliably clean up its created users —
so the names persisted in the IAM users page and looked confusingly like
real teammates' accounts.
Files changed:
- `tests/api-testing/helpers/workflow/pages/user_page.py`
- `tests/api-testing/helpers/workflow/pages/serviceaccount_page.py`
Both now use:
```
first_name: "Automation"
last_name: "TestUser"
```
Convention going forward: any account named `Automation TestUser` in the
IAM users page is automatically identifiable as test-created and safe to
clean up. Email addresses still use random suffixes
(`d4m21_{random_int}`) so uniqueness isn't affected — only the display
name standardizes.
## Test plan
- [ ] CI green on this branch
- [ ] Companion ENT PR 1841 (which depends on the `pytest-xdist` dep
added here) stays green
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
ktx-akshay
|
47b3944774 |
test: restore missing visualize e2e test cases (#12347)
## Test Inspector Review ### Summary - 0 Critical issues - 0 Selector mismatches (14/14 validated against Vue source) - 0 CI/Workflow issues - 11/11 tests passed (localhost) --- ### ✅ Framework Compliance — PASS | Check | Status | |-------|--------| | Raw selectors in spec | 0 critical (1 acceptable: dynamic panel name) | | console.log | None | | Vacuous pass / soft assertions | None | | testLogger | ✅ 14 calls | | Tags | ✅ All 11 tests | | waitForTimeout | 0 | ### ✅ Selector Cross-Validation — PASS 14/14 page object selectors verified against Vue source. 0 mismatches. ### ✅ CI/Workflow — PASS All spec files present in OSS `.github/workflows/playwright.yml`. ### ✅ Test Execution — 11/11 PASSED (localhost, 1.2m) *Reviewed by Test Inspector* |
||
|
ktx-akshay
|
e890b83d2c | test: fix flaky dashboard end-to-end tests (#12375) | ||
|
Shrinath RaoandClaude Opus 4.7
|
61aa772d57 |
test: revamp api-testing suite (framework + 16 file rewrites + CI matrix) (#12372)
## Summary
Comprehensive rework of the `tests/api-testing/` suite. 30+ commits
implementing all 6 phases of the [revamp
plan](../../tests/ui-testing/MD_Files/api%20tests%20revamp/API_TESTS_REVAMP.md).
**Before → After:**
| | Baseline | Final | Δ |
|---|---|---|---|
| passed | 421 | **443** | **+22 ✓** |
| failed | 3 | **2** | -1 (only same pre-existing) |
| skipped | 53 | 39 | -14 (unskipped dead tests now run) |
| xfailed | 1 | 2 | +1 (new visible backend-bug signal) |
| runtime | 588s | **387s** | **-201s (34% faster)** |
**Zero tests that passed in baseline are now failing.** The 2 remaining
failures are the exact pre-existing ones
(`test_enrichment_table_join.py` backend panic + `test_workflow.py` SMTP
env-miss).
## What's in the diff
- **New `support/` framework** — `OpenObserveClient`, endpoint wrappers
(`alerts`, `dashboards`, `streams`, `search`, etc.), `wait_until()`,
payload factories, pytest fixtures, SSE parser. 14 framework selfcheck
tests.
- **16 of 16 OLD-tier test files rewritten** — using the new framework,
unique-suffix names, body validation, `wait_until` instead of
`time.sleep`, parametrize where appropriate.
- **29 files moved into 12 feature folders** mirroring the UI suite
layout
(`tests/{alerts,dashboards,ingest,orgs,pipelines,regression,rum,search,streams,users,workflow}/`).
- **−6,500 LOC net** (copy-paste + dead code removal; new framework adds
back ~1,500).
- **Ruff config** in pyproject.toml: T201 ban on `print()`, PT pytest
rules, B bugbear, UP pyupgrade.
## CI workflow changes
- New job `api_regression_tests`: `strategy.matrix` over the 4 env-flag
combinations (UTF8_VIEW × JOIN_MATCH_ONE) — now runs in parallel instead
of as 4 sequential server reboots in the main job.
- New job `api_fuzz_tests`: schemathesis OpenAPI-driven fuzz against
`/api-doc/openapi.json` (214 endpoints tested). Informational
(`continue-on-error: true`). Each run surfaces **7–10 distinct 5xx
panics** (e.g., empty `sql` body returning 500 instead of 400, traces
endpoints crashing on unescaped path params that get interpolated into
SQL). Filed to backend team for separate triage.
- Server `o2_*.log` files uploaded as artifacts on failure (was:
silently lost).
- `api_tests_summary` now requires BOTH integration + regression to
pass.
### CI infrastructure fixes
A short hardening pass landed after the initial CI run surfaced several
issues:
- **pytest-sugar count inflation** — dropped `--force-sugar` from both
pytest invocations. The flag double-registered sugar on top of the
auto-loaded entry point, inflating the CI test counter ~3x (display
showed 1364 when collection was 486). Counter is now honest.
- **schemathesis 4.x flag migration** — `--base-url` → `-u`,
`--hypothesis-max-examples` → `-n`, `--hypothesis-deadline` →
`--request-timeout`, `--data-generation-method` → `-m`, dropped
`--auth-type basic`. The fuzz job was failing immediately on flag-parse
before this.
- **Regression matrix binary path** — used `${{ github.workspace
}}/target/debug/openobserve` so the absolute path resolves correctly
under `working-directory: tests/api-testing/` (was: every matrix job
failed with "No such file or directory").
- **`permissions: contents: read`** added to all 4 workflow jobs —
closes CodeQL alert #369.
- **Removed redundant server boot** in `api_integration_tests` (leftover
from main's structure after the regression matrix split; was silently
failing to bind to :5080 and getting masked by the first server).
## Backend bugs surfaced (for backend team to track)
The rework uncovered and visibly flagged these as `xfail`/documented
test comments:
1. `DELETE /api/organizations/{id}` returns 404 unconditionally even for
just-created orgs — endpoint appears unimplemented.
2. `test_dashboard_missing_required_fields` triggers backend panic
(returns 500 instead of 400/422).
3. JS `getEnrichmentTableData()` restricted to `_meta` org AND not
exposed in the `/functions/test` endpoint runtime — `ReferenceError`
even with explicit `enrichment_tables: [...]`.
4. Bulk API silently sanitizes invalid `_index` values (`****` → `_`)
without flagging — user data lands in unexpected index.
5. `AVG` of a single value drifts by ~1e-14 (likely float reduction
order; documented tolerance in `test_advanced_sql.py`).
6. Org name validation rejects hyphens (only
`A-Z/a-z/0-9/spaces/underscores`) — not in spec.
Plus 7–10 additional 5xx panics surfaced per fuzz run (input validation
gaps — separate triage list in fuzz job logs).
## Test bugs uncovered + fixed
- `test_pipeline.py`: had two function definitions named
`test_pipeline_creation_and_action_with_schedule` — Python's last-wins
meant the first test's body never executed for years.
- `test_alertsV2.py`: `assert alert_disabled == "False"` was indented
inside an `else:` branch that never executed because the response is
always a dict; AND `enabled` is a bool, not the string `"False"`. Two
layers of broken.
- `test_bulk.py`: variable typo (`valid_bulk_payload` reused on the POST
call) made the invalid-index test silently duplicate the valid case.
- `test_organisations.py`: `test_create_organization` used pre-pytest-4
`yield`-style fixtures — silently ignored by pytest 8.
- `test_js_result_array.py`: 8 tests skipped with `"Temporarily disabled
- failing test"` — the actual cause was that JS functions are restricted
to `_meta` org. One-line fix unskipped all 8.
- `test_fuzzy.py`: 4 tests `@pytest.mark.skip`-ed because they used a
1-minute time window against older static test data → 0 hits → flake →
skip. Wider window unskips them.
- 547 LOC of zombie commented-out test blocks deleted (15 commented `#
def test_` definitions, including one orphan case where the `def` was
commented but the indented body wasn't).
## Folder layout
```
tests/api-testing/
├── support/ # NEW framework
│ ├── client.py # OpenObserveClient
│ ├── endpoints/ # 7 per-resource wrappers
│ ├── factories.py # unique_name, time_window, *_payload
│ ├── wait.py # wait_until + WaitTimeout
│ ├── sse.py # read_sse_response
│ └── fixtures.py # client, temp_user_email, temp_stream_name, ...
├── helpers/ # Non-test shared code (was scattered in tests/)
│ ├── sourcemap_helpers.py
│ └── workflow/ # workflow chain + pages/ live here now
├── fixtures/
│ └── rum/ # Standalone RUM ingest scripts (was in tests/)
└── tests/
├── alerts/ # 3 files
├── dashboards/ # 4 files
├── ingest/ # 4 files
├── orgs/ # 2 files
├── pipelines/ # 10 files
├── regression/ # 2 files (env-flag matrix targets)
├── rum/ # 3 files
├── search/ # 8 files
├── streams/ # 1 file
├── users/ # 1 file
├── workflow/ # 3 files
├── conftest.py
└── test_framework_selfcheck.py
```
## Test plan
- [x] `rye run pytest --collect-only` — 486 tests discovered
- [x] Full default-set run (`--ignore=tests/regression`) — 443 pass / 2
fail / 39 skip / 2 xfail in 387s
- [x] `rye run pytest -m framework` — 14/14 selfcheck tests pass
- [x] `rye run ruff check support/ tests/test_framework_selfcheck.py` —
clean
- [x] CI run on this PR — `api_integration_tests` + all 4
`api_regression_tests` matrix jobs green
- [x] CI run — `api_fuzz_tests` runs cleanly (informational; surfaces
7–10 unique 5xx panics per run)
## Note on test count
The `test_streaming.py` rewrite folded 7 "tests" (`test_ingest_data`,
`test_ingest_join`, `test_disable_streaming`,
`test_update_max_query_range`, `test_enable_streaming`,
`test_delete_stream`, `test_delete_stream_join`) into pytest fixtures.
Each only checked `status_code == 200` of a setup/teardown operation —
they were imperative steps masquerading as tests. The fixtures still
perform the same operations and assert their success; they're just no
longer counted as discrete pytest items. **Functional coverage is
identical.**
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Subhra264
|
9a1ed606aa |
feat: org level ingestion token (#11691)
Design at: observability/org-level-ingestion-token |
||
|
Shrinath RaoandClaude Sonnet 4.6
|
859f5be016 |
test: fix clickScheduledSearchRow selector for logs-regression-ent tests (#12371)
## Summary
- `clickScheduledSearchRow` in `logsPage.js` was waiting for
`[data-test="search-scheduler-table-{traceId}-row"]` which never exists
in the DOM
- `OTableBodyRow` renders rows as `data-test="o2-table-row-{index}"`
(zero-based integer), not by trace_id
- Fixed to wait for `o2-table-row-0` (first row) and click it — the
table has `expand-on-row-click="true"` so no separate expand button
click is needed
- Timeout bumped from 10 s → 30 s to allow the API data fetch to
complete
## Failing tests fixed
- `logs-regression-ent.spec.js` › No Undefined Length error after
navigating Logs -> Scheduled Search -> Streams -> Logs (`@P0`)
- `logs-regression-ent.spec.js` › No console errors after navigating the
reproduction path (`@P1`)
## Test plan
- [x] Both tests run and pass locally against the enterprise binary (`2
passed (1.0m)`)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
|
||
|
Shrinath RaoandClaude Sonnet 4.6
|
3e172523d5 |
test: sync regression workflow runners and TestDino with playwright.yml (#12370)
## Summary - **Runners**: `merge_and_upload_reports` and `playwright_summary` now use `ubicloud-standard-4` (was `-8`), matching `playwright.yml` - **TestDino rerun optimization**: added `check_rerun` step + full `last-failed` logic (same as `playwright.yml`) — on re-run-failed, only the previously failed test in each shard is re-executed - **Upload step**: added failures-only filter (`node -e` filterSuite), `startTime` ISO 8601 conversion, and `--verbose`; now uploads `report-failures-only.json` instead of the full report ## Test plan - [ ] Trigger a regression run and confirm it completes with the new runner sizes - [ ] Intentionally fail a test, re-run failed jobs, and verify TestDino optimization runs only the failed test - [ ] Confirm Upload to TestDino step produces a filtered report 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
 yujinqiu
|
833b2bdc8e | feat(security): enforce strong password policy (#12327) | ||
|
Shrinath RaoandClaude Sonnet 4.6
|
3f3248fe64 |
fix(metrics): poll for invalid PromQL error indicator (enterprise CI fix) (#12368)
## Summary
- Fixes the `Invalid PromQL syntax error handling` test failing in
enterprise CI
- Enterprise CI renders the no-data/error state slower than the 3s fixed
timeout used previously
- Replaced `isVisible({ timeout: 3000})` with `expect.poll` (10s window,
500ms–2s intervals) so the assertion is real but tolerates CI rendering
lag
## Root cause
When `sum(rate(` is entered and applied, the UI renders a no-data
indicator asynchronously. Locally (enterprise) this appears in ~2s. In
enterprise CI (slower hardware, more features enabled) it takes longer
than 3s, causing all three indicator checks to return `false` before
they appear.
## Test plan
- [x] Passes locally: `hasNoData=true` confirmed
- [ ] Enterprise CI Metrics shard goes green
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
|
||
|
Shrinath RaoandClaude Sonnet 4.6
|
0929fa5678 |
test: Sentinel audit fixes + parallelize Metrics E2E shard (~22 min → ~5-7 min) (#12367)
## Summary Two independent improvements to the Playwright E2E test suite. --- ### Part 1 — Sentinel audit fixes (original PR scope) Fixes from the Sentinel audit review of PR #12358 (selector drift stabilization). - **Dead code removal**: delete `clickJobID()` from `logsPageEP.js` (unreachable code containing `console.log`); remove unused `chromium` import - **Critical fix**: replace `console.log`/`console.error` in `decryptLogSQL()` with `testLogger` equivalents - **POM compliance**: replace raw `page.locator()` call in `alerts-ui-operations.spec.js` `beforeEach` with `pm.alertsPage.isIncidentsFeatureEnabled()`; add that method to `alertsPage.js` - **POM compliance**: move per-test `const pm = new PageManager(page)` into `beforeEach` in `service-graph.spec.js`; replace raw `page.waitForLoadState()` with `pm.serviceGraphPage.waitForGraphReload()`; add `waitForGraphReload()` to `serviceGraphPage.js` - **Inline selector cleanup**: move metrics tab inline string literals in `serviceGraphPage.js` into constructor properties - **Tagging**: add `@cipherKeys/@enterprise/@P1/@P2/@negative/@validation` to all 6 tests in `cipherKeys.spec.js` (were untagged); add `@P1/@P2` to all 5 tests in `regexPatternManagement.spec.js` - **ESM fix**: correct `export const path = require('path')` → `import path from 'path'` in `logoManagementPage.js` - **Deduplication**: consolidate `createFirstFunction`/`createSecondFunction` into shared `createFunction(name, vrlCode)` in `pipelinesEP.js` --- ### Part 2 — Parallelize Metrics E2E shard **Problem**: The `e2e / Metrics` CI shard was consistently the slowest job at ~22 minutes per PR run, while all other shards complete in ~5-7 minutes. **Root cause**: 9 of 11 Metrics spec files used `test.describe.configure({ mode: 'serial' })` because they shared a describe-scope `let pm` variable reassigned in `beforeEach`. This forced every test in those files onto a single Playwright worker, eliminating all parallelism despite `fullyParallel: true` and 5 workers being configured. **Fix applied to all 9 affected files** (`metrics.spec.js`, `metrics-config.spec.js`, `metrics-config-tabs.spec.js`, `metrics-visualizations.spec.js`, `metrics-promql-query-persistence.spec.js`, `metrics-table-column-order.spec.js`, `metrics-queries.spec.js`, `metrics-aggregations.spec.js`, `metrics-advanced.spec.js`): - Remove `test.describe.configure({ mode: 'serial' })` - Replace shared `let pm` + `beforeEach` pattern with a `setupTest(page, testInfo)` helper that creates and returns a fresh `PageManager` per test — the same pattern already used in `metrics-promql-builder.spec.js` (the one file that was already parallel) - `metrics-table-column-order.spec.js`: pass `pm` explicitly into `setupTableChart(pm, page, mode)` since it can no longer close over a describe-scope variable - Split consolidated mega-tests (for-loops over multiple independent scenarios) in `metrics-queries`, `metrics-aggregations`, and `metrics-advanced` into individual `test()` blocks so Playwright can distribute them across workers **No test assertions, selectors, query logic, or validation changed.** | File | Tests before | Tests after | |---|---|---| | `metrics.spec.js` | 10 (serial) | 10 (parallel) | | `metrics-config.spec.js` | 2 (serial) | 2 (parallel) | | `metrics-config-tabs.spec.js` | 3 (serial) | 3 (parallel) | | `metrics-visualizations.spec.js` | 4 (serial) | 4 (parallel) | | `metrics-promql-query-persistence.spec.js` | 6 (serial) | 6 (parallel) | | `metrics-table-column-order.spec.js` | 20 (serial) | 20 (parallel) | | `metrics-queries.spec.js` | 4 mega-tests (serial) | 12 individual (parallel) | | `metrics-aggregations.spec.js` | 3 mega-tests (serial) | 10 individual (parallel) | | `metrics-advanced.spec.js` | 4 mega-tests (serial) | 12 individual (parallel) | **Expected impact**: ~1300s (~22 min) → ~300-400s (~5-7 min) wall-clock time. --- ## Test plan - [ ] All 40 tests in the 4 Sentinel-audit spec files enumerate cleanly via `npx playwright test --list` - [ ] Tag filters `@cipherKeys`, `@P2`, `@sdr` match expected tests - [ ] All Metrics spec files enumerate cleanly with no serial mode warnings - [ ] Metrics shard CI time drops to ~5-7 min on next PR run 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
 dependabot[bot]andHengfei Yang
|
48ff43f776 |
chore(deps): bump qs from 6.15.1 to 6.15.2 in /tests/ui-testing (#12168)
Bumps [qs](https://github.com/ljharb/qs) from 6.15.1 to 6.15.2. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/ljharb/qs/blob/main/CHANGELOG.md">qs's changelog</a>.</em></p> <blockquote> <h2><strong>6.15.2</strong></h2> <ul> <li>[Fix] <code>stringify</code>: skip null/undefined entries in <code>arrayFormat: 'comma'</code> + <code>encodeValuesOnly</code> instead of crashing in <code>encoder</code></li> <li>[Fix] <code>stringify</code>: use configured <code>delimiter</code> after <code>charsetSentinel</code> (<a href="https://redirect.github.com/ljharb/qs/issues/555">#555</a>)</li> <li>[Fix] <code>stringify</code>: apply <code>formatter</code> to encoded key under <code>strictNullHandling</code> (<a href="https://redirect.github.com/ljharb/qs/issues/554">#554</a>)</li> <li>[Fix] <code>stringify</code>: skip null/undefined filter-array entries instead of crashing in <code>encoder</code> (<a href="https://redirect.github.com/ljharb/qs/issues/551">#551</a>)</li> <li>[Fix] <code>parse</code>: handle nested bracket groups and add regression tests (<a href="https://redirect.github.com/ljharb/qs/issues/530">#530</a>)</li> <li>[readme] fix grammar (<a href="https://redirect.github.com/ljharb/qs/issues/550">#550</a>)</li> <li>[Dev Deps] update <code>@ljharb/eslint-config</code></li> <li>[Tests] add regression tests for keys containing percent-encoded bracket text</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/ljharb/qs/commit/9aca4076fe788338c67cf7e115f0be6bc58d85a8"><code>9aca407</code></a> v6.15.2</li> <li><a href="https://github.com/ljharb/qs/commit/5e33d33447ed0bf1ddab9abc41d27dea4687d992"><code>5e33d33</code></a> [Dev Deps] update <code>@ljharb/eslint-config</code></li> <li><a href="https://github.com/ljharb/qs/commit/21f80b33e5c8b3f7eba1034fff0da4a4a37a1d41"><code>21f80b3</code></a> [Fix] <code>stringify</code>: skip null/undefined entries in <code>arrayFormat: 'comma'</code> + `e...</li> <li><a href="https://github.com/ljharb/qs/commit/a0a81ea2071acce3eff41a040f719ac8f5c4f64c"><code>a0a81ea</code></a> [Fix] <code>stringify</code>: use configured <code>delimiter</code> after <code>charsetSentinel</code></li> <li><a href="https://github.com/ljharb/qs/commit/e3062f78f5233b338ceeb8e8dfa5a07dea4b32a8"><code>e3062f7</code></a> [Fix] <code>stringify</code>: apply <code>formatter</code> to encoded key under <code>strictNullHandling</code></li> <li><a href="https://github.com/ljharb/qs/commit/0c180a40adb8c6703fffc85b2ff06ca209f5c1e0"><code>0c180a4</code></a> [Fix] <code>stringify</code>: skip null/undefined filter-array entries instead of crashi...</li> <li><a href="https://github.com/ljharb/qs/commit/3a8b94aec19bd664720f6f6b1e66c4a0dfe4b656"><code>3a8b94a</code></a> [Tests] add regression tests for keys containing percent-encoded bracket text</li> <li><a href="https://github.com/ljharb/qs/commit/96755abd357c0e534dd3442a84a04d08864bfe0d"><code>96755ab</code></a> [readme] fix grammar</li> <li><a href="https://github.com/ljharb/qs/commit/a419ce5bbfcdb98a299f1a0bb47ea055baef20e6"><code>a419ce5</code></a> [Fix] <code>parse</code>: handle nested bracket groups and add regression tests</li> <li>See full diff in <a href="https://github.com/ljharb/qs/compare/v6.15.1...v6.15.2">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/openobserve/openobserve/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com> |
||
|
|
a749599898 |
fix: update e2e selectors for O2 component migration (#12346)
Replace Quasar-based selectors with O2 component equivalents across
alerts, dashboard, logs, and streams page objects. Key changes:
- OSwitch: target -btn suffix on inner button
- OSelect: use -trigger/-popover/-option suffixes, fallback to
[role="option"][data-test-value] when parent has no data-test
- OCombobox: use placeholder-based locators where no data-test exists
- OInput: descendant input selectors, pagination text format handling
- OTable: custom bottom slot pagination format ("N Stream(s)")
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: ktx-akshay <akshay.patil@kiara.tech>
|
||
|
ktx-vaidehiandktx-riya
|
b7ffc10a5d |
fix: ux revamp logs stream single or multi select (#12350)
Co-authored-by: ktx-riya <riya.kukadiya@kiara.tech> |
||
|
Shrinath RaoandClaude Sonnet 4.6
|
cc1f09c5c1 |
test: fix selector drift across ENT specs, re-enable enterprise tests, and improve CI (#12358)
## Summary Broad test stabilization pass covering Quasar component selector drift in ENT UI specs, re-enabling previously disabled enterprise tests, and CI infrastructure fixes. ### UI test fixes (selector drift) Quasar's O2 component library wraps native elements — `OInput`, `OTextarea`, `ODropdown`, `OToast`, `OToggleGroup`, `OToggleGroupItem`, `ODrawer`, `ODialog`, `OFile` — in ways that break naive attribute/role selectors. This PR updates affected specs to use the correct locators. | Spec | Fixes | |------|-------| | `service-graph.spec.js` | `OToggleGroupItem` selector, `ODrawer` close btn, `beforeAll` timeout | | `alerts-ui-operations.spec.js` | Remove `waitForResponse` overhead from enterprise block | | `logoManagement.spec.js` | `OInput` / `OFile` / `ODialog` / `OToast` locators | | `jobSearch.spec.js` | `OInput` fill targets wrapper div, `OTable` index-based rows | | `searchPatterns.spec.js` | `OToggleGroup` same-element selector, pattern details dialog | | `pipelineImport.spec.js` | `OInput` / `ODropdown` / Monaco / `OToast` selector drift | | `regexPatternManagement.spec.js` | Correct test-data import file path | | `cipherKeys.spec.js` | `OInput`/`OTextarea` `-field` suffix, `OToast` selector, 403 skip guard, all-toasts polling | ### Enterprise tests re-enabled - Removed `/* ... */` block comment disabling `"Alerts & Incidents Page Navigation"` describe block (`@enterprise`) in `alerts-ui-operations.spec.js` - 3 tests restored: page navigation, element visibility, round-trip nav back to Alerts - Requires `O2_INCIDENTS_ENABLED=true` (set in ENT CI) ### CI improvements - **playwright.yml**: Install `ffmpeg` via local Playwright binary (required for `video: retain-on-failure` recording) - **api-testing.yml**: Reduce runner from `ubicloud-standard-8` → `ubicloud-standard-4` - **test_stream_settings_dedup.py**: Fix field-filtering logic and assertion count in `test_multiple_distinct_fields_no_duplicates` ## Paired ENT PR Full ENT-only test restoration (SDR, RUM, cipherKeys, etc.): https://github.com/openobserve/o2-enterprise/pull/1835 ## Test plan - [ ] ENT CI: `"Alerts & Incidents Page Navigation"` enterprise block passes - [ ] Playwright CI: video artifacts generated on failure (ffmpeg install works) - [ ] API tests: `test_multiple_distinct_fields_no_duplicates` passes 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
ktx-riyaandktx-vaidehi
|
2af558441a |
fix: theme switching in revamped (#12340)
Co-authored-by: ktx-vaidehi <vaidehi.akhani@kiara.tech> |
||
|
|
1afabd35ce |
feat: bloom filter pruning layer above tantivy (#12121)
## Summary Design at: https://github.com/openobserve/designs/blob/main/infrastructure/bloomfilter/DESIGN.md A bloom-filter pruning layer that sits **between `file_list` and tantivy** in the search path. It drops candidate files before any `.ttv` is opened, for **positive equality / IN** lookups on high-cardinality fields (`trace_id`, `request_id`, …). It is built **on top of** tantivy: the `.bf` is constructed by iterating the tantivy term dictionary, so a field must be a tantivy secondary index (`index_fields`) **and** opted into `bloom_filter_fields`. Only the intersection is built — a bloom-only field produces nothing. To make that intersection non-empty by construction, enabling a field for bloom now folds it into `index_fields` automatically (see Config). Values are not inspected; the operator chooses which fields are worth it (see guidance below). Design notes live in the module-level docs under `src/infra/src/bloom/` (`mod.rs`, `writer.rs`, `reader.rs`, `sbbf.rs`). ## Design - **Transposed (block-major) `.bf` layout.** All files in a `.bf` share `num_blocks = B`, so `block_index = fastmap(hash(value), B)` is identical across files and the blocks a query needs form **one contiguous row**. The search side reads **one `M × 32`-byte row per `.bf`** to answer membership for all `M` files — prune cost is `O(groups)`, not `O(files)`. (`infra::bloom`, own ~120-LOC SBBF over gxhash64.) - **Sub-grouping.** A hour's new files are sorted by record count and split into chunks of ≤ `ZO_BLOOM_FILTER_MAX_FILES_PER_BF` (default 256), one `.bf` each. Bounds compactor build memory (`M × B × 32`) and `.bf` object size; query reads become `ceil(files / 256)` rows. - **Footer-only reader + batched fetch.** Footer (≈ `M × 12` B) comes from `BLOOM_FOOTER_CACHE` or one 16 KB suffix GET (with a precise re-read when a large footer overflows the probe); rows are pulled with one `cache_storage::get_ranges` call. `cache_storage` was reworked to do real batched cache reads (memory: 1 lookup + N slices; disk: 1 `File::open` + N `pread`). - **B sizing.** `num_blocks_for(max_records_in_chunk, ZO_BLOOM_FILTER_FPP)` (default fpp `0.01`) — record count is a safe NDV upper bound, so the SBBF never saturates for high-cardinality fields. - **Versioned, with orphan cleanup.** A file's `bloom_ver` is write-once — a row is never re-stamped, so dumped rows keep pointing at the `.bf` they were built into. But `bloom_ver` is **not** append-only at the bucket level: a merge produces new output files with a fresh `bloom_ver` and deletes the small inputs, so the old inputs' `.bf` can become orphaned. After each merge round the compactor retires any `.bf` whose `bloom_ver` is no longer referenced by a live `file_list` row in the bucket (enqueued through the existing `file_list_deleted` path). `bloom_ver` is preserved through every file_list INSERT/SELECT path (incl. the SQLite read-cache) and the dump parquet. - **Observability.** Prometheus metrics for prune (`bloom_prune_keep_ratio`, `bloom_prune_duration_seconds`); `bloom-inspect` CLI dumps a `.bf` footer (fields, B, row bytes, file names); footer-cache hit/miss via the shared `BytesCache` tagged metrics; `Local::get_ranges` now reports `STORAGE_READ_*` (previously bypassed). ## When it helps (measured, S3, no disk cache) Same stream `benchtest`, 170 files / 14.64 GB index, single matching row. The only variable is whether the field's values are time-correlated: | Field | Type | tantivy only | bloom + tantivy | Outcome | |---|---|---|---|---| | `trace_id` (16-byte random) | random | **2584 ms** (opens all 170 `.ttv`) | 24 ms prune + 65 ms = **89 ms** | **~29× faster** | | `request_id` (`…-7e74-…`, UUIDv7) | time-ordered | **154 ms** (sparse index range-prunes) | 42 ms + 154 ms = **196 ms** | ~1.3× **slower** | The whole 170-file group was answered by **one row read of 5440 bytes** (`170 × 32`) + a cached footer. **Guidance — enable only when all three hold (advisory; not enforced):** 1. **High cardinality** field. 2. **Random values** (not time-ordered) — so tantivy can't range-prune and must scan every file. Time-ordered ids (UUIDv7, snowflake) are already pruned by tantivy's sparse index in memory → bloom is pure overhead there. 3. **Many files per hour** — hourly volume ≳ 10 × `ZO_COMPACT_MAX_FILE_SIZE` (≳10 `.ttv` files), so there's something to prune. Bloom turns N file opens into one row read; at 1–2 files/hour it's net overhead. The 29× result above had 170 files/hour all answered by one `.bf`. ❌ Low-cardinality / non-indexed / range / `LIKE` / regex / `!=` / OR: no effect. ## Config - `ZO_BLOOM_FILTER_ENABLED` (default true) — global gate. - Per stream: a field must be in **both** `index_fields` and `bloom_filter_fields`. Adding a field to `bloom_filter_fields` auto-adds it to `index_fields` (bloom is built on the secondary index), and the default index fields include the default bloom fields. - `ZO_BLOOM_FILTER_FPP` (default 0.01) — target false-positive probability; sizes the SBBF block count (smaller fpp = fewer false survivors, larger `.bf`). Out-of-range falls back to 0.01. - `ZO_BLOOM_FILTER_MAX_FILES_PER_BF` (default 256) — files per `.bf`; trades query reads vs build memory. - `ZO_BLOOM_FILTER_DEFAULT_FIELDS` (default empty) — fields bloomed for every stream. - `ZO_BLOOM_FOOTER_CACHE_MAX_SIZE` (default 1% mem, clamped 32–256 MB). Removed in this PR: `ZO_BLOOM_FILTER_DISABLED_ON_SEARCH`, `ZO_BLOOM_FILTER_NDV_RATIO` (B is now sized from record count + fpp, not an ndv ratio). ## Test plan - [x] `cargo build`, `cargo clippy -p infra -p openobserve --lib -- -D warnings`, `cargo fmt --check` clean - [x] Unit tests: `cargo test -p infra --lib bloom`, `cargo test -p openobserve --lib bloom` - [x] E2E on S3 (results above): random field prunes 170→1 and is ~29× faster; transposed read = 1 request / 5440 B per group - [x] Reviewer: confirm gating expectations (operator opts fields in; no auto value detection) - [x] Test: orphan bloom: 2 * 256MB -> merge to 512MB -> merge to 1024MB, the first .bf should be deleted - [x] Test: orphan bloom: 6 * 256MB -> merge to 1GB + 512MB -> 512 MB merge to 1024MB, the first .bf include 2 files(1GB + 512MB), after merge again, the first still have 2 files but one of them already deleted. we should keep the first .bf, because still have files in use. ## Notes - `.bf` format is `VERSION 0x01` (transposed). Not compatible with any pre-merge experimental `.bf`; rebuild by wiping the `bloom/` dir and resetting `bloom_ver=0`. - Hash is gxhash64 (project-wide `config::utils::hash`), not the Parquet spec's xxhash64 — `.bf` is internal, never read by external tools. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |