mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-04 16:09:05 +00:00
b0118090e142102aea3fc4aa9074fe7f7499b402
153
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Shrinath RaoandClaude Opus 4.7
|
b0118090e1 |
test: api-testing follow-ups — pytest-xdist dep + standardize test user display name (#12393)
## Summary
Companion PR to o2-enterprise PR 1841 (api-testing follow-ups). All
ENT-side test/CI work lands there; this OSS PR carries the two changes
that must live on the OSS side because they touch shared code.
Both PRs live on the same-named branch (`test/api-test-followups`) so
ENT CI's cp-merge picks up this branch and sees both changes together.
## What's in this PR
### 1 — Add `pytest-xdist` dependency
Adds `pytest-xdist>=3.0` to `tests/api-testing/pyproject.toml`. Needed
by the ENT api-testing CI's `api_integration / ent_rbac` shard to run
RBAC tests in parallel (`-n 4`).
CI-verified speedup: **~70 s → 21.7 s (~69% reduction)** on the ENT RBAC
shard's 280+ tests, with zero failures.
Foundation only — no test code in this repo uses xdist yet. Available
for any future shard that wants it.
### 2 — Standardize automated-test user display name to "Automation
TestUser"
Two helper files in `tests/api-testing/helpers/workflow/pages/` had a
personal name (`"Shyam P"` / `"Shyam Panjiyar"`) hardcoded in their user
/ service-account create payloads. Those helpers are consumed by the
workflow chain (`helpers/workflow/create_objects.py` +
`edit_objects.py`), which doesn't reliably clean up its created users —
so the names persisted in the IAM users page and looked confusingly like
real teammates' accounts.
Files changed:
- `tests/api-testing/helpers/workflow/pages/user_page.py`
- `tests/api-testing/helpers/workflow/pages/serviceaccount_page.py`
Both now use:
```
first_name: "Automation"
last_name: "TestUser"
```
Convention going forward: any account named `Automation TestUser` in the
IAM users page is automatically identifiable as test-created and safe to
clean up. Email addresses still use random suffixes
(`d4m21_{random_int}`) so uniqueness isn't affected — only the display
name standardizes.
## Test plan
- [ ] CI green on this branch
- [ ] Companion ENT PR 1841 (which depends on the `pytest-xdist` dep
added here) stays green
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Shrinath RaoandClaude Opus 4.7
|
61aa772d57 |
test: revamp api-testing suite (framework + 16 file rewrites + CI matrix) (#12372)
## Summary
Comprehensive rework of the `tests/api-testing/` suite. 30+ commits
implementing all 6 phases of the [revamp
plan](../../tests/ui-testing/MD_Files/api%20tests%20revamp/API_TESTS_REVAMP.md).
**Before → After:**
| | Baseline | Final | Δ |
|---|---|---|---|
| passed | 421 | **443** | **+22 ✓** |
| failed | 3 | **2** | -1 (only same pre-existing) |
| skipped | 53 | 39 | -14 (unskipped dead tests now run) |
| xfailed | 1 | 2 | +1 (new visible backend-bug signal) |
| runtime | 588s | **387s** | **-201s (34% faster)** |
**Zero tests that passed in baseline are now failing.** The 2 remaining
failures are the exact pre-existing ones
(`test_enrichment_table_join.py` backend panic + `test_workflow.py` SMTP
env-miss).
## What's in the diff
- **New `support/` framework** — `OpenObserveClient`, endpoint wrappers
(`alerts`, `dashboards`, `streams`, `search`, etc.), `wait_until()`,
payload factories, pytest fixtures, SSE parser. 14 framework selfcheck
tests.
- **16 of 16 OLD-tier test files rewritten** — using the new framework,
unique-suffix names, body validation, `wait_until` instead of
`time.sleep`, parametrize where appropriate.
- **29 files moved into 12 feature folders** mirroring the UI suite
layout
(`tests/{alerts,dashboards,ingest,orgs,pipelines,regression,rum,search,streams,users,workflow}/`).
- **−6,500 LOC net** (copy-paste + dead code removal; new framework adds
back ~1,500).
- **Ruff config** in pyproject.toml: T201 ban on `print()`, PT pytest
rules, B bugbear, UP pyupgrade.
## CI workflow changes
- New job `api_regression_tests`: `strategy.matrix` over the 4 env-flag
combinations (UTF8_VIEW × JOIN_MATCH_ONE) — now runs in parallel instead
of as 4 sequential server reboots in the main job.
- New job `api_fuzz_tests`: schemathesis OpenAPI-driven fuzz against
`/api-doc/openapi.json` (214 endpoints tested). Informational
(`continue-on-error: true`). Each run surfaces **7–10 distinct 5xx
panics** (e.g., empty `sql` body returning 500 instead of 400, traces
endpoints crashing on unescaped path params that get interpolated into
SQL). Filed to backend team for separate triage.
- Server `o2_*.log` files uploaded as artifacts on failure (was:
silently lost).
- `api_tests_summary` now requires BOTH integration + regression to
pass.
### CI infrastructure fixes
A short hardening pass landed after the initial CI run surfaced several
issues:
- **pytest-sugar count inflation** — dropped `--force-sugar` from both
pytest invocations. The flag double-registered sugar on top of the
auto-loaded entry point, inflating the CI test counter ~3x (display
showed 1364 when collection was 486). Counter is now honest.
- **schemathesis 4.x flag migration** — `--base-url` → `-u`,
`--hypothesis-max-examples` → `-n`, `--hypothesis-deadline` →
`--request-timeout`, `--data-generation-method` → `-m`, dropped
`--auth-type basic`. The fuzz job was failing immediately on flag-parse
before this.
- **Regression matrix binary path** — used `${{ github.workspace
}}/target/debug/openobserve` so the absolute path resolves correctly
under `working-directory: tests/api-testing/` (was: every matrix job
failed with "No such file or directory").
- **`permissions: contents: read`** added to all 4 workflow jobs —
closes CodeQL alert #369.
- **Removed redundant server boot** in `api_integration_tests` (leftover
from main's structure after the regression matrix split; was silently
failing to bind to :5080 and getting masked by the first server).
## Backend bugs surfaced (for backend team to track)
The rework uncovered and visibly flagged these as `xfail`/documented
test comments:
1. `DELETE /api/organizations/{id}` returns 404 unconditionally even for
just-created orgs — endpoint appears unimplemented.
2. `test_dashboard_missing_required_fields` triggers backend panic
(returns 500 instead of 400/422).
3. JS `getEnrichmentTableData()` restricted to `_meta` org AND not
exposed in the `/functions/test` endpoint runtime — `ReferenceError`
even with explicit `enrichment_tables: [...]`.
4. Bulk API silently sanitizes invalid `_index` values (`****` → `_`)
without flagging — user data lands in unexpected index.
5. `AVG` of a single value drifts by ~1e-14 (likely float reduction
order; documented tolerance in `test_advanced_sql.py`).
6. Org name validation rejects hyphens (only
`A-Z/a-z/0-9/spaces/underscores`) — not in spec.
Plus 7–10 additional 5xx panics surfaced per fuzz run (input validation
gaps — separate triage list in fuzz job logs).
## Test bugs uncovered + fixed
- `test_pipeline.py`: had two function definitions named
`test_pipeline_creation_and_action_with_schedule` — Python's last-wins
meant the first test's body never executed for years.
- `test_alertsV2.py`: `assert alert_disabled == "False"` was indented
inside an `else:` branch that never executed because the response is
always a dict; AND `enabled` is a bool, not the string `"False"`. Two
layers of broken.
- `test_bulk.py`: variable typo (`valid_bulk_payload` reused on the POST
call) made the invalid-index test silently duplicate the valid case.
- `test_organisations.py`: `test_create_organization` used pre-pytest-4
`yield`-style fixtures — silently ignored by pytest 8.
- `test_js_result_array.py`: 8 tests skipped with `"Temporarily disabled
- failing test"` — the actual cause was that JS functions are restricted
to `_meta` org. One-line fix unskipped all 8.
- `test_fuzzy.py`: 4 tests `@pytest.mark.skip`-ed because they used a
1-minute time window against older static test data → 0 hits → flake →
skip. Wider window unskips them.
- 547 LOC of zombie commented-out test blocks deleted (15 commented `#
def test_` definitions, including one orphan case where the `def` was
commented but the indented body wasn't).
## Folder layout
```
tests/api-testing/
├── support/ # NEW framework
│ ├── client.py # OpenObserveClient
│ ├── endpoints/ # 7 per-resource wrappers
│ ├── factories.py # unique_name, time_window, *_payload
│ ├── wait.py # wait_until + WaitTimeout
│ ├── sse.py # read_sse_response
│ └── fixtures.py # client, temp_user_email, temp_stream_name, ...
├── helpers/ # Non-test shared code (was scattered in tests/)
│ ├── sourcemap_helpers.py
│ └── workflow/ # workflow chain + pages/ live here now
├── fixtures/
│ └── rum/ # Standalone RUM ingest scripts (was in tests/)
└── tests/
├── alerts/ # 3 files
├── dashboards/ # 4 files
├── ingest/ # 4 files
├── orgs/ # 2 files
├── pipelines/ # 10 files
├── regression/ # 2 files (env-flag matrix targets)
├── rum/ # 3 files
├── search/ # 8 files
├── streams/ # 1 file
├── users/ # 1 file
├── workflow/ # 3 files
├── conftest.py
└── test_framework_selfcheck.py
```
## Test plan
- [x] `rye run pytest --collect-only` — 486 tests discovered
- [x] Full default-set run (`--ignore=tests/regression`) — 443 pass / 2
fail / 39 skip / 2 xfail in 387s
- [x] `rye run pytest -m framework` — 14/14 selfcheck tests pass
- [x] `rye run ruff check support/ tests/test_framework_selfcheck.py` —
clean
- [x] CI run on this PR — `api_integration_tests` + all 4
`api_regression_tests` matrix jobs green
- [x] CI run — `api_fuzz_tests` runs cleanly (informational; surfaces
7–10 unique 5xx panics per run)
## Note on test count
The `test_streaming.py` rewrite folded 7 "tests" (`test_ingest_data`,
`test_ingest_join`, `test_disable_streaming`,
`test_update_max_query_range`, `test_enable_streaming`,
`test_delete_stream`, `test_delete_stream_join`) into pytest fixtures.
Each only checked `status_code == 200` of a setup/teardown operation —
they were imperative steps masquerading as tests. The fixtures still
perform the same operations and assert their success; they're just no
longer counted as discrete pytest items. **Functional coverage is
identical.**
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Shrinath RaoandClaude Sonnet 4.6
|
3e172523d5 |
test: sync regression workflow runners and TestDino with playwright.yml (#12370)
## Summary - **Runners**: `merge_and_upload_reports` and `playwright_summary` now use `ubicloud-standard-4` (was `-8`), matching `playwright.yml` - **TestDino rerun optimization**: added `check_rerun` step + full `last-failed` logic (same as `playwright.yml`) — on re-run-failed, only the previously failed test in each shard is re-executed - **Upload step**: added failures-only filter (`node -e` filterSuite), `startTime` ISO 8601 conversion, and `--verbose`; now uploads `report-failures-only.json` instead of the full report ## Test plan - [ ] Trigger a regression run and confirm it completes with the new runner sizes - [ ] Intentionally fail a test, re-run failed jobs, and verify TestDino optimization runs only the failed test - [ ] Confirm Upload to TestDino step produces a filtered report 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Shrinath RaoandClaude Sonnet 4.6
|
cc1f09c5c1 |
test: fix selector drift across ENT specs, re-enable enterprise tests, and improve CI (#12358)
## Summary Broad test stabilization pass covering Quasar component selector drift in ENT UI specs, re-enabling previously disabled enterprise tests, and CI infrastructure fixes. ### UI test fixes (selector drift) Quasar's O2 component library wraps native elements — `OInput`, `OTextarea`, `ODropdown`, `OToast`, `OToggleGroup`, `OToggleGroupItem`, `ODrawer`, `ODialog`, `OFile` — in ways that break naive attribute/role selectors. This PR updates affected specs to use the correct locators. | Spec | Fixes | |------|-------| | `service-graph.spec.js` | `OToggleGroupItem` selector, `ODrawer` close btn, `beforeAll` timeout | | `alerts-ui-operations.spec.js` | Remove `waitForResponse` overhead from enterprise block | | `logoManagement.spec.js` | `OInput` / `OFile` / `ODialog` / `OToast` locators | | `jobSearch.spec.js` | `OInput` fill targets wrapper div, `OTable` index-based rows | | `searchPatterns.spec.js` | `OToggleGroup` same-element selector, pattern details dialog | | `pipelineImport.spec.js` | `OInput` / `ODropdown` / Monaco / `OToast` selector drift | | `regexPatternManagement.spec.js` | Correct test-data import file path | | `cipherKeys.spec.js` | `OInput`/`OTextarea` `-field` suffix, `OToast` selector, 403 skip guard, all-toasts polling | ### Enterprise tests re-enabled - Removed `/* ... */` block comment disabling `"Alerts & Incidents Page Navigation"` describe block (`@enterprise`) in `alerts-ui-operations.spec.js` - 3 tests restored: page navigation, element visibility, round-trip nav back to Alerts - Requires `O2_INCIDENTS_ENABLED=true` (set in ENT CI) ### CI improvements - **playwright.yml**: Install `ffmpeg` via local Playwright binary (required for `video: retain-on-failure` recording) - **api-testing.yml**: Reduce runner from `ubicloud-standard-8` → `ubicloud-standard-4` - **test_stream_settings_dedup.py**: Fix field-filtering logic and assertion count in `test_multiple_distinct_fields_no_duplicates` ## Paired ENT PR Full ENT-only test restoration (SDR, RUM, cipherKeys, etc.): https://github.com/openobserve/o2-enterprise/pull/1835 ## Test plan - [ ] ENT CI: `"Alerts & Incidents Page Navigation"` enterprise block passes - [ ] Playwright CI: video artifacts generated on failure (ffmpeg install works) - [ ] API tests: `test_multiple_distinct_fields_no_duplicates` passes 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
 
|
1afabd35ce |
feat: bloom filter pruning layer above tantivy (#12121)
## Summary Design at: https://github.com/openobserve/designs/blob/main/infrastructure/bloomfilter/DESIGN.md A bloom-filter pruning layer that sits **between `file_list` and tantivy** in the search path. It drops candidate files before any `.ttv` is opened, for **positive equality / IN** lookups on high-cardinality fields (`trace_id`, `request_id`, …). It is built **on top of** tantivy: the `.bf` is constructed by iterating the tantivy term dictionary, so a field must be a tantivy secondary index (`index_fields`) **and** opted into `bloom_filter_fields`. Only the intersection is built — a bloom-only field produces nothing. To make that intersection non-empty by construction, enabling a field for bloom now folds it into `index_fields` automatically (see Config). Values are not inspected; the operator chooses which fields are worth it (see guidance below). Design notes live in the module-level docs under `src/infra/src/bloom/` (`mod.rs`, `writer.rs`, `reader.rs`, `sbbf.rs`). ## Design - **Transposed (block-major) `.bf` layout.** All files in a `.bf` share `num_blocks = B`, so `block_index = fastmap(hash(value), B)` is identical across files and the blocks a query needs form **one contiguous row**. The search side reads **one `M × 32`-byte row per `.bf`** to answer membership for all `M` files — prune cost is `O(groups)`, not `O(files)`. (`infra::bloom`, own ~120-LOC SBBF over gxhash64.) - **Sub-grouping.** A hour's new files are sorted by record count and split into chunks of ≤ `ZO_BLOOM_FILTER_MAX_FILES_PER_BF` (default 256), one `.bf` each. Bounds compactor build memory (`M × B × 32`) and `.bf` object size; query reads become `ceil(files / 256)` rows. - **Footer-only reader + batched fetch.** Footer (≈ `M × 12` B) comes from `BLOOM_FOOTER_CACHE` or one 16 KB suffix GET (with a precise re-read when a large footer overflows the probe); rows are pulled with one `cache_storage::get_ranges` call. `cache_storage` was reworked to do real batched cache reads (memory: 1 lookup + N slices; disk: 1 `File::open` + N `pread`). - **B sizing.** `num_blocks_for(max_records_in_chunk, ZO_BLOOM_FILTER_FPP)` (default fpp `0.01`) — record count is a safe NDV upper bound, so the SBBF never saturates for high-cardinality fields. - **Versioned, with orphan cleanup.** A file's `bloom_ver` is write-once — a row is never re-stamped, so dumped rows keep pointing at the `.bf` they were built into. But `bloom_ver` is **not** append-only at the bucket level: a merge produces new output files with a fresh `bloom_ver` and deletes the small inputs, so the old inputs' `.bf` can become orphaned. After each merge round the compactor retires any `.bf` whose `bloom_ver` is no longer referenced by a live `file_list` row in the bucket (enqueued through the existing `file_list_deleted` path). `bloom_ver` is preserved through every file_list INSERT/SELECT path (incl. the SQLite read-cache) and the dump parquet. - **Observability.** Prometheus metrics for prune (`bloom_prune_keep_ratio`, `bloom_prune_duration_seconds`); `bloom-inspect` CLI dumps a `.bf` footer (fields, B, row bytes, file names); footer-cache hit/miss via the shared `BytesCache` tagged metrics; `Local::get_ranges` now reports `STORAGE_READ_*` (previously bypassed). ## When it helps (measured, S3, no disk cache) Same stream `benchtest`, 170 files / 14.64 GB index, single matching row. The only variable is whether the field's values are time-correlated: | Field | Type | tantivy only | bloom + tantivy | Outcome | |---|---|---|---|---| | `trace_id` (16-byte random) | random | **2584 ms** (opens all 170 `.ttv`) | 24 ms prune + 65 ms = **89 ms** | **~29× faster** | | `request_id` (`…-7e74-…`, UUIDv7) | time-ordered | **154 ms** (sparse index range-prunes) | 42 ms + 154 ms = **196 ms** | ~1.3× **slower** | The whole 170-file group was answered by **one row read of 5440 bytes** (`170 × 32`) + a cached footer. **Guidance — enable only when all three hold (advisory; not enforced):** 1. **High cardinality** field. 2. **Random values** (not time-ordered) — so tantivy can't range-prune and must scan every file. Time-ordered ids (UUIDv7, snowflake) are already pruned by tantivy's sparse index in memory → bloom is pure overhead there. 3. **Many files per hour** — hourly volume ≳ 10 × `ZO_COMPACT_MAX_FILE_SIZE` (≳10 `.ttv` files), so there's something to prune. Bloom turns N file opens into one row read; at 1–2 files/hour it's net overhead. The 29× result above had 170 files/hour all answered by one `.bf`. ❌ Low-cardinality / non-indexed / range / `LIKE` / regex / `!=` / OR: no effect. ## Config - `ZO_BLOOM_FILTER_ENABLED` (default true) — global gate. - Per stream: a field must be in **both** `index_fields` and `bloom_filter_fields`. Adding a field to `bloom_filter_fields` auto-adds it to `index_fields` (bloom is built on the secondary index), and the default index fields include the default bloom fields. - `ZO_BLOOM_FILTER_FPP` (default 0.01) — target false-positive probability; sizes the SBBF block count (smaller fpp = fewer false survivors, larger `.bf`). Out-of-range falls back to 0.01. - `ZO_BLOOM_FILTER_MAX_FILES_PER_BF` (default 256) — files per `.bf`; trades query reads vs build memory. - `ZO_BLOOM_FILTER_DEFAULT_FIELDS` (default empty) — fields bloomed for every stream. - `ZO_BLOOM_FOOTER_CACHE_MAX_SIZE` (default 1% mem, clamped 32–256 MB). Removed in this PR: `ZO_BLOOM_FILTER_DISABLED_ON_SEARCH`, `ZO_BLOOM_FILTER_NDV_RATIO` (B is now sized from record count + fpp, not an ndv ratio). ## Test plan - [x] `cargo build`, `cargo clippy -p infra -p openobserve --lib -- -D warnings`, `cargo fmt --check` clean - [x] Unit tests: `cargo test -p infra --lib bloom`, `cargo test -p openobserve --lib bloom` - [x] E2E on S3 (results above): random field prunes 170→1 and is ~29× faster; transposed read = 1 request / 5440 B per group - [x] Reviewer: confirm gating expectations (operator opts fields in; no auto value detection) - [x] Test: orphan bloom: 2 * 256MB -> merge to 512MB -> merge to 1024MB, the first .bf should be deleted - [x] Test: orphan bloom: 6 * 256MB -> merge to 1GB + 512MB -> 512 MB merge to 1024MB, the first .bf include 2 files(1GB + 512MB), after merge again, the first still have 2 files but one of them already deleted. we should keep the first .bf, because still have files in use. ## Notes - `.bf` format is `VERSION 0x01` (transposed). Not compatible with any pre-merge experimental `.bf`; rebuild by wiping the `bloom/` dir and resetting `bloom_ver=0`. - Hash is gxhash64 (project-wide `config::utils::hash`), not the Parquet spec's xxhash64 — `.bf` is internal, never read by external tools. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
 
|
e9266c4510 |
fix: restore retry guard in enrichment table URL validation (#12218)
Fixes https://github.com/openobserve/o2-enterprise/issues/1796 (item 1: "Reload Existing URLs" returns error). ## Summary Commit [`85d261b5d0`](https://github.com/openobserve/openobserve/commit/85d261b5d0e4b68dbc79ca18b844fc53e2368ee9) (vulnerability scan fixes, Apr 14 2026) removed the `!retry` guard from enrichment table URL validation. This broke the **reload** update mode because the frontend sends an empty URL when retrying/reprocessing existing URLs — the backend already has them stored and just needs to re-trigger processing. ## Root Cause Before the breaking commit, URL validation was guarded by `!retry`: ```rust if !retry && let Err(err_msg) = validate_enrichment_url(&request_body.url) { return MetaHttpResponse::bad_request(err_msg); } ``` The security fix removed the guard under the assumption that skipping validation on retry was an SSRF bypass vector. However, the retry flow sends `url=""` intentionally — the existing URLs are in the database. ## Fix Re-adds the `!retry` guard around URL validation. In retry mode, URL validation is skipped entirely (the URLs come from DB). All other modes (create, append, replace, replace_failed) still require a non-empty, valid URL with full SSRF protection. ## Test Plan - [ ] Create enrichment table from URL - [ ] Edit table → Reload existing URLs → Should succeed (no "URL cannot be empty") - [ ] Edit table → Add new URL → Should still validate URL format - [ ] Edit table → Replace all URLs → Should still validate URL format - [ ] Edit table → Replace failed URL → Should still validate URL format 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com> |
||
|
Subhra264
|
6ec195ac81 |
fix: revamp rbac (#11610)
Revamps the complex rbac logic to make it simpler. Also, - Adds RBAC for anomaly apis - Fixes https://github.com/openobserve/openobserve/issues/11710 |
||
 Ashish Kolhe
|
3161d13785 | fix: refactor ssrf (#11734) | ||
|
Neha PandClaude
|
031bad113f |
test: add API tests for stream settings deduplication (#11516)
## Summary - Add API tests verifying that stream settings fields (index_fields, bloom_filter_fields, full_text_search_keys) are deduplicated when the same field is added multiple times - Tests cover: single field dedup, multiple fields, repeated adds (4x), remove+re-add, empty payloads, and partial updates - All tests use existing schema fields from the seed data stream (`stream_pytest_data`) ## Test plan - [x] All 10 tests pass against testing environment - Tests run via `cd tests/api-testing && rye run pytest tests/test_stream_settings_dedup.py -v` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
 udev
|
5665a2059f | ci: fix flaky tests (#11504) | ||
 ktx-kirtan
|
8bdd333a0f |
Revert "feat: dashboard validation sync from BE" (#11437)
Reverts openobserve/openobserve#11034 |
||
   
|
78b7b7b622 |
feat: dashboard validation sync from BE (#11034)
Design at: https://github.com/openobserve/designs/blob/main/dashboards/dashboard-validation-sync --------- Co-authored-by: ktx-vaidehi <vaidehi.akhani@kiara.tech> Co-authored-by: udev <uddhav@openobserve.ai> Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Abhay Padamani <undefined@users.noreply.github.com> |
||
 ByteBaker
|
d25fbf97c1 |
fix: do not show service account token post creation (#11110)
So far, service account token could be viewed and copied even after an SA was created. This was bad design and posed risk of impersonation. The changes prevent this from happening by returning only the redacted token in the API response. Full token with create API itself, and removing GET API to get the service account details (which returned raw token). The same affects token rotation as well. |
||
|
|
b1fe77871a |
test: add E2E tests for RUM sourcemap feature (#11014)
## Summary - Add comprehensive E2E test coverage for RUM sourcemap functionality (26 total tests) - API tests: 20 pytest tests (upload/list/delete/resolve operations + RBAC) - UI tests: 6 Playwright tests for error tracking UI - Helper scripts for RUM data ingestion ## Implementation Details ### API Tests (pytest) - **test_sourcemap_api.py**: 15 tests covering: - Sourcemap upload (multipart/form-data) - List sourcemaps - Delete sourcemaps - Stacktrace resolution with sourcemaps - **test_sourcemap_rbac.py**: 5 tests for role-based access control - **ingest_rum_errors.py**: Helper to inject synthetic RUM error data for UI tests - **ingest_rum_performance.py**: Helper to enable RUM feature in UI ### UI Tests (Playwright) - **sourcemap-ui.spec.js**: 6 tests validating: - Error tracking page loads - Query execution - Error list display - Error detail navigation - Metadata display - Stack trace display - **rumPage.js**: Updated page object with API response interception - Workaround for Vue 3 @click handlers not responding to Playwright clicks - Navigate via URL using error IDs extracted from API responses - Tests run in **parallel mode** with proper assertions (no graceful skips) ## Test Execution ```bash # API tests (pytest) cd tests/api-testing pytest tests/test_sourcemap_api.py -v # 15 tests pytest tests/test_sourcemap_rbac.py -v # 5 tests # UI tests (Playwright) cd tests/ui-testing npx playwright test playwright-tests/RUM/sourcemap-ui.spec.js --workers=3 # 6 tests ``` ## Technical Notes - Vue 3 production builds don't respond to Playwright's synthetic click events on @click handlers - Solution: Intercept API responses to extract error IDs, then navigate directly via URL - All tests use hard assertions - tests fail if data doesn't exist (no graceful skips) --------- Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com> Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Shrinath Rao <undefined@users.noreply.github.com> |
||
 ktx-akshay
|
6c5500eed3 | test: add API tests for dashboard multi-window VRL functionality (#10864) | ||
|
Subhra264
|
3760b55cf9 |
fix: multi-window dashboard should support vrl functions (#10860)
Currently dashboard with Multi-window set up does not really run the result array vrl functions. This pr adds support for vrl function when `/_search_multi_stream` is used. When `per_query_response` is true, and the vrl function is result array vrl function, the function is applied on the whole combined response (including both the time windows). The input of the result array vrl function becomes a 2d array where each child array represent a time window. Otherwise, if `per_query_response` is false, or the vrl function is not result array vrl function, then the function is applied individually on every result of every time-window queries. |
||
|
Neha PandClaude
|
33608a09b2 |
test: add max_query_range enforcement tests (#10907)
Add comprehensive pytest tests for max_query_range stream setting that silently truncates queries exceeding the configured time limit. - TestLogsMaxQueryRange: 4 tests verifying truncation for logs streams - TestMetricsMaxQueryRange: 1 test for metrics stream truncation - TestTracesMaxQueryRange: 1 test for traces stream truncation - TestMaxQueryRangeNegativeCases: 7 edge case tests (zero value, negative values, inverted time range, non-existent stream, boundary conditions) Tests inject backdated data (3h, 2h, 90min, 45min, 30min, 15min, 5min, now) and verify that with max_query_range=1h, only data within 1 hour is returned. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Neha PandClaude
|
4f931b2d9d |
test: add Alert History API tests with trigger setup fixture (#10812)
Adds comprehensive tests for GET /api/{org_id}/alerts/history:
- Contract validation for response schema
- Sort parameter tests (13 sort_by fields, asc/desc)
- Error path tests (invalid params, unauthorized)
- Pagination edge cases
Includes setup fixture that creates and triggers an alert to ensure the
triggers stream exists in fresh CI environments.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
Neha PandClaude
|
fef55301ab |
test: add time_shift dashboard and multi_time_range alert methods (#10782)
- Add create_dashboard_with_time_shift() to dashboard_page.py - Add create_scheduled_sql_alert_multi_time_range() to alertV2_page.py Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Neha PandClaude
|
f7b63ec883 |
test: add URL-based enrichment table methods to page object (#10770)
- create_enrichment_table_from_url(): Create table from public CSV URL
- get_enrichment_table_url_status(): Check async job status
- wait_for_url_enrichment_job(): Wait for job completion
API: POST /api/{org}/enrichment_tables/{table_name}/url Returns 202 (job
runs async)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
Neha PandClaude
|
4ff58d0510 |
test: add VRL alert test cases (#10759)
- Add 3 new VRL test functions in test_alertsV2.py: - test_new_alert_create_with_vrl_no_trigger - test_new_alert_create_with_vrl_trigger - test_update_alert_vrl_function - Fix base64 encoding in alertV2_page.py VRL methods - Add optional webhook_url param to destination_page.py --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Neha PandClaude
|
da65a0ec8d |
test: add broadcast joins tests (#10557)
Add comprehensive API tests for logs-to-logs joins with asymmetric data sizes: - 29 test cases covering INNER/LEFT/RIGHT/FULL joins, aggregations, filters - Subquery tests with IN/NOT IN, substr, length, CONCAT functions - Tests run with both ZO_FEATURE_JOIN_MATCH_ONE_ENABLED=true and false - Test data: logs_small.json (10 records) and logs_large.json (50 records) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
ByteBakerandgithub-actions[bot]
|
cf98c39c81 |
fix: semantic groups, service graph, dedup, dashboard cleanup (#10502)
## Summary
- **Semantic Groups**: add JSON export button, fix broken JSON import
(`handleJsonUpdate` was a no-op), fix "Applied 0 changes" toast bug
- **Service Graph**: remove unused `availableStreams` from
`build_topology` return and API response, skip `_`-prefixed orgs in
processor
- **Dedup Config**: change `time_window_minutes` from `Option<i64>` to
`Option<i32>`, add `deserialize_optional_i32` for defense-in-depth,
sanitize empty/NaN values on frontend before sending
- **Dashboard**: remove dead `/export` endpoint (identical to GET,
unused by frontend)
## Test plan
- [x] Semantic groups: export downloads JSON, import with file upload
works, import shows correct change count
- [x] Service graph: topology loads, stream dropdown works (populated by
`loadTraceStreams`)
- [x] Dedup: clearing time window field and saving no longer returns 422
- [x] Dashboard: GET `/{dashboard_id}` still works, `/export` route
removed
- [x] `cargo check` clean, `cargo clippy` clean
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
|
||
|
Hengfei Yang
|
f00fe6b1ec | fix: cache with query size (#10384) | ||
|
Ashish Kolheandgithub-actions[bot]
|
c34988a3ea |
fix: insights traces changes (#10304)
### **PR Type** Enhancement ___ ### **Description** - Introduce `timeStart`/`timeEnd` in filters - Enable baseline-only and comparison modes - Conditional single vs comparison query logic - Always show Insights button and update translations ___ ### Diagram Walkthrough ```mermaid flowchart LR start["Open analysis"] start -- "Has time filters?" --> comp["Comparison mode"] start -- "No time filters" --> base["Baseline-only mode"] comp -- "Query both ranges" --> merge["Merge & score"] base -- "Use baseline only" --> sort["Sort by baseline percent"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>3 files</summary><table> <tr> <td><strong>useLatencyInsightsAnalysis.ts</strong><dd><code>Support time-based filters and baseline-only mode</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-343474bc1f120f5ecebfc595bf0d3151b2db096d9226369c496b7a0cfaa33899">+87/-54</a> </td> </tr> <tr> <td><strong>useLatencyInsightsDashboard.ts</strong><dd><code>Conditional single/comparison query generation</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-960dec4a653d7c079157499a2473d0e87bd8b61463e1c3f633e7682ba763a4b1">+58/-7</a> </td> </tr> <tr> <td><strong>TracesMetricsDashboard.vue</strong><dd><code>Always show Insights button and update labels</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-7b6f29a2737d9501b18dbd8622eba4965e4bc2b92115627ae7d2e70cc3382087">+68/-54</a> </td> </tr> </table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>11 files</summary><table> <tr> <td><strong>de.json</strong><dd><code>Add `volumeInsights.insightsButtonLabel` translation</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-78f24ecb2dbcfc38fdf3ec22b2559a5b40d236ea689d1c55f5d3e87168310264">+3/-1</a> </td> </tr> <tr> <td><strong>en.json</strong><dd><code>Add insights translation keys to English</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-892aa3d75ab6e129ef9116aa21bec1b9a15ba69ce5366d2a19128078c690d824">+2/-0</a> </td> </tr> <tr> <td><strong>es.json</strong><dd><code>Add insights translation keys to Spanish</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-ac2a2167b8b973e3f776f9fe3b43bef4576470d8ad33f3d7cfaff6edff0e5778">+3/-1</a> </td> </tr> <tr> <td><strong>fr.json</strong><dd><code>Add insights translation keys to French</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-db80e762b88ddf183820c050cdc6b215e6b02b365b5e73f5152af46b0d26f1a2">+3/-1</a> </td> </tr> <tr> <td><strong>it.json</strong><dd><code>Add insights translation keys to Italian</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-2de78074a5f887afddce60d694ead9322e6ab223eb70251c13819f3028bdc70d">+3/-1</a> </td> </tr> <tr> <td><strong>ja.json</strong><dd><code>Add insights translation keys to Japanese</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-092e49c5f9fbc1b539ab9c2866d108d0ce1d314022b4c294eb4611b285e7bbd3">+3/-1</a> </td> </tr> <tr> <td><strong>ko.json</strong><dd><code>Add insights translation keys to Korean</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-50e27ca1b4825a5026f3a251bfcf79c97cab07cfade2ebd512d332453d707c48">+3/-1</a> </td> </tr> <tr> <td><strong>nl.json</strong><dd><code>Add insights translation keys to Dutch</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-a1706021687ae5d8a1c006c49eab3946174fecd895f9b70a89c2e76d97250906">+3/-1</a> </td> </tr> <tr> <td><strong>pt.json</strong><dd><code>Add insights translation keys to Portuguese</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-38e804f64cb5e0e451849fc84e9b46e53c954a4a4263d51c18665ad45f4b4815">+3/-1</a> </td> </tr> <tr> <td><strong>tr.json</strong><dd><code>Add insights translation keys to Turkish</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-6f0f131e9e2591480c4c5d1eaffc72d02dc31cbd8d3f90d8f012f08cabac8818">+3/-1</a> </td> </tr> <tr> <td><strong>zh.json</strong><dd><code>Add insights translation keys to Chinese</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-8818055db58f83230aeea4ebfd4cec31d924e8c47b6f91298e356959ab25339e">+3/-1</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
|
Subhra264
|
7ebb4c9940 |
fix: backfill icons and edit timerange (#10209)
- Use consistent styling for backfill UI. - The timerange picker on edit page should display the backfill timerange by default. |
||
|
Neha PandClaude
|
59f86c491d |
test: add API tests for pipeline backfill functionality (#10177)
### **User description**
Add comprehensive API tests for backfill core functionality:
- Basic backfill job creation and status retrieval
- Backfill execution with data verification
- List all backfill jobs for organization
- Pause and resume backfill jobs
- Delete backfill jobs
- Invalid time range validation
- Non-existent pipeline handling
- Progress tracking verification
- Non-existent job 404 handling
- Realtime pipeline rejection (only scheduled pipelines allowed)
___
### **PR Type**
Tests, Enhancement
___
### **Description**
- Add comprehensive backfill pipeline API tests
- Creation, execution, listing, pause/resume, delete, validation
- Provide test utilities for ingestion and pipelines
- Configure CI env var for backdated ingestion
___
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>test_backfill.py</strong><dd><code>Add pipeline
backfill API tests</code>
</dd></summary>
<hr>
tests/api-testing/tests/test_backfill.py
<ul><li>Add a suite of 10+ backfill API tests<br> <li> Implement helpers
for log ingestion and pipelines<br> <li> Test job lifecycle: create,
status, execute, pause/resume, delete<br> <li> Validate time range,
non-existent resources, progress tracking</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/10177/files#diff-644fee0c3348626de59a5b38627f499f0c57afcffcc1a203533ff5fefa5df6dd">+706/-0</a>
</td>
</tr>
</table></td></tr><tr><td><strong>Configuration
changes</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>api-testing.yml</strong><dd><code>Allow backdated
ingestion in CI</code>
</dd></summary>
<hr>
.github/workflows/api-testing.yml
<ul><li>Add ZO_INGEST_ALLOWED_UPTO env var<br> to allow backdated data
ingestion</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/10177/files#diff-bf87f46624aaa9c4f1d9faa033baf531a543f2d4ac188cd64e9ab3febfee8f9e">+1/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
Neha PandClaude
|
93d4914b52 |
test: add API tests for cross-stream type JOIN with enrichment tables (#10173)
### **User description** Tests the feature from PR #5452 - joining logs with enrichment tables using both enrichment_tables.{table} and enrich.{table} prefixes. Test cases: - JOIN with enrichment_tables prefix - JOIN with enrich prefix (shorter alias) - LEFT JOIN preserves all rows - JOIN with GROUP BY aggregation - JOIN with WHERE clause filtering - Verify enriched columns appear in results - Both prefixes return identical results 🤖 Generated with [Claude Code](https://claude.com/claude-code) ___ ### **PR Type** Tests ___ ### **Description** - Add API tests for enrichment table cross-stream JOIN - Test both `enrichment_tables` and `enrich` prefixes - Include LEFT JOIN, aggregation, and WHERE filter scenarios - Validate enriched column presence and prefix equivalence ___ <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table> <tr> <td> <details> <summary><strong>test_enrichment_table_join.py</strong><dd><code>Tests for enrichment table JOIN suite</code> </dd></summary> <hr> tests/api-testing/tests/test_enrichment_table_join.py <ul><li>New test suite <code>TestEnrichmentTableJoin</code> with 7 methods<br> <li> Fixture setup/teardown for enrichment table creation<br> <li> Tests cover prefix variants, LEFT JOIN, aggregation, filtering<br> <li> Utility helpers for multipart upload and SQL search</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/10173/files#diff-59300e59ac91f497cbad0e9df7ee3c47ade382b0f37382da4f25861eac3e2628">+338/-0</a> </td> </tr> </table></td></tr></tr></tbody></table> </details> ___ Co-authored-by: Claude <noreply@anthropic.com> |
||
|
d904f2eb26 |
fix: histogram interval validation for multi day factors of 24hrs (#10167)
### Problem
When using `histogram(_timestamp, '7 days')` in SQL queries, the
interval was incorrectly adjusted to 86400 seconds (1 day) instead of
the correct 604800 seconds (7 days).
### Root Cause
The `validate_and_adjust_histogram_interval` function only checked if
intervals were **factors** of 24 hours (values that divide evenly into
24 hours), but didn't check if they were **multiples** of 24 hours (2
days, 7 days, 30 days, etc.).
### Fix
Added a check for intervals that are multiples of 24 hours **before**
checking for factors:
```rust
// Check if the interval is a multiple of 24 hours (like 2 days, 7 days, etc.)
if interval_seconds % TWENTY_FOUR_HOURS_SECONDS == 0 {
return interval_seconds;
}
// Check if the interval can divide 24 hours evenly (is a factor of 24h)
if TWENTY_FOUR_HOURS_SECONDS % interval_seconds == 0 {
return interval_seconds;
}
```
___
### **PR Type**
Bug fix, Enhancement, Tests
___
### **Description**
- Validate interval multiples of 24h first
- Factor-of-24h intervals returned unchanged
- Round other intervals to nearest factor
- Add tests for multiples and edge cases
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["Interval seconds"]
B{"Multiple of 24h?"}
E{"Factor of 24h?"}
F["Round up to nearest factor"]
C["Return adjusted interval"]
A -- "input" --> B
B -- "yes" --> C
B -- "no" --> E
E -- "yes" --> C
E -- "no" --> F
F --> C
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>histogram_interval.rs</strong><dd><code>Enhance
histogram interval validation logic</code>
</dd></summary>
<hr>
src/service/search/sql/visitor/histogram_interval.rs
<ul><li>Added doc comments describing new validation logic<br> <li>
Inserted multiple-of-24h check before factor check<br> <li> Retained
factor check and added rounding fallback<br> <li> Extended tests for
multiples and edge scenarios</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/10167/files#diff-0442907b4d8a3b5c4bd9553fd5475c3044fe74a2853cda8e4017bf295d714b4b">+60/-3</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
---------
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
|
||
|
Neha PandClaude
|
87a507f83a |
test: add API tests for bulk enable, functions, kvstore, multi-stream search, passcode (#10155)
### **User description** - Add test_alert_bulk_enable.py: bulk enable/disable alerts (5 tests) - Add test_function_test_endpoint.py: VRL function testing (7 tests) - Add test_kvstore.py: KV store CRUD operations (8 tests) - Add test_multi_stream_search.py: multi-stream SSE search (6 tests) - Add test_org_passcode.py: organization passcode API (4 tests) 🤖 Generated with [Claude Code](https://claude.com/claude-code) ___ ### **PR Type** Tests ___ ### **Description** - Bulk alert enable/disable tests for existing alerts - VRL/JS function test endpoint happy and error flows - KV store create, read, list, update, delete tests - Multi-stream SSE search tests for various scenarios - Organization passcode get, update, persistence tests ___ <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table> <tr> <td> <details> <summary><strong>test_alert_bulk_enable.py</strong><dd><code>Bulk alert enable/disable tests</code> </dd></summary> <hr> tests/api-testing/tests/test_alert_bulk_enable.py <ul><li>Fixture setup for session and existing alerts retrieval<br> <li> Tests bulk disable and enable single alerts<br> <li> Tests bulk disable for multiple alerts<br> <li> Idempotency and response structure verification</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/10155/files#diff-bb438bbffe5aad82142e50aa7765cb47c711463ffc0fc1ce72b221cbcadafe9a">+241/-0</a> </td> </tr> <tr> <td> <details> <summary><strong>test_function_test_endpoint.py</strong><dd><code>Function test endpoint tests</code> </dd></summary> <hr> tests/api-testing/tests/test_function_test_endpoint.py <ul><li>Fixture setup for VRL/JS function test endpoint<br> <li> Tests basic function and value modification<br> <li> Multi-event and conditional logic tests<br> <li> Invalid syntax and response structure verification</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/10155/files#diff-8f8ba09e2a96c246c77ae4512443842d0e8accb855bac0f949971941b9956b6c">+230/-0</a> </td> </tr> <tr> <td> <details> <summary><strong>test_kvstore.py</strong><dd><code>KV store CRUD API tests</code> </dd></summary> <hr> tests/api-testing/tests/test_kvstore.py <ul><li>Added KV store CRUD tests (create, read, list)<br> <li> Tests prefix filtering and update behavior<br> <li> Delete operations, 404 and idempotency tests</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/10155/files#diff-b353df992b5db6b02190e9232e077967b8c30d702f24bb1903c8d5c8c32fd14e">+226/-0</a> </td> </tr> <tr> <td> <details> <summary><strong>test_multi_stream_search.py</strong><dd><code>Multi-stream search SSE tests</code> </dd></summary> <hr> tests/api-testing/tests/test_multi_stream_search.py <ul><li>Fixture setup and SSE response parser added<br> <li> Tests single and multi-stream search queries<br> <li> SSE format and metadata structure verification<br> <li> Size limit and invalid stream handling tests</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/10155/files#diff-c20d33fef67f42eb651e7a0d5d97d053c1d0d58307c13fb57ea0a19ffeedb329">+223/-0</a> </td> </tr> <tr> <td> <details> <summary><strong>test_org_passcode.py</strong><dd><code>Organization passcode API tests</code> </dd></summary> <hr> tests/api-testing/tests/test_org_passcode.py <ul><li>Tests organization passcode get and update endpoints<br> <li> Verifies persistence and response structure</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/10155/files#diff-819841cc5f7f2475248a5edd31f9d44c79159fa6fb97a9c2db9ea70139904379">+126/-0</a> </td> </tr> </table></td></tr></tr></tbody></table> </details> ___ --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Hengfei Yang
|
91e2f06895 | fix: pipeline destinations incorrectly saved as alert destinations (#10079) | ||
 priyabrata-stackandClaude Sonnet 4.5
|
dacc8f6365 |
chore: remove documentation files and test artifacts from PR
Removed temporary documentation and test result files: - METRICS_CLEANUP_SUMMARY.md - METRICS_TESTS_AUDIT_REPORT.md - POM_COMPLIANCE_FIXES_SUMMARY.md - PR_STRATEGY.md - PR_SUMMARY.md - TRACE_TESTS_FALSE_POSITIVE_ANALYSIS.md - test-results/.last-run.json - tests/test-results/.last-run.json - tests/test-logs/test-run-2025-12-15T17-25-37.log - tests/api-testing/RUM_TEST_README.md - tests/api-testing/rum-node-test.js - tests/api-testing/rum-test.html - tests/ui-testing/METRICS_INGESTION_README.md These files were used for development/documentation purposes and should not be included in the PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> |
||
|
priyabrata-stackandClaude Sonnet 4.5
|
0cf0e8937c |
fix(tests): implement POM compliance for metrics tests (72% complete)
Major refactoring to fix Page Object Model violations in metrics test suite. This addresses critical code quality issues identified in the Sentinel audit. ## Changes ### Page Objects Enhanced - **metricsPage.js**: Added 350+ lines of new methods - Data visualization verification methods - Config sidebar operations - Stream selection, date ranges, notifications - Chart types, inputs, and generic UI helpers ### Test Files Fixed (5/8 - 62.5%) - ✅ **metrics-advanced.spec.js**: Fixed 13 POM violations - ✅ **metrics-aggregations.spec.js**: Fixed 8 POM violations - ✅ **metrics-config-tabs.spec.js**: Fixed 31/33 POM violations - ✅ **metrics-queries.spec.js**: Fixed 28 POM violations - ✅ **metrics-visualizations.spec.js**: Fixed 44 POM violations ### Additional Fixes - **apiCleanup.js**: Fixed error handling consistency - Unified error response format (message vs error field) - Added explicit code: 200 for successful responses ## Statistics - **Violations Fixed**: 132/184 (72%) - **Helper Functions Refactored**: 3 (eliminated 200+ lines of duplication) - **All fixed files**: ✓ Syntax validated ## Remaining Work - metrics-config.spec.js: 28 violations - metrics.spec.js: 30 violations - See POM_COMPLIANCE_FIXES_SUMMARY.md for completion guide 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> |
||
|
Hengfei Yang
|
8b69f8f151 | fix: Axum migration (#10040) | ||
|
Hengfei Yang
|
fdcb375256 | fix: unit tests (#10032) | ||
|
Hengfei Yang
|
1fc4270f54 |
fix: payload limit (#10021)
1. We only need `ZO_PAYLOAD_LIMIT` now, removed `ZO_JSON_LIMIT` 2. Config axum log level `RUST_LOG = "info,config::axum=warn"` |
||
|
Shrinath Rao
|
8bf9eac8a2 |
test: Functions test suite rework - 67% reduction with full validation coverage (#9957)
## Summary Rework of the Functions test suite to reduce test count while maintaining 100% validation coverage. | Metric | Before | After | Change | |--------|--------|-------|--------| | **UI Tests** | 36 | 12 | **-67%** | | **API Tests (Python)** | 0 | 16 | +16 new | | **Validation coverage** | 100% | 100% | Preserved | | **CI Integration** | ❌ | ✅ | Added | **Cleanup performed:** - 3 placeholder tests removed (only had logging, no assertions) - 1 duplicate test removed - 4 cross-file duplicates deduplicated ## What Changed ### Test File Consolidation | Original Spec Files | Final Spec File | Original | Final | Reduction | |---------------------|-----------------|----------|-------|-----------| | `meta-org-js-restriction.spec.js` + `javascript-functions.spec.js` | `js-transform-type.spec.js` | 22 | 6 | 73% | | `result-array-functions.spec.js` | `row-expansion.spec.js` | 14 | 6 | 57% | | **Total** | **2 files** | **36** | **12** | **67%** | ### New Python API Tests Added 16 comprehensive API tests in `tests/api-testing/tests/test_functions_meta_org_restriction.py`: | Category | Count | Tests | |----------|-------|-------| | JS Restriction (blocked in default org) | 5 | create/test/update blocked | | VRL Functions (work in default org) | 4 | create/test/update works | | Auto-Detection | 2 | trans_type auto-detection | | Error Messages | 2 | content and consistency | | Edge Cases | 3 | empty/invalid/case-sensitivity | ### Critical Bugs Fixed 1. **Missing `_meta` org navigation** - JavaScript functions are ONLY allowed in `_meta` organization. Tests were navigating to default org. 2. **Python API tests using wrong field names** - Create/Update use `transType` (camelCase), Test endpoint uses `trans_type` (snake_case). 3. **Missing pipeline cleanup pattern** - Added `realfile*` pattern that was causing 409 errors. ## Files Modified ### UI Tests (Final) - `tests/ui-testing/playwright-tests/Functions/js-transform-type.spec.js` (6 tests) - `tests/ui-testing/playwright-tests/Functions/row-expansion.spec.js` (6 tests) ### UI Tests (Removed/Consolidated) - `meta-org-js-restriction.spec.js` → merged into `js-transform-type.spec.js` - `javascript-functions.spec.js` → merged into `js-transform-type.spec.js` - `result-array-functions.spec.js` → renamed to `row-expansion.spec.js` ### Cleanup Infrastructure - `tests/ui-testing/playwright-tests/cleanup.spec.js` - Consolidated function cleanup, added realfile* pattern - `tests/ui-testing/pages/apiCleanup.js` - Added org-aware cleanup methods ### API Tests (New) - `tests/api-testing/tests/test_functions_meta_org_restriction.py` - 16 new tests ### CI Integration - `.github/workflows/playwright.yml` - Added Functions test matrix entry ## Final Test List (12 UI Tests) ### js-transform-type.spec.js (6 tests) | # | Test | Priority | |---|------|----------| | 1 | P0: JavaScript function creation flow in _meta org | @smoke, @P0 | | 2 | P0: JS function creation and validation flow | @smoke, @P0 | | 3 | P1: JS function execution and error handling | @P1 | | 4 | P1: JS function error cases | @P1 | | 5 | P0: Default org - JS hidden, VRL works | @smoke, @P0 | | 6 | P1: Radio visibility across orgs | @P1 | ### row-expansion.spec.js (6 tests) | # | Test | Priority | |---|------|----------| | 1 | P0: VRL row expansion creation and validation flow | @smoke, @P0 | | 2 | P2: VRL row expansion edge cases | @P2 | | 3 | P0: JS row expansion creation and validation flow | @smoke, @P0 | | 4 | P1: JS row expansion edge cases and transformations | @P1 | | 5 | P1: Compare VRL and JS row expansion output | @P1 | | 6 | P1: Pipeline integration and error handling | @P1 | ## Test plan - [x] All 12 UI tests passing locally - [x] All 16 Python API tests passing locally - [x] Sentinel audit passed (POM compliant, no raw selectors) - [x] Hard assertions applied (no soft assertions) - [ ] CI pipeline runs successfully |
||
|
Ashish KolheandClaude Opus 4.5
|
ce09cd0a4d |
feat: add js for claim parser (#9873)
### **User description** Design at : security/sso-claim-parser/configuration.md ___ ### **PR Type** Enhancement, Tests ___ ### **Description** - Introduce rquickjs-based JS runtime for functions - Add compile_js_function and apply_js_fn utilities - Extend pipeline to support JS via CompiledFunctionRuntime - Update web UI: select JS/VRL transforms, show JS errors ___ ### Diagram Walkthrough ```mermaid flowchart LR A["FunctionNode"] -->|JS| B["compile_js_function"] A -->|VRL| C["compile_vrl_function"] B --> D["apply_js_fn"] C --> E["apply_vrl_fn"] D --> F["send to children"] E --> F ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>13 files</summary><table> <tr> <td><strong>function.ts</strong><dd><code>Add optional `trans_type` to TestFunctionPayload</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-c6766f6c671adce05f9eef5d4a57a32371fa7da35628ef5c142fa503d191cb16">+16/-0</a> </td> </tr> <tr> <td><strong>js.rs</strong><dd><code>Introduce JS runtime, compile and apply functions</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-88e490bed5fa6c9c8e0aa2738b95c5284bb97899298f54b9cd587713e3ef0109">+756/-0</a> </td> </tr> <tr> <td><strong>batch_execution.rs</strong><dd><code>Extend pipeline execution to handle JS functions</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-3f6824a6eddaf1603e5cb0f1a4311ac6a45e1ffb967266ccc1f0bba944625e69">+465/-96</a></td> </tr> <tr> <td><strong>functions.rs</strong><dd><code>Validate and test-run JS transforms, restrict to _meta</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-ed5611a9aa6b5c033e70d7631ba72fafad3cc842672077a7fcf965d11a4fef23">+319/-35</a></td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Forbid JavaScript functions in pipelines globally</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-c186c796a7bdfd9d1301b1429df11456430f1d0d4758783810b9e2e44384e3b2">+39/-1</a> </td> </tr> <tr> <td><strong>jwt.rs</strong><dd><code>Add JS claim parser execution function</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-232cab74f4cbf135af424744272e17732fbda8ac005d33c0acb4597e65479d02">+25/-2</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Wrap JS compile/apply in ingestion API</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-7c7b5bca85f0ced652965cae66540c96f56dfcca4f4c16b77bf7b45aac203d1b">+24/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Include `trans_type` in test_function API payload</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-a5e1b3644cef9feb689dfe66382e44e5434ecadbff3469127346a333d4733102">+8/-3</a> </td> </tr> <tr> <td><strong>AddFunction.vue</strong><dd><code>UI: add JS/VRL transform type selector</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-ebfd6c538f3cd3ecb9ab17550fd76e2e89e7a1abccc83a7505077e4af3eee110">+42/-25</a> </td> </tr> <tr> <td><strong>FunctionsToolbar.vue</strong><dd><code>Toolbar: transform type radio buttons</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-8283d2ff9c4a66b4b59eac306672aca2d54655a462e2ddb88e14e039d79dcdaf">+51/-3</a> </td> </tr> <tr> <td><strong>DomainManagement.vue</strong><dd><code>Track claim-parser function changes for save button</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-5e47cd01fcb63f7ef27dc64d2ecf66c39e17097e29944951c3ae1bfe715bb97c">+17/-9</a> </td> </tr> <tr> <td><strong>TestFunction.vue</strong><dd><code>Support JS error messaging in test UI</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-b9af9a2d1bf04d23422c26a13e6a763c6953c8713357a2f64ed8073ce6c926a3">+15/-5</a> </td> </tr> <tr> <td><strong>PipelineEditor.vue</strong><dd><code>Filter out JS functions in pipeline editor UI</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-68c21f647336067eb79737ce9aca2d49632f2fa1c0878bcfdae55607e3473a15">+12/-4</a> </td> </tr> </table></details></td></tr><tr><td><strong>Configuration changes</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>function.rs</strong><dd><code>Extend `Transform` model for JS support</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-27c9f3d93723397a0e234bc45c9f543b43eff5fd518d76e735f78d95a58f3f1c">+16/-6</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Register new `js` utils module</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-2aa59ce4464e7dbabdb9367fc8bb8c1b489c914b8690577d4916870463ab12db">+1/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Bug fix</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>errors.rs</strong><dd><code>Rename `MemoryTableOverflowError` display message</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-bf45932466265b732f13d0cd9b9cffbaefbe37c5861ace9315449eeca9250713">+1/-1</a> </td> </tr> </table></details></td></tr><tr><td><strong>Tests</strong></td><td><details><summary>3 files</summary><table> <tr> <td><strong>result-array-functions.spec.js</strong><dd><code>Add E2E tests for `#ResultArray#` functions in UI</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-e9911625ad2bab76f2b760f0ad254791331993a6a874b96f6a11fc6d88038b10">+470/-0</a> </td> </tr> <tr> <td><strong>cleanup.spec.js</strong><dd><code>Clean up JS function test data in UI tests</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-c214275a512fa285a7e4f2ab61dfcb42d71a0436b0e4241f94d7b45693587983">+31/-0</a> </td> </tr> <tr> <td><strong>page-manager.js</strong><dd><code>Register `FunctionsPage` in `PageManager`</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-565eab8b3c767ae50c9d05e591621d14f66b81af84d6b5ea6efe136c1c3933df">+6/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Dependencies</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>Cargo.toml</strong><dd><code>Add `rquickjs` dependency for JS runtime</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542">+2/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>18 files</summary><table> <tr> <td><strong>test_functions_meta_org_restriction.py</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-99ef1207d6ae6d7eeaed37d70c8cf0c4770ba78f6f51fbdc067d35492d2fe9ea">+401/-0</a> </td> </tr> <tr> <td><strong>test_js_enrichment.py</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-c5547d46bec60b888d947d1e9b7c5445aee72cf8e837182f7138e0fc903eedbb">+356/-0</a> </td> </tr> <tr> <td><strong>test_js_result_array.py</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-9ed1363a1f58c94e018bb2b0aaa9d08fa71f89e7e55a6ff6be4a82bbeed8985c">+390/-0</a> </td> </tr> <tr> <td><strong>functionsPage.js</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-7bab9cc2eb8f36d3ee5d0aab453c353ec6d6dfecbc401cf401322ff57faa8440">+326/-0</a> </td> </tr> <tr> <td><strong>javascript-functions.spec.js</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-bdae6529346a7222bdec19f75f490cb9d0f3578a260ba835225efcd27db3f64e">+314/-0</a> </td> </tr> <tr> <td><strong>meta-org-js-restriction.spec.js</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-16cc2373f6f1955f8344a59d5f2edcd6ce121c4fdee1d432b1f0ec8e27f25881">+483/-0</a> </td> </tr> <tr> <td><strong>FunctionList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-6ff9af50a1597504b7f0dca91e61cf30c800b4a17b4be2ed2c3a4f7a7b2d7068">+1/-0</a> </td> </tr> <tr> <td><strong>de.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-78f24ecb2dbcfc38fdf3ec22b2559a5b40d236ea689d1c55f5d3e87168310264">+30/-18</a> </td> </tr> <tr> <td><strong>en.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-892aa3d75ab6e129ef9116aa21bec1b9a15ba69ce5366d2a19128078c690d824">+15/-7</a> </td> </tr> <tr> <td><strong>es.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-ac2a2167b8b973e3f776f9fe3b43bef4576470d8ad33f3d7cfaff6edff0e5778">+30/-18</a> </td> </tr> <tr> <td><strong>fr.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-db80e762b88ddf183820c050cdc6b215e6b02b365b5e73f5152af46b0d26f1a2">+30/-18</a> </td> </tr> <tr> <td><strong>it.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-2de78074a5f887afddce60d694ead9322e6ab223eb70251c13819f3028bdc70d">+30/-18</a> </td> </tr> <tr> <td><strong>ja.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-092e49c5f9fbc1b539ab9c2866d108d0ce1d314022b4c294eb4611b285e7bbd3">+30/-18</a> </td> </tr> <tr> <td><strong>ko.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-50e27ca1b4825a5026f3a251bfcf79c97cab07cfade2ebd512d332453d707c48">+30/-18</a> </td> </tr> <tr> <td><strong>nl.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-a1706021687ae5d8a1c006c49eab3946174fecd895f9b70a89c2e76d97250906">+30/-18</a> </td> </tr> <tr> <td><strong>pt.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-38e804f64cb5e0e451849fc84e9b46e53c954a4a4263d51c18665ad45f4b4815">+30/-18</a> </td> </tr> <tr> <td><strong>tr.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-6f0f131e9e2591480c4c5d1eaffc72d02dc31cbd8d3f90d8f012f08cabac8818">+30/-18</a> </td> </tr> <tr> <td><strong>zh.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-8818055db58f83230aeea4ebfd4cec31d924e8c47b6f91298e356959ab25339e">+254/-242</a></td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
  Bhargavandomkark06
|
271ff27fac |
fix: remove traces error and rate value filters (#9617)
### **PR Type**
Bug fix
___
### **Description**
- Merge Rate and Errors filter chip displays
- Remove redundant Errors start-value span
- Always display Duration time range when set
___
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Bug
fix</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>TracesMetricsDashboard.vue</strong><dd><code>Unify
filter chip display logic</code>
</dd></summary>
<hr>
web/src/plugins/traces/metrics/TracesMetricsDashboard.vue
<ul><li>Merged Rate and Errors chips into single time-range
indicator<br> <li> Removed specific Errors value display span<br> <li>
Changed Duration chip to always use time-range v-if</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9617/files#diff-7b6f29a2737d9501b18dbd8622eba4965e4bc2b92115627ae7d2e70cc3382087">+2/-5</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
---------
Co-authored-by: omkark06 <omkar@zinclabs.io>
|
||
|
BhargavandClaude
|
e7703e86ec |
fix: upgrader two eslint packages (#9765)
### **PR Type**
Enhancement
___
### **Description**
- Upgrade eslint-plugin-cypress to ^5.2.0
- Upgrade eslint-plugin-vue to ^10.6.2
___
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Dependencies</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>package.json</strong><dd><code>ESLint plugin versions
bump</code>
</dd></summary>
<hr>
web/package.json
<ul><li>Bump <code>eslint-plugin-cypress</code> from ^4.1.0 to
^5.2.0<br> <li> Bump <code>eslint-plugin-vue</code> from ^9.32.0 to
^10.6.2</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9765/files#diff-b861012a5dd72b8a9f3281b7cf09f5a779c98569d040b1bbc1db50f1b15e7cce">+2/-2</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
Bhargav
|
50402cf46a |
fix: batch 3 package upgrade (#9755)
### **PR Type**
Enhancement
___
### **Description**
- Adjust Vitest config imports and coverage settings
- Refine coverage: add V8 provider and include patterns
- Bump Vitest, Vitest UI, and canvas mock versions
- Upgrade dotenv and remove unused type definitions
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["web/vitest.config.ts"] --> B["Import types corrected"]
A --> C["Coverage provider & include updated"]
D["web/package.json"] --> E["Bump Vitest & mocks versions"]
D --> F["Upgrade dotenv & remove types"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Configuration
changes</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>vitest.config.ts</strong><dd><code>Adjust Vitest config
imports and coverage</code>
</dd></summary>
<hr>
web/vitest.config.ts
<ul><li>Replace type import from <code>vitest/config</code> to
<code>vite</code><br> <li> Remove <code>VitestUserConfig</code> cast for
mergeConfig<br> <li> Add coverage provider <code>v8</code><br> <li>
Update coverage include pattern to
<code>src/**/*.{js,ts,vue}</code></ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9755/files#diff-c3fc010b682e68f2ed0b7b1485b8aa513443a66d500903b8dff1d65908d02aae">+4/-3</a>
</td>
</tr>
</table></td></tr><tr><td><strong>Dependencies</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>package.json</strong><dd><code>Upgrade devDependencies
versions</code>
</dd></summary>
<hr>
web/package.json
<ul><li>Bump <code>@vitest/coverage-v8</code> and
<code>@vitest/ui</code> to 4.x<br> <li> Upgrade <code>vitest</code> and
<code>vitest-canvas-mock</code> versions<br> <li> Update
<code>dotenv</code> to <code>^17.2.3</code><br> <li> Remove unused
<code>@types/dompurify</code></ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9755/files#diff-b861012a5dd72b8a9f3281b7cf09f5a779c98569d040b1bbc1db50f1b15e7cce">+5/-6</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
|
|
ac60e181dd |
feat: dashboard join support for auto sql (#9343)
# Nested Functions and Joins - User Guide ## Overview This guide covers two powerful features that enable sophisticated data analysis in OpenObserve: - **Nested Functions**: Combine multiple functions together to create complex calculations - **Joins**: Merge data from multiple streams (data sources) based on common fields These features work in the **Auto SQL** mode of dashboard panels, allowing you to write advanced queries without deep SQL knowledge. --- ## What's New ### Why These Features? **Before:** - Limited function composition - you could only use basic, single-level functions - No way to combine data from multiple streams in one query - Complex analytics required exporting data and manual processing **Now:** - Compose unlimited nested functions with full DataFusion support - Join multiple streams directly in the dashboard panel editor - Create sophisticated analytics queries with a visual interface ### Key Benefits ✓ Calculate complex metrics (error rates, percentiles, conditional aggregations) ✓ Build advanced dashboards without custom code ✓ Support for all DataFusion aggregate and scalar functions --- ## Nested Functions ### What Are Nested Functions? Nested functions let you combine multiple functions together - using the output of one function as input to another. Think of it like a cooking recipe where each step builds on the previous one. **Simple Example:** - Basic: `AVG(response_time)` - calculates average response time - Nested: `ROUND(AVG(response_time), 2)` - calculates average AND rounds it to 2 decimal places ### Supported Functions OpenObserve supports **all DataFusion functions**, giving you powerful analytics capabilities: **Aggregate Functions** (work across multiple rows): - `count`, `sum`, `avg`, `min`, `max` - basic aggregations - `median`, `stddev`, `variance` - statistical functions - `percentile_cont` - calculate percentiles (e.g., p95, p99) - `string_agg` - combine text values **Mathematical Functions**: - `round`, `ceil`, `floor` - rounding operations - `abs`, `power`, `sqrt` - mathematical operations - `log`, `log10`, `log2` - logarithmic functions - `sin`, `cos`, `tan` - trigonometric functions **String Functions**: - `concat`, `upper`, `lower`, `trim` - `substring`, `length`, `replace` **Conditional Functions**: - `CASE WHEN ... THEN ... ELSE ... END` - conditional logic - `COALESCE` - return first non-null value 📖 **Full Reference**: [DataFusion Aggregate Functions](https://datafusion.apache.org/user-guide/sql/aggregate_functions.html) | [DataFusion Scalar Functions](https://datafusion.apache.org/user-guide/sql/scalar_functions.html) --- ### Using Nested Functions (Step-by-Step) #### Step 1: Create or Edit a Dashboard Panel Open your dashboard and create a new panel or edit an existing one. Make sure you're in **Auto SQL** mode (not Custom SQL). #### Step 2: Select a Function In the Y-Axis or column configuration, click **Select Function** to choose your base function. <img width="795" alt="Nested Function - Adding Layers" src="https://github.com/user-attachments/assets/0b6b134e-59b8-48b1-a4ae-52fb4800d772" /> In this example, we're using the `Concat` function to combine multiple fields (namespace and pod name). #### Step 3: Add Nested Layers Click the function icon to add another function layer. You can nest functions multiple times. <img width="793" alt="Nested Function - Select Base Function" src="https://github.com/user-attachments/assets/4bd9e03e-1104-4e82-b2ec-a34e6bb64870" /> Here we're building a nested function: - Inner layer: `Avg(severity)` - calculates average severity - Outer layer: `Ceil()` - rounds up the average to the nearest integer #### Step 4: Build Complex Expressions For advanced analytics, you can combine nested functions with conditional logic using `CASE WHEN` statements. <img width="1340" alt="Complex Nested Function with CASE WHEN" src="https://github.com/user-attachments/assets/4c3e3623-bfad-44c4-a760-e10cc94d4a95" /> This example shows a sophisticated calculation: ```sql ROUND(100.0 * SUM(CASE WHEN COALESCE(severity, 0) >= 4 THEN 1 ELSE 0 END) / COUNT(_timestamp), 2) ``` **What this does:** 1. `COALESCE(severity, 0)` - replaces null severity with 0 2. `CASE WHEN severity >= 4 THEN 1 ELSE 0` - returns 1 for high severity, 0 otherwise 3. `SUM(...)` - counts high severity events 4. Divide by `COUNT(_timestamp)` - calculates percentage 5. Multiply by `100.0` - converts to percentage 6. `ROUND(..., 2)` - rounds to 2 decimal places **Result**: Percentage of high-severity events, rounded to 2 decimals #### Step 5: View the Generated SQL Switch to the **Raw** tab to see the complete SQL query that was generated from your nested function configuration. <img width="1388" alt="View Generated SQL" src="https://github.com/user-attachments/assets/d7f6111a-26c6-4dd0-9e9f-e26092fe6368" /> You can review the auto-generated SQL in the Query section at the bottom of the panel editor. The chart visualization updates in real-time as you build your nested functions. --- ## Joins ### What Are Joins? Joins allow you to combine data from two or more different streams (data sources) based on common fields between them. **Example:** **employee_data_join** stream: ```json [ {"employee_id": 1, "name": "Alice", "department_id": 1}, {"employee_id": 2, "name": "Bob", "department_id": 2}, {"employee_id": 3, "name": "Charlie", "department_id": null}, {"employee_id": 4, "name": "David", "department_id": 1}, {"employee_id": 5, "name": "Eve", "department_id": 3} ] ``` **dept_data_join** stream: ```json [ {"department_id": 1, "department_name": "HR"}, {"department_id": 2, "department_name": "Engineering"}, {"department_id": 3, "department_name": "Sales"}, {"department_id": 4, "department_name": "Marketing"} ] ``` A join lets you combine these streams to see the department name alongside each employee, matched by `department_id`. ### Types of Joins OpenObserve supports three types of joins: | Join Type | Description | When to Use | |-----------|-------------|-------------| | **INNER JOIN** | Returns only records with matches in both streams | When you need data that exists in both sources | | **LEFT JOIN** | Returns all records from left stream + matching records from right | When you want all data from your primary source, with supplemental info where available | | **RIGHT JOIN** | Returns all records from right stream + matching records from left | When you want all data from your secondary source, with supplemental info where available | > **Note:** UNION and CROSS joins are not currently supported. --- ### Using Joins (Step-by-Step) #### Step 1: Access the Joins Section In your dashboard panel editor, locate the **Joins** section. Click the **+** button to add a join. <img width="1725" alt="Access Joins Interface" src="https://github.com/user-attachments/assets/35db4f28-c526-43a2-bdd1-470e188f53d7" /> In this example: - **Primary stream**: `employee_data_join` (contains employee names) - We'll join it with `dept_data_join` (contains department information) - The interface shows available fields from both streams in the left panel #### Step 2: Configure the Join Click on the **Joins** dropdown and select the stream you want to join. Then configure: 1. **Joining Stream**: Select the stream to join (e.g., `employee_data_join`) 2. **Join Type**: Choose from LEFT, INNER, or RIGHT using the visual toggle 3. **On Stream**: Select the target stream (e.g., `dept_data_join`) <img width="1725" alt="Configure Join Type" src="https://github.com/user-attachments/assets/35db4f28-c526-43a2-bdd1-470e188f53d7" /> The interface displays: **"Performing a Left Join between employee_data_join and dept_data_join"** Notice the visual toggle showing three join types: - **Left** (selected in this example) - **Inner** - **Right** #### Step 3: Define the Joining Clause Set up the join condition by specifying which fields to match between the two streams: **Clause 1:** - **Select Field** (Left): Choose a field from the first stream (e.g., `department_id`) - **Select Operator**: Choose comparison operator (usually `=`) - **Select Field** (Right): Choose the matching field from the second stream (e.g., `department_id`) You can add multiple clauses using the **+** button for complex join conditions (e.g., joining on multiple fields or adding additional filters). #### Step 4: View the Results After configuring the join, the data table will show combined results from both streams. <img width="1724" alt="Join Results" src="https://github.com/user-attachments/assets/94ca9f33-6cfa-4ad1-9b84-bee9db34f912" /> In this example: - **Name** column from `employee_data_join` (Alice, Bob, Charlie, David, Eve) - **Department Name** column from `dept_data_join` (HR, Engineering, HR, Sales) - Data successfully joined on `department_id` #### Step 5: Review the Generated SQL Check the **Query** section at the bottom to see the generated SQL: ```sql SELECT employee_data_join.name as "x_axis_1", stream_0.department_name as "x_axis_2" FROM "employee_data_join" LEFT JOIN "dept_data_join" AS stream_0 ON employee_data_join.department_id = stream_0.department_id ``` The SQL query shows exactly how OpenObserve is combining your data streams. --- --- ## Examples ### Example 1: String Concatenation with Nested Functions **Use Case**: Combine multiple fields into a single display value ```sql Concat(k8s_namespace_name, '-', k8s_pod_name) ``` This joins namespace and pod names with a hyphen separator. ### Example 2: Statistical Analysis with Nested Math Functions **Use Case**: Calculate rounded average of a metric ```sql Ceil(Avg(severity)) ``` This calculates the average severity level and rounds up to the nearest whole number. ### Example 3: Conditional Percentage Calculation **Use Case**: Calculate percentage of high-severity events ```sql ROUND(100.0 * SUM(CASE WHEN COALESCE(severity, 0) >= 4 THEN 1 ELSE 0 END) / COUNT(_timestamp), 2) ``` **Breakdown:** - Handles null values with `COALESCE` - Counts events where severity >= 4 - Calculates percentage of total events - Rounds to 2 decimal places ### Example 4: Employee-Department Join **Use Case**: Show employee names with their department information ```sql SELECT employee_data_join.name as "Name", dept_data_join.department_name as "Department Name" FROM employee_data_join LEFT JOIN dept_data_join ON employee_data_join.department_id = dept_data_join.department_id ``` This displays all employees with their department names, showing null for employees without a department assignment. ### Example 5: Complex Analytics - Joins + Nested Functions **Use Case**: Calculate error rate per service with deployment information ```sql SELECT logs.service_name, deployments.version, COUNT(*) as total_requests, ROUND((COUNT(CASE WHEN logs.status_code >= 500 THEN 1 END) * 100.0 / COUNT(*)), 2) as error_rate_percent FROM service_logs as logs LEFT JOIN deployments ON logs.service_name = deployments.service_name GROUP BY logs.service_name, deployments.version ORDER BY error_rate_percent DESC ``` This combines: - Join between service logs and deployment data - Nested functions for conditional counting and percentage calculation - Grouping and sorting for analysis --- ## Tips and Best Practices ### Nested Functions Best Practices ✅ **Start Simple**: Build your function one layer at a time - Begin with inner function: `AVG(response_time)` - Add outer layers: `ROUND(AVG(response_time), 2)` ✅ **Test Each Layer**: Verify results before adding complexity - Test the inner function first - Add one layer at a time and check output ✅ **Handle Null Values**: Use `COALESCE` to avoid unexpected nulls ```sql COALESCE(AVG(response_time), 0) ``` ✅ **Use Clear Aliases**: Name your calculated fields descriptively ```sql ROUND(AVG(response_time), 2) as avg_response_ms ``` ✅ **Check for Division by Zero**: Protect against zero denominators ```sql CASE WHEN COUNT(*) > 0 THEN (SUM(errors) * 100.0 / COUNT(*)) ELSE 0 END as error_rate ``` ### Joins Best Practices ✅ **Choose the Right Join Type**: - **INNER JOIN**: Both streams must have matching data - **LEFT JOIN**: Keep all records from primary stream - **RIGHT JOIN**: Keep all records from secondary stream ✅ **Verify Field Compatibility**: - Match data types (don't join string to number) - Ensure field names are correct - Check that values actually match between streams ✅ **Consider Performance**: - Join on indexed fields when possible - Test with limited data first --- ## FAQ ### Q: Can I nest more than 2 functions? **A**: Yes! You can nest as many functions as needed. Example: ```sql ROUND(CEIL(AVG(COALESCE(response_time, 0))), 2) ``` ### Q: What's the difference between LEFT and RIGHT joins? **A**: - **LEFT JOIN**: Keeps all records from the first (left) stream - **RIGHT JOIN**: Keeps all records from the second (right) stream - Choose based on which dataset you want to preserve completely ### Q: Can I join more than 2 streams? **A**: Yes! Click the **+** button in the Joins section to add additional joins. You can chain multiple joins together. ### Q: Do nested functions impact performance? **A**: Complex nested functions may take longer to compute, especially on large datasets. Test with small time ranges first and optimize your expressions. ### Q: Can I use Custom SQL instead of Auto SQL? **A**: Yes! If you prefer writing raw SQL, switch to **Custom SQL** mode. However, Auto SQL provides a visual interface that's easier for building queries. --- ## Additional Resources 📚 **DataFusion Documentation**: - [Aggregate Functions](https://datafusion.apache.org/user-guide/sql/aggregate_functions.html) - COUNT, SUM, AVG, etc. - [Scalar Functions](https://datafusion.apache.org/user-guide/sql/scalar_functions.html) - ROUND, CONCAT, CASE, etc. 🔗 **OpenObserve Resources**: - [GitHub Repository](https://github.com/openobserve/openobserve) - [PR #9343 - Feature Implementation](https://github.com/openobserve/openobserve/pull/9343) - [OpenObserve Documentation](https://openobserve.ai/docs/) 💬 **Get Help**: - [GitHub Issues](https://github.com/openobserve/openobserve/issues) - Report bugs or request features - [Community Discussions](https://github.com/openobserve/openobserve/discussions) - Ask questions and share ideas --- Design at: https://github.com/openobserve/designs/tree/main/joins --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Shrinath Rao <shnath@openobserve.ai> |
||
|
Neha PandClaude
|
52b2759bfa |
test: add INTERSECT and EXCEPT query tests for PR #9544 (#9601)
Added 4 new test cases to verify the join optimization fix: - INTERSECT query: Common user_ids in both streams - EXCEPT query: Users without orders - INTERSECT with WHERE clause: Active users with orders - EXCEPT with multiple columns: Users details without orders Also improved test_joinqueries.py by adding a 3-second sleep after data ingestion to allow proper indexing before querying. Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Neha PandClaude
|
be9aceb881 |
fix: add unique names for alert test cleanup (#9591)
- Add 2 second wait after alert deletion before deleting destination to allow backend cleanup to propagate - Use timestamp-based unique names for template, destination, and alert to avoid conflicts between parallel test runs - Fixes 409 error: "Destination is currently used by alert" --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
priyabrata-stack
|
2a03b2a57d | add pipeline null condition test | ||
|
Shrinath Rao
|
ffc265ceba | test: added auto rerun step to e2e yml (#9466) | ||
|
udev
|
aff58b33a4 | fix: use chrono apis to validate time (#9422) | ||
|
Neha P
|
8a158884c8 |
test: match_all_camelcase_tests (#9345)
- Verify full camel case token search: DbException should be found when searching for complete camel case identifiers - Verify atomic token search: UserAccountService should be searchable and return matching results for full service class names - Verify XML acronym tokenization: XMLHttpRequest should be properly tokenized and discoverable through match_all queries - Verify camel case with numbers: OAuth2TokenHandlershould handle mixed alphanumeric camel case tokens correctly - Verify multi-word service search:UserManagementService should support complex multi-word camel case service names |
||
|
Shrinath Rao
|
a4dd576df4 | test: unskipped 2 pipeline-core tests (#9085) | ||
|
Subhra264andShrinath Rao
|
683bf89c7b |
test: Add tests for alerts and pipelines history (#9064)
Co-authored-by: Shrinath Rao <shnath@openobserve.ai> |