starred/openobserve
1
0
Fork 0
mirror of https://github.com/openobserve/openobserve.git synced 2026-11-06 00:09:22 +00:00
Code Issues Packages Projects Releases Wiki Activity
8,345 Commits 629 Branches 244 Tags
database_docker_k8s_setup
Commit Graph
7 Commits
Author SHA1 Message Date
Omkar Kesarkhane d522a37112 test: Playwright e2e suite for Synthetics (#14583) 2026-10-14 10:37:03 +00:00
Vamsi Karthik RandShrinath Rao 88e7dabb24 test(alpha1): document how the alpha cloud matrix is decided (#14778) 
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
 2026-09-23 08:24:01 +00:00
Ashish KolheandShrinath Rao 226b401da8 ci: changed-path smoke selection for Playwright PR runs (#14107) 
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
 2026-09-02 13:14:04 +00:00
Vamsi Karthik R 7b79fdcdbe ci(playwright): stop SLO-Measurement backfill from serializing 21 daily chunks (#13870) 
## What this fixes

The `SLO-Measurement` shard (`slo-timeslice.spec.js` +
`slo-alerts.spec.js`) has been taking **~26 minutes**

## Root cause

Both specs seed **8 days of history** and create SLOs with a **7-day
rolling window**, then block in `beforeAll` until the SLO is actually
measured (`waitForSloMeasured`). The server backfills history in
`ZO_SLO_BACKFILL_CHUNK_SECS` chunks — default **86400s (1 day)** — and
only **one chunk at a time** (`ZO_SCHEDULER_SLO_BACKFILL_CONCURRENCY=1`,
by design, so bulk backfill never starves real-time alerts).

The shard backfills **3 SLOs** across the two specs, each needing 7
daily chunks to cover its 7-day window:

- `slo-timeslice.spec.js` `beforeAll`: 2 SLOs (`sloLt`, `sloGt`) × 7
chunks = 14
- `slo-alerts.spec.js` `beforeAll`: 1 SLO × 7 chunks = 7

**21 chunks total, strictly serial**, at roughly 40s/chunk ≈ **~14
minutes of pure backfill wait**, before any UI assertion runs. This is
the server behaving correctly and exactly as designed for production —
the test is just paying for it the slow way.

## Fix

Neither spec asserts on per-chunk timing or partial coverage, only on
the final measured state, so raising the chunk size to cover the whole
window in one pass is free wall-clock, not a behavior change.

- Threaded `slo_backfill_chunk_secs` through `build-ci-matrix.js`'s
existing per-shard passthrough, same pattern already used for
`ingest_allowed_upto` / `quick_mode_enabled`.
- Set it to `604800` (7 days) for the `SLO-Measurement` shard only — the
server-wide default (`86400`) is untouched everywhere else.
- Documented the new knob in `tests/ui-testing/ci-matrix/README.md`.

## Validation

**Synthetic (API-only), same seed shape as the real specs** (8 days,
1-min interval, deterministic 3h fault window), against a local build:

| | Default (86400s chunk) | Fixed (604800s chunk) |
|---|---|---|
| Time to measured | 246.9s (~4.1 min) for **one** SLO, stepping 1/7 →
2/7 → ... every ~40s | **3.0s** |
| Final SLI | 98.21428571428571% | 98.21428571428571% (**identical**) |

Same math, same result — only the number of scheduler round-trips
changes.

**Real shard, end-to-end**, run exactly as CI invokes it (`npx
playwright test slo-timeslice.spec.js slo-alerts.spec.js --workers=1`)
against a server mirroring this shard's full env including the fix:

- **25/25 tests passed**, 0 regressions
- Total wall-clock (server start → global teardown): **4m Avg**

Local timing won't map 1:1 onto the GitHub Actions runner (different
disk/scheduler characteristics — even the "before" baseline was faster
locally than the ~26min/~5min-per-spec seen in CI), so the authoritative
number is whatever this PR's own CI run reports. But the structural win
— 1 chunk instead of 7 per SLO — is verified correct and regression-free
independent of machine.

## Companion PR

The enterprise repo's `playwright.yml` independently consumes per-shard
matrix fields (it explicitly mirrors OSS's `ingest_allowed_upto`
handling for the same reason), so it needs the equivalent env
passthrough or it'll keep paying the old cost even after this merges.
Companion PR: openobserve/o2-enterprise#2460
 2026-08-19 00:58:33 +00:00
Vamsi Karthik R 99b1a23773 test(slo): E2E + API suite for SLOs (#13836) 
Adds automated test coverage for **SLOs** (#13547, #13577, #13579,
#13704, #13761, #13784), which shipped with none. **149 tests** — 77
Playwright + 72 pytest — all three SLI types, both query languages,
happy / negative / edge paths. Clean branch off `main`.

**UI (Playwright · `tests/ui-testing`)**
- `pages/sloPages/` — 4 page objects (verified `data-test` selectors,
hydration + Monaco-sync waits, OSelect commit-retry, virtualized-list
filtering).
- `playwright-tests/SLO/slo-crud.spec.js` (24) — create
count/time-slice/grouped through the form, detail, edit, SLI-type
change, tags, pause, delete, filters; negative: dismissed delete, empty
search, cancel, unmeasured SLI is absent not zero.
- `playwright-tests/SLO/slo-form-validation.spec.js` (20) — server
errors surfacing in the form, conditional fields per SLI type, count +
time-slice previews, **PromQL count and time-slice created end to end**;
negative: no stream, invalid aggregate, over-long name, target
boundaries.
- `playwright-tests/SLO/slo-alert-sli.spec.js` (8) — **the `alert` SLI
type**: source picker, create, config carries only the alert id, detail
names the source; negative: ineligible source offered-but-disabled,
grouped alert SLI, slice finer than source cadence.
- `playwright-tests/SLO/slo-timeslice.spec.js` (10) — **#13761 P0
regression** (measured SLI, not the preview), PromQL, previews, gaps,
language-flip clearing.
- `playwright-tests/SLO/slo-alerts.spec.js` (15) — burn-rate +
error-budget kinds, presets, edit, freeze door, burndown; negative: no
destination, cancel, blank/invalid name.
- `playwright-tests/utils/slo-seed.js` — backdated seeding, metrics
seeding, eligible-source-alert fixture, ground-truth query, fail-fast
env guard, transient-store retry, full fixture teardown (alerts →
streams → destinations → templates).

**API (pytest · `tests/api-testing`)**
- `tests/slos/test_slo_validation.py` (72) — full REST surface
(create/get/update/enable/delete/groups/slo-eligible), validation on
**both create and update** (name, target, window, slice, config shapes,
comparators, float-threshold round-trip), boundaries (target
0.001/99.999, name 256 vs 257, threshold 0/negative/1e12/NaN, unicode
names), alert-source eligibility (silence-gated, too-infrequent, cron),
PromQL config shapes, auth 401/404.

**CI**
- Two shards: `SLO` (`workers: 3`) and `SLO-Measurement` (`workers: 1`).
Split because the backfill job runs with
`ZO_SCHEDULER_SLO_BACKFILL_CONCURRENCY=1`, so measurement specs sharing
an instance starve each other's waits — `fullyParallel` races spec
*files*, which `mode:'serial'` cannot prevent. The 3-worker pin is
SQLite: at 5, concurrent SLO/alert/destination writes trip `database is
locked` on the single-node meta store.
- New optional per-shard `ingest_allowed_upto` (`240`) alongside the
existing `quick_mode_enabled`: the workflow default of 5 **hours**
silently drops the 8 days of history an SLO window needs, while still
answering `200`. Other shards keep the default. Same precedent as
`api-testing.yml`'s `48`.

**Feature code**: 9 additive `data-test`/`dataTest` attributes in
`SloDetail.vue`, `SloList.vue`, `SloAlertCondition.vue` — no logic
changes, `vue-tsc` clean. The stat tiles, tabs and burn-window inputs
had no stable selectors; `OStatStrip`/`OTab` already forward the
attribute.

**Behaviour**: self-cleaning (`e2e_slo_*`, worker-scoped since
`afterAll` runs per worker).

**Verified in CI** on this branch: `e2e / SLO` **52 passed** (1.8m,
workers pinned to 3), `e2e / SLO-Measurement` **25 passed** (11.3m,
workers pinned to 1) — the measurement shard passing confirms the
per-shard `ingest_allowed_upto` took effect, since those tests are
impossible under the 5-hour default. pytest **72 passed** locally; in
CI's 599-test run one setup delete hit a transient 500 (see findings)
and now retries.

**Findings for the feature owner** (reported, not fixed here):
- A blank "good when" shows `Request failed with status code 422` —
`pruned()` drops the empty key, so the API's real 400 message never
reaches `save()`.
- A rejected alert name shows no banner: `SloAlertForm.submit()` returns
on `nameError` before setting `saveError`.
- `avg(latency_ms); DROP TABLE users--` is accepted at create time
(parsed only at measurement).
- The name-length message says "less than 256 characters" but 256 is
accepted — 257 is the first rejection.
- Duplicate alert names on one SLO are accepted (duplicate SLO names are
a 409), and the panel lists them by name, so they are indistinguishable.
- SLO names accept characters that break the list's own
`slos-slolist-edit-${name}` selectors; alert names correctly reject
them.
- All chart/preview root `data-test` ids are unreachable — a parent's
fallthrough attribute overrides the child root's.
- Both edit forms hydrate asynchronously, so a blank form can flash
before the stored values appear.
- `DELETE /slos/{id}` on a freshly-created SLO returned **500** once
under CI load (not reproducible locally in 12 rapid iterations).
Contention is arguably a retryable 503, and the body carried no
actionable message.
- **A scheduled alert whose source stream is deleted is retried
forever** (`Stream "…" not found in schema`), and the SLO backfill lane
shares the scheduler's worker budget — so accumulated dead alerts
*starve SLO measurement*. Observed: 65 of them held coverage at 1 of 7
chunks indefinitely; deleting them restored 0.143 → 1.0 in 240s.
Harmless in CI (fresh instance per shard) but it degrades any long-lived
environment. The suite now cleans up after itself; the retry-forever
behaviour itself may deserve a look.

**Paired ENT PR: openobserve/o2-enterprise#2434.** ENT's matrix is this
repo's manifest plus an overlay, merged by this repo's
`build-ci-matrix.js`, so both SLO shards reach ENT automatically —
verified: 31 merged shards, both SLO entries carrying
`ingest_allowed_upto=240` and their worker pins, no other shard
affected. What ENT was missing is *consuming* those two fields, which
#2434 adds. Merge the two together: this one alone leaves ENT booting
the SLO shards on the 5-hour ingest window, and #2434 alone is inert.

*(An earlier review reported ENT as using an inline matrix with no SLO
shard — that turned out not to be the case.)*
 2026-08-17 18:50:29 +00:00
ktx-akshay 7999fd33f3 test: enhance logs visualization error handling and persistence tests (#13591) 
## Summary

Adds **10 E2E tests** covering the two defects fixed in #13244, plus the
`SELECT *` behaviour change that shipped alongside them. Also introduces
per-shard backend configuration so the `SELECT *` tests can actually run
in CI.

## Test coverage

### `logsVisualizePersistence.spec.js` — 7 tests (quick mode **off**)

| # | Test | Defect | Fix code exercised |
|---|---|---|---|
| 1 | Renders without the false required-fields error | A |
`convertPanelData.ts` `!query?.customQuery` |
| 2 | Restores chart type, query and chart after URL refresh (h-bar) | B
| `setupLogsTab` re-run + `isInitialVisualizeRestore` |
| 3 | Re-renders after reload with the default chart type | B |
permanent No-Data half |
| 4 | Restores a non-default chart type (area) after refresh | B |
`restoreVisualizationFromUrlOnLoad()` |
| 5 | Chart still renders after reload with a VRL function applied | B |
`prepareVisualizeMode()` VRL copy |
| 6 | Never shows required-fields error toggling Visualize → Logs →
Visualize | A | manual-toggle rebuild path |
| 7 | No `SELECT *` toast opening Visualize with no stream in the URL |
B | page-load ordering guard |

**Defect A** — false "Please select required fields to render the chart"
on the custom-query Timechart panel.
**Defect B** — URL refresh produced a `SELECT *` toast plus permanent No
Data.

### `logsVisualizeSelectStar.spec.js` — 3 tests (quick mode **on**,
dedicated shard)

| # | Test | Behaviour verified | Call site |
|---|---|---|---|
| 8 | Table chart on `SELECT *` → toast shown **and** chart type reverts
| Table is blocked | `Index.vue:2265` (watcher) |
| 9 | Histogram chart on `SELECT *` → no toast, renders |
Line/bar/area/scatter allowed | entry path |
| 10 | `SELECT *` line chart survives browser refresh → no toast, state
restored | Restore path doesn't re-emit | `Index.vue:2465` (restore) |

## Infrastructure changes

| File | Change | Why |
|---|---|---|
| `ci_matrix.json` | New `Logs-SelectStar` shard with
`quick_mode_enabled: "true"` | `isSelectStarForTable()` only exists when
the server runs with quick mode on |
| `playwright.yml` | Per-shard `ZO_QUICK_MODE_ENABLED` env on `Start
OpenObserve` | The flag is read once at server start — a shard is the
smallest unit that can vary it |
| `build-ci-matrix.js` | Emit `quick_mode_enabled` (defaults `"false"`)
| The field was being silently stripped by the emit whitelist |
| `logsPage.js` | 13 assertion helpers + fixed chart-type detection |
Detection read `data-selected` off the wrong element and matched retired
CSS classes, so it always returned `null` |

All 24 existing shards are unaffected — they omit the field and default
to `false`.

## Notes on test quality

- Every negative assertion ("no toast", "no error") is sequenced after a
deterministic positive end-state, and gated on a precondition so it
cannot pass vacuously.
- Toast absence is proved with a MutationObserver recorder installed
before the navigation under test, since toasts auto-dismiss.
- The quick-mode precondition is a hard assertion, not `test.skip` — a
misconfigured shard fails loudly instead of silently reporting
"skipped".
 2026-08-04 12:09:26 +00:00
Shrinath Rao d76c0a6bbb ci(playwright): single-source ALL shard matrices via ci-matrix/ manifests (#13606) 
## Why
The OSS and ENT Playwright shard matrices (`run_files` per `testfolder`)
were hand-maintained inline in each `playwright.yml` and had to be
synced on every spec add — error-prone and drift-prone (ENT was even
missing `dashboard-favorites`). This makes the shard list a **single
source of truth** for **every** Playwright workflow.

## What (OSS side)
All matrices are generated at run time from JSON manifests in
`tests/ui-testing/ci-matrix/`:

- **`ci-matrix/ci_matrix.json`** — PR-gate base (24 shards). The only
place shared shards live.
- **`ci-matrix/ci_matrix_regression.json`** — regression base (10
shards); `RegressionSet/<Feature>` mapping now carried in
`actual_folder`.
- **`.github/scripts/build-ci-matrix.js`** — merges base (+ optional ENT
overlay) → GitHub matrix; validates uniqueness / no dup specs / no spec
both active+disabled. Reused by ENT from its OSS checkout.
- **`ci-matrix/README.md`** — manifest index + how to add/move/disable a
spec.
- Workflows `playwright.yml` + `playwright_regression.yml`: new
`generate_matrix` job → `strategy.matrix: fromJSON(...)`; `case` blocks
→ `${{ matrix.actual_folder }}`.

Disabling a spec (JSON has no `//`): move it into that shard's
`disabled: [{file, reason}]` array — never emitted (won't run), record
survives. Every shard ships a `"disabled": []` placeholder.

## Safety (verified)
- **OSS `playwright.yml` and `playwright_regression.yml` matrices are
byte-identical to before** — zero behavior change.
- Companion ENT PR (same branch slug
`test/playwright-matrix-single-source`) derives its matrices from these
manifests + overlays.

## Validate
Add the `e2e` label; confirm `generate_matrix` prints the shard count
and the `e2e /` jobs fan out.
 2026-08-03 17:36:29 +00:00