Automated coverage for **email alert destinations and distribution
lists**, and unblocks the email tests that have been skipped since they
were written. **32 cases** — 27 active, 5 skipped regressions tracking
open defects (openobserve/o2-enterprise#2471).
**The unblocking.** The two skipped tests said *"the test environment
does not have email infrastructure set up"*. That over-stated it:
creating, editing, validating and deleting an email destination needs
SMTP switched **on**, and never reads a mailbox. Only delivery
assertions do. The suite is tiered on that:
| Tier | Needs | Cases |
|---|---|---|
| A | SMTP enabled | 24 |
| B | a readable sink | 2 |
| C | an alias that fans out | 1 |
**Tests**
- `Alerts/dl-email-destinations.spec.js` (32) — the DL half (alias
accepted, addressed, fanned out) and the recipient-list half, where our
code actually runs: splitting/trimming/lowercasing, the org-membership
gate, edit round-trips, add/remove/delete, both form paradigms (prebuilt
free-text vs the custom path's org-user picker), template auto-attach,
duplicate names, cancel, type-switching, keyboard, dark mode at 420px.
- `utils/mail-sink.js` — pluggable sink: Mailpit, a Mailinator inbox, or
a clean skip with a stated reason.
- `utils/o2-api.js` — storage is asserted from the API, never the form
field: an input can echo what you typed while the stored value differs.
- `scripts/dl-relay.js` — SMTP relay that expands one alias into
per-member deliveries, so fan-out runs with no external provider. Same
pattern as the in-test HTTP receiver in
`alerts-content-templates.spec.js`.
- `alertDestinationsPage.js` — +17 methods; the spec carries zero raw
locators.
**Replaces, without losing coverage.** The two skipped tests in
`alerts-destinations-prebuilt.spec.js` are removed for a pointer mapping
their intent to its new home: create (`D-01`), edit-add (`D-07`),
edit-remove (`ML-08`), delete (`UI-12`), format validation (`UI-02`).
Three of those did not exist before. That file now has no skips.
**CI.** `ZO_SMTP_ENABLED: true` was already set in `playwright.yml` but
had no host or port, so it fell back to `localhost:25` where nothing
listens — destinations could be **saved** while every **send** failed
silently. The workflow now starts Mailpit and a DL relay as static
binaries before the app (no Docker-in-Docker on these runners) and
points `ZO_SMTP_*` at them.
All three tiers run with no secrets and no egress. Verified in the CI
topology: **27 passed, 5 skipped**, `D-02` confirmed reaching all three
member mailboxes.
**Behaviour.** Self-cleaning via the `auto_dest_dl_*` prefix
`cleanup.spec.js` already sweeps, plus an API `afterAll`. `mode:
'parallel'` — each test owns its fixtures.
**Findings** are filed in openobserve/o2-enterprise#2471, not fixed
here. Two P1s: the *"Show matching rows"* toggle is ignored so log rows
are always embedded, and a user removed from the org keeps receiving
alerts while the destination can no longer be saved. The regression
tests for each are in this PR, skipped until the fix lands.
**Paired ENT PR: openobserve/o2-enterprise#2472.** ENT's workflow sets
`ZO_SMTP_ENABLED: false`, so without it the suite skips all 27 cases
there. The shard matrix needs no ENT change — ENT builds it from this
repo's `ci_matrix.json` plus its overlay, so the new spec is picked up
automatically. Merge the two together: this one alone leaves enterprise
with no email coverage.
## What
Heals the two Traces-Regression tests that broke after #13852
(\`fix(ui): clean up SLO detail and traces navigation\`).
Failing scheduled runs:
- ENT:
https://github.com/openobserve/o2-enterprise/actions/runs/32213206401/job/95953026991
— \`traces-ent.spec.js\` Service Graph test
- OSS:
https://github.com/openobserve/openobserve/actions/runs/32213278081 —
\`traces-regression.spec.js\` Service Catalog test
Both failed all 4 retries with \`locator.click: Timeout 45000ms
exceeded\` on the left-rail flyout item.
## Root cause
#13852 rerouted the Traces rail-flyout children through the Traces
page's in-page \`?tab=\` views. That changed **both**:
| | before | after |
|---|---|---|
| flyout \`data-test\` | \`nav-group-item-serviceGraph\` |
\`nav-group-item-traces-service-graph\` |
| | \`nav-group-item-servicesCatalog\` |
\`nav-group-item-traces-services-catalog\` |
| URL | \`/traces/service-graph\` | \`/traces?tab=service-graph\` |
| | \`/traces/services\` | \`/traces?tab=services-catalog\` |
(\`childDataTest = nav-group-item-\${child.name}-\${child.tab}\`; legacy
paths now only exist as redirects.) The stale \`data-test\` made
\`.click()\` wait the full 45s action timeout.
## Fix
\`serviceGraphPage.navigateToServiceGraph()\` and
\`servicesCatalogPage.clickServiceCatalogTab()\`:
- point at the new flyout \`data-test\` ids
- \`waitFor\` the teleported flyout item to be **visible** before
clicking — the flyout opens after ONavGroup's 120ms \`OPEN_DELAY\`, so
the item isn't in the DOM the instant \`hover()\` resolves
- match the new \`/traces?tab=\` URL
Engine/other nav helpers untouched.
## Verification
Run locally against a #13852 frontend (localhost:5080, enterprise
build):
- ✅ Live flyout click →
\`/web/traces?org_identifier=default&tab=service-graph\` and
\`...&tab=services-catalog\`
- ✅ **OSS** \`traces-regression.spec.js\` "Clearing search filter should
not blank out Service Catalog table" — **passes**
- ✅ **ENT** \`traces-ent.spec.js\` Service Graph nav now works — test
reaches the chart assertion (line 54); the only residual local failure
is absent service-graph topology (0 nodes locally; CI seeds it via
ingestion + the daemon)
## Follow-up (not in this PR / not in any CI matrix)
\`Traces/traces-toolbar-tabs.spec.js\` +
\`tracesPage.switchToServiceMaps()\` still assert the pre-#13852 "flyout
→ standalone route" dual-semantics that #13852 removed. Not run by any
CI matrix, so left for a separate change aligned with the new
single-\`?tab=\` semantics.
## What this fixes
The `SLO-Measurement` shard (`slo-timeslice.spec.js` +
`slo-alerts.spec.js`) has been taking **~26 minutes**
## Root cause
Both specs seed **8 days of history** and create SLOs with a **7-day
rolling window**, then block in `beforeAll` until the SLO is actually
measured (`waitForSloMeasured`). The server backfills history in
`ZO_SLO_BACKFILL_CHUNK_SECS` chunks — default **86400s (1 day)** — and
only **one chunk at a time** (`ZO_SCHEDULER_SLO_BACKFILL_CONCURRENCY=1`,
by design, so bulk backfill never starves real-time alerts).
The shard backfills **3 SLOs** across the two specs, each needing 7
daily chunks to cover its 7-day window:
- `slo-timeslice.spec.js` `beforeAll`: 2 SLOs (`sloLt`, `sloGt`) × 7
chunks = 14
- `slo-alerts.spec.js` `beforeAll`: 1 SLO × 7 chunks = 7
**21 chunks total, strictly serial**, at roughly 40s/chunk ≈ **~14
minutes of pure backfill wait**, before any UI assertion runs. This is
the server behaving correctly and exactly as designed for production —
the test is just paying for it the slow way.
## Fix
Neither spec asserts on per-chunk timing or partial coverage, only on
the final measured state, so raising the chunk size to cover the whole
window in one pass is free wall-clock, not a behavior change.
- Threaded `slo_backfill_chunk_secs` through `build-ci-matrix.js`'s
existing per-shard passthrough, same pattern already used for
`ingest_allowed_upto` / `quick_mode_enabled`.
- Set it to `604800` (7 days) for the `SLO-Measurement` shard only — the
server-wide default (`86400`) is untouched everywhere else.
- Documented the new knob in `tests/ui-testing/ci-matrix/README.md`.
## Validation
**Synthetic (API-only), same seed shape as the real specs** (8 days,
1-min interval, deterministic 3h fault window), against a local build:
| | Default (86400s chunk) | Fixed (604800s chunk) |
|---|---|---|
| Time to measured | 246.9s (~4.1 min) for **one** SLO, stepping 1/7 →
2/7 → ... every ~40s | **3.0s** |
| Final SLI | 98.21428571428571% | 98.21428571428571% (**identical**) |
Same math, same result — only the number of scheduler round-trips
changes.
**Real shard, end-to-end**, run exactly as CI invokes it (`npx
playwright test slo-timeslice.spec.js slo-alerts.spec.js --workers=1`)
against a server mirroring this shard's full env including the fix:
- **25/25 tests passed**, 0 regressions
- Total wall-clock (server start → global teardown): **4m Avg**
Local timing won't map 1:1 onto the GitHub Actions runner (different
disk/scheduler characteristics — even the "before" baseline was faster
locally than the ~26min/~5min-per-spec seen in CI), so the authoritative
number is whatever this PR's own CI run reports. But the structural win
— 1 chunk instead of 7 per SLO — is verified correct and regression-free
independent of machine.
## Companion PR
The enterprise repo's `playwright.yml` independently consumes per-shard
matrix fields (it explicitly mirrors OSS's `ingest_allowed_upto`
handling for the same reason), so it needs the equivalent env
passthrough or it'll keep paying the old cost even after this merges.
Companion PR: openobserve/o2-enterprise#2460
Adds automated test coverage for **SLOs** (#13547, #13577, #13579,
#13704, #13761, #13784), which shipped with none. **149 tests** — 77
Playwright + 72 pytest — all three SLI types, both query languages,
happy / negative / edge paths. Clean branch off `main`.
**UI (Playwright · `tests/ui-testing`)**
- `pages/sloPages/` — 4 page objects (verified `data-test` selectors,
hydration + Monaco-sync waits, OSelect commit-retry, virtualized-list
filtering).
- `playwright-tests/SLO/slo-crud.spec.js` (24) — create
count/time-slice/grouped through the form, detail, edit, SLI-type
change, tags, pause, delete, filters; negative: dismissed delete, empty
search, cancel, unmeasured SLI is absent not zero.
- `playwright-tests/SLO/slo-form-validation.spec.js` (20) — server
errors surfacing in the form, conditional fields per SLI type, count +
time-slice previews, **PromQL count and time-slice created end to end**;
negative: no stream, invalid aggregate, over-long name, target
boundaries.
- `playwright-tests/SLO/slo-alert-sli.spec.js` (8) — **the `alert` SLI
type**: source picker, create, config carries only the alert id, detail
names the source; negative: ineligible source offered-but-disabled,
grouped alert SLI, slice finer than source cadence.
- `playwright-tests/SLO/slo-timeslice.spec.js` (10) — **#13761 P0
regression** (measured SLI, not the preview), PromQL, previews, gaps,
language-flip clearing.
- `playwright-tests/SLO/slo-alerts.spec.js` (15) — burn-rate +
error-budget kinds, presets, edit, freeze door, burndown; negative: no
destination, cancel, blank/invalid name.
- `playwright-tests/utils/slo-seed.js` — backdated seeding, metrics
seeding, eligible-source-alert fixture, ground-truth query, fail-fast
env guard, transient-store retry, full fixture teardown (alerts →
streams → destinations → templates).
**API (pytest · `tests/api-testing`)**
- `tests/slos/test_slo_validation.py` (72) — full REST surface
(create/get/update/enable/delete/groups/slo-eligible), validation on
**both create and update** (name, target, window, slice, config shapes,
comparators, float-threshold round-trip), boundaries (target
0.001/99.999, name 256 vs 257, threshold 0/negative/1e12/NaN, unicode
names), alert-source eligibility (silence-gated, too-infrequent, cron),
PromQL config shapes, auth 401/404.
**CI**
- Two shards: `SLO` (`workers: 3`) and `SLO-Measurement` (`workers: 1`).
Split because the backfill job runs with
`ZO_SCHEDULER_SLO_BACKFILL_CONCURRENCY=1`, so measurement specs sharing
an instance starve each other's waits — `fullyParallel` races spec
*files*, which `mode:'serial'` cannot prevent. The 3-worker pin is
SQLite: at 5, concurrent SLO/alert/destination writes trip `database is
locked` on the single-node meta store.
- New optional per-shard `ingest_allowed_upto` (`240`) alongside the
existing `quick_mode_enabled`: the workflow default of 5 **hours**
silently drops the 8 days of history an SLO window needs, while still
answering `200`. Other shards keep the default. Same precedent as
`api-testing.yml`'s `48`.
**Feature code**: 9 additive `data-test`/`dataTest` attributes in
`SloDetail.vue`, `SloList.vue`, `SloAlertCondition.vue` — no logic
changes, `vue-tsc` clean. The stat tiles, tabs and burn-window inputs
had no stable selectors; `OStatStrip`/`OTab` already forward the
attribute.
**Behaviour**: self-cleaning (`e2e_slo_*`, worker-scoped since
`afterAll` runs per worker).
**Verified in CI** on this branch: `e2e / SLO` **52 passed** (1.8m,
workers pinned to 3), `e2e / SLO-Measurement` **25 passed** (11.3m,
workers pinned to 1) — the measurement shard passing confirms the
per-shard `ingest_allowed_upto` took effect, since those tests are
impossible under the 5-hour default. pytest **72 passed** locally; in
CI's 599-test run one setup delete hit a transient 500 (see findings)
and now retries.
**Findings for the feature owner** (reported, not fixed here):
- A blank "good when" shows `Request failed with status code 422` —
`pruned()` drops the empty key, so the API's real 400 message never
reaches `save()`.
- A rejected alert name shows no banner: `SloAlertForm.submit()` returns
on `nameError` before setting `saveError`.
- `avg(latency_ms); DROP TABLE users--` is accepted at create time
(parsed only at measurement).
- The name-length message says "less than 256 characters" but 256 is
accepted — 257 is the first rejection.
- Duplicate alert names on one SLO are accepted (duplicate SLO names are
a 409), and the panel lists them by name, so they are indistinguishable.
- SLO names accept characters that break the list's own
`slos-slolist-edit-${name}` selectors; alert names correctly reject
them.
- All chart/preview root `data-test` ids are unreachable — a parent's
fallthrough attribute overrides the child root's.
- Both edit forms hydrate asynchronously, so a blank form can flash
before the stored values appear.
- `DELETE /slos/{id}` on a freshly-created SLO returned **500** once
under CI load (not reproducible locally in 12 rapid iterations).
Contention is arguably a retryable 503, and the body carried no
actionable message.
- **A scheduled alert whose source stream is deleted is retried
forever** (`Stream "…" not found in schema`), and the SLO backfill lane
shares the scheduler's worker budget — so accumulated dead alerts
*starve SLO measurement*. Observed: 65 of them held coverage at 1 of 7
chunks indefinitely; deleting them restored 0.143 → 1.0 in 240s.
Harmless in CI (fresh instance per shard) but it degrades any long-lived
environment. The suite now cleans up after itself; the retry-forever
behaviour itself may deserve a look.
**Paired ENT PR: openobserve/o2-enterprise#2434.** ENT's matrix is this
repo's manifest plus an overlay, merged by this repo's
`build-ci-matrix.js`, so both SLO shards reach ENT automatically —
verified: 31 merged shards, both SLO entries carrying
`ingest_allowed_upto=240` and their worker pins, no other shard
affected. What ENT was missing is *consuming* those two fields, which
#2434 adds. Merge the two together: this one alone leaves ENT booting
the SLO shards on the 5-hour ingest window, and #2434 alone is inert.
*(An earlier review reported ENT as using an inline matrix with no SLO
shard — that turned out not to be the case.)*
## What
Hide **LLM Model Pricing** on OSS builds and make it an
enterprise/cloud-only feature, end to end (UI + e2e coverage). Paired
with o2-enterprise PR on branch `fix/hide-llm-model-pricing-oss`.
## Why
Model Pricing was surfaced on OSS (settings rail + navbar flyout), gated
only on the `model_pricing_enabled` backend flag with **no**
enterprise/cloud check — unlike its sibling AI feature **LLM Providers**
(`(isEnterprise || isCloud) && online_evals_enabled`). Product decision:
it should not appear on OSS. This brings it in line with that sibling.
## Changes
**Web (hide on OSS)**
- `web/src/components/settings/index.vue` — settings rail item now
`visible: (isEnt || isCloud) && !!z.model_pricing_enabled`.
- `web/src/lib/core/Navbar/navGroups.ts` — `modelPricing` gate predicate
now requires enterprise/cloud (mirrors `llmProviders`), so the
collapsed-navbar flyout matches the rail.
- `web/src/composables/shared/useManagementRoutes.ts` — `model_pricing`
+ `model_pricing/edit` routes registered only inside the `isEnterprise
|| isCloud` block (defense-in-depth; matches `llm_providers`), so the
route isn't reachable by direct URL on OSS.
**E2E (move coverage to ENT)**
- `tests/ui-testing/ci-matrix/ci_matrix.json` — drop
`model-pricing.spec.js` from the OSS `GeneralTests` shard. It's added
back on ENT via the enterprise overlay (`ci_matrix.ent.json`, in the
paired ENT PR), so the merged ENT matrix runs it exactly once (verified
with `build-ci-matrix.js`).
- `.github/workflows/playwright.yml` — remove `ZO_MODEL_PRICING_ENABLED:
true` (OSS no longer exercises the feature; ENT `playwright.yml` keeps
it).
-
`tests/ui-testing/playwright-tests/GeneralTests/settings-form-validation.spec.js`
— gate the *ModelPricingEditor* describe block to a clean SKIP on OSS,
mirroring the existing Domain Management / Org Storage / Correlation
blocks.
- `tests/ui-testing/pages/generalPages/settingsFormValidationPage.js` —
`navigateToModelPricing()` now returns a boolean so the suite can detect
the OSS build and skip (mirrors `navigateToDomainManagement()`).
## Not changed (deliberately)
The backend `ZO_MODEL_PRICING_ENABLED` default (`true`,
[config.rs](src/config/src/config.rs)) is **left as-is**. It lives in
the shared config crate used by both editions, so flipping it to `false`
would also regress real ENT/Cloud deployments (feature off by default).
The UI gate already ensures the page never shows on OSS regardless of
the flag. Making the backend itself enterprise-only would be a separate,
backend-owned change.
## Behavior
| Build | Model Pricing UI | e2e |
|-------|------------------|-----|
| OSS | Hidden | `model-pricing.spec.js` not run;
settings-form-validation block SKIPPED |
| Enterprise / Cloud | Shown (if flag set) — unchanged |
`model-pricing.spec.js` runs (via ENT overlay) |
## Stabilize the Alpha1 (cloud) e2e suite + regression carryover
Root-caused and fixed the failures surfacing on the Alpha1 cloud runs —
investigated live via Playwright MCP against alpha and, where possible,
a local build. Grouped by theme. Every fix is non-masking (still fails
if the underlying is genuinely broken). Pairs with ENT #2351
(`max-parallel` + Logs-SelectStar `quick_mode`).
### Runtime / infra resilience (whole-suite)
- **Auth self-heal on 401 / session expiry** — the shared cloud session
+ per-org passcode are minted once and reused all run; on
load/contention they expire mid-run (`User not authenticated`, `HTTP 401
Unauthorized`). `cloud-auth.authedRequest` refreshes the passcode and
retries on 401 (escalating to a full Dex re-login if the session is
dead); `navigateToBase` re-authenticates and resumes instead of
throwing. Also routed the alert **template-creation** API through it
(401 there cascaded into "Failed to navigate to templates"). Gated on
`isCloudEnvironment()` — no self-hosted change.
- **`--disable-dev-shm-usage`** on the alpha1 Chromium launch — the EKS
pods used the default 64MB `/dev/shm`; under 5 workers on long/heavy
shards Chromium exhausted it and the pod OOM-killed → "runner lost
communication" (frozen step + 404 logs). Route shared memory to /tmp;
eliminates the infra deaths.
- **Service graph queries a wide window** (`period=6h`) — the tab
defaulted to Past 15 Minutes and rendered "No service graph data" before
the daemon-processed topology fell inside the window.
### Pipelines
- `queryStream(stream, expectedMinCount)` **polls until the record count
lands** (pipeline output arrives async) — was single-shot ("Expected 5,
got 3").
- `openPipelineForEdit` **searches + polls-with-reload** for the edit
button (list not refreshed after save → 45s click timeout).
- `exploreStreamAndNavigateToPipeline` **reloads the streams list**
while polling for Explore, and **falls back to a direct logs URL** when
a freshly-ingested source stream never surfaces in the list
(stream-stats lag).
### Alerts
- Alert-destination URL → **public `example.com` webhook** (self-host
URL hit the cloud **SSRF guard** → form never closed).
- **Force-click the virtualized folder item** (#9484) — the row
re-rendered mid-click → 45s "not stable".
- **Export button readiness** — it's rendered only on selection; wait
for visible+enabled (re-selecting if needed) before clicking.
- **`deleteAlertByRow` cleanup readiness** — the just-created alert's
more-options kebab needs longer than 5s under load.
### RegressionSet
- **Use the shard's org** (`getOrgIdentifier()`), not hardcoded
`org_identifier=default`, in `logs-bugs` — on cloud the data lives in
`auto_org_N`, so the logs page never rendered.
- **Unique stream names** for `logs-9754` / `logs-bugs:1419` — fixed
names collided with concurrent shared-org runs ("stream is being
deleted").
### Traces / correlation
- **Migrated `logs-bugs:995` (bug-11469)** to the redesigned correlation
UI — the old `log-correlation-btn` / `correlation-dashboard-close` are
gone; repointed to the inline `correlated-traces-tab`, assertion
(`expectNoContextMenuVisible`) unchanged and **verified live on alpha**
(no fake-pass).
### Workflows (ENT-only) + misc
- **Healed the workflows destination flow** and forced-type
Connection-step assertion; alert/report list-readiness hardening.
- **Enrichment tables list** readiness (30s + re-nav retry) — "slow
under load".
### Notes
- Validated live via MCP (alpha) and a local build where the feature
exists.
- Alpha1 config already runs `retries: 2`; the remaining first-attempt
flakes are shared-env load, addressed on the ENT side via
`max-parallel`.
## Summary
Adds **10 E2E tests** covering the two defects fixed in #13244, plus the
`SELECT *` behaviour change that shipped alongside them. Also introduces
per-shard backend configuration so the `SELECT *` tests can actually run
in CI.
## Test coverage
### `logsVisualizePersistence.spec.js` — 7 tests (quick mode **off**)
| # | Test | Defect | Fix code exercised |
|---|---|---|---|
| 1 | Renders without the false required-fields error | A |
`convertPanelData.ts` `!query?.customQuery` |
| 2 | Restores chart type, query and chart after URL refresh (h-bar) | B
| `setupLogsTab` re-run + `isInitialVisualizeRestore` |
| 3 | Re-renders after reload with the default chart type | B |
permanent No-Data half |
| 4 | Restores a non-default chart type (area) after refresh | B |
`restoreVisualizationFromUrlOnLoad()` |
| 5 | Chart still renders after reload with a VRL function applied | B |
`prepareVisualizeMode()` VRL copy |
| 6 | Never shows required-fields error toggling Visualize → Logs →
Visualize | A | manual-toggle rebuild path |
| 7 | No `SELECT *` toast opening Visualize with no stream in the URL |
B | page-load ordering guard |
**Defect A** — false "Please select required fields to render the chart"
on the custom-query Timechart panel.
**Defect B** — URL refresh produced a `SELECT *` toast plus permanent No
Data.
### `logsVisualizeSelectStar.spec.js` — 3 tests (quick mode **on**,
dedicated shard)
| # | Test | Behaviour verified | Call site |
|---|---|---|---|
| 8 | Table chart on `SELECT *` → toast shown **and** chart type reverts
| Table is blocked | `Index.vue:2265` (watcher) |
| 9 | Histogram chart on `SELECT *` → no toast, renders |
Line/bar/area/scatter allowed | entry path |
| 10 | `SELECT *` line chart survives browser refresh → no toast, state
restored | Restore path doesn't re-emit | `Index.vue:2465` (restore) |
## Infrastructure changes
| File | Change | Why |
|---|---|---|
| `ci_matrix.json` | New `Logs-SelectStar` shard with
`quick_mode_enabled: "true"` | `isSelectStarForTable()` only exists when
the server runs with quick mode on |
| `playwright.yml` | Per-shard `ZO_QUICK_MODE_ENABLED` env on `Start
OpenObserve` | The flag is read once at server start — a shard is the
smallest unit that can vary it |
| `build-ci-matrix.js` | Emit `quick_mode_enabled` (defaults `"false"`)
| The field was being silently stripped by the emit whitelist |
| `logsPage.js` | 13 assertion helpers + fixed chart-type detection |
Detection read `data-selected` off the wrong element and matched retired
CSS classes, so it always returned `null` |
All 24 existing shards are unaffected — they omit the field and default
to `false`.
## Notes on test quality
- Every negative assertion ("no toast", "no error") is sequenced after a
deterministic positive end-state, and gated on a precondition so it
cannot pass vacuously.
- Toast absence is proved with a MutationObserver recorder installed
before the navigation under test, since toasts auto-dismiss.
- The quick-mode precondition is a hard assertion, not `test.skip` — a
misconfigured shard fails loudly instead of silently reporting
"skipped".
## Why
The OSS and ENT Playwright shard matrices (`run_files` per `testfolder`)
were hand-maintained inline in each `playwright.yml` and had to be
synced on every spec add — error-prone and drift-prone (ENT was even
missing `dashboard-favorites`). This makes the shard list a **single
source of truth** for **every** Playwright workflow.
## What (OSS side)
All matrices are generated at run time from JSON manifests in
`tests/ui-testing/ci-matrix/`:
- **`ci-matrix/ci_matrix.json`** — PR-gate base (24 shards). The only
place shared shards live.
- **`ci-matrix/ci_matrix_regression.json`** — regression base (10
shards); `RegressionSet/<Feature>` mapping now carried in
`actual_folder`.
- **`.github/scripts/build-ci-matrix.js`** — merges base (+ optional ENT
overlay) → GitHub matrix; validates uniqueness / no dup specs / no spec
both active+disabled. Reused by ENT from its OSS checkout.
- **`ci-matrix/README.md`** — manifest index + how to add/move/disable a
spec.
- Workflows `playwright.yml` + `playwright_regression.yml`: new
`generate_matrix` job → `strategy.matrix: fromJSON(...)`; `case` blocks
→ `${{ matrix.actual_folder }}`.
Disabling a spec (JSON has no `//`): move it into that shard's
`disabled: [{file, reason}]` array — never emitted (won't run), record
survives. Every shard ships a `"disabled": []` placeholder.
## Safety (verified)
- **OSS `playwright.yml` and `playwright_regression.yml` matrices are
byte-identical to before** — zero behavior change.
- Companion ENT PR (same branch slug
`test/playwright-matrix-single-source`) derives its matrices from these
manifests + overlays.
## Validate
Add the `e2e` label; confirm `generate_matrix` prints the shard count
and the `e2e /` jobs fan out.
Design at: alerts_2.md
Per-group and per-series alerting, SLO measurement, and the UI for both.
## Alerts
**Multi-alerts (Feature 3).** A grouped alert can opt in to evaluating
and paging **per group** rather than collapsing to one verdict. Opt-in
only (`multi_alert`), so no existing alert changes behaviour: the flag
cannot be present in JSON written before it existed. Per-group state,
transitions, silence fingerprints, disappearance/reaping and a
cardinality cap, with the count-gate rules that make "most severe group"
and the legacy collapsed level provably the same verdict.
**Per-series alerting for PromQL.** The same feature for the metrics
family, where a group is one returned **series** keyed by its full label
set — that is a series' identity in Prometheus, and the expression's own
`by (…)` clause is already where the label set is chosen, so a second
picker could only disagree with it. One additive nullable column; NULL
means off, so stored alerts are untouched.
**Alert detail page**, replacing the row-click side panel: stat strip,
evaluation chart with threshold marklines, per-group table and history.
Simple alerts get a real history too — evaluations (one record per run)
by default, level changes alongside.
## SLOs
Count / time-slice / alert SLIs, error-budget and burn-rate arithmetic,
rolling windows, per-group slices with an exact rollup, coverage gating,
backfill, and SLO-based alerts.
Two capabilities the model could not express, both added tests-first:
- **Metrics-native counting** (`CountSource::PromQl`) — counter metrics
have no rows for a predicate to classify, so "good" exists only as
arithmetic between series, and correct counter arithmetic is
`increase()`.
- **Freshness semantics** (`absent_is_bad`) — for a pipeline SLO,
silence IS the failure, which the default gap rule can never report. A
*failed* query still freezes rather than pages; only a query that PROVED
emptiness counts as bad.
**SLO form**: live good/bad preview so a wrong predicate is caught
before saving rather than days later, real query editors with field
autocomplete, folders shared with alerts, and a move endpoint.
## Notes for review
- The SLO and composite features are **deferred (TODO)**:
`ZO_SLO_ENABLED` stays default **false** and the SLOs menu entry is
hidden until they ship; the SLO backend and UI remain in the tree behind
the flag, and composites stay pure logic + tests.
- One migration (schema 61): a nullable column for the PromQL opt-in. No
backfill.
- Two fixes outside the feature that it surfaced: query-editor
completions were duplicated per mounted editor (Monaco registers
providers per *language*), and the alert detail chart had no PromQL
branch, so it charted row counts against the series-count gate.
- Lands together with the matching o2-enterprise PR — `QueryType::Slo`
is new here and the enterprise dedup matches are exhaustive over it.
- `ratelimit::test_add_batch_missing_rule_id` fails intermittently on
`main` as well; unrelated to this branch.
---------
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
Co-authored-by: hengfeiyang <hengfei.yang@gmail.com>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
## What
Moves the Kinora upload **into the `merge_reports` job** instead of a
separate `report_kinora` job, and renames it to **`Merge Reports &
Report to Kinora`** so it's obvious the upload happens there.
## Why
The standalone `report_kinora` job re-downloaded every `blob-report-*`
artifact and re-ran `playwright merge-reports` before uploading —
repeating exactly what `merge_reports` already did. On large runs that
step was **hitting the 20-min timeout** (the merge + sequential trace
uploads).
Now the Kinora upload + sticky PR comment run as the **last two steps of
`merge_reports`**, reusing the `report.json` + extracted `trace.zip`s
already on that runner. **No second download, no second merge** — the
upload just piggybacks on the merge that already happened.
## Details
- Renamed job `Merge Reports` → **`Merge Reports & Report to Kinora`**;
removed the separate `report_kinora` job.
- Added `pull-requests: write` to the job (for the sticky comment).
- Still **non-fatal** (`continue-on-error`) — can't fail the build;
still gated by `UPLOAD_TO_KINORA`.
- Reuses `playwright-results/report.json` (traces resolve because they
were extracted on this same runner).
Companion enterprise PR uses the same branch name
(`test/kinora-in-merge-job`).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
Adds a Playwright E2E test suite covering the dashboard Favorites
feature — favoriting/unfavoriting a dashboard, the `Favorites`
pseudo-folder view, and its interaction with single and bulk delete
flows.
## What's covered
New `dashboard-favorites.spec.js` (6 tests) + `DashboardFavorites` page
object:
- Add a dashboard to favorites and verify it appears in the Favorites
view
- Remove a dashboard from favorites — confirms it drops from the
Favorites view while the dashboard itself remains untouched
- Favorite state persists across a page reload
- Bulk-delete a favorited dashboard from the Favorites view completes
successfully with no error
- Deleting a dashboard directly also removes it from Favorites
automatically
- Bulk-deleting a dashboard from the Favorites view doesn't leave a
stale copy behind in its original folder
## CI
`dashboard-favorites.spec.js` runs in its own dedicated
`Dashboards-Isolated` shard with a fresh backend instance, keeping it
independent from the rest of the dashboard test suite.
🤖 Auto-generated E2E tests for #13025 — **healed to passing** against a
live OpenObserve before this PR was opened.
- Branch `autoe2e/pr-13025` (stacked on `issue-ent-2038`), so the tests
run against the feature code and pass before it merges.
- Spec: `tests/ui-testing/playwright-tests/Logs/logsQuickPick.spec.js`
The repo Playwright CI re-validates here. Review is for **quality +
coverage** — the tests already pass.
_Generated by the E2E Council pipeline._
**Test cases added / changed:**
| Test case | Type | What it verifies |
|---|---|---|
| `should quick pick a stream from the sidebar` | new | clicking a
sidebar quick pick button selects the stream and loads the field list |
| `should show hero state on fresh arrival with no stream selected` |
new | the main content area shows both hero action cards when no stream
is selected |
| `should open stream dropdown when clicking select stream card` | new |
clicking the "Select a stream" hero card opens the stream dropdown
popover |
| `should reappear quick pick after deselecting and re-select works` |
new | after deselecting a stream the quick pick reappears and
re-selection works correctly |
| `should show more footer when org has more than 8 streams` | new | the
quick pick more footer is visible with the correct remaining count when
org has > 8 streams (skips otherwise) |
| `should select stream via hero recent chip` | new | clicking a
recent-stream chip in the hero state selects the stream and triggers
field list loading |
| `should open query guide documentation in new tab` | new | clicking
the "Read the query guide" card opens the OpenObserve docs in a new
browser tab |
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## What
Removes **all TestDino integration** from the Playwright e2e CI
workflows.
### Removed
- **Reporting/upload to TestDino** — the `upload_testdino` job, the
`UPLOAD_TO_TESTDINO` env/var, and all `TESTDINO_API_TOKEN` usages.
- **Rerun optimization** — the `check_rerun` step and the `tdpw
last-failed` `RERUN_MODE` dispatch.
### Files
- `.github/workflows/playwright.yml` (−444)
- `.github/workflows/playwright_regression.yml` (−350)
- `.github/workflows/playwright-auto-rerun.yml` (stale notice line)
- `tests/ui-testing/playwright.config.js` (comment reword only)
### Behavior change
"Re-run failed jobs" no longer targets only the previously-failed tests
— a re-run now runs the **full matrix job**. Native GitHub
re-run-failed-*jobs* still works.
### Kept
`report.json` / merge-reports generation is untouched — still consumed
by the Kinora and OpenObserve reporters.
### Follow-up (after merge)
Revoke `TESTDINO_API_TOKEN` (repo secret) + delete the
`UPLOAD_TO_TESTDINO` repo variable, and revoke the token in the TestDino
dashboard. Paired ENT PR uses the same branch `test/remove-testdino`.
### Validation
`actionlint` clean (no new findings) and zero TestDino references
remain.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
# Design System Unification — one visual language, enforced end-to-end
Establishes **one source of truth for every visual decision**, wires the
whole app to it, and **locks it in with CI** so it can't drift back.
### Foundations
- **Design-token system ("one knob per decision").** Every colour, text
size, spacing, corner radius, and shadow is a named token defined in
exactly one place — change the token, every screen updates. Collapsed
hundreds of near-duplicate definitions and removed "fake" duplicates
that pretended to be different but weren't.
- **Shared component library (O2).** Buttons, tables, dialogs, drawers,
form fields, badges, etc. already know the correct colours, spacing, and
dark-mode behaviour; new screens compose them instead of hand-styling.
Added new primitives on this branch (`OContent`, `KpiCard`) and
standardized existing ones.
- **One page structure everywhere** (`PageLayout` / `AppPageHeader` /
`AppPage` / `OContent`) — titles, breadcrumbs, spacing, and content
edges line up identically across every screen.
- **One table implementation** — all tabular data uses the same table
(sorting, density, sticky headers, empty states).
- **Styling consolidated onto Tailwind; old systems removed** —
hand-written CSS driven to the practical minimum, the legacy **Quasar**
framework fully removed, surviving stylesheets required to justify
themselves.
- **Two-tier corner radius** — a tight **4px** for controls and a soft
**12px** for surfaces (dialogs, cards, the app shell), plus a
full-circle option — replacing five radius names that all secretly meant
the same thing.
### Detailed improvements across screens
**Page structure & alignment** — unified page shell rolled across ~40+
screens (IAM, Settings, Alerts, Dashboards, Functions, Pipelines, RUM,
Metrics, Synthetics, Billing, editors, lists); one header height; one
page-edge grid driving headers, tables, rails, and tabs; unified
side-rail widths; inset rail selection/hover pills; every tab strip on
the grid line; tightened title/subtitle spacing.
**Tables (cross-screen)** — action columns consistently centered (incl.
Organizations); unified row-number column with no "#" truncation;
non-standard tables restyled to match; loading skeletons made
geometrically identical to the loaded table; symmetric edge inset; IAM
first-column grid alignment fixed.
**Traces** — flame graph + span-detail correlation panel fixes;
correlated logs/metrics placeholder fit; sticky headers; removed a
double scrollbar + a 1px sliver; aligned table fonts/badges; polished
the service detail drawer; stream-selector + correlation alignment;
aligned loading spinners; floating overlay arrows for the
disabled-services strip.
**Logs** — field-list controls aligned (focus ring no longer clipped);
detail-drawer spacing + metrics-tab overflow fixed; Run Query toolbar
tidied; listing sizing + stray border fixed.
**Metrics** — search added to the "Type" facet rail and aligned with the
other rails; proper empty states for the explorer and facet rail.
**Dashboards** — removed inconsistent card shadows; tightened
borders/resizers/field-list spacing; aligned View Dashboard chrome to
the grid; chart colours via the token helper.
**Home** — redesigned the Usage tab; added reusable KPI stat cards so
summary strips stop drifting.
**Dark mode** — new theme-switch animation; fixed a 1px content shift on
toggle and a white flash on load; fixed AI-chat text and assorted
light/dark mismatches; completed the token-driven dark migration.
**Empty states, controls & small polish** — restored 3 broken
empty-state animations; adopted the standard empty state; added the
sliding indicator to the toggle control; fixed the Quality-page inset +
drawer badge; refreshed Search History / Schedulers; fixed the
alert-history drawer scrollbar + timeline hover; removed a stray
menu-bar border.
### Dead code & cruft removal
- **Deleted the entire legacy `utilities.css`** global stylesheet (well
over a thousand lines) — evacuated in waves, roughly half of it dead
once traced.
- **Removed Quasar entirely**, including dead `.q-*` styles and ~25
phantom `--q-*` variables that were silently breaking bits of the UI.
- Stripped hundreds of **dead CSS classes and tokens that compiled to
nothing** — e.g. 46 uses of a colour class with no backing token, a
no-op shadow class on 51 elements, dead layout/utility classes, a
leftover `card-container` block, a dead table-actions token, and a
mis-registered gradient that rendered nothing.
- Cut **unused keyframe animations (19 → 3)** and removed **unused
components** + a stray build artifact.
- Net effect: a much smaller, leaner styling surface — less to maintain,
fewer dead ends to copy.
## Safety & verification
Large but low-risk: this is overwhelmingly a **styling and structure**
change, not product behaviour or data. Guarded by the
design-token/consistency/type checks above plus the existing test suite,
and reviewed screen-by-screen.
---------
Co-authored-by: Kirtan Thakkar <kirtan@kiara.tech>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: ktx-mihir <mihir.patel@kiara.tech>
## Problem
Every rebuild of `openobserve-api` re-runs the `RustEmbed` macro
expansion, which embeds `web/dist` (53MB, 1035 files) and compresses
every file with gzip + brotli(q11) — single-threaded, non-incremental,
and it peaks at 8GB RSS. `-Ztime-passes` attributes ~540s of the 769s
api compile to `expand_crate`. Net effect: touching one file in
`openobserve-core` costs a **16m45s** rebuild on an M-series laptop, and
neither incremental compilation nor codegen-unit tuning can help,
because macro expansion happens before both.
## Fix
Move the `#[derive(RustEmbed)]` into a new `src/web` leaf crate with no
workspace dependencies. The embed + compression now only recompiles when
`web/dist` changes; edits anywhere in the backend no longer pay for it.
**Measured (same machine, default profile settings, `--profile
release-profiling`):**
| | before | after |
|---|---|---|
| `openobserve-api` compile | 769s | 231s |
| touch `src/core` → binary | 16m45s | **6m44s** |
## Also in this PR
- **`src/web/build.rs`**: the derive has no file tracking, so a rebuilt
`web/dist` silently shipped stale embedded assets until something else
dirtied the api crate. `rerun-if-changed` on the embed folder fixes
that. The folder is parsed from the `#[folder]` attribute rather than
hardcoded, so it stays correct when the e2e workflows rewrite it. A
missing `index.html` now prints a build warning instead of silently
embedding nothing (the #13197 failure mode).
- **Playwright workflows**: the embed-folder `sed` now targets
`src/web/src/lib.rs` and is guarded with `grep -q`, so a future path
drift fails at the sed step instead of silently shipping an empty UI and
failing at the login page (see openobserve/o2-enterprise#2175 for that
incident).
- **`deny.toml`**: allow the new crate's AGPL-3.0 license (CI runs
`cargo deny check licenses`, which requires a per-crate exception).
## Validation
- `cargo deny check licenses` passes
- Smoke-tested the binary: `/healthz` ok, `/web/` serves `index.html`,
JS assets return 200 from the embedded bundle
- Enterprise overlay resolves: `cargo metadata` passes with the
companion `Cargo.toml.openobserve` change
## Companion
openobserve/o2-enterprise#2184 carries the overlay + e2e workflow sync
on the **same branch name** (`perf/web-embed-crate`); the two should
merge together.
## Problem
The web UI returns **HTTP 404 for all `/web/*` paths** on builds from
current `main`. The router process itself is healthy (`/config`,
`/healthz` return 200; `/` redirects to `/web/`) — only the UI is
missing.
## Root cause
The static UI assets are embedded at compile time in
`src/api/src/handler/http/router/ui.rs`:
```rust
#[derive(RustEmbed)]
#[folder = "web/dist/"]
struct WebAssets;
```
`rust-embed-for-web` resolves a relative `folder` against the crate's
`CARGO_MANIFEST_DIR`. When the HTTP handlers were moved into the new
`src/api/` sub-crate, this path was left as `web/dist/`, so it now
resolves to `src/api/web/dist/` — which does not exist.
`rust-embed-for-web` does not fail the build on a missing folder; it
simply embeds **zero** assets. The binary compiles cleanly and ships
with an empty UI, so `WebAssets::get()` returns `None` and the handler
returns `404 Not Found` for every asset (including `index.html`).
Previously (before the `src/api/` move) this file lived in the root
crate, where `web/dist/` correctly resolved to the workspace-root
`web/dist/`.
## Fix
Point the embed folder back at the workspace-root `web/dist` relative to
`src/api/`:
```rust
#[folder = "../../web/dist/"]
```
## Notes
- The Playwright workflows already `sed` this path to an absolute
`/tmp/dist/`, which is why CI didn't catch the regression — the
production build relies on the relative path.
Shrinath Rao
Harsh Mahajan
Vamsi Karthik RandAshish Kolhe
dependabot[bot]
Omkar Kesarkhane
Loakesh Indiran
Thibaud-Vdb
ktx-akshay
Huaijin
