Automated coverage for **email alert destinations and distribution
lists**, and unblocks the email tests that have been skipped since they
were written. **32 cases** — 27 active, 5 skipped regressions tracking
open defects (openobserve/o2-enterprise#2471).
**The unblocking.** The two skipped tests said *"the test environment
does not have email infrastructure set up"*. That over-stated it:
creating, editing, validating and deleting an email destination needs
SMTP switched **on**, and never reads a mailbox. Only delivery
assertions do. The suite is tiered on that:
| Tier | Needs | Cases |
|---|---|---|
| A | SMTP enabled | 24 |
| B | a readable sink | 2 |
| C | an alias that fans out | 1 |
**Tests**
- `Alerts/dl-email-destinations.spec.js` (32) — the DL half (alias
accepted, addressed, fanned out) and the recipient-list half, where our
code actually runs: splitting/trimming/lowercasing, the org-membership
gate, edit round-trips, add/remove/delete, both form paradigms (prebuilt
free-text vs the custom path's org-user picker), template auto-attach,
duplicate names, cancel, type-switching, keyboard, dark mode at 420px.
- `utils/mail-sink.js` — pluggable sink: Mailpit, a Mailinator inbox, or
a clean skip with a stated reason.
- `utils/o2-api.js` — storage is asserted from the API, never the form
field: an input can echo what you typed while the stored value differs.
- `scripts/dl-relay.js` — SMTP relay that expands one alias into
per-member deliveries, so fan-out runs with no external provider. Same
pattern as the in-test HTTP receiver in
`alerts-content-templates.spec.js`.
- `alertDestinationsPage.js` — +17 methods; the spec carries zero raw
locators.
**Replaces, without losing coverage.** The two skipped tests in
`alerts-destinations-prebuilt.spec.js` are removed for a pointer mapping
their intent to its new home: create (`D-01`), edit-add (`D-07`),
edit-remove (`ML-08`), delete (`UI-12`), format validation (`UI-02`).
Three of those did not exist before. That file now has no skips.
**CI.** `ZO_SMTP_ENABLED: true` was already set in `playwright.yml` but
had no host or port, so it fell back to `localhost:25` where nothing
listens — destinations could be **saved** while every **send** failed
silently. The workflow now starts Mailpit and a DL relay as static
binaries before the app (no Docker-in-Docker on these runners) and
points `ZO_SMTP_*` at them.
All three tiers run with no secrets and no egress. Verified in the CI
topology: **27 passed, 5 skipped**, `D-02` confirmed reaching all three
member mailboxes.
**Behaviour.** Self-cleaning via the `auto_dest_dl_*` prefix
`cleanup.spec.js` already sweeps, plus an API `afterAll`. `mode:
'parallel'` — each test owns its fixtures.
**Findings** are filed in openobserve/o2-enterprise#2471, not fixed
here. Two P1s: the *"Show matching rows"* toggle is ignored so log rows
are always embedded, and a user removed from the org keeps receiving
alerts while the destination can no longer be saved. The regression
tests for each are in this PR, skipped until the fix lands.
**Paired ENT PR: openobserve/o2-enterprise#2472.** ENT's workflow sets
`ZO_SMTP_ENABLED: false`, so without it the suite skips all 27 cases
there. The shard matrix needs no ENT change — ENT builds it from this
repo's `ci_matrix.json` plus its overlay, so the new spec is picked up
automatically. Merge the two together: this one alone leaves enterprise
with no email coverage.
Adds automated test coverage for **SLOs** (#13547, #13577, #13579,
#13704, #13761, #13784), which shipped with none. **149 tests** — 77
Playwright + 72 pytest — all three SLI types, both query languages,
happy / negative / edge paths. Clean branch off `main`.
**UI (Playwright · `tests/ui-testing`)**
- `pages/sloPages/` — 4 page objects (verified `data-test` selectors,
hydration + Monaco-sync waits, OSelect commit-retry, virtualized-list
filtering).
- `playwright-tests/SLO/slo-crud.spec.js` (24) — create
count/time-slice/grouped through the form, detail, edit, SLI-type
change, tags, pause, delete, filters; negative: dismissed delete, empty
search, cancel, unmeasured SLI is absent not zero.
- `playwright-tests/SLO/slo-form-validation.spec.js` (20) — server
errors surfacing in the form, conditional fields per SLI type, count +
time-slice previews, **PromQL count and time-slice created end to end**;
negative: no stream, invalid aggregate, over-long name, target
boundaries.
- `playwright-tests/SLO/slo-alert-sli.spec.js` (8) — **the `alert` SLI
type**: source picker, create, config carries only the alert id, detail
names the source; negative: ineligible source offered-but-disabled,
grouped alert SLI, slice finer than source cadence.
- `playwright-tests/SLO/slo-timeslice.spec.js` (10) — **#13761 P0
regression** (measured SLI, not the preview), PromQL, previews, gaps,
language-flip clearing.
- `playwright-tests/SLO/slo-alerts.spec.js` (15) — burn-rate +
error-budget kinds, presets, edit, freeze door, burndown; negative: no
destination, cancel, blank/invalid name.
- `playwright-tests/utils/slo-seed.js` — backdated seeding, metrics
seeding, eligible-source-alert fixture, ground-truth query, fail-fast
env guard, transient-store retry, full fixture teardown (alerts →
streams → destinations → templates).
**API (pytest · `tests/api-testing`)**
- `tests/slos/test_slo_validation.py` (72) — full REST surface
(create/get/update/enable/delete/groups/slo-eligible), validation on
**both create and update** (name, target, window, slice, config shapes,
comparators, float-threshold round-trip), boundaries (target
0.001/99.999, name 256 vs 257, threshold 0/negative/1e12/NaN, unicode
names), alert-source eligibility (silence-gated, too-infrequent, cron),
PromQL config shapes, auth 401/404.
**CI**
- Two shards: `SLO` (`workers: 3`) and `SLO-Measurement` (`workers: 1`).
Split because the backfill job runs with
`ZO_SCHEDULER_SLO_BACKFILL_CONCURRENCY=1`, so measurement specs sharing
an instance starve each other's waits — `fullyParallel` races spec
*files*, which `mode:'serial'` cannot prevent. The 3-worker pin is
SQLite: at 5, concurrent SLO/alert/destination writes trip `database is
locked` on the single-node meta store.
- New optional per-shard `ingest_allowed_upto` (`240`) alongside the
existing `quick_mode_enabled`: the workflow default of 5 **hours**
silently drops the 8 days of history an SLO window needs, while still
answering `200`. Other shards keep the default. Same precedent as
`api-testing.yml`'s `48`.
**Feature code**: 9 additive `data-test`/`dataTest` attributes in
`SloDetail.vue`, `SloList.vue`, `SloAlertCondition.vue` — no logic
changes, `vue-tsc` clean. The stat tiles, tabs and burn-window inputs
had no stable selectors; `OStatStrip`/`OTab` already forward the
attribute.
**Behaviour**: self-cleaning (`e2e_slo_*`, worker-scoped since
`afterAll` runs per worker).
**Verified in CI** on this branch: `e2e / SLO` **52 passed** (1.8m,
workers pinned to 3), `e2e / SLO-Measurement` **25 passed** (11.3m,
workers pinned to 1) — the measurement shard passing confirms the
per-shard `ingest_allowed_upto` took effect, since those tests are
impossible under the 5-hour default. pytest **72 passed** locally; in
CI's 599-test run one setup delete hit a transient 500 (see findings)
and now retries.
**Findings for the feature owner** (reported, not fixed here):
- A blank "good when" shows `Request failed with status code 422` —
`pruned()` drops the empty key, so the API's real 400 message never
reaches `save()`.
- A rejected alert name shows no banner: `SloAlertForm.submit()` returns
on `nameError` before setting `saveError`.
- `avg(latency_ms); DROP TABLE users--` is accepted at create time
(parsed only at measurement).
- The name-length message says "less than 256 characters" but 256 is
accepted — 257 is the first rejection.
- Duplicate alert names on one SLO are accepted (duplicate SLO names are
a 409), and the panel lists them by name, so they are indistinguishable.
- SLO names accept characters that break the list's own
`slos-slolist-edit-${name}` selectors; alert names correctly reject
them.
- All chart/preview root `data-test` ids are unreachable — a parent's
fallthrough attribute overrides the child root's.
- Both edit forms hydrate asynchronously, so a blank form can flash
before the stored values appear.
- `DELETE /slos/{id}` on a freshly-created SLO returned **500** once
under CI load (not reproducible locally in 12 rapid iterations).
Contention is arguably a retryable 503, and the body carried no
actionable message.
- **A scheduled alert whose source stream is deleted is retried
forever** (`Stream "…" not found in schema`), and the SLO backfill lane
shares the scheduler's worker budget — so accumulated dead alerts
*starve SLO measurement*. Observed: 65 of them held coverage at 1 of 7
chunks indefinitely; deleting them restored 0.143 → 1.0 in 240s.
Harmless in CI (fresh instance per shard) but it degrades any long-lived
environment. The suite now cleans up after itself; the retry-forever
behaviour itself may deserve a look.
**Paired ENT PR: openobserve/o2-enterprise#2434.** ENT's matrix is this
repo's manifest plus an overlay, merged by this repo's
`build-ci-matrix.js`, so both SLO shards reach ENT automatically —
verified: 31 merged shards, both SLO entries carrying
`ingest_allowed_upto=240` and their worker pins, no other shard
affected. What ENT was missing is *consuming* those two fields, which
#2434 adds. Merge the two together: this one alone leaves ENT booting
the SLO shards on the 5-hour ingest window, and #2434 alone is inert.
*(An earlier review reported ENT as using an inline matrix with no SLO
shard — that turned out not to be the case.)*
## What
**Phase 2** of the Playwright→pytest API-test migration (Phase 1 =
Correlation, #13788 / o2-enterprise#2419). Ports the 4 **pure-API**
Alerts 4.0 (multi-alert) specs into pytest and deletes the Playwright
versions — they launched a browser only to assert the raw API.
## Changes — `tests/api-testing/tests/alerts/`
| File | Ported from |
|---|---|
| `multialert_helpers.py` | `utils/alerts-api-helpers.js` (v1/v2
endpoints, canonical payloads, seed fixtures, ingest, scheduler-outcome
polling) |
| `conftest.py` | `alerts` fixture — seeds template + destination +
stream, auto-cleans created alerts |
| `test_multialert_api.py` | `alerts-multialert-api.spec.js` — 8 tests
(save-time guardrails, round-trip type contract, B1 group-endpoint RBAC
guard, tag/priority facets) |
| `test_multialert_regression.py` |
`alerts-multialert-regression.spec.js` — 4 tests (zero-behaviour-change
guards) |
| `test_multialert_firing.py` | `alerts-multialert-firing.spec.js` — 4
tests (fires at critical/warning, per-group fan-out, simple rollup) |
| `test_multialert_transitions.py` |
`alerts-multialert-transitions.spec.js` — 1 test (Ok→Warning→Critical
history) |
**17 tests, 1:1 port.** Deletes the 4 API specs and trims them from
`ci_matrix.json`'s Alerts `run_files`. Keeps
`alerts-multialert-ui.spec.js` (UI) and `alerts-api-helpers.js` (still
used by the UI + priority-tags specs).
## No workflow change needed
Unlike the Correlation pilot (which needed a dedicated server env), the
main `api_integration_tests` job **already** runs `tests/alerts/`, and
its server env **already** sets `ZO_ALERT_SCHEDULE_INTERVAL: 3` — the
exact scheduler cadence the firing/transitions tests need. So these
tests are picked up automatically with no CI edits.
## Verified locally
Full suite against a fresh single-node enterprise build: **17/17
passed** (~52s), including the timing-dependent firing
(critical/warning/per-group/rollup) and level-transition history tests.
## What & why
Begins consolidating the **correlation (service-streams) e2e suite**
onto the standard `tests/ui-testing` framework. Today those specs live
in a self-contained parallel harness (own configs, own auth, own `O2_*`
env vars, per-test orgs with no teardown, non-standard names) and are
**not registered in any CI matrix** — so they never run automatically.
This is **Phase 0**: land the shared standard-shaped API helper and
convert the first spec as the **reference pattern** for review, before
batch-converting the rest.
## Changes
- **`utils/correlation-api-helpers.js`** (new) — single source of truth
for correlation API plumbing. Auth via `authedRequest`/`getAuthHeaders`
(framework session + 401/403 self-heal), per-test **fresh-org create +
teardown**. No `test`/`expect`.
- **`Correlation/correlation-discovery.spec.js`** (new) — converted from
`journeyA`: `enhanced-baseFixtures`, `@correlation`/`@P0` tags,
Copyright + JSDoc header, `testLogger.testStart`, standard `ZO_*` env
and naming. Assertions ported **1:1** (DISC-01/02/03).
- Remove `Correlation/journeyA-discovery.api.spec.js` (superseded).
## Deliberate deviation (documented)
Correlation discovery / identity config / `_reset` are **org-global and
destructive**, so each test provisions a **fresh org** and tears it down
in `afterEach`. This keeps the isolation the feature genuinely needs
while fixing the real defect in the old suite: **orgs were created but
never deleted** (leak).
## Validation
- `node --check` clean; requires resolve.
- Collects cleanly under `playwright.config.js` (`--list` → 3 tests,
tagged).
- ⚠️ **Green run pending an enterprise env** (service_streams 403s on
OSS). Not wired into CI yet, so `main` stays green. **Draft** until
validated.
## Follow-ups (later phases)
Batch-convert remaining 4 API + 3 UI specs · revive
`correlationSettings.spec.js` against the new UI · register a dedicated
**Correlation** shard in the ENT `ci_matrix.ent.json` · delete
`corr-*.config.js`.
## What
#13659 converted Service Graph and Services Catalog from in-page tabs on
the Traces page into standalone routes reached from the nav-rail flyout.
This PR restores the in-page tabs as a **second access path** — both now
coexist, with distinct semantics:
| Path | Semantics |
|---|---|
| Toolbar tabs on `/traces` | In-page view switch (`?tab=service-graph`
/ `?tab=services-catalog`), shares the page's stream + time context —
the pre-#13659 UX, including old `?tab=` bookmarks |
| Nav-rail flyout | Navigates to the standalone routes
`/traces/service-graph`, `/traces/services` — unchanged from #13659 |
Both paths render the same inner components (`ServiceGraph.vue`,
`ServicesCatalog.vue`); only toolbar chrome differs.
## How
A guided **reverse-apply of #13659's diff to `Index.vue` +
`SearchBar.vue` only** (verified byte-identical to the pre-PR files —
both were untouched on main since), with four deliberate adaptations:
- **Re-added `applyHandoffFilter`** + its two hook call sites, which the
revert would have deleted — the standalone pages' "View Traces" handoff
(`?filter=` landing) depends on it. Covered by a new unit regression
test and verified live.
- The restored in-page `view-traces` handlers call
`viewTracesHandoff.ts` (`buildViewTracesFilter` /
`normalizeViewTracesPayload`) instead of resurrecting the inline filter
builders that #13659 extracted from them — no re-duplication.
- **Fixed a latent pre-#13659 bug**: the graph tab's refresh button
called `serviceGraphRef?.loadServiceGraph()`, which was never in
`ServiceGraph.vue`'s `expose()` — now `refresh()`.
- **Flyout highlight now follows the visible view** on both paths: new
`SubnavChild.activeOnTab` alias marks the Service Graph / Service
Catalog flyout items active when `/traces?tab=…` shows their view
in-page. Previously the same view highlighted different flyout items
depending on how you reached it — the only submenu that deviated.
Also:
- Renamed the "Services" submenu / page header / route title to
**"Service Catalog"**, matching the toolbar tab label (single
`menu.services` locale value covers all).
- `global-setup.js` login verification moved to `navbar-main-nav` — the
same stale-selector fix#13659 applied to `enhanced-baseFixtures.js`;
suites fail setup on main without it.
## Tests
| Suite | Result |
|---|---|
| Unit: `SearchBar` 69, `Index` 76, `ONavGroup` 18 (5 new `activeOnTab`
cases), `viewTracesHandoff` 21, `navGroups` 28 | all pass (233 total, 2
pre-existing skips) |
| E2E (new) `Traces/traces-toolbar-tabs.spec.js` — in-page tab switches,
shared-datetime carry, `?tab=` bookmark reload, flyout→standalone-route
coexistence | 4/4 vs live enterprise build |
| E2E `Traces/service-catalog.spec.js` (standalone-route regression) |
18 pass |
| `type-check`, `eslint` (0 errors), `prettier`, `lint:design:strict` |
clean |
Not run: `Traces/service-graph.spec.js` — its `beforeAll` ingests via
`ZO_BASE_URL`, which requires a single-origin instance (CI); the dev
vite+backend split can't satisfy it. Nothing in this PR touches
`ServiceGraph.vue` or the standalone views.
## Notes for review
- `Index.vue` / `SearchBar.vue` restored regions are intentionally
verbatim pre-#13659 (including its 16 pre-existing lint *warnings*) —
deviations are limited to the four adaptations above, so the
diff-vs-pre-PR is auditable.
- Two known-inert quirks restored deliberately, not bugs to fix here:
`hide-splitter-separator` has no CSS rule (was a no-op pre-#13659 too),
and `@update:activeTab` writes to a getter-only computed.
---------
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
## What
Service Graph and Services Catalog were tabs on the Traces page,
switched via a `?tab=` query param on a single keep-alive route. They
are now real routes, reached from a hover flyout on the Traces rail
tile:
| Route | Notes |
|---|---|
| `/traces` | Traces search — keeps the **Spans / Traces** toggle |
| `/traces/service-graph` | enterprise-gated; redirects to Traces on OSS
|
| `/traces/services` | |
**Spans stays an in-page toggle** on purpose. It's a view *granularity*
that shares the query, stream and time range — you switch to it
mid-investigation and expect context to carry over. Datadog's Trace
Explorer does the same. Service Graph and Services are destinations;
Datadog, New Relic and Grafana all treat them that way.
## Why this wasn't just cosmetic
As tabs, all three views shared one `meta.title`, one history entry, and
the traces page's `searchObj`. That last part is why Service Graph and
Services Catalog inherited the traces page's selected stream — the
coupling this PR removes structurally.
## UI
Both pages now use the Agent Graph header pattern (`OPageLayout` +
title/subtitle/icon, date picker and last-run refresh in `#actions`),
with a subnav row carrying stream selector + search on the left and
view-type + layout on the right.
## Handoff
`view-traces` no longer mutates the shared store from a sibling tab.
Both views build the same filter string — extracted verbatim into
`viewTracesHandoff.ts` — and navigate to Traces with it as query params,
so the URL survives a reload. (Traces then canonicalises `filter=` to
its own base64 `query=` via existing code in `useTraces.ts:290`.)
## Two bugs this surfaced
Both were things the tab views inherited for free and now have to set
themselves — and both would have hit real users deep-linking these
pages:
- **`organizationIdentifier`** was only assigned inside the Traces
page's `loadPageData()`. Without it the catalog issued requests to
`/api//_search_stream` (empty org segment) and rendered nothing.
- **`?period=` / `?from=`+`?to=`** were only applied by the Traces
page's `restoreUrlQueryParams()`. Without it a shared link carrying a
time range silently opened on the 15-minute default.
## Tests
| Suite | Result |
|---|---|
| `Traces/service-graph.spec.js` | 14 / 14 |
| `Traces/service-catalog.spec.js` | 10 / 10 |
| `RegressionSet/Traces/traces-ent.spec.js` | 1 / 1 |
| Unit (`Navbar` + `plugins/traces`) | 2514 pass |
Also clean: `type-check`, `eslint` (0 errors), `lint:design:strict`.
Test-side changes: navigation POMs go through the rail flyout instead of
the removed toggles; the graph POM pins an explicit window, since on its
own route the graph no longer inherits the traces page's range. The
shared auth fixture now verifies against `navbar-main-nav` — it keyed
off `menu-link-/-item`, which the current rail no longer renders, so
suites failed setup even after a successful login.
Filter-building coverage moved with the code into
`viewTracesHandoff.spec.ts` (21 tests) rather than being deleted
alongside the component tests that asserted the old in-page handlers.
## Known / follow-up
- `RegressionSet/Traces/traces-regression.spec.js` has 2 failures (trace
column sorting, pagination). **Pre-existing** — they fail identically
against production on `main`, and the Service Catalog test in that file
passes.
- Both views still default their stream picker to the traces page's
selected stream. Defaulting Service Graph and Services to **all
streams** is a deliberate follow-up: on our own instance the `default`
stream shows 22 topology nodes vs 71 across all streams, and services
like `o2-ai` span 4 streams. Graph is a one-line change (the API already
supports it); the catalog needs a backend decision, so it's out of scope
here.
## Alpha1 e2e robustness — self-healing auth + service-graph window
Builds on **#13621** (re-auth self-heal). Both branches share the slug
`test/alpha1-e2e-robustness` is stacked on `test/alpha1-reauth-on-401`;
review/merge #13621 first (or merge this, which contains it).
Root-caused against alpha1 runs 30892800970 / 30900015839 / 30902942856
(each investigated live via Playwright MCP).
### 1. Auth self-heal (from #13621, included here)
Shared cloud session + per-org passcode are minted once and reused all
run; on long runs / org-contention they expire mid-run → `User not
authenticated` (UI) and `HTTP 401 Unauthorized` (API). Now:
`authedRequest` refreshes the passcode and retries on 401;
`navigateToBase` re-authenticates and resumes instead of throwing.
**Verified working** in run 30902942856 — the auth storm cleared
(Metrics/SDR/Streams recovered, zero `User not authenticated`), and
`[auth] Refreshed cloud passcode after 401` fired and recovered mid-run.
### 2. Service graph queries a wide window
All 9 `service-graph.spec.js` chart tests failed with
`service-graph-chart` "element(s) not found" **despite** `beforeAll`
confirming topology via API (25 nodes). **Reproduced live**: the
service-graph tab defaults to **Past 15 Minutes** and shows *"No service
graph data / 0 entities"* (no chart element) when the daemon-processed
topology sits outside that window. The tab **honours the `period` URL
param** (verified: label → "Past 6 Hours").
→ `navigateToServiceGraphUrl` now navigates with `period=6h`; added
`navigateToServiceGraphAndWaitForChart()` (poll + refresh, tolerates
daemon lag). Non-masking.
### Honest scope
- **Not included (can't validate locally / not test-code):**
- **Pipelines ×5** (`Explore`/`update-pipeline` 45s timeout) —
persistent across *every* run; pipeline/stream data unavailable.
Env/data, needs CI-side repro.
- **OTLP metrics ingestion 401** (a PromQL-prereq `beforeAll`) — a
node-`fetch` ingest path with no page context to re-fetch the passcode;
narrow, needs a small refactor to self-heal.
- Singles: `logoManagement` (@enterprise `_meta`), `shareLink` (known
race), `alerts-ui-operations:131` (#9484) — pre-existing flaky/env.
- Service-graph is **enterprise/feature-gated** (403 in my org), so its
fix is evidence-reasoned + validated at the URL-param level, but final
proof is the seeded CI run.
## What
Lets alpha1 shards log in as **different Dex users** instead of all
hammering the cloud env as one user. Paired with
**openobserve/o2-enterprise#2333** (the workflow wiring).
## Why
- **Load distribution** — spreads API/session load across users if the
env rate-limits per user.
- **Blast radius** — one user's account/session issue stops ~1/3 of
shards, not all.
- Test **data isolation is unchanged** (still per-org); this is purely
about the login user.
## Changes (OSS)
- `global-setup-alpha1.js` — new `ALPHA1_USER_INDEX` (1|2|3) selects the
Dex user: email resolves from `ALPHA1_USER_EMAIL_<N>`, **falling back to
`ALPHA1_USER_EMAIL`** when the indexed secret isn't set (safe
incremental rollout). Password shared (`ALPHA1_USER_PASSWORD`).
- Auth **filenames stay canonical** (`user.json` / `cloud-config.json`)
— ~20 specs + shared utils read those exact paths. The per-user split
happens at the CI layer (each shard downloads only its user's artifact
into the canonical path), so no spec changes needed.
- New `auth-warm.spec.js` — a login-only spec the barrier uses to mint
auth artifacts for users 2..N **without** running the heavy org-wide
cleanup.
## Default behaviour
With no index set, everything resolves to user 1 — identical to today.
## Enterprise PR
openobserve/o2-enterprise#2333
Design at: alerts_2.md
Per-group and per-series alerting, SLO measurement, and the UI for both.
## Alerts
**Multi-alerts (Feature 3).** A grouped alert can opt in to evaluating
and paging **per group** rather than collapsing to one verdict. Opt-in
only (`multi_alert`), so no existing alert changes behaviour: the flag
cannot be present in JSON written before it existed. Per-group state,
transitions, silence fingerprints, disappearance/reaping and a
cardinality cap, with the count-gate rules that make "most severe group"
and the legacy collapsed level provably the same verdict.
**Per-series alerting for PromQL.** The same feature for the metrics
family, where a group is one returned **series** keyed by its full label
set — that is a series' identity in Prometheus, and the expression's own
`by (…)` clause is already where the label set is chosen, so a second
picker could only disagree with it. One additive nullable column; NULL
means off, so stored alerts are untouched.
**Alert detail page**, replacing the row-click side panel: stat strip,
evaluation chart with threshold marklines, per-group table and history.
Simple alerts get a real history too — evaluations (one record per run)
by default, level changes alongside.
## SLOs
Count / time-slice / alert SLIs, error-budget and burn-rate arithmetic,
rolling windows, per-group slices with an exact rollup, coverage gating,
backfill, and SLO-based alerts.
Two capabilities the model could not express, both added tests-first:
- **Metrics-native counting** (`CountSource::PromQl`) — counter metrics
have no rows for a predicate to classify, so "good" exists only as
arithmetic between series, and correct counter arithmetic is
`increase()`.
- **Freshness semantics** (`absent_is_bad`) — for a pipeline SLO,
silence IS the failure, which the default gap rule can never report. A
*failed* query still freezes rather than pages; only a query that PROVED
emptiness counts as bad.
**SLO form**: live good/bad preview so a wrong predicate is caught
before saving rather than days later, real query editors with field
autocomplete, folders shared with alerts, and a move endpoint.
## Notes for review
- The SLO and composite features are **deferred (TODO)**:
`ZO_SLO_ENABLED` stays default **false** and the SLOs menu entry is
hidden until they ship; the SLO backend and UI remain in the tree behind
the flag, and composites stay pure logic + tests.
- One migration (schema 61): a nullable column for the PromQL opt-in. No
backfill.
- Two fixes outside the feature that it surfaced: query-editor
completions were duplicated per mounted editor (Monaco registers
providers per *language*), and the alert detail chart had no PromQL
branch, so it charted row counts against the series-count gate.
- Lands together with the matching o2-enterprise PR — `QueryType::Slo`
is new here and the enterprise dedup matches are exhaustive over it.
- `ratelimit::test_add_batch_missing_rule_id` fails intermittently on
`main` as well; unrelated to this branch.
---------
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
Co-authored-by: hengfeiyang <hengfei.yang@gmail.com>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
Companion OSS change to enterprise 2278 (alpha1 workflow). Same branch
slug on both repos.
## What
Supports running the Alpha1 (cloud) e2e matrix with its shards **divided
across multiple orgs** for data isolation, on top of the shared-auth
login (one Dex login, reused by all shards). Both changes are in
\`global-setup-alpha1.js\`.
## Changes
1. **Ingest into the shard's own org (\`ORGNAME\`)** instead of the org
baked into \`cloud-config.json\`. The shared-auth barrier logs into one
org; without this, every shard ingested into that one org even when
\`ORGNAME\` differed.
2. **Re-fetch the shard's own org passcode in the shared-auth path.**
Each org has its own \`o2oi_\` default ingestion token (see
\`core/organization.rs\` \`get_passcode\`). The barrier captured only
its org's token, so shards on other orgs got \`401 Unauthorized Access\`
on ingest. \`verifySharedAuth\` now calls \`fetchCloudConfig(page)\` to
overwrite \`cloud-config.json\` with the current shard's passcode
(fetched via the session, which is a member of every org). Keeps the
single shared login — no login storm — while giving each shard the
correct per-org credential.
## Context
Part of the alpha1 parallelization effort:
- Shared-auth artifact removes the 13-way concurrent Dex login storm
(enterprise 2278).
- Shards divided across 6 orgs (enterprise 2278 matrix) for data
isolation.
- These two OSS fixes make per-org ingestion actually work.
## Testing
Validated by dispatching the enterprise \`playwright_alpha1.yml\` with
\`o2_opensource_repo=test/alpha1-parallel-shards\`. Verifying each shard
now ingests \`e2e_automate\` into its own org with \`status:200\`
(previously \`401\` for non-barrier orgs).
## What
Adds a small **log-only** Playwright reporter that prints a LOUD,
unmissable banner to stdout **the moment any test starts a retry
attempt**, plus an end-of-shard summary listing every test that was
retried.
## Why
In CI the reporter is `blob`, which writes nothing to stdout. So when a
failed test is retried (`retries: process.env.CI ? 3 : 0` in
`playwright.config.js`) there is **no signal in the GitHub Actions logs
that a retry is even happening** — a test silently re-runs up to 3 more
times and the log just shows the next `Test Started` line. This makes
flaky-vs-real failures hard to spot when scanning CI output.
The retry is internal to `npx playwright test`, so the workflow YAML
can't see it — it has to be emitted from the Playwright reporter layer.
This is that layer.
## What it looks like
```
🔁🔁🔁══════════════════════════════════════════════════════════🔁🔁🔁🔁 R E T R Y # 1 · attempt 2 of 4 (test previously failed — re-running)
🔁 ▶ chromium › logsqueries.cte.spec.js › should query with simple TotalCols CTE
🔁 ▶ Logs/logsqueries.cte.spec.js
🔁🔁🔁══════════════════════════════════════════════════════════🔁🔁🔁
```
…and once per shard at the end:
```
🔁 RETRY SUMMARY — N test(s) were retried in this shard:
🔁 • [2 retries] chromium › … › should query … (Logs/logsqueries.cte.spec.js)
```
## Scope / safety
- Added to the **CI** reporter list only, alongside `blob` (multiple
reporters are fine). It only `console.log`s — never sends anything,
never fails the run.
- **ENT gets this for free**: the ENT Playwright workflow checks out and
runs the OSS `tests/ui-testing` tree, so this one OSS change makes
retries loud in both OSS and ENT CI. There is no ENT test tree to
change.
- Verified locally with a forced-failure probe (`CI=true`, `retries:
2`): the per-retry banner fires on attempt 2 and 3, and the summary
lists the retried test.
## Files
- `tests/ui-testing/playwright-tests/utils/retry-banner-reporter.js` —
new reporter
- `tests/ui-testing/playwright.config.js` — wire it into the CI reporter
array
## What
Stabilizes the **Traces** and **Pipelines** shards of the alpha1 cloud
E2E suite
(`playwright_alpha1.yml`) to green. Rebased on `main` — builds on
#13191's
protocol-aware ingestion agent (`selectAgent`), so no duplicate
transport fix here.
The failing run had 1 hard failure + 1 flake in Traces and 7 failures +
1 flake in
Pipelines. Root causes: a data dependency (error traces not guaranteed),
hover-flyout
navigation force-clicks that miss, a refresh button disabled while
loading, and logs
indexing latency. All fixed with deterministic data/waits — no
`waitForTimeout` as a
fix, no `force:true`, no retry-count bumps, no weakened assertions.
## Root cause & fixes
**Traces**
1. **`tracesAnalyzeDimensions` "P0: Error-only toggle filters trace
results"** (hard fail,
both attempts). The error-only toggle is the error-count badge in
`SearchResult.vue`,
rendered `v-if="errorCount != null && errorCount > 0"` — it exists only
when the search
returns an error span. The shared `default` trace stream has no
guaranteed error traces
(global-setup ingests only logs), so the badge was present or absent by
luck of the
window (hence the intermittent CI greens). Fix: `beforeAll` seeds
guaranteed error
traces via a new `ingestTraces(page, n, { forceScenario: 'error' })`
option in
`trace-ingestion.js`, and `tracesPage.waitForErrorBadgeAfterSearch()`
re-runs the search
until the seeded errors are indexed — a bounded wait for backend
processing, not a sleep.
2. **`traceQueryEditor` "P2: View error spans and their details"**
(flaky). `clickFirstErrorTrace()`
used a blind `waitForTimeout(2000)` that raced the trace-details render
before the test
asserted on it. Now waits for `trace-details-tree` to be visible.
**Pipelines** (`pipelinesPage.js`)
3. **`pipeline-section-tab-streamPipelines` 45s timeout** (5 tests:
`pipeline-core:56/159/238`,
`pipelines:479`, `pipeline-conditions:297`). `openPipelineMenu()` /
`navigateToStreams()`
navigate via `openNavFlyoutChild`, which hovers a nav group tile and
force-clicks the
teleported flyout child. The flyout self-closes on settle, so the click
can miss and the
page never navigates — the follow-up tab/refresh click then dead-waits
the 45s action
timeout. Both now verify the target page rendered (`pipeline-list-page`
/ `log-stream-table`)
and fall back to a direct `page.goto()` of the route otherwise. (A
failure screenshot showed
exactly this: the Data flyout open over a blank page.)
4. **`log-stream-refresh-stats-btn` 45s timeout** (3 tests:
`pipeline-dynamic:72/97/113`). The
refresh button is an `OButton` with `:loading` bound to the stream-stats
fetch, and `OButton`
renders `:disabled="disabled || loading"`. Clicking during the (slow, on
loaded alpha) stats
load waited on actionability until timeout. `refreshStreamStats()` now
waits `toBeEnabled()`
before clicking; `navigateToStreams()` waits for the table instead of a
blind 1s.
5. **`openTimestampMenu` empty-table dead-wait.** After exploring the
destination stream, a bare
`.click()` on the first logs row timed out 45s when the freshly-ingested
rows weren't indexed
yet (high indexing latency on loaded envs) — a latent flake exposed once
fix 3 let the test
reach this step. Now re-runs the logs query until a row appears, then
clicks.
## Validation
- **CI — both shards green (workers=3, alpha cloud):**
https://github.com/openobserve/o2-enterprise/actions/runs/29563931877
(`alpha1-e2e / Traces` ✅ · `alpha1-e2e / Pipelines` ✅)
- **Original failing run:**
https://github.com/openobserve/o2-enterprise/actions/runs/29481437880
- **Local (alpha, CI conditions):** each previously-failing test re-run
to green —
Traces error-toggle & error-spans; `pipeline-core:56/159/238`;
`pipeline-dynamic:72`.
## Notes
- Only 4 files change (`pipelinesPage.js`, `tracesPage.js`,
`tracesAnalyzeDimensions.spec.js`,
`trace-ingestion.js`). The https-ingestion agent fix these depend on
already landed on `main`
in #13191 — this branch relies on it rather than re-adding it.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What
Stabilizes the **Alerts** shard of the alpha1 cloud E2E suite to green,
running **fully
parallel** (`workers:3`, `fullyParallel:true`) — no serialization.
Test-infra +
page-object changes only; **no product code**. The one product-side
defect surfaced (the
prebuilt destination form unmounts its fields on template auto-load) is
worked around in
the test layer and written up separately for a frontend fix.
## Approach change since the first revision
The first version serialized the contended tests (`@serialAlpha1` + a
dedicated
`chromium-serial` project). Per review feedback ("solve it in
parallel"), that was
**removed** — the shard now runs **fully parallel**, and the flakiness
is fixed at the
source (fixture / registration / render races) instead of by
serializing.
## Root causes & fixes
### Streams / fixtures
- **`auto_playwright_stream` "being deleted" race** — the 3
`alerts-ui-operations` tests
now use the shared, always-protected **`e2e_automate`** stream
(pre-ingested by
global-setup, never deleted by cleanup), eliminating the shared
fixed-name delete race
entirely. `cleanup.spec.js` no longer deletes `auto_playwright_stream`;
`commonActions.ingestTestData` checks `response.ok()`, retries the "is
being deleted"
race, and polls the streams API until the stream is registered before
the wizard opens.
### Alert wizard
- **Dedup alert blocked at save** —
`createScheduledAlertWithDeduplication` typed its SQL
into the Monaco editor without clearing the default `SELECT * FROM
"<stream>"`, producing
malformed SQL → "Please fix the SQL error before saving." Now clears the
editor first.
- **Stream-select ordering + registration poll** — drive the stream
dropdown only after its
list fetch resolves; poll registration; page-reload recovery when the
cached list is stale.
### Prebuilt destinations (the deep one)
- **App defect:** on a *reused* form (2nd+ destination), selecting a
prebuilt type
intermittently **unmounts the credential/name fields ~0.5 s later**
(template auto-load
re-render) and they stay gone. Confirmed via DOM timeline (`count:0` for
25 s+). Since a
wait can't recover an unmounted element, `selectDestinationType` clicks
the card once,
waits for the template network to settle, confirms the fields are
**stable** (not just
present), and — if they unmount — **remounts the form** (cancel+reopen →
full page reload)
and re-selects. Written up separately for a frontend fix.
- **List verifications raced the OTable loading skeleton** —
`expectDestinationInList` / `verifyDestinationExists` /
`verifyAlertCreated` now wait for
`o2-table-skeleton-body` to clear and poll the **list** endpoint the UI
actually reads
(not just the single-GET) before asserting the row.
### Form-validation specs
- `alerts-form-validation` + `anomaly-detection-form-validation` flipped
serial→parallel and
made self-sufficient: `beforeAll` seeds a worker-scoped
template+destination via API (the
"Add Alert" button is hard-disabled with zero destinations), `afterAll`
removes them,
cleanup patterns registered.
### Misc
- Cloud-correct (https) ingestion agent; alert search-input +
`updateAlert` edit-form wait
hardening for shared-org load.
## Cloud environment gates (legitimate — not coverage drops)
`test.skip(isCloudEnvironment())` (dedup validation), `if
(!isCloudEnvironment())` (import
trigger round-trip), and external destination URLs — all because cloud's
**SSRF guard**
blocks self-referencing destinations by design. All still run fully on
OSS/self-hosted.
## Scope / safety
- 15 files: Alerts page-objects, alpha1 config + global-setup, Alerts
specs, cleanup. **No product code.**
- **No tests deleted, no `.only`, no unconditional skips, no assertions
removed.**
- The temporary Alerts-only playwright matrix (added for faster
iteration) has been
**reverted** — the full OSS e2e matrix is restored.
## Validation
- Prebuilt P1 validated locally on cloud alpha (repeated clean passes;
every field-unmount
recovered by the remount).
- alpha1 Alerts CI:
https://github.com/openobserve/o2-enterprise/actions/runs/29391293698🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
## What
Repairs the shared setup + ingestion path that was failing **every**
module of the alpha1 cloud E2E suite (`playwright_alpha1.yml`) after the
UX revamp, **and stabilizes the first full test shard — Functions — to
green** (validated in CI, run link below). No changes to the test
matrix.
Per-module fixes for the remaining shards land in separate follow-up
PRs.
## Root cause — shared setup (blocked all 13 modules)
1. **Automation login user routes through Okta.** After the revamp,
`@openobserve.ai` logins on alpha redirect to Okta SSO and 404 —
automation can't complete login. The local-auth `e2etest@o2-qa.com` user
still uses the Dex password flow.
2. **Org-switch selectors changed** (`global-setup-alpha1.js`). The org
dropdown now opens via a dedicated trigger button; the old inline
`arrow_drop_down` text is an SVG icon, the search box's `data-test`
moved to the `OSearchInput` wrapper `<div>`, and rows carry a
`data-test-org-identifier`.
3. **Ingestion helper rejected https.** `ingestionPage.js` hardcoded an
`http.Agent` for node-fetch — throws `Protocol "https:" not supported`
against alpha's https ingestion URL (killed ~27 tests/run across
modules).
## Root cause — Functions shard
4. **Tests created enrichment tables in the wrong org.**
`navigateToBase` passes `?org_identifier=` on the root URL; the redirect
to `/web/` **drops the query param**, so the app fell back to the user's
default org and enrichment tables were created there. Fixed by
deep-navigating to the enrichment page with the org param and asserting
the active org stuck (same pattern js-transform tests already use).
5. **Backend bug (reported separately): enrichment URL-jobs fail for
table names starting with `schema`.** 100% reproducible,
org/session/timing-independent: the job's save step fails with `Failed
to save batch 1: not an ingester` (`schema_probe1` ❌ / `view_probe1` ✅,
same request otherwise). Test tables renamed `schema_view_*` →
`view_schema_*` and `schema_mismatch_*` → `mismatch_schema_*`; cleanup
patterns cover both prefixes. The product bug itself still needs a
backend fix — it affects any user naming a table `schema…`.
Plus hardening while there: URL-job completion is now decided from the
backend status API instead of inferring from row-button counts (which
misread a failed job as success), the job's `error_message` is logged on
failure, and the create form verifies field values + Save enabled +
create-API response before proceeding.
## Validation
- **CI (this PR's branch): [run
#168](https://github.com/openobserve/o2-enterprise/actions/runs/29315765982)
— Cloud ✅ (3m31s) + Functions ✅ (13m14s), `playwright_summary` ✅**
- Local (alpha, CI conditions): previously-failing schema-view +
schema-mismatch tests 2/2 green; earlier full-shard run 24/25 with the
same single failure now fixed.
## Notes
- `ALPHA1_USER_*` secrets: CI already uses a working non-Okta user
(Cloud job green).
- Follow-ups: backend `schema`-prefix bug (separate report), per-module
PRs for remaining shards.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Reduce `Logs-Builder-Basic` e2e shard runtime — ~10 min → ~5 min
### 🎯 Result (measured on CI)
| Metric | Before | After | Saved |
|--------|--------|-------|-------|
| **`Logs-Builder-Basic` shard** | **10m 05s** | **5m 12s** | **~4m 53s
(≈48% faster)** |
The shard runtime is cut roughly in half. Most of the old time was fixed
sleeps, waits that always ran their full timeout, and duplicated
per-test setup — not actual test logic. **No test cases, steps, or
assertions were removed.**
### What we changed
| # | Area | Before | After | Saving |
|---|------|--------|-------|--------|
| 1 | `initQueryBuilderTest` setup | Logs page loaded twice (helper
`goto` + `selectStream` navigates again) | Navigate once | 1 page load ×
~41 tests |
| 2 | `waitForBuildTabLoaded` | Always waited up to 10s for editor to
fill, even when it's meant to be empty | Skip when SQL mode is off |
~10s dead wait on many tests |
| 3 | `enableSqlModeIfNeeded` | 2s watcher wait + 500ms sleep, even
where the watcher never fires | Gated to Build tab; waits on real signal
| ~2.5s × ~35 tests |
| 4 | `disableSqlModeIfNeeded` | Fixed 500–600ms sleeps | Deterministic
wait on editor update | Blind sleeps removed |
| 5 | `applyQueryButton` | Clicked Run blindly | Waits for button ready
first | Fewer early-click retries |
| 6 | `pagination.spec` | Fresh stream ingest + index-wait per test (9×)
| One ingest per worker, reused | 8 redundant ingests + waits |
| 7 | `pagination.spec` | Dead `waitForTimeout(2000)` per streaming test
| Removed | ~2s × 5 tests |
| 8 | `unflattened.spec` | Search run twice with 3s + 1s sleeps | Single
run | ~5s × ~8 calls |
| 9 | `unflattened` row scan | Fixed 3s sleep per row | Wait for content
to render | 15-row scan: ~97s → seconds |
| 10 | `unflattened.spec` | Blind sleep after ingest | Poll until stream
queryable | No over-waiting |
### Bonus
Items 2–5 live in the shared `logsPage` page object, so
`Logs-Builder-Advanced`, `Logs-Core`, and `Logs-Features` shards get
faster for free too.
### Notes
| Item | Detail |
|------|--------|
| `playwright.yml` | Unchanged — shard layout stays as-is |
| Assertions | Preserved everywhere (e.g. pagination still checks the
same record counts) |
| Validation | All 52 tests still discovered by `playwright --list`;
shard confirmed green on CI at 5m 12s |
## What
Heals the flaky **\`should fetch fresh field values when switching
streams\`** (`@fieldValuesCache @P1 @regression`) test in
`RegressionSet/Logs/logs-bugs.spec.js`, which failed the
`Logs-Regression` job in [o2-enterprise run
28454057830](https://github.com/openobserve/o2-enterprise/actions/runs/28454057830/job/84328391696).
## Root cause
The test ingests a unique `service_name=svc-automate-<runId>` into the
**shared** `e2e_automate` stream, then immediately drives the UI
field-values panel — with **no readiness gate** confirming that value is
searchable yet. Freshly-ingested data isn't queryable instantly (WAL →
index lag), so the `_values` call returns `svc-automate-*` values from
earlier runs/retries but misses the just-ingested one:
```
Error: Expected stream1 values to contain svc-automate-1782832273253, got: svc-automate-17828322504262
```
Playwright retries amplify it — each retry adds another `svc-automate`
value to the shared stream while still missing the current run's value,
so all retries fail.
> Note: the CI job's attempt-2 then crashed on a `jq: Cannot iterate
over null` error in the rerun-detection step. That is a **separate,
already-fixed** issue — current `main` already has `actions: read` on
the job + jq guards; the failing run executed a stale pre-fix workflow
snapshot (GitHub re-runs reuse the original attempt's YAML). No workflow
change is needed here.
## Fix
- Add `waitForFieldValueSearchable()` to `utils/data-ingestion.js` —
polls `_search` until the exact field value is returned (the proven
readiness-gate pattern, value-level rather than row-count level).
- Gate both the stream1 and stream2 ingested values before the UI
assertions.
## Verification
Run locally against a local OpenObserve with `--repeat-each=3`:
```
Stream1 field values: ["svc-automate-...0702","svc-automate-...4762"] # accumulates (reproduces CI pollution)
✓ Stream1 shows correct value: svc-automate-1782833946762 # current value always present now
...
3 passed (1.0m)
```
The accumulating `svc-automate-*` values across repeats reproduce
exactly the shared-stream pollution seen in CI, and the gate ensures the
current run's value is always present.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
- Adds `alerts-metrics-notification.spec.js` — a 3-test UI suite
verifying the full metrics alert chain
- Test 1 (P1): PromQL alert triggers via scheduler, verified in history
UI — self-contained using `ensureMetricsIngested()`
- Test 2 (P2): Template body contains required alert variables
(`{alert_url}`, `{alert_name}`, `{stream_name}`)
- Test 3 (P1): Manual trigger from alert list more-options menu sends
notification
- Includes `alert-metrics-notification-findings.md` documenting manual
testing results (API schema, PromQL vs SQL, trigger indicators)
## Test plan
- [ ] Test 1: Alert triggers on freshly ingested `cpu_usage` metrics and
history rows appear in UI
- [ ] Test 2: Template editor (Monaco) shows expected variables
- [ ] Test 3: Manual trigger menu option calls PATCH /trigger and
succeeds
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Vamsi Karthik R
Prabhat Sharmaandgithub-actions[bot]
Shrinath RaoandShrinath-O2
ktx-akshay
Neha P
