Automated coverage for **email alert destinations and distribution
lists**, and unblocks the email tests that have been skipped since they
were written. **32 cases** — 27 active, 5 skipped regressions tracking
open defects (openobserve/o2-enterprise#2471).
**The unblocking.** The two skipped tests said *"the test environment
does not have email infrastructure set up"*. That over-stated it:
creating, editing, validating and deleting an email destination needs
SMTP switched **on**, and never reads a mailbox. Only delivery
assertions do. The suite is tiered on that:
| Tier | Needs | Cases |
|---|---|---|
| A | SMTP enabled | 24 |
| B | a readable sink | 2 |
| C | an alias that fans out | 1 |
**Tests**
- `Alerts/dl-email-destinations.spec.js` (32) — the DL half (alias
accepted, addressed, fanned out) and the recipient-list half, where our
code actually runs: splitting/trimming/lowercasing, the org-membership
gate, edit round-trips, add/remove/delete, both form paradigms (prebuilt
free-text vs the custom path's org-user picker), template auto-attach,
duplicate names, cancel, type-switching, keyboard, dark mode at 420px.
- `utils/mail-sink.js` — pluggable sink: Mailpit, a Mailinator inbox, or
a clean skip with a stated reason.
- `utils/o2-api.js` — storage is asserted from the API, never the form
field: an input can echo what you typed while the stored value differs.
- `scripts/dl-relay.js` — SMTP relay that expands one alias into
per-member deliveries, so fan-out runs with no external provider. Same
pattern as the in-test HTTP receiver in
`alerts-content-templates.spec.js`.
- `alertDestinationsPage.js` — +17 methods; the spec carries zero raw
locators.
**Replaces, without losing coverage.** The two skipped tests in
`alerts-destinations-prebuilt.spec.js` are removed for a pointer mapping
their intent to its new home: create (`D-01`), edit-add (`D-07`),
edit-remove (`ML-08`), delete (`UI-12`), format validation (`UI-02`).
Three of those did not exist before. That file now has no skips.
**CI.** `ZO_SMTP_ENABLED: true` was already set in `playwright.yml` but
had no host or port, so it fell back to `localhost:25` where nothing
listens — destinations could be **saved** while every **send** failed
silently. The workflow now starts Mailpit and a DL relay as static
binaries before the app (no Docker-in-Docker on these runners) and
points `ZO_SMTP_*` at them.
All three tiers run with no secrets and no egress. Verified in the CI
topology: **27 passed, 5 skipped**, `D-02` confirmed reaching all three
member mailboxes.
**Behaviour.** Self-cleaning via the `auto_dest_dl_*` prefix
`cleanup.spec.js` already sweeps, plus an API `afterAll`. `mode:
'parallel'` — each test owns its fixtures.
**Findings** are filed in openobserve/o2-enterprise#2471, not fixed
here. Two P1s: the *"Show matching rows"* toggle is ignored so log rows
are always embedded, and a user removed from the org keeps receiving
alerts while the destination can no longer be saved. The regression
tests for each are in this PR, skipped until the fix lands.
**Paired ENT PR: openobserve/o2-enterprise#2472.** ENT's workflow sets
`ZO_SMTP_ENABLED: false`, so without it the suite skips all 27 cases
there. The shard matrix needs no ENT change — ENT builds it from this
repo's `ci_matrix.json` plus its overlay, so the new spec is picked up
automatically. Merge the two together: this one alone leaves enterprise
with no email coverage.
Adds automated test coverage for **SLOs** (#13547, #13577, #13579,
#13704, #13761, #13784), which shipped with none. **149 tests** — 77
Playwright + 72 pytest — all three SLI types, both query languages,
happy / negative / edge paths. Clean branch off `main`.
**UI (Playwright · `tests/ui-testing`)**
- `pages/sloPages/` — 4 page objects (verified `data-test` selectors,
hydration + Monaco-sync waits, OSelect commit-retry, virtualized-list
filtering).
- `playwright-tests/SLO/slo-crud.spec.js` (24) — create
count/time-slice/grouped through the form, detail, edit, SLI-type
change, tags, pause, delete, filters; negative: dismissed delete, empty
search, cancel, unmeasured SLI is absent not zero.
- `playwright-tests/SLO/slo-form-validation.spec.js` (20) — server
errors surfacing in the form, conditional fields per SLI type, count +
time-slice previews, **PromQL count and time-slice created end to end**;
negative: no stream, invalid aggregate, over-long name, target
boundaries.
- `playwright-tests/SLO/slo-alert-sli.spec.js` (8) — **the `alert` SLI
type**: source picker, create, config carries only the alert id, detail
names the source; negative: ineligible source offered-but-disabled,
grouped alert SLI, slice finer than source cadence.
- `playwright-tests/SLO/slo-timeslice.spec.js` (10) — **#13761 P0
regression** (measured SLI, not the preview), PromQL, previews, gaps,
language-flip clearing.
- `playwright-tests/SLO/slo-alerts.spec.js` (15) — burn-rate +
error-budget kinds, presets, edit, freeze door, burndown; negative: no
destination, cancel, blank/invalid name.
- `playwright-tests/utils/slo-seed.js` — backdated seeding, metrics
seeding, eligible-source-alert fixture, ground-truth query, fail-fast
env guard, transient-store retry, full fixture teardown (alerts →
streams → destinations → templates).
**API (pytest · `tests/api-testing`)**
- `tests/slos/test_slo_validation.py` (72) — full REST surface
(create/get/update/enable/delete/groups/slo-eligible), validation on
**both create and update** (name, target, window, slice, config shapes,
comparators, float-threshold round-trip), boundaries (target
0.001/99.999, name 256 vs 257, threshold 0/negative/1e12/NaN, unicode
names), alert-source eligibility (silence-gated, too-infrequent, cron),
PromQL config shapes, auth 401/404.
**CI**
- Two shards: `SLO` (`workers: 3`) and `SLO-Measurement` (`workers: 1`).
Split because the backfill job runs with
`ZO_SCHEDULER_SLO_BACKFILL_CONCURRENCY=1`, so measurement specs sharing
an instance starve each other's waits — `fullyParallel` races spec
*files*, which `mode:'serial'` cannot prevent. The 3-worker pin is
SQLite: at 5, concurrent SLO/alert/destination writes trip `database is
locked` on the single-node meta store.
- New optional per-shard `ingest_allowed_upto` (`240`) alongside the
existing `quick_mode_enabled`: the workflow default of 5 **hours**
silently drops the 8 days of history an SLO window needs, while still
answering `200`. Other shards keep the default. Same precedent as
`api-testing.yml`'s `48`.
**Feature code**: 9 additive `data-test`/`dataTest` attributes in
`SloDetail.vue`, `SloList.vue`, `SloAlertCondition.vue` — no logic
changes, `vue-tsc` clean. The stat tiles, tabs and burn-window inputs
had no stable selectors; `OStatStrip`/`OTab` already forward the
attribute.
**Behaviour**: self-cleaning (`e2e_slo_*`, worker-scoped since
`afterAll` runs per worker).
**Verified in CI** on this branch: `e2e / SLO` **52 passed** (1.8m,
workers pinned to 3), `e2e / SLO-Measurement` **25 passed** (11.3m,
workers pinned to 1) — the measurement shard passing confirms the
per-shard `ingest_allowed_upto` took effect, since those tests are
impossible under the 5-hour default. pytest **72 passed** locally; in
CI's 599-test run one setup delete hit a transient 500 (see findings)
and now retries.
**Findings for the feature owner** (reported, not fixed here):
- A blank "good when" shows `Request failed with status code 422` —
`pruned()` drops the empty key, so the API's real 400 message never
reaches `save()`.
- A rejected alert name shows no banner: `SloAlertForm.submit()` returns
on `nameError` before setting `saveError`.
- `avg(latency_ms); DROP TABLE users--` is accepted at create time
(parsed only at measurement).
- The name-length message says "less than 256 characters" but 256 is
accepted — 257 is the first rejection.
- Duplicate alert names on one SLO are accepted (duplicate SLO names are
a 409), and the panel lists them by name, so they are indistinguishable.
- SLO names accept characters that break the list's own
`slos-slolist-edit-${name}` selectors; alert names correctly reject
them.
- All chart/preview root `data-test` ids are unreachable — a parent's
fallthrough attribute overrides the child root's.
- Both edit forms hydrate asynchronously, so a blank form can flash
before the stored values appear.
- `DELETE /slos/{id}` on a freshly-created SLO returned **500** once
under CI load (not reproducible locally in 12 rapid iterations).
Contention is arguably a retryable 503, and the body carried no
actionable message.
- **A scheduled alert whose source stream is deleted is retried
forever** (`Stream "…" not found in schema`), and the SLO backfill lane
shares the scheduler's worker budget — so accumulated dead alerts
*starve SLO measurement*. Observed: 65 of them held coverage at 1 of 7
chunks indefinitely; deleting them restored 0.143 → 1.0 in 240s.
Harmless in CI (fresh instance per shard) but it degrades any long-lived
environment. The suite now cleans up after itself; the retry-forever
behaviour itself may deserve a look.
**Paired ENT PR: openobserve/o2-enterprise#2434.** ENT's matrix is this
repo's manifest plus an overlay, merged by this repo's
`build-ci-matrix.js`, so both SLO shards reach ENT automatically —
verified: 31 merged shards, both SLO entries carrying
`ingest_allowed_upto=240` and their worker pins, no other shard
affected. What ENT was missing is *consuming* those two fields, which
#2434 adds. Merge the two together: this one alone leaves ENT booting
the SLO shards on the 5-hour ingest window, and #2434 alone is inert.
*(An earlier review reported ENT as using an inline matrix with no SLO
shard — that turned out not to be the case.)*
## What
**Phase 2** of the Playwright→pytest API-test migration (Phase 1 =
Correlation, #13788 / o2-enterprise#2419). Ports the 4 **pure-API**
Alerts 4.0 (multi-alert) specs into pytest and deletes the Playwright
versions — they launched a browser only to assert the raw API.
## Changes — `tests/api-testing/tests/alerts/`
| File | Ported from |
|---|---|
| `multialert_helpers.py` | `utils/alerts-api-helpers.js` (v1/v2
endpoints, canonical payloads, seed fixtures, ingest, scheduler-outcome
polling) |
| `conftest.py` | `alerts` fixture — seeds template + destination +
stream, auto-cleans created alerts |
| `test_multialert_api.py` | `alerts-multialert-api.spec.js` — 8 tests
(save-time guardrails, round-trip type contract, B1 group-endpoint RBAC
guard, tag/priority facets) |
| `test_multialert_regression.py` |
`alerts-multialert-regression.spec.js` — 4 tests (zero-behaviour-change
guards) |
| `test_multialert_firing.py` | `alerts-multialert-firing.spec.js` — 4
tests (fires at critical/warning, per-group fan-out, simple rollup) |
| `test_multialert_transitions.py` |
`alerts-multialert-transitions.spec.js` — 1 test (Ok→Warning→Critical
history) |
**17 tests, 1:1 port.** Deletes the 4 API specs and trims them from
`ci_matrix.json`'s Alerts `run_files`. Keeps
`alerts-multialert-ui.spec.js` (UI) and `alerts-api-helpers.js` (still
used by the UI + priority-tags specs).
## No workflow change needed
Unlike the Correlation pilot (which needed a dedicated server env), the
main `api_integration_tests` job **already** runs `tests/alerts/`, and
its server env **already** sets `ZO_ALERT_SCHEDULE_INTERVAL: 3` — the
exact scheduler cadence the firing/transitions tests need. So these
tests are picked up automatically with no CI edits.
## Verified locally
Full suite against a fresh single-node enterprise build: **17/17
passed** (~52s), including the timing-dependent firing
(critical/warning/per-group/rollup) and level-transition history tests.
## What & why
Begins consolidating the **correlation (service-streams) e2e suite**
onto the standard `tests/ui-testing` framework. Today those specs live
in a self-contained parallel harness (own configs, own auth, own `O2_*`
env vars, per-test orgs with no teardown, non-standard names) and are
**not registered in any CI matrix** — so they never run automatically.
This is **Phase 0**: land the shared standard-shaped API helper and
convert the first spec as the **reference pattern** for review, before
batch-converting the rest.
## Changes
- **`utils/correlation-api-helpers.js`** (new) — single source of truth
for correlation API plumbing. Auth via `authedRequest`/`getAuthHeaders`
(framework session + 401/403 self-heal), per-test **fresh-org create +
teardown**. No `test`/`expect`.
- **`Correlation/correlation-discovery.spec.js`** (new) — converted from
`journeyA`: `enhanced-baseFixtures`, `@correlation`/`@P0` tags,
Copyright + JSDoc header, `testLogger.testStart`, standard `ZO_*` env
and naming. Assertions ported **1:1** (DISC-01/02/03).
- Remove `Correlation/journeyA-discovery.api.spec.js` (superseded).
## Deliberate deviation (documented)
Correlation discovery / identity config / `_reset` are **org-global and
destructive**, so each test provisions a **fresh org** and tears it down
in `afterEach`. This keeps the isolation the feature genuinely needs
while fixing the real defect in the old suite: **orgs were created but
never deleted** (leak).
## Validation
- `node --check` clean; requires resolve.
- Collects cleanly under `playwright.config.js` (`--list` → 3 tests,
tagged).
- ⚠️ **Green run pending an enterprise env** (service_streams 403s on
OSS). Not wired into CI yet, so `main` stays green. **Draft** until
validated.
## Follow-ups (later phases)
Batch-convert remaining 4 API + 3 UI specs · revive
`correlationSettings.spec.js` against the new UI · register a dedicated
**Correlation** shard in the ENT `ci_matrix.ent.json` · delete
`corr-*.config.js`.
## What
#13659 converted Service Graph and Services Catalog from in-page tabs on
the Traces page into standalone routes reached from the nav-rail flyout.
This PR restores the in-page tabs as a **second access path** — both now
coexist, with distinct semantics:
| Path | Semantics |
|---|---|
| Toolbar tabs on `/traces` | In-page view switch (`?tab=service-graph`
/ `?tab=services-catalog`), shares the page's stream + time context —
the pre-#13659 UX, including old `?tab=` bookmarks |
| Nav-rail flyout | Navigates to the standalone routes
`/traces/service-graph`, `/traces/services` — unchanged from #13659 |
Both paths render the same inner components (`ServiceGraph.vue`,
`ServicesCatalog.vue`); only toolbar chrome differs.
## How
A guided **reverse-apply of #13659's diff to `Index.vue` +
`SearchBar.vue` only** (verified byte-identical to the pre-PR files —
both were untouched on main since), with four deliberate adaptations:
- **Re-added `applyHandoffFilter`** + its two hook call sites, which the
revert would have deleted — the standalone pages' "View Traces" handoff
(`?filter=` landing) depends on it. Covered by a new unit regression
test and verified live.
- The restored in-page `view-traces` handlers call
`viewTracesHandoff.ts` (`buildViewTracesFilter` /
`normalizeViewTracesPayload`) instead of resurrecting the inline filter
builders that #13659 extracted from them — no re-duplication.
- **Fixed a latent pre-#13659 bug**: the graph tab's refresh button
called `serviceGraphRef?.loadServiceGraph()`, which was never in
`ServiceGraph.vue`'s `expose()` — now `refresh()`.
- **Flyout highlight now follows the visible view** on both paths: new
`SubnavChild.activeOnTab` alias marks the Service Graph / Service
Catalog flyout items active when `/traces?tab=…` shows their view
in-page. Previously the same view highlighted different flyout items
depending on how you reached it — the only submenu that deviated.
Also:
- Renamed the "Services" submenu / page header / route title to
**"Service Catalog"**, matching the toolbar tab label (single
`menu.services` locale value covers all).
- `global-setup.js` login verification moved to `navbar-main-nav` — the
same stale-selector fix#13659 applied to `enhanced-baseFixtures.js`;
suites fail setup on main without it.
## Tests
| Suite | Result |
|---|---|
| Unit: `SearchBar` 69, `Index` 76, `ONavGroup` 18 (5 new `activeOnTab`
cases), `viewTracesHandoff` 21, `navGroups` 28 | all pass (233 total, 2
pre-existing skips) |
| E2E (new) `Traces/traces-toolbar-tabs.spec.js` — in-page tab switches,
shared-datetime carry, `?tab=` bookmark reload, flyout→standalone-route
coexistence | 4/4 vs live enterprise build |
| E2E `Traces/service-catalog.spec.js` (standalone-route regression) |
18 pass |
| `type-check`, `eslint` (0 errors), `prettier`, `lint:design:strict` |
clean |
Not run: `Traces/service-graph.spec.js` — its `beforeAll` ingests via
`ZO_BASE_URL`, which requires a single-origin instance (CI); the dev
vite+backend split can't satisfy it. Nothing in this PR touches
`ServiceGraph.vue` or the standalone views.
## Notes for review
- `Index.vue` / `SearchBar.vue` restored regions are intentionally
verbatim pre-#13659 (including its 16 pre-existing lint *warnings*) —
deviations are limited to the four adaptations above, so the
diff-vs-pre-PR is auditable.
- Two known-inert quirks restored deliberately, not bugs to fix here:
`hide-splitter-separator` has no CSS rule (was a no-op pre-#13659 too),
and `@update:activeTab` writes to a getter-only computed.
---------
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
## What
Service Graph and Services Catalog were tabs on the Traces page,
switched via a `?tab=` query param on a single keep-alive route. They
are now real routes, reached from a hover flyout on the Traces rail
tile:
| Route | Notes |
|---|---|
| `/traces` | Traces search — keeps the **Spans / Traces** toggle |
| `/traces/service-graph` | enterprise-gated; redirects to Traces on OSS
|
| `/traces/services` | |
**Spans stays an in-page toggle** on purpose. It's a view *granularity*
that shares the query, stream and time range — you switch to it
mid-investigation and expect context to carry over. Datadog's Trace
Explorer does the same. Service Graph and Services are destinations;
Datadog, New Relic and Grafana all treat them that way.
## Why this wasn't just cosmetic
As tabs, all three views shared one `meta.title`, one history entry, and
the traces page's `searchObj`. That last part is why Service Graph and
Services Catalog inherited the traces page's selected stream — the
coupling this PR removes structurally.
## UI
Both pages now use the Agent Graph header pattern (`OPageLayout` +
title/subtitle/icon, date picker and last-run refresh in `#actions`),
with a subnav row carrying stream selector + search on the left and
view-type + layout on the right.
## Handoff
`view-traces` no longer mutates the shared store from a sibling tab.
Both views build the same filter string — extracted verbatim into
`viewTracesHandoff.ts` — and navigate to Traces with it as query params,
so the URL survives a reload. (Traces then canonicalises `filter=` to
its own base64 `query=` via existing code in `useTraces.ts:290`.)
## Two bugs this surfaced
Both were things the tab views inherited for free and now have to set
themselves — and both would have hit real users deep-linking these
pages:
- **`organizationIdentifier`** was only assigned inside the Traces
page's `loadPageData()`. Without it the catalog issued requests to
`/api//_search_stream` (empty org segment) and rendered nothing.
- **`?period=` / `?from=`+`?to=`** were only applied by the Traces
page's `restoreUrlQueryParams()`. Without it a shared link carrying a
time range silently opened on the 15-minute default.
## Tests
| Suite | Result |
|---|---|
| `Traces/service-graph.spec.js` | 14 / 14 |
| `Traces/service-catalog.spec.js` | 10 / 10 |
| `RegressionSet/Traces/traces-ent.spec.js` | 1 / 1 |
| Unit (`Navbar` + `plugins/traces`) | 2514 pass |
Also clean: `type-check`, `eslint` (0 errors), `lint:design:strict`.
Test-side changes: navigation POMs go through the rail flyout instead of
the removed toggles; the graph POM pins an explicit window, since on its
own route the graph no longer inherits the traces page's range. The
shared auth fixture now verifies against `navbar-main-nav` — it keyed
off `menu-link-/-item`, which the current rail no longer renders, so
suites failed setup even after a successful login.
Filter-building coverage moved with the code into
`viewTracesHandoff.spec.ts` (21 tests) rather than being deleted
alongside the component tests that asserted the old in-page handlers.
## Known / follow-up
- `RegressionSet/Traces/traces-regression.spec.js` has 2 failures (trace
column sorting, pagination). **Pre-existing** — they fail identically
against production on `main`, and the Service Catalog test in that file
passes.
- Both views still default their stream picker to the traces page's
selected stream. Defaulting Service Graph and Services to **all
streams** is a deliberate follow-up: on our own instance the `default`
stream shows 22 topology nodes vs 71 across all streams, and services
like `o2-ai` span 4 streams. Graph is a one-line change (the API already
supports it); the catalog needs a backend decision, so it's out of scope
here.
## Alpha1 e2e robustness — self-healing auth + service-graph window
Builds on **#13621** (re-auth self-heal). Both branches share the slug
`test/alpha1-e2e-robustness` is stacked on `test/alpha1-reauth-on-401`;
review/merge #13621 first (or merge this, which contains it).
Root-caused against alpha1 runs 30892800970 / 30900015839 / 30902942856
(each investigated live via Playwright MCP).
### 1. Auth self-heal (from #13621, included here)
Shared cloud session + per-org passcode are minted once and reused all
run; on long runs / org-contention they expire mid-run → `User not
authenticated` (UI) and `HTTP 401 Unauthorized` (API). Now:
`authedRequest` refreshes the passcode and retries on 401;
`navigateToBase` re-authenticates and resumes instead of throwing.
**Verified working** in run 30902942856 — the auth storm cleared
(Metrics/SDR/Streams recovered, zero `User not authenticated`), and
`[auth] Refreshed cloud passcode after 401` fired and recovered mid-run.
### 2. Service graph queries a wide window
All 9 `service-graph.spec.js` chart tests failed with
`service-graph-chart` "element(s) not found" **despite** `beforeAll`
confirming topology via API (25 nodes). **Reproduced live**: the
service-graph tab defaults to **Past 15 Minutes** and shows *"No service
graph data / 0 entities"* (no chart element) when the daemon-processed
topology sits outside that window. The tab **honours the `period` URL
param** (verified: label → "Past 6 Hours").
→ `navigateToServiceGraphUrl` now navigates with `period=6h`; added
`navigateToServiceGraphAndWaitForChart()` (poll + refresh, tolerates
daemon lag). Non-masking.
### Honest scope
- **Not included (can't validate locally / not test-code):**
- **Pipelines ×5** (`Explore`/`update-pipeline` 45s timeout) —
persistent across *every* run; pipeline/stream data unavailable.
Env/data, needs CI-side repro.
- **OTLP metrics ingestion 401** (a PromQL-prereq `beforeAll`) — a
node-`fetch` ingest path with no page context to re-fetch the passcode;
narrow, needs a small refactor to self-heal.
- Singles: `logoManagement` (@enterprise `_meta`), `shareLink` (known
race), `alerts-ui-operations:131` (#9484) — pre-existing flaky/env.
- Service-graph is **enterprise/feature-gated** (403 in my org), so its
fix is evidence-reasoned + validated at the URL-param level, but final
proof is the seeded CI run.
## What
Lets alpha1 shards log in as **different Dex users** instead of all
hammering the cloud env as one user. Paired with
**openobserve/o2-enterprise#2333** (the workflow wiring).
## Why
- **Load distribution** — spreads API/session load across users if the
env rate-limits per user.
- **Blast radius** — one user's account/session issue stops ~1/3 of
shards, not all.
- Test **data isolation is unchanged** (still per-org); this is purely
about the login user.
## Changes (OSS)
- `global-setup-alpha1.js` — new `ALPHA1_USER_INDEX` (1|2|3) selects the
Dex user: email resolves from `ALPHA1_USER_EMAIL_<N>`, **falling back to
`ALPHA1_USER_EMAIL`** when the indexed secret isn't set (safe
incremental rollout). Password shared (`ALPHA1_USER_PASSWORD`).
- Auth **filenames stay canonical** (`user.json` / `cloud-config.json`)
— ~20 specs + shared utils read those exact paths. The per-user split
happens at the CI layer (each shard downloads only its user's artifact
into the canonical path), so no spec changes needed.
- New `auth-warm.spec.js` — a login-only spec the barrier uses to mint
auth artifacts for users 2..N **without** running the heavy org-wide
cleanup.
## Default behaviour
With no index set, everything resolves to user 1 — identical to today.
## Enterprise PR
openobserve/o2-enterprise#2333
Design at: alerts_2.md
Per-group and per-series alerting, SLO measurement, and the UI for both.
## Alerts
**Multi-alerts (Feature 3).** A grouped alert can opt in to evaluating
and paging **per group** rather than collapsing to one verdict. Opt-in
only (`multi_alert`), so no existing alert changes behaviour: the flag
cannot be present in JSON written before it existed. Per-group state,
transitions, silence fingerprints, disappearance/reaping and a
cardinality cap, with the count-gate rules that make "most severe group"
and the legacy collapsed level provably the same verdict.
**Per-series alerting for PromQL.** The same feature for the metrics
family, where a group is one returned **series** keyed by its full label
set — that is a series' identity in Prometheus, and the expression's own
`by (…)` clause is already where the label set is chosen, so a second
picker could only disagree with it. One additive nullable column; NULL
means off, so stored alerts are untouched.
**Alert detail page**, replacing the row-click side panel: stat strip,
evaluation chart with threshold marklines, per-group table and history.
Simple alerts get a real history too — evaluations (one record per run)
by default, level changes alongside.
## SLOs
Count / time-slice / alert SLIs, error-budget and burn-rate arithmetic,
rolling windows, per-group slices with an exact rollup, coverage gating,
backfill, and SLO-based alerts.
Two capabilities the model could not express, both added tests-first:
- **Metrics-native counting** (`CountSource::PromQl`) — counter metrics
have no rows for a predicate to classify, so "good" exists only as
arithmetic between series, and correct counter arithmetic is
`increase()`.
- **Freshness semantics** (`absent_is_bad`) — for a pipeline SLO,
silence IS the failure, which the default gap rule can never report. A
*failed* query still freezes rather than pages; only a query that PROVED
emptiness counts as bad.
**SLO form**: live good/bad preview so a wrong predicate is caught
before saving rather than days later, real query editors with field
autocomplete, folders shared with alerts, and a move endpoint.
## Notes for review
- The SLO and composite features are **deferred (TODO)**:
`ZO_SLO_ENABLED` stays default **false** and the SLOs menu entry is
hidden until they ship; the SLO backend and UI remain in the tree behind
the flag, and composites stay pure logic + tests.
- One migration (schema 61): a nullable column for the PromQL opt-in. No
backfill.
- Two fixes outside the feature that it surfaced: query-editor
completions were duplicated per mounted editor (Monaco registers
providers per *language*), and the alert detail chart had no PromQL
branch, so it charted row counts against the series-count gate.
- Lands together with the matching o2-enterprise PR — `QueryType::Slo`
is new here and the enterprise dedup matches are exhaustive over it.
- `ratelimit::test_add_batch_missing_rule_id` fails intermittently on
`main` as well; unrelated to this branch.
---------
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
Co-authored-by: hengfeiyang <hengfei.yang@gmail.com>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
ktx-akshay
Shrinath Rao
Omkar Kesarkhane
Vamsi Karthik R
Hengfei Yang
Prabhat Sharmaandgithub-actions[bot]
Neha P
