## Alert Library
A curated catalog at `/alert-library`: browse packs of ready-made
alerts, see
which ones can actually run against the streams this org already
receives,
preview a verdict against real data, and install one or many.
The library is a **peer page** of Alerts / Notification Destinations /
Templates
— flat routing, presented as tabs via a shared `AlertSectionTabs` in
`OPageLayout #header-tabs` (the pattern `PipelineSectionTabs` already
ships).
### What landed, by phase
| Phase | What |
| --- | --- |
| Data layer | S3 manifest fetch, cache, readiness resolution
(`useAlertLibrary`, `constants/alertLibrary`, `types/alertLibrary`) |
| Gallery | Rail (packs / categories, multi-select, segmented switch),
stat-strip availability facets, cards |
| Drawer | Detail view with query preview and customizable tunables |
| Install | Bulk-install wizard — destination selection, per-alert
tunables, stale-session hardening |
| Prefill | Carry a library alert (query, trigger, threshold, severity)
into the alert editor |
### Design decisions worth knowing
- **Rail navigates, strip filters.** Availability is deliberately not in
the rail
— carrying both overflowed it and pushed a facet group below the fold.
- **Colour marks the exception only.** No "Ready" pill (it would
describe ~75 of
86 cards and signal nothing); only unusable cards carry a label.
- **A failed `/streams` call leaves readiness unknown**, rather than
claiming
"none of your data matches" — that claim would be about the request, not
the
org, and it is the most alarming thing this page can say. Counts read
`—`
until there is an answer, and a failed *refresh* keeps the verdict it
already had.
- **No "Installed" tile** — installed detection doesn't exist yet, and a
tile
reading a permanent zero is dishonest UI.
- **`severity: warning` renders as `medium`** — the badge group's scale
is
critical/high/medium/low/info, and medium is the amber it also gives P3,
which
is exactly the priority a warning alert installs as.
- Pack and category names come from the manifest, so they render as
`raw()` data —
a pack added upstream needs no frontend release.
### Design-system changes (used by the library, applied app-wide)
- **`warning-quiet` badge tone** — one variant whose values differ by
theme: the
soft amber wash in light, a neutral chip in dark. Neither the filled
amber chip
nor the neutral one was right in both, because the two ramps aren't
symmetric.
- **Border tokens now meet WCAG 2.1 SC 1.4.11.** Every border token
measured
under 3:1 (`border-strong` was 2.52:1). `border-default` firms to
grey-300 and
controls get a token that clears 3:1.
- Nav: Reliability's flyout groups its four alerting entries under one
**Alerts**
header, declared in the same order as the tab strip those pages carry.
Flyout
levels are sized by depth.
- `sensors-off` icon added to the `OIcon` registry.
### Fixes that rode along
- **metrics**: a shared URL whose `metrics_data` blob omits panel query
fields no
longer blanks the whole metrics page — blob queries are filled out to
the panel
schema before being laid over.
- **dashboards**: a PromQL series with no legend template was named
`JSON.stringify(metric)`, so fifty series read identically for the width
of the
chip. Series are now named by the labels that actually differ,
preferring the
ones that say *which workload* it is (namespace / service / pod / host).
Correlation's semantic groups refine that when loaded, but nothing is
fetched
to render a chart — that endpoint is enterprise-only and a chart must
render
the same on OSS.
- **alerts**: preview evaluates against the alert's real trigger;
deleting a row
no longer reloads the list behind it; peer tabs no longer shift when
switching
alerting pages.
- **ui**: badge-only select options no longer render bold.
### Tests
- `test(search)`: pins the streaming-aggregate **wire contract** at the
HTTP
boundary. Under streaming aggregates each partition carries the
cumulative
merged state (last frame is the answer) while ordinary partitions are
disjoint
and must be summed. No pytest had ever received `streaming_aggs: true` —
the
emitter could have flipped to per-partition deltas with the whole suite
green
while dashboards, Logs and the alert preview all silently over-counted.
- `test(e2e)`: four helpers that passed only while an org stayed small —
an
assertion reading past a 20-row page, an unfiltered `OSelect` that
virtualises
past 50 entries, a tab click landing on the rename pencil, and two
builder
tests asserting on whichever metric sorted first.
- New unit specs across the library components, composable, facets,
tunables and
install path.
### Notes
- **i18n**: en-US only; CI generates the other 14 locales.
- **No backend changes** — this branch is entirely `web/` plus test
helpers.
Adds automated test coverage for **SLOs** (#13547, #13577, #13579,
#13704, #13761, #13784), which shipped with none. **149 tests** — 77
Playwright + 72 pytest — all three SLI types, both query languages,
happy / negative / edge paths. Clean branch off `main`.
**UI (Playwright · `tests/ui-testing`)**
- `pages/sloPages/` — 4 page objects (verified `data-test` selectors,
hydration + Monaco-sync waits, OSelect commit-retry, virtualized-list
filtering).
- `playwright-tests/SLO/slo-crud.spec.js` (24) — create
count/time-slice/grouped through the form, detail, edit, SLI-type
change, tags, pause, delete, filters; negative: dismissed delete, empty
search, cancel, unmeasured SLI is absent not zero.
- `playwright-tests/SLO/slo-form-validation.spec.js` (20) — server
errors surfacing in the form, conditional fields per SLI type, count +
time-slice previews, **PromQL count and time-slice created end to end**;
negative: no stream, invalid aggregate, over-long name, target
boundaries.
- `playwright-tests/SLO/slo-alert-sli.spec.js` (8) — **the `alert` SLI
type**: source picker, create, config carries only the alert id, detail
names the source; negative: ineligible source offered-but-disabled,
grouped alert SLI, slice finer than source cadence.
- `playwright-tests/SLO/slo-timeslice.spec.js` (10) — **#13761 P0
regression** (measured SLI, not the preview), PromQL, previews, gaps,
language-flip clearing.
- `playwright-tests/SLO/slo-alerts.spec.js` (15) — burn-rate +
error-budget kinds, presets, edit, freeze door, burndown; negative: no
destination, cancel, blank/invalid name.
- `playwright-tests/utils/slo-seed.js` — backdated seeding, metrics
seeding, eligible-source-alert fixture, ground-truth query, fail-fast
env guard, transient-store retry, full fixture teardown (alerts →
streams → destinations → templates).
**API (pytest · `tests/api-testing`)**
- `tests/slos/test_slo_validation.py` (72) — full REST surface
(create/get/update/enable/delete/groups/slo-eligible), validation on
**both create and update** (name, target, window, slice, config shapes,
comparators, float-threshold round-trip), boundaries (target
0.001/99.999, name 256 vs 257, threshold 0/negative/1e12/NaN, unicode
names), alert-source eligibility (silence-gated, too-infrequent, cron),
PromQL config shapes, auth 401/404.
**CI**
- Two shards: `SLO` (`workers: 3`) and `SLO-Measurement` (`workers: 1`).
Split because the backfill job runs with
`ZO_SCHEDULER_SLO_BACKFILL_CONCURRENCY=1`, so measurement specs sharing
an instance starve each other's waits — `fullyParallel` races spec
*files*, which `mode:'serial'` cannot prevent. The 3-worker pin is
SQLite: at 5, concurrent SLO/alert/destination writes trip `database is
locked` on the single-node meta store.
- New optional per-shard `ingest_allowed_upto` (`240`) alongside the
existing `quick_mode_enabled`: the workflow default of 5 **hours**
silently drops the 8 days of history an SLO window needs, while still
answering `200`. Other shards keep the default. Same precedent as
`api-testing.yml`'s `48`.
**Feature code**: 9 additive `data-test`/`dataTest` attributes in
`SloDetail.vue`, `SloList.vue`, `SloAlertCondition.vue` — no logic
changes, `vue-tsc` clean. The stat tiles, tabs and burn-window inputs
had no stable selectors; `OStatStrip`/`OTab` already forward the
attribute.
**Behaviour**: self-cleaning (`e2e_slo_*`, worker-scoped since
`afterAll` runs per worker).
**Verified in CI** on this branch: `e2e / SLO` **52 passed** (1.8m,
workers pinned to 3), `e2e / SLO-Measurement` **25 passed** (11.3m,
workers pinned to 1) — the measurement shard passing confirms the
per-shard `ingest_allowed_upto` took effect, since those tests are
impossible under the 5-hour default. pytest **72 passed** locally; in
CI's 599-test run one setup delete hit a transient 500 (see findings)
and now retries.
**Findings for the feature owner** (reported, not fixed here):
- A blank "good when" shows `Request failed with status code 422` —
`pruned()` drops the empty key, so the API's real 400 message never
reaches `save()`.
- A rejected alert name shows no banner: `SloAlertForm.submit()` returns
on `nameError` before setting `saveError`.
- `avg(latency_ms); DROP TABLE users--` is accepted at create time
(parsed only at measurement).
- The name-length message says "less than 256 characters" but 256 is
accepted — 257 is the first rejection.
- Duplicate alert names on one SLO are accepted (duplicate SLO names are
a 409), and the panel lists them by name, so they are indistinguishable.
- SLO names accept characters that break the list's own
`slos-slolist-edit-${name}` selectors; alert names correctly reject
them.
- All chart/preview root `data-test` ids are unreachable — a parent's
fallthrough attribute overrides the child root's.
- Both edit forms hydrate asynchronously, so a blank form can flash
before the stored values appear.
- `DELETE /slos/{id}` on a freshly-created SLO returned **500** once
under CI load (not reproducible locally in 12 rapid iterations).
Contention is arguably a retryable 503, and the body carried no
actionable message.
- **A scheduled alert whose source stream is deleted is retried
forever** (`Stream "…" not found in schema`), and the SLO backfill lane
shares the scheduler's worker budget — so accumulated dead alerts
*starve SLO measurement*. Observed: 65 of them held coverage at 1 of 7
chunks indefinitely; deleting them restored 0.143 → 1.0 in 240s.
Harmless in CI (fresh instance per shard) but it degrades any long-lived
environment. The suite now cleans up after itself; the retry-forever
behaviour itself may deserve a look.
**Paired ENT PR: openobserve/o2-enterprise#2434.** ENT's matrix is this
repo's manifest plus an overlay, merged by this repo's
`build-ci-matrix.js`, so both SLO shards reach ENT automatically —
verified: 31 merged shards, both SLO entries carrying
`ingest_allowed_upto=240` and their worker pins, no other shard
affected. What ENT was missing is *consuming* those two fields, which
#2434 adds. Merge the two together: this one alone leaves ENT booting
the SLO shards on the 5-hour ingest window, and #2434 alone is inert.
*(An earlier review reported ENT as using an inline matrix with no SLO
shard — that turned out not to be the case.)*
## What
First test coverage for the dashboard **panel** endpoints and the
**dashboard
search path**, running CTE/subquery `match_all` queries as real
dashboard panels.
Nothing in the repo previously touched either:
`grep -rn "/panels\|search_type=dashboards" tests/api-testing/` returned
zero hits.
## Why
Panels query `_search?type=logs&search_type=dashboards&dashboard_id=…` —
a
different route into the engine than the Logs page (`search_type=ui`).
The
`match_all` + FST projection rewrite fixed in #13808 runs inside it, so
a query
that passes via Logs can still regress on a dashboard.
The rewrite has to preserve the projection of the block the predicate
sits in.
When that block is the whole statement, the projection is obvious. When
it's a
CTE body feeding a JOIN, a window, or a four-branch UNION ALL, it has to
preserve
the *CTE's* output schema while the outer query reads different columns
from it —
which is where it went wrong.
## Coverage
**71 distinct queries across 29 combination families**, all with
`match_all`
inside a `WITH …` body or a derived table:
| Axis | Variants |
|---|---|
| Nesting depth | 1, 2, 3, and 5-level CTE chains |
| Nesting kind | CTEs and derived tables (and one query using both) |
| UNION ALL | 2, 3, and 4 separate `match_all` branches |
| JOIN | CTE⋈CTE, derived⋈derived, cross-stream |
| Window | `ROW_NUMBER`, `RANK`, `DENSE_RANK`, partitioned aggregates |
| Negation | `NOT match_all`, and mixed with positive `match_all` |
| Combined with | `COUNT(DISTINCT)`, `CASE`, `HAVING`, `histogram()` |
Each case creates a real panel, runs it via the dashboard search path,
then
asserts row count against the DuckDB oracle and that no expected column
was
dropped from the projection. A further 30 cases run the same SQL through
**both**
Logs and dashboard and diff the results — that's what catches a rewrite
firing on
only one path.
Queries are reused from the existing query-agent corpus rather than
duplicated,
so the two suites can't drift: query-agent verifies the values via the
Logs path,
this verifies the same SQL survives the dashboard path.
Cases are **discovered by classifying the SQL**, not from a hand-written
ID list,
so new combinations added to the corpus are picked up automatically.
Test ids
name the shape (`Q1022-cte1_union_cdist_4ma`), so a failure identifies
the
combination without opening the corpus.
Plus 14 tests for the panel endpoints themselves — AddPanel /
UpdatePanel /
DeletePanel, contract + CRUD lifecycle + 7 error paths derived from
`DashboardError` in `src/core/src/http.rs`.
## Changes
| File | |
|---|---|
| `tests/dashboards/test_dashboard_cte_panels.py` | new — 103 tests |
| `tests/dashboards/test_dashboard_panels.py` | new — 14 tests |
| `tests/dashboards/conftest.py` | new — shared fixtures |
| `support/panel_queries.py` | new — discovery + panel helpers |
| `support/endpoints/dashboards.py` | +78, additive |
| `support/endpoints/search.py` | +36, additive |
| `support/factories.py` | +73, additive |
The three shared-file edits are **187 added lines, 0 modified** —
nothing
existing can break.
## Results
168 tests in tests/api-testing/tests/dashboards/ (117 new + 51
pre-existing)
166 passed, 1 skipped, 1 xfailed # full nested-query corpus present
74 passed, 2 skipped, 1 xfailed # current corpus on main
Repeated 2× and 4× respectively, no flakes.
Discovery scales with the corpus: on main it finds 8 nested `match_all`
cases,
and 71 once the larger sentinel set is present. Cases that aren't in the
corpus
skip with an explanatory message instead of failing, so this is safe to
merge in
either order.
Hengfei Yang
Prabhat Sharmaandgithub-actions[bot]
Ashish Kolheandgithub-actions[bot]
Vamsi Karthik R
Neha P
Amey PawarandHengfei Yang
uddhavdave
Bhargav
