design at /slo
## What
Adds **SLO burn-rate alerts** — an alert family that fires off an SLO's
error budget rather than a raw query — and finishes the SLO surface
around it: navigation entry, detail-page fixes, and the flag removal now
that SLOs have shipped.
## Changes
**SLO burn-rate alerts (backend)**
- New burn-rate evaluation (`config/src/meta/slo/burn.rs`,
`condition.rs`) with multi-window burn conditions, and
`core/src/slo/evaluate.rs` extended to evaluate them on the SLO job.
- Alerts carry an `slo_id`; `infra/src/table/alerts` gains the column,
filtering, and query support so an SLO's alerts can be listed and
deleted with it.
- Alert levels (`meta/alerts/level.rs`) and tags extended for the SLO
family; scheduler handlers route SLO alerts through the same
notification path.
**SLO alert UI**
- `SloAlertForm` + `AlertDetailSlo`: create, edit and view burn-rate
alerts from the SLO detail page and the alert list.
- Payload/routing helpers (`utils/alerts/sloAlertPayload.ts`,
`sloAlertRouting.ts`, `alertPayload.ts`) with unit tests.
- `AlertDestinationsField` extracted as the complete Destination form
field — side label, tooltip, combined destinations/workflows picker,
inline error, refresh, and create-in-new-tab. It replaces two verbatim
copies in the generic alert form and is reused by the SLO form and
synthetics `CheckAlerts` (which sets `supportsWorkflows: false`, since a
check has no workflow routing). Fixes the SLO form's previously unwired
refresh and Add Destination buttons.
**SLO detail page fixes**
- Burndown and burn-rate charts drew as bare axes when coverage was
sparse: `showSymbol: false` means an isolated measured bucket between
gaps has no segment and paints nothing. Symbols are now sized per point,
and a sparse-coverage note states how many buckets actually recorded
events.
- Nothing scrolled below ~700px of viewport height — `OPageLayout`'s
fixed flex body plus `OTabPanels` `scroll="none"` clipped the panels.
Panels now grow and scroll; the stat strip and tab bar stay pinned.
Verified at 1024x500, 1280x620, 1512x945.
**Other**
- `ZO_SLO_ENABLED` removed — SLOs have shipped, so the flag and its
status/route gating are gone.
- SLOs surfaced in the main nav.
- Dashboard chart panel header gets its own tint via new
`--color-panel-bar-bg` / `--color-panel-bar-border` tokens, and the bar
is extracted into `PanelBar.vue` (it existed as five hand-copied class
strings that had already drifted).
- `tsconfig.vitest.tsbuildinfo` untracked; third-party vendor name
removed from comments and test names.
## Testing
- New Rust tests across `meta/slo/burn.rs`, `condition.rs`,
`core/src/alerts/alert.rs` and `infra/src/table/alerts`.
- New frontend suites: `sloAlertPayload.spec.ts`,
`sloAlertRouting.spec.ts`, `alertPayload.spec.ts`,
`AlertDetailSlo.spec.ts`, `SloDetailAlerts.spec.ts`,
`SloListDelete.spec.ts`, `SloTimeSlicePreview.spec.ts`.
- `cargo clippy --all-targets` and `web` `lint:ci` clean.
Design at: alerts_2.md
Per-group and per-series alerting, SLO measurement, and the UI for both.
## Alerts
**Multi-alerts (Feature 3).** A grouped alert can opt in to evaluating
and paging **per group** rather than collapsing to one verdict. Opt-in
only (`multi_alert`), so no existing alert changes behaviour: the flag
cannot be present in JSON written before it existed. Per-group state,
transitions, silence fingerprints, disappearance/reaping and a
cardinality cap, with the count-gate rules that make "most severe group"
and the legacy collapsed level provably the same verdict.
**Per-series alerting for PromQL.** The same feature for the metrics
family, where a group is one returned **series** keyed by its full label
set — that is a series' identity in Prometheus, and the expression's own
`by (…)` clause is already where the label set is chosen, so a second
picker could only disagree with it. One additive nullable column; NULL
means off, so stored alerts are untouched.
**Alert detail page**, replacing the row-click side panel: stat strip,
evaluation chart with threshold marklines, per-group table and history.
Simple alerts get a real history too — evaluations (one record per run)
by default, level changes alongside.
## SLOs
Count / time-slice / alert SLIs, error-budget and burn-rate arithmetic,
rolling windows, per-group slices with an exact rollup, coverage gating,
backfill, and SLO-based alerts.
Two capabilities the model could not express, both added tests-first:
- **Metrics-native counting** (`CountSource::PromQl`) — counter metrics
have no rows for a predicate to classify, so "good" exists only as
arithmetic between series, and correct counter arithmetic is
`increase()`.
- **Freshness semantics** (`absent_is_bad`) — for a pipeline SLO,
silence IS the failure, which the default gap rule can never report. A
*failed* query still freezes rather than pages; only a query that PROVED
emptiness counts as bad.
**SLO form**: live good/bad preview so a wrong predicate is caught
before saving rather than days later, real query editors with field
autocomplete, folders shared with alerts, and a move endpoint.
## Notes for review
- The SLO and composite features are **deferred (TODO)**:
`ZO_SLO_ENABLED` stays default **false** and the SLOs menu entry is
hidden until they ship; the SLO backend and UI remain in the tree behind
the flag, and composites stay pure logic + tests.
- One migration (schema 61): a nullable column for the PromQL opt-in. No
backfill.
- Two fixes outside the feature that it surfaced: query-editor
completions were duplicated per mounted editor (Monaco registers
providers per *language*), and the alert detail chart had no PromQL
branch, so it charted row counts against the series-count gate.
- Lands together with the matching o2-enterprise PR — `QueryType::Slo`
is new here and the enterprise dedup matches are exhaustive over it.
- `ratelimit::test_add_batch_missing_rule_id` fails intermittently on
`main` as well; unrelated to this branch.
---------
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
Co-authored-by: hengfeiyang <hengfei.yang@gmail.com>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
Hengfei Yang
Loakesh Indiranandgithub-actions[bot]
Ashish Kolhe
Prabhat Sharma
Huaijin