mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-04 08:09:27 +00:00
4772df1723
Design at: alerts_2.md Per-group and per-series alerting, SLO measurement, and the UI for both. ## Alerts **Multi-alerts (Feature 3).** A grouped alert can opt in to evaluating and paging **per group** rather than collapsing to one verdict. Opt-in only (`multi_alert`), so no existing alert changes behaviour: the flag cannot be present in JSON written before it existed. Per-group state, transitions, silence fingerprints, disappearance/reaping and a cardinality cap, with the count-gate rules that make "most severe group" and the legacy collapsed level provably the same verdict. **Per-series alerting for PromQL.** The same feature for the metrics family, where a group is one returned **series** keyed by its full label set — that is a series' identity in Prometheus, and the expression's own `by (…)` clause is already where the label set is chosen, so a second picker could only disagree with it. One additive nullable column; NULL means off, so stored alerts are untouched. **Alert detail page**, replacing the row-click side panel: stat strip, evaluation chart with threshold marklines, per-group table and history. Simple alerts get a real history too — evaluations (one record per run) by default, level changes alongside. ## SLOs Count / time-slice / alert SLIs, error-budget and burn-rate arithmetic, rolling windows, per-group slices with an exact rollup, coverage gating, backfill, and SLO-based alerts. Two capabilities the model could not express, both added tests-first: - **Metrics-native counting** (`CountSource::PromQl`) — counter metrics have no rows for a predicate to classify, so "good" exists only as arithmetic between series, and correct counter arithmetic is `increase()`. - **Freshness semantics** (`absent_is_bad`) — for a pipeline SLO, silence IS the failure, which the default gap rule can never report. A *failed* query still freezes rather than pages; only a query that PROVED emptiness counts as bad. **SLO form**: live good/bad preview so a wrong predicate is caught before saving rather than days later, real query editors with field autocomplete, folders shared with alerts, and a move endpoint. ## Notes for review - The SLO and composite features are **deferred (TODO)**: `ZO_SLO_ENABLED` stays default **false** and the SLOs menu entry is hidden until they ship; the SLO backend and UI remain in the tree behind the flag, and composites stay pure logic + tests. - One migration (schema 61): a nullable column for the PromQL opt-in. No backfill. - Two fixes outside the feature that it surfaced: query-editor completions were duplicated per mounted editor (Monaco registers providers per *language*), and the alert detail chart had no PromQL branch, so it charted row counts against the series-count gate. - Lands together with the matching o2-enterprise PR — `QueryType::Slo` is new here and the enterprise dedup matches are exhaustive over it. - `ratelimit::test_add_batch_missing_rule_id` fails intermittently on `main` as well; unrelated to this branch. --------- Co-authored-by: Shrinath Rao <shnath@openobserve.ai> Co-authored-by: hengfeiyang <hengfei.yang@gmail.com> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>