2 Commits
Author SHA1 Message Date
Prabhat Sharma fea71e02d4 feat(alerts): alert library — browse, preview and install curated alerts (#13922)
## Alert Library

A curated catalog at `/alert-library`: browse packs of ready-made
alerts, see
which ones can actually run against the streams this org already
receives,
preview a verdict against real data, and install one or many.

The library is a **peer page** of Alerts / Notification Destinations /
Templates
— flat routing, presented as tabs via a shared `AlertSectionTabs` in
`OPageLayout #header-tabs` (the pattern `PipelineSectionTabs` already
ships).

### What landed, by phase

| Phase | What |
| --- | --- |
| Data layer | S3 manifest fetch, cache, readiness resolution
(`useAlertLibrary`, `constants/alertLibrary`, `types/alertLibrary`) |
| Gallery | Rail (packs / categories, multi-select, segmented switch),
stat-strip availability facets, cards |
| Drawer | Detail view with query preview and customizable tunables |
| Install | Bulk-install wizard — destination selection, per-alert
tunables, stale-session hardening |
| Prefill | Carry a library alert (query, trigger, threshold, severity)
into the alert editor |

### Design decisions worth knowing

- **Rail navigates, strip filters.** Availability is deliberately not in
the rail
  — carrying both overflowed it and pushed a facet group below the fold.
- **Colour marks the exception only.** No "Ready" pill (it would
describe ~75 of
  86 cards and signal nothing); only unusable cards carry a label.
- **A failed `/streams` call leaves readiness unknown**, rather than
claiming
"none of your data matches" — that claim would be about the request, not
the
org, and it is the most alarming thing this page can say. Counts read
`—`
until there is an answer, and a failed *refresh* keeps the verdict it
already had.
- **No "Installed" tile** — installed detection doesn't exist yet, and a
tile
  reading a permanent zero is dishonest UI.
- **`severity: warning` renders as `medium`** — the badge group's scale
is
critical/high/medium/low/info, and medium is the amber it also gives P3,
which
  is exactly the priority a warning alert installs as.
- Pack and category names come from the manifest, so they render as
`raw()` data —
  a pack added upstream needs no frontend release.

### Design-system changes (used by the library, applied app-wide)

- **`warning-quiet` badge tone** — one variant whose values differ by
theme: the
soft amber wash in light, a neutral chip in dark. Neither the filled
amber chip
nor the neutral one was right in both, because the two ramps aren't
symmetric.
- **Border tokens now meet WCAG 2.1 SC 1.4.11.** Every border token
measured
under 3:1 (`border-strong` was 2.52:1). `border-default` firms to
grey-300 and
  controls get a token that clears 3:1.
- Nav: Reliability's flyout groups its four alerting entries under one
**Alerts**
header, declared in the same order as the tab strip those pages carry.
Flyout
  levels are sized by depth.
- `sensors-off` icon added to the `OIcon` registry.

### Fixes that rode along

- **metrics**: a shared URL whose `metrics_data` blob omits panel query
fields no
longer blanks the whole metrics page — blob queries are filled out to
the panel
  schema before being laid over.
- **dashboards**: a PromQL series with no legend template was named
`JSON.stringify(metric)`, so fifty series read identically for the width
of the
chip. Series are now named by the labels that actually differ,
preferring the
ones that say *which workload* it is (namespace / service / pod / host).
Correlation's semantic groups refine that when loaded, but nothing is
fetched
to render a chart — that endpoint is enterprise-only and a chart must
render
  the same on OSS.
- **alerts**: preview evaluates against the alert's real trigger;
deleting a row
no longer reloads the list behind it; peer tabs no longer shift when
switching
  alerting pages.
- **ui**: badge-only select options no longer render bold.

### Tests

- `test(search)`: pins the streaming-aggregate **wire contract** at the
HTTP
boundary. Under streaming aggregates each partition carries the
cumulative
merged state (last frame is the answer) while ordinary partitions are
disjoint
and must be summed. No pytest had ever received `streaming_aggs: true` —
the
emitter could have flipped to per-partition deltas with the whole suite
green
while dashboards, Logs and the alert preview all silently over-counted.
- `test(e2e)`: four helpers that passed only while an org stayed small —
an
assertion reading past a 20-row page, an unfiltered `OSelect` that
virtualises
past 50 entries, a tab click landing on the rename pencil, and two
builder
  tests asserting on whichever metric sorted first.
- New unit specs across the library components, composable, facets,
tunables and
  install path.

### Notes

- **i18n**: en-US only; CI generates the other 14 locales.
- **No backend changes** — this branch is entirely `web/` plus test
helpers.
2026-08-23 20:32:28 +00:00
Shrinath RaoandClaude Opus 4.7 61aa772d57 test: revamp api-testing suite (framework + 16 file rewrites + CI matrix) (#12372)
## Summary

Comprehensive rework of the `tests/api-testing/` suite. 30+ commits
implementing all 6 phases of the [revamp
plan](../../tests/ui-testing/MD_Files/api%20tests%20revamp/API_TESTS_REVAMP.md).

**Before → After:**

| | Baseline | Final | Δ |
|---|---|---|---|
| passed | 421 | **443** | **+22 ✓** |
| failed | 3 | **2** | -1 (only same pre-existing) |
| skipped | 53 | 39 | -14 (unskipped dead tests now run) |
| xfailed | 1 | 2 | +1 (new visible backend-bug signal) |
| runtime | 588s | **387s** | **-201s (34% faster)** |

**Zero tests that passed in baseline are now failing.** The 2 remaining
failures are the exact pre-existing ones
(`test_enrichment_table_join.py` backend panic + `test_workflow.py` SMTP
env-miss).

## What's in the diff

- **New `support/` framework** — `OpenObserveClient`, endpoint wrappers
(`alerts`, `dashboards`, `streams`, `search`, etc.), `wait_until()`,
payload factories, pytest fixtures, SSE parser. 14 framework selfcheck
tests.
- **16 of 16 OLD-tier test files rewritten** — using the new framework,
unique-suffix names, body validation, `wait_until` instead of
`time.sleep`, parametrize where appropriate.
- **29 files moved into 12 feature folders** mirroring the UI suite
layout
(`tests/{alerts,dashboards,ingest,orgs,pipelines,regression,rum,search,streams,users,workflow}/`).
- **−6,500 LOC net** (copy-paste + dead code removal; new framework adds
back ~1,500).
- **Ruff config** in pyproject.toml: T201 ban on `print()`, PT pytest
rules, B bugbear, UP pyupgrade.

## CI workflow changes

- New job `api_regression_tests`: `strategy.matrix` over the 4 env-flag
combinations (UTF8_VIEW × JOIN_MATCH_ONE) — now runs in parallel instead
of as 4 sequential server reboots in the main job.
- New job `api_fuzz_tests`: schemathesis OpenAPI-driven fuzz against
`/api-doc/openapi.json` (214 endpoints tested). Informational
(`continue-on-error: true`). Each run surfaces **7–10 distinct 5xx
panics** (e.g., empty `sql` body returning 500 instead of 400, traces
endpoints crashing on unescaped path params that get interpolated into
SQL). Filed to backend team for separate triage.
- Server `o2_*.log` files uploaded as artifacts on failure (was:
silently lost).
- `api_tests_summary` now requires BOTH integration + regression to
pass.

### CI infrastructure fixes

A short hardening pass landed after the initial CI run surfaced several
issues:

- **pytest-sugar count inflation** — dropped `--force-sugar` from both
pytest invocations. The flag double-registered sugar on top of the
auto-loaded entry point, inflating the CI test counter ~3x (display
showed 1364 when collection was 486). Counter is now honest.
- **schemathesis 4.x flag migration** — `--base-url` → `-u`,
`--hypothesis-max-examples` → `-n`, `--hypothesis-deadline` →
`--request-timeout`, `--data-generation-method` → `-m`, dropped
`--auth-type basic`. The fuzz job was failing immediately on flag-parse
before this.
- **Regression matrix binary path** — used `${{ github.workspace
}}/target/debug/openobserve` so the absolute path resolves correctly
under `working-directory: tests/api-testing/` (was: every matrix job
failed with "No such file or directory").
- **`permissions: contents: read`** added to all 4 workflow jobs —
closes CodeQL alert #369.
- **Removed redundant server boot** in `api_integration_tests` (leftover
from main's structure after the regression matrix split; was silently
failing to bind to :5080 and getting masked by the first server).

## Backend bugs surfaced (for backend team to track)

The rework uncovered and visibly flagged these as `xfail`/documented
test comments:

1. `DELETE /api/organizations/{id}` returns 404 unconditionally even for
just-created orgs — endpoint appears unimplemented.
2. `test_dashboard_missing_required_fields` triggers backend panic
(returns 500 instead of 400/422).
3. JS `getEnrichmentTableData()` restricted to `_meta` org AND not
exposed in the `/functions/test` endpoint runtime — `ReferenceError`
even with explicit `enrichment_tables: [...]`.
4. Bulk API silently sanitizes invalid `_index` values (`****` → `_`)
without flagging — user data lands in unexpected index.
5. `AVG` of a single value drifts by ~1e-14 (likely float reduction
order; documented tolerance in `test_advanced_sql.py`).
6. Org name validation rejects hyphens (only
`A-Z/a-z/0-9/spaces/underscores`) — not in spec.

Plus 7–10 additional 5xx panics surfaced per fuzz run (input validation
gaps — separate triage list in fuzz job logs).

## Test bugs uncovered + fixed

- `test_pipeline.py`: had two function definitions named
`test_pipeline_creation_and_action_with_schedule` — Python's last-wins
meant the first test's body never executed for years.
- `test_alertsV2.py`: `assert alert_disabled == "False"` was indented
inside an `else:` branch that never executed because the response is
always a dict; AND `enabled` is a bool, not the string `"False"`. Two
layers of broken.
- `test_bulk.py`: variable typo (`valid_bulk_payload` reused on the POST
call) made the invalid-index test silently duplicate the valid case.
- `test_organisations.py`: `test_create_organization` used pre-pytest-4
`yield`-style fixtures — silently ignored by pytest 8.
- `test_js_result_array.py`: 8 tests skipped with `"Temporarily disabled
- failing test"` — the actual cause was that JS functions are restricted
to `_meta` org. One-line fix unskipped all 8.
- `test_fuzzy.py`: 4 tests `@pytest.mark.skip`-ed because they used a
1-minute time window against older static test data → 0 hits → flake →
skip. Wider window unskips them.
- 547 LOC of zombie commented-out test blocks deleted (15 commented `#
def test_` definitions, including one orphan case where the `def` was
commented but the indented body wasn't).

## Folder layout

```
tests/api-testing/
├── support/                    # NEW framework
│   ├── client.py               # OpenObserveClient
│   ├── endpoints/              # 7 per-resource wrappers
│   ├── factories.py            # unique_name, time_window, *_payload
│   ├── wait.py                 # wait_until + WaitTimeout
│   ├── sse.py                  # read_sse_response
│   └── fixtures.py             # client, temp_user_email, temp_stream_name, ...
├── helpers/                    # Non-test shared code (was scattered in tests/)
│   ├── sourcemap_helpers.py
│   └── workflow/               # workflow chain + pages/ live here now
├── fixtures/
│   └── rum/                    # Standalone RUM ingest scripts (was in tests/)
└── tests/
    ├── alerts/                 # 3 files
    ├── dashboards/             # 4 files
    ├── ingest/                 # 4 files
    ├── orgs/                   # 2 files
    ├── pipelines/              # 10 files
    ├── regression/             # 2 files (env-flag matrix targets)
    ├── rum/                    # 3 files
    ├── search/                 # 8 files
    ├── streams/                # 1 file
    ├── users/                  # 1 file
    ├── workflow/               # 3 files
    ├── conftest.py
    └── test_framework_selfcheck.py
```

## Test plan

- [x] `rye run pytest --collect-only` — 486 tests discovered
- [x] Full default-set run (`--ignore=tests/regression`) — 443 pass / 2
fail / 39 skip / 2 xfail in 387s
- [x] `rye run pytest -m framework` — 14/14 selfcheck tests pass
- [x] `rye run ruff check support/ tests/test_framework_selfcheck.py` —
clean
- [x] CI run on this PR — `api_integration_tests` + all 4
`api_regression_tests` matrix jobs green
- [x] CI run — `api_fuzz_tests` runs cleanly (informational; surfaces
7–10 unique 5xx panics per run)

## Note on test count

The `test_streaming.py` rewrite folded 7 "tests" (`test_ingest_data`,
`test_ingest_join`, `test_disable_streaming`,
`test_update_max_query_range`, `test_enable_streaming`,
`test_delete_stream`, `test_delete_stream_join`) into pytest fixtures.
Each only checked `status_code == 200` of a setup/teardown operation —
they were imperative steps masquerading as tests. The fixtures still
perform the same operations and assert their success; they're just no
longer counted as discrete pytest items. **Functional coverage is
identical.**

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-06-01 07:19:48 +00:00