## Alert Library
A curated catalog at `/alert-library`: browse packs of ready-made
alerts, see
which ones can actually run against the streams this org already
receives,
preview a verdict against real data, and install one or many.
The library is a **peer page** of Alerts / Notification Destinations /
Templates
— flat routing, presented as tabs via a shared `AlertSectionTabs` in
`OPageLayout #header-tabs` (the pattern `PipelineSectionTabs` already
ships).
### What landed, by phase
| Phase | What |
| --- | --- |
| Data layer | S3 manifest fetch, cache, readiness resolution
(`useAlertLibrary`, `constants/alertLibrary`, `types/alertLibrary`) |
| Gallery | Rail (packs / categories, multi-select, segmented switch),
stat-strip availability facets, cards |
| Drawer | Detail view with query preview and customizable tunables |
| Install | Bulk-install wizard — destination selection, per-alert
tunables, stale-session hardening |
| Prefill | Carry a library alert (query, trigger, threshold, severity)
into the alert editor |
### Design decisions worth knowing
- **Rail navigates, strip filters.** Availability is deliberately not in
the rail
— carrying both overflowed it and pushed a facet group below the fold.
- **Colour marks the exception only.** No "Ready" pill (it would
describe ~75 of
86 cards and signal nothing); only unusable cards carry a label.
- **A failed `/streams` call leaves readiness unknown**, rather than
claiming
"none of your data matches" — that claim would be about the request, not
the
org, and it is the most alarming thing this page can say. Counts read
`—`
until there is an answer, and a failed *refresh* keeps the verdict it
already had.
- **No "Installed" tile** — installed detection doesn't exist yet, and a
tile
reading a permanent zero is dishonest UI.
- **`severity: warning` renders as `medium`** — the badge group's scale
is
critical/high/medium/low/info, and medium is the amber it also gives P3,
which
is exactly the priority a warning alert installs as.
- Pack and category names come from the manifest, so they render as
`raw()` data —
a pack added upstream needs no frontend release.
### Design-system changes (used by the library, applied app-wide)
- **`warning-quiet` badge tone** — one variant whose values differ by
theme: the
soft amber wash in light, a neutral chip in dark. Neither the filled
amber chip
nor the neutral one was right in both, because the two ramps aren't
symmetric.
- **Border tokens now meet WCAG 2.1 SC 1.4.11.** Every border token
measured
under 3:1 (`border-strong` was 2.52:1). `border-default` firms to
grey-300 and
controls get a token that clears 3:1.
- Nav: Reliability's flyout groups its four alerting entries under one
**Alerts**
header, declared in the same order as the tab strip those pages carry.
Flyout
levels are sized by depth.
- `sensors-off` icon added to the `OIcon` registry.
### Fixes that rode along
- **metrics**: a shared URL whose `metrics_data` blob omits panel query
fields no
longer blanks the whole metrics page — blob queries are filled out to
the panel
schema before being laid over.
- **dashboards**: a PromQL series with no legend template was named
`JSON.stringify(metric)`, so fifty series read identically for the width
of the
chip. Series are now named by the labels that actually differ,
preferring the
ones that say *which workload* it is (namespace / service / pod / host).
Correlation's semantic groups refine that when loaded, but nothing is
fetched
to render a chart — that endpoint is enterprise-only and a chart must
render
the same on OSS.
- **alerts**: preview evaluates against the alert's real trigger;
deleting a row
no longer reloads the list behind it; peer tabs no longer shift when
switching
alerting pages.
- **ui**: badge-only select options no longer render bold.
### Tests
- `test(search)`: pins the streaming-aggregate **wire contract** at the
HTTP
boundary. Under streaming aggregates each partition carries the
cumulative
merged state (last frame is the answer) while ordinary partitions are
disjoint
and must be summed. No pytest had ever received `streaming_aggs: true` —
the
emitter could have flipped to per-partition deltas with the whole suite
green
while dashboards, Logs and the alert preview all silently over-counted.
- `test(e2e)`: four helpers that passed only while an org stayed small —
an
assertion reading past a 20-row page, an unfiltered `OSelect` that
virtualises
past 50 entries, a tab click landing on the rename pencil, and two
builder
tests asserting on whichever metric sorted first.
- New unit specs across the library components, composable, facets,
tunables and
install path.
### Notes
- **i18n**: en-US only; CI generates the other 14 locales.
- **No backend changes** — this branch is entirely `web/` plus test
helpers.
Adds automated test coverage for **SLOs** (#13547, #13577, #13579,
#13704, #13761, #13784), which shipped with none. **149 tests** — 77
Playwright + 72 pytest — all three SLI types, both query languages,
happy / negative / edge paths. Clean branch off `main`.
**UI (Playwright · `tests/ui-testing`)**
- `pages/sloPages/` — 4 page objects (verified `data-test` selectors,
hydration + Monaco-sync waits, OSelect commit-retry, virtualized-list
filtering).
- `playwright-tests/SLO/slo-crud.spec.js` (24) — create
count/time-slice/grouped through the form, detail, edit, SLI-type
change, tags, pause, delete, filters; negative: dismissed delete, empty
search, cancel, unmeasured SLI is absent not zero.
- `playwright-tests/SLO/slo-form-validation.spec.js` (20) — server
errors surfacing in the form, conditional fields per SLI type, count +
time-slice previews, **PromQL count and time-slice created end to end**;
negative: no stream, invalid aggregate, over-long name, target
boundaries.
- `playwright-tests/SLO/slo-alert-sli.spec.js` (8) — **the `alert` SLI
type**: source picker, create, config carries only the alert id, detail
names the source; negative: ineligible source offered-but-disabled,
grouped alert SLI, slice finer than source cadence.
- `playwright-tests/SLO/slo-timeslice.spec.js` (10) — **#13761 P0
regression** (measured SLI, not the preview), PromQL, previews, gaps,
language-flip clearing.
- `playwright-tests/SLO/slo-alerts.spec.js` (15) — burn-rate +
error-budget kinds, presets, edit, freeze door, burndown; negative: no
destination, cancel, blank/invalid name.
- `playwright-tests/utils/slo-seed.js` — backdated seeding, metrics
seeding, eligible-source-alert fixture, ground-truth query, fail-fast
env guard, transient-store retry, full fixture teardown (alerts →
streams → destinations → templates).
**API (pytest · `tests/api-testing`)**
- `tests/slos/test_slo_validation.py` (72) — full REST surface
(create/get/update/enable/delete/groups/slo-eligible), validation on
**both create and update** (name, target, window, slice, config shapes,
comparators, float-threshold round-trip), boundaries (target
0.001/99.999, name 256 vs 257, threshold 0/negative/1e12/NaN, unicode
names), alert-source eligibility (silence-gated, too-infrequent, cron),
PromQL config shapes, auth 401/404.
**CI**
- Two shards: `SLO` (`workers: 3`) and `SLO-Measurement` (`workers: 1`).
Split because the backfill job runs with
`ZO_SCHEDULER_SLO_BACKFILL_CONCURRENCY=1`, so measurement specs sharing
an instance starve each other's waits — `fullyParallel` races spec
*files*, which `mode:'serial'` cannot prevent. The 3-worker pin is
SQLite: at 5, concurrent SLO/alert/destination writes trip `database is
locked` on the single-node meta store.
- New optional per-shard `ingest_allowed_upto` (`240`) alongside the
existing `quick_mode_enabled`: the workflow default of 5 **hours**
silently drops the 8 days of history an SLO window needs, while still
answering `200`. Other shards keep the default. Same precedent as
`api-testing.yml`'s `48`.
**Feature code**: 9 additive `data-test`/`dataTest` attributes in
`SloDetail.vue`, `SloList.vue`, `SloAlertCondition.vue` — no logic
changes, `vue-tsc` clean. The stat tiles, tabs and burn-window inputs
had no stable selectors; `OStatStrip`/`OTab` already forward the
attribute.
**Behaviour**: self-cleaning (`e2e_slo_*`, worker-scoped since
`afterAll` runs per worker).
**Verified in CI** on this branch: `e2e / SLO` **52 passed** (1.8m,
workers pinned to 3), `e2e / SLO-Measurement` **25 passed** (11.3m,
workers pinned to 1) — the measurement shard passing confirms the
per-shard `ingest_allowed_upto` took effect, since those tests are
impossible under the 5-hour default. pytest **72 passed** locally; in
CI's 599-test run one setup delete hit a transient 500 (see findings)
and now retries.
**Findings for the feature owner** (reported, not fixed here):
- A blank "good when" shows `Request failed with status code 422` —
`pruned()` drops the empty key, so the API's real 400 message never
reaches `save()`.
- A rejected alert name shows no banner: `SloAlertForm.submit()` returns
on `nameError` before setting `saveError`.
- `avg(latency_ms); DROP TABLE users--` is accepted at create time
(parsed only at measurement).
- The name-length message says "less than 256 characters" but 256 is
accepted — 257 is the first rejection.
- Duplicate alert names on one SLO are accepted (duplicate SLO names are
a 409), and the panel lists them by name, so they are indistinguishable.
- SLO names accept characters that break the list's own
`slos-slolist-edit-${name}` selectors; alert names correctly reject
them.
- All chart/preview root `data-test` ids are unreachable — a parent's
fallthrough attribute overrides the child root's.
- Both edit forms hydrate asynchronously, so a blank form can flash
before the stored values appear.
- `DELETE /slos/{id}` on a freshly-created SLO returned **500** once
under CI load (not reproducible locally in 12 rapid iterations).
Contention is arguably a retryable 503, and the body carried no
actionable message.
- **A scheduled alert whose source stream is deleted is retried
forever** (`Stream "…" not found in schema`), and the SLO backfill lane
shares the scheduler's worker budget — so accumulated dead alerts
*starve SLO measurement*. Observed: 65 of them held coverage at 1 of 7
chunks indefinitely; deleting them restored 0.143 → 1.0 in 240s.
Harmless in CI (fresh instance per shard) but it degrades any long-lived
environment. The suite now cleans up after itself; the retry-forever
behaviour itself may deserve a look.
**Paired ENT PR: openobserve/o2-enterprise#2434.** ENT's matrix is this
repo's manifest plus an overlay, merged by this repo's
`build-ci-matrix.js`, so both SLO shards reach ENT automatically —
verified: 31 merged shards, both SLO entries carrying
`ingest_allowed_upto=240` and their worker pins, no other shard
affected. What ENT was missing is *consuming* those two fields, which
#2434 adds. Merge the two together: this one alone leaves ENT booting
the SLO shards on the 5-hour ingest window, and #2434 alone is inert.
*(An earlier review reported ENT as using an inline matrix with no SLO
shard — that turned out not to be the case.)*
## What
First test coverage for the dashboard **panel** endpoints and the
**dashboard
search path**, running CTE/subquery `match_all` queries as real
dashboard panels.
Nothing in the repo previously touched either:
`grep -rn "/panels\|search_type=dashboards" tests/api-testing/` returned
zero hits.
## Why
Panels query `_search?type=logs&search_type=dashboards&dashboard_id=…` —
a
different route into the engine than the Logs page (`search_type=ui`).
The
`match_all` + FST projection rewrite fixed in #13808 runs inside it, so
a query
that passes via Logs can still regress on a dashboard.
The rewrite has to preserve the projection of the block the predicate
sits in.
When that block is the whole statement, the projection is obvious. When
it's a
CTE body feeding a JOIN, a window, or a four-branch UNION ALL, it has to
preserve
the *CTE's* output schema while the outer query reads different columns
from it —
which is where it went wrong.
## Coverage
**71 distinct queries across 29 combination families**, all with
`match_all`
inside a `WITH …` body or a derived table:
| Axis | Variants |
|---|---|
| Nesting depth | 1, 2, 3, and 5-level CTE chains |
| Nesting kind | CTEs and derived tables (and one query using both) |
| UNION ALL | 2, 3, and 4 separate `match_all` branches |
| JOIN | CTE⋈CTE, derived⋈derived, cross-stream |
| Window | `ROW_NUMBER`, `RANK`, `DENSE_RANK`, partitioned aggregates |
| Negation | `NOT match_all`, and mixed with positive `match_all` |
| Combined with | `COUNT(DISTINCT)`, `CASE`, `HAVING`, `histogram()` |
Each case creates a real panel, runs it via the dashboard search path,
then
asserts row count against the DuckDB oracle and that no expected column
was
dropped from the projection. A further 30 cases run the same SQL through
**both**
Logs and dashboard and diff the results — that's what catches a rewrite
firing on
only one path.
Queries are reused from the existing query-agent corpus rather than
duplicated,
so the two suites can't drift: query-agent verifies the values via the
Logs path,
this verifies the same SQL survives the dashboard path.
Cases are **discovered by classifying the SQL**, not from a hand-written
ID list,
so new combinations added to the corpus are picked up automatically.
Test ids
name the shape (`Q1022-cte1_union_cdist_4ma`), so a failure identifies
the
combination without opening the corpus.
Plus 14 tests for the panel endpoints themselves — AddPanel /
UpdatePanel /
DeletePanel, contract + CRUD lifecycle + 7 error paths derived from
`DashboardError` in `src/core/src/http.rs`.
## Changes
| File | |
|---|---|
| `tests/dashboards/test_dashboard_cte_panels.py` | new — 103 tests |
| `tests/dashboards/test_dashboard_panels.py` | new — 14 tests |
| `tests/dashboards/conftest.py` | new — shared fixtures |
| `support/panel_queries.py` | new — discovery + panel helpers |
| `support/endpoints/dashboards.py` | +78, additive |
| `support/endpoints/search.py` | +36, additive |
| `support/factories.py` | +73, additive |
The three shared-file edits are **187 added lines, 0 modified** —
nothing
existing can break.
## Results
168 tests in tests/api-testing/tests/dashboards/ (117 new + 51
pre-existing)
166 passed, 1 skipped, 1 xfailed # full nested-query corpus present
74 passed, 2 skipped, 1 xfailed # current corpus on main
Repeated 2× and 4× respectively, no flakes.
Discovery scales with the corpus: on main it finds 8 nested `match_all`
cases,
and 71 once the larger sentinel set is present. Cases that aren't in the
corpus
skip with an explanatory message instead of failing, so this is safe to
merge in
either order.
Design at: #13822
## Summary
`distinct_values_*` metadata streams existed to serve one reader: the
`_values` API. They are a redundant — and lossier — materialization of
what the tantivy secondary index already holds, and the index has the
collectors needed to replace them outright (`SimpleTopN` /
`SimpleDistinct`). This PR removes the distinct streams from the write
and read paths and lets the `_values` API be served by the tantivy index
optimizer.
### 1. Treat distinct value fields as secondary index fields (read-time
union)
- `get_stream_setting_index_fields()` — the single accessor used by both
the index build side (ingester parquet/pack jobs, compactor merge) and
the query side (leader/follower index optimizers) — now unions
`distinct_value_fields` into the effective index field set, gated on
`enable_distinct_fields`. Fields that cannot be secondary-indexed (full
text search keys, partition keys, reserved columns) are skipped; the
values API falls back to a normal scan for those.
- The distinct list is deliberately **not merged into
`settings.index_fields`**: the two stay independent, so removing a
distinct value field (or disabling `enable_distinct_fields`) cleanly
stops its indexing with no residue, and the user-managed `index_fields`
list is never mutated behind their back.
- Correctness gate: a distinct-only field is considered indexed since
its `added_ts` (`get_stream_setting_index_updated_at_for_fields`), so
parquet files older than that are served by DataFusion scan instead of
the index (`split_file_list_by_time_range`). `added_ts` is set on add
and reset when `enable_distinct_fields` is re-enabled (existing
behavior). For streams configured before this release the fields only
start being indexed at upgrade time, so `added_ts` is additionally
clamped by a release-wide cutoff `/tantivy/distinct_fields/updated_at` —
a lazy marker initialized to the first access after upgrade, following
the existing `ttv_secondary_index_updated_at` pattern. No data migration
and no `DB_SCHEMA_VERSION` bump needed.
### 2. `_values` API always queries the original stream
- Removed `can_use_distinct_stream()` and the `distinct_values_*` /
`SUM(count)` / `StreamType::Metadata` rewrite from `values_inner` (GET
`/_values`, formerly `values_v1`) and `build_search_request_per_field`
(POST `/_values_stream`).
- The generated SQL shapes are exactly what the index optimizer detects:
- `SELECT f AS zo_sql_key, COUNT(*) AS zo_sql_num FROM s GROUP BY
zo_sql_key ORDER BY zo_sql_num DESC, zo_sql_key ASC` + limit →
`SimpleTopN`
- `no_count`: `SELECT f AS zo_sql_key FROM s GROUP BY zo_sql_key ORDER
BY zo_sql_key ASC` + limit → `SimpleDistinct`
- The `keyword` filter (`str_match_ignore_case`) is a valid index
condition, so keyword-filtered values queries stay on the index path
too.
### 3. Remove distinct values generation
- Dropped the per-record distinct value collection in logs and traces
ingestion and the whole `core::metadata` module (channel, mem-table,
flush job, shutdown hook).
- Removed the now-unused `DISTINCT_FIELDS` static and the
`ZO_FEATURE_DISTINCT_EXTRA_FIELDS` / `ZO_DISTINCT_VALUES_INTERVAL` /
`ZO_DISTINCT_VALUES_HOURLY` config knobs, including the `SUM(count)`
merge rollup the last one gated — legacy `distinct_values_*` streams now
compact as ordinary metadata streams. The default fields
`DISTINCT_FIELDS` carried (`service_name`, `operation_name`) are already
default secondary index fields via `SQL_SECONDARY_INDEX_SEARCH_FIELDS`,
so traces `_values` queries on them remain index-served.
- Renamed `values_v1` to `values_inner`: the `v2` variant was removed
long ago (#5087) and the distinct stream switch inside `v1` is gone now
too, so the version suffix was pure history.
### 4. Default fields after this change
- `service_name`, `operation_name`: default secondary index fields
(unchanged, via `SQL_SECONDARY_INDEX_SEARCH_FIELDS`).
- `trace_id`: now also a built-in default **bloom filter** field
(`BLOOM_FILTER_DEFAULT_FIELDS`, gated on
`ZO_FEATURE_DEFAULT_INDEX_FIELDS_ENABLED`), in addition to being a
default index field — bloom sits on top of the secondary index and
prunes files for high-cardinality equality lookups.
- There is no "default distinct fields" concept anymore.
### 5. Fix surfaced by e2e: SimpleDistinct returned null values as `""`
The tantivy index encodes null/missing field values as empty strings
(`index_builder` adds `""` for null docs), so the `SimpleDistinct`
term-dictionary walk returned `""` as a distinct value — sorted first in
ascending order, it surfaced as a blank first option in dashboard
variable dropdowns, which the DataFusion `GROUP BY` path never returns
(its null group sorts last). This was exposed once distinct fields
became index-served; `handle_simple_distinct` now skips empty terms
(with a unit test).
### 6. Drive-by: fix the failing `api_fuzz_tests` CI job
`CreateAlertRequestSchema` was an untagged documentation union with two
identical variants, producing `oneOf: [CreateAlertRequestBody,
CreateAlertRequestBody]` in the OpenAPI spec. Under `oneOf` semantics
(exactly one branch must match) no instance can ever validate, so
schemathesis filtered out every generated example and `api_fuzz_tests`
failed on `POST /api/v2/{org_id}/alerts` on every CI run since that
schema landed. The handler deserializes `CreateAlertRequestBody`
directly, so the annotation now documents exactly that type; runtime
behavior is unchanged.
## What stays (intentionally)
- `distinct_value_fields` / `enable_distinct_fields` in stream settings:
still the user-facing way to declare values-API fields; the list now
drives the read-time index-field union described above.
- The `distinct_value_fields` DB table that tracks dashboard/report
usage of these fields (still gates removal in the settings API).
- `DISTINCT_STREAM_PREFIX` helpers, retention sync and stream-deletion
cleanup for existing `distinct_values_*` streams — legacy stream data
keeps compacting (as ordinary metadata streams) and ages out via
retention; no new records are written to them.
## Testing
- `cargo check --workspace --all-targets` and `cargo clippy --workspace
--all-targets` clean.
- Enterprise build verified with `Cargo.toml.openobserve` (`cargo check
--all-targets`).
- New unit tests for the index-field union (skip rules for
FTS/partition/reserved fields, `enable_distinct_fields` gating) and the
`added_ts` / release-cutoff indexed-since resolution; schema, infra,
config and api-search unit tests pass. A few pre-existing failures on a
clean tree (config `ssrf_guard`, one flaky sqlite-backed destinations
test, one `.env`-dependent index-fields test) are local-environment
issues and unrelated.
Closes#13822
## What
**Phase 2** of the Playwright→pytest API-test migration (Phase 1 =
Correlation, #13788 / o2-enterprise#2419). Ports the 4 **pure-API**
Alerts 4.0 (multi-alert) specs into pytest and deletes the Playwright
versions — they launched a browser only to assert the raw API.
## Changes — `tests/api-testing/tests/alerts/`
| File | Ported from |
|---|---|
| `multialert_helpers.py` | `utils/alerts-api-helpers.js` (v1/v2
endpoints, canonical payloads, seed fixtures, ingest, scheduler-outcome
polling) |
| `conftest.py` | `alerts` fixture — seeds template + destination +
stream, auto-cleans created alerts |
| `test_multialert_api.py` | `alerts-multialert-api.spec.js` — 8 tests
(save-time guardrails, round-trip type contract, B1 group-endpoint RBAC
guard, tag/priority facets) |
| `test_multialert_regression.py` |
`alerts-multialert-regression.spec.js` — 4 tests (zero-behaviour-change
guards) |
| `test_multialert_firing.py` | `alerts-multialert-firing.spec.js` — 4
tests (fires at critical/warning, per-group fan-out, simple rollup) |
| `test_multialert_transitions.py` |
`alerts-multialert-transitions.spec.js` — 1 test (Ok→Warning→Critical
history) |
**17 tests, 1:1 port.** Deletes the 4 API specs and trims them from
`ci_matrix.json`'s Alerts `run_files`. Keeps
`alerts-multialert-ui.spec.js` (UI) and `alerts-api-helpers.js` (still
used by the UI + priority-tags specs).
## No workflow change needed
Unlike the Correlation pilot (which needed a dedicated server env), the
main `api_integration_tests` job **already** runs `tests/alerts/`, and
its server env **already** sets `ZO_ALERT_SCHEDULE_INTERVAL: 3` — the
exact scheduler cadence the firing/transitions tests need. So these
tests are picked up automatically with no CI edits.
## Verified locally
Full suite against a fresh single-node enterprise build: **17/17
passed** (~52s), including the timing-dependent firing
(critical/warning/per-group/rollup) and level-transition history tests.
This PR adds a fix so that workflows can be tested without having to
save.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
Design at: design/mobile-rum-sdk-evaluation/design-docs
Backend changes exist to make the OpenObserve RUM intake accept the
mobile SDK wire format alongside the existing browser format.
src/api/src/handler/http/request/rum/ingest.rs
200 → 202 Accepted on /rum, /logs, /replay | ---> OpenObserve-family
SDKs (which the mobile forks derive from) treat any non-202 as an
unexpected error and drop the batch.
#[serde(default)] on Event ---> Mobile replay omits creation_reason /
index_in_view that the browser format sends.
Multi-segment replay handling ---> Browser sends 1 segment + 1 event
object; mobile sends N segment parts + one event JSON array. Handler now
collects all segments, parses event as array-or-object, pairs them,
emits NDJSON → one _sessionreplay row per segment.
MAX_SEGMENTS=200, 16 MB decompressed cap via .take() --->
Decompression-bomb guard, newly needed now that N segments are accepted.
Replaces the `skip` band-aids that PR #13156 ("workflows v1") left on
the **Functions** tests with real, running coverage of the current
behaviour — on the correct edition — plus new tests for behaviour #13156
introduced but never covered. Test files only, no feature-code churn.
Clean branch off `main`.
## Background — what #13156 actually changed
#13156 bundled a Functions backend change unrelated to Workflows, and
the old tests still asserted the *pre*-#13156 rules, so they were
silenced with `pytest.skip(...)` / `test.skip(...)`. Two rules changed:
1. **JS is no longer `_meta`-only at the backend.** `save_function` /
`test_run_function` have **no org gate on any edition** — JS
(`transType:1`) is accepted in every org, including `default`, on **both
OSS and Enterprise**. The `_meta`-only rule now lives **only in the
frontend** (`AddFunction.vue` gates the JS radio on `isEnterprise ||
isCloud || org === "_meta"`, baked at build time by
`VITE_OPENOBSERVE_ENTERPRISE`).
2. **The compile-check reports only *syntax* errors, not *runtime*
errors.** A syntax error still returns **400**; code that compiles but
throws at runtime now returns **200** with the error surfaced
**per-event** in `results[].message` — for **both VRL and JS**.
Because the API tests hit the backend directly, there is **no edition
difference to assert for them** — the earlier "OSS asserts 400 / ENT
asserts 200" framing was incorrect. Only the **e2e** (frontend JS-radio)
is genuinely edition-dependent.
## API tests (pytest · `tests/api-testing`)
**`test_functions.py`**
- Keep the compile-error → 400 test
(`test_test_endpoint_with_invalid_vrl_returns_400`) — it was never
actually failing; `test`/`=====` are genuine VRL *compile* errors
(`undefined variable` / syntax) and still 400.
- Add `test_test_endpoint_with_js_runtime_error_returns_200` and
`test_test_endpoint_with_vrl_runtime_error_returns_200` — runtime errors
now return 200 with a per-event `results[].message`.
**`test_functions_meta_org_restriction.py`**
- Flip the 3 previously-skipped "blocked in default" tests → assert
**200** (JS allowed in every org): `..._blocked` → `..._allowed`.
- Repurpose `TestErrorMessages` (its old restriction-message assertions
no longer exist) into per-event runtime-error coverage: JS
`ReferenceError` content, VRL + JS surfacing consistency,
partial-failure (only the failing event carries a message), and a
syntax-error → 400 regression guard.
- Update the class/module docstrings to the real contract.
**`test_js_enrichment.py`**
- Flip `test_js_function_in_default_org_returns_400` →
`test_js_function_in_default_org_allowed` (200, transform runs).
## e2e (Playwright ·
`tests/ui-testing/Functions/js-transform-type.spec.js`)
The JS-radio rule *is* edition-dependent. Instead of a skip (or an
env-var hack), the two tests now detect edition at runtime via
`EditionFeaturesPage.detectEdition()` — which reads the header edition
button, driven by the **same** build flag that gates the JS radio, so
it's the exact source of truth and needs **no CI change**:
- **OSS build** → assert JS radio **hidden** in a non-meta org.
- **Enterprise / cloud build** → assert JS radio **visible** in every
org.
The edition-agnostic steps that were trapped inside the skipped tests
now **always run**: VRL-function create in a non-meta org (a P0 smoke
path), and JS-visible in `_meta`. Added `P1: VRL runtime error surfaces
in error-details` (mirrors the existing JS `ReferenceError` case; covers
the VRL half of the per-event-error change).
## Skips removed (before → after)
| Old (skipped) | New (runs & passes) |
|---|---|
| `test_create_js_function_in_default_org_blocked` |
`..._default_org_allowed` |
| `test_test_js_function_in_default_org_blocked` |
`..._default_org_allowed` |
| `test_update_vrl_to_js_in_default_org_blocked` |
`..._default_org_allowed` |
| `test_error_message_content` / `_consistency` | per-event
runtime-error tests |
| `test_js_function_in_default_org_returns_400` (enrichment) |
`..._default_org_allowed` |
| e2e `P0: Non-meta org - JS hidden, VRL works` | `P0: Non-meta org - JS
radio edition-gated, VRL works` |
| e2e `P1: Radio visibility across orgs` (skipped) | same name,
edition-aware, no `test.skip` |
`grep -rn "JS fns are now allowed" tests/` and `grep -rn "JS is allowed
in all orgs on enterprise" tests/` → **both empty**. (The 7 remaining
pytest skips are the unrelated `JS_HELPER_UNAVAILABLE` enrichment-helper
cases — a separate pre-existing limitation, not part of this change.)
## Verification (local, against current `main` source)
Built the current-`main` OSS binary and ran everything live:
- **API:** `32 passed`, `0` target skips (7 unrelated
`JS_HELPER_UNAVAILABLE` skips remain).
- **e2e OSS build:** `7/7` — `detectEdition=opensource`, JS hidden in
non-meta.
- **e2e Enterprise-frontend build:** `3/3` edition tests —
`detectEdition=enterprise`, JS visible everywhere, VRL runtime error
renders in error-details.
All backend behaviour was confirmed by probing the live `/functions` +
`/functions/test` endpoints, not assumed.
## What CI proves, and where
- **This PR (openobserve):** `api-testing.yml` runs the API tests;
`playwright.yml` (`Functions` shard) runs the e2e on the **OSS** edition
(JS-hidden branch).
- **Enterprise edition:** dispatch `o2-enterprise` → `playwright.yml`
(workflow_dispatch) with `o2_opensource_repo =
test/functions-skipped-tests-13318` to run the same specs on the
**enterprise** build (JS-visible branch). Pairs with the ENT Functions
work in #13318 / o2-enterprise#2217.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
## Summary
- Adds `TestVortexUtf8Int64CastPushdown` in
[tests/api-testing/tests/query_agent/test_3_vortex.py](tests/api-testing/tests/query_agent/test_3_vortex.py)
— regression coverage for the bug fixed by
[o2-enterprise#2179](https://github.com/openobserve/o2-enterprise/pull/2179).
- The fixture ingests a batch as **i64** and then a batch as **utf8** on
the same field into a fresh stream, flushes to force vortex file
generation, then runs `SELECT * FROM {stream} WHERE value > 0`.
- The bug: DataFusion coerces the predicate to `CAST(value AS Int64) >
0`; vortex used to accept the cast for pushdown but had no `utf8→i64`
cast kernel, so execution failed with `No CastReduce to cast constant
array from utf8? to i64?` / `No CastKernel …`.
- Assertion: status 200 + empty `partial_error` — the query must not 5xx
or partially fail.
## Where this runs
- Runs in the enterprise `api_query_agent_vortex` job (server booted
with `ZO_FILE_FORMAT=vortex`).
- Not enabled in OSS `api-testing.yml` (test file is enterprise-only per
the file header).
## Test plan
- [ ] `api_query_agent_vortex` on the paired ent PR passes
- [ ] Ent PR (same branch name) needed to fire ent's `api-testing.yml` —
the OSS `ent-build-status-check` only runs `cargo check/test`, not the
Python vortex suite
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- **Phase 4 — Error Paths** (`test_4_error_paths.py`): 28 cases across 8
categories that send intentionally invalid SQL and assert 4xx not 5xx.
Self-contained, runs in ~24s without data dependency.
- **Phase 5 — Concurrency Stress** (`test_5_concurrency.py`): 26 queries
(2 per category) run simultaneously at 5/10/15 worker levels via
ThreadPoolExecutor. Each thread gets its own client. Catches deadlocks
and thread-safety bugs.
- **EXPLAIN ANALYZE** (`test_explain_analyze.py`): Verifies
EXPLAIN/EXPLAIN ANALYZE return 200 with plan output and no
`partial_error`. Also validates EXPLAIN ANALYZE on cross-schema
dot-notation queries returns 4xx (not 500 crash).
- **Fix**: `load_all_queries()` now skips JSON files without a
`"queries"` key.
## Test plan
- [x] Error path tests pass on pentest (28/28, all return 400)
- [ ] CI run: `rye run pytest tests/query_agent/test_4_error_paths.py
-v`
- [ ] CI run: `rye run pytest tests/query_agent/test_5_concurrency.py
-v`
- [ ] CI run: `rye run pytest tests/query_agent/test_explain_analyze.py
-v`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## What
Extends `tests/api-testing/tests/vortex/test_schema_evolution.py` with
**34 new tests** across 10 new classes covering all realistic field
drift scenarios for petabyte-scale schema evolution.
Grows the schema evolution suite from 13 scenarios (14–25) to **46
scenarios (14–59)**.
## The scenario (shared across all groups)
Think of a post office logbook. In January every letter has *From, To,
Weight*. In February the form changes — Weight is dropped, Stamp Color
is added. In March they go back plus add more fields.
```
Batch 1 [fa, fb, fc] — initial deployment
Batch 2 [fa, fb, fd] — update: fc dropped, fd introduced
Batch 3 [fa, fb, fc, fd, fe] — redeployment: fc back, fd kept, fe added
```
Each batch is flushed to a separate on-disk file before the next is
ingested, forcing the query planner to reconcile three distinct schemas
— the exact code path touched by `perf: Batch schema evolution checks`
(v0.80.0).
## Test classes and scenarios added
| Class | Scenarios | What it checks |
|---|---|---|
| `TestSchemaDivergenceConvergence` | 26–28 | Nullability per batch,
cross-batch aggregation, field coexistence isolation |
| `TestNullHandling` | 29–31 | IS NULL / IS NOT NULL exhaustive for
absent fields |
| `TestAggregationsOnPartialFields` | 32–36 | COUNT, SUM, MIN/MAX, COUNT
DISTINCT on partial fields |
| `TestGroupByOrderByEvolved` | 37–40 | GROUP BY / ORDER BY when grouped
field is absent in some batches |
| `TestTypeChangeOnReappearance` | 41–44 | Field returns as different
type; OO type-preservation behavior documented |
| `TestMultipleDivergenceCycles` | 45–46 | Field disappears and
reappears multiple times |
| `TestComplexQueriesEvolved` | 47–51 | SELECT *, CASE WHEN, subquery,
HAVING on evolved schemas |
| `TestTimeRangeEvolved` | 52–55 | Time-range filters correctly isolate
individual batch windows |
| `TestFTSEvolved` | 56–58 | Full-text search across batches with absent
fields |
| `TestHighFieldCountDivergence` | 59 | Wide schema: 20-field batch
diverging to 10-field batch |
## Key findings documented in tests
- **OO string-null behavior**: absent string fields are stored as `''`
not SQL NULL — correct check is `IS NOT NULL AND != ''`; numeric fields
use `IS NOT NULL` only
- **OO type preservation**: when a field returns as a different numeric
type, OO preserves the original type on disk; test_42 documents this
behavior explicitly
## Code review fixes included
- Added missing `fb` assertion in test_26
- Removed incorrect `!= ''` check on numeric `latency_ms` field (tests
27, 32–35)
- Added comment in test_28 explaining the OO string-null quirk
- Updated module docstring to list all 9 scenario groups
- Added docstrings to all `_ingest` / `_ingest_three_batches` helper
methods
## Why
The v0.80.0 commit changed how OpenObserve batches schema reconciliation
across multiple on-disk files. A bug in that path can:
- Silently return fewer records than expected (data loss with no error)
- Return wrong nullability for fields that reappear after being absent
- Mis-count aggregations over fields that span only some of the files
None of the 13 existing tests covered the diverge-then-converge pattern.
This PR closes that gap across all query types.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
- Adds a matrix strategy to the query agent SQL regression suite jobs in
both `api-testing.yml` and `query-agent.yml`
- Each job now runs with `ZO_FEATURE_SINGLE_NODE_OPTIMIZE_ENABLED=false`
and `ZO_FEATURE_SINGLE_NODE_OPTIMIZE_ENABLED=true`
- Verifies both DataFusion physical optimizer paths produce identical
query results
- Matrix legs run in parallel with `fail-fast: false`
- All artifact/log names include the matrix label to avoid collisions
## Test plan
- [ ] PR CI shows two `api_query_agent` matrix legs
(single_node_opt_false and single_node_opt_true), both pass
- [ ] `/query-agent` PR comment triggers both modes and posts per-mode
results comments
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
Fixes a test gap where `match_all` Tantivy queries could return wrong
results after a DataFusion upgrade without being caught by existing
assertions.
### Problem
After a DataFusion upgrade merge, the DataFusion access plan was no
longer used for `match_all` queries, causing Tantivy to return wrong
results. The existing tests only compared row counts against a DuckDB
oracle that emulates `match_all` with `LIKE '%term%'` — so wrong rows
that still matched the LIKE pattern passed undetected.
Additionally, `node/flush` does not generate Tantivy indexes
immediately. The default `ZO_MAX_FILE_RETENTION_TIME=600s` means
WAL→parquet→Tantivy conversion can lag, so FTS queries may fall back to
a non-Tantivy path during tests.
### Changes
1. **Content verification**
([conftest.py](tests/api-testing/tests/query_agent/conftest.py)): Every
hit from a `match_all` query that returns the `log` field is now checked
— each search term word must be present in `log`. A hard assertion fails
if Tantivy returns rows that don't contain the search terms.
2. **Tantivy readiness gate**
([test_2_parquet.py](tests/api-testing/tests/query_agent/test_2_parquet.py)):
After `node/flush`, the test now polls with a `match_all('warehouse')
LIMIT 1` query (up to 120s timeout) to confirm Tantivy indexes are built
before running FTS assertions. Previously only COUNT(*) was checked.
3. **Multi-word match_all emulation**
([compute_counts.py](tests/test-data/query-agent/compute_counts.py)):
`match_all('ACK batch')` now translates to `(log LIKE '%ACK%' AND log
LIKE '%batch%')` — per-token AND conditions matching Tantivy's
token-level search semantics, instead of the old phrase match `log LIKE
'%ACK batch%'`.
4. **Regenerated expected results** across all query JSON files.
## Test plan
- [ ] Run query agent test suite against an environment with Tantivy
enabled
- [ ] Verify `match_all` content assertions catch intentionally
corrupted results
- [ ] Confirm post-flush Tantivy readiness poll works within the timeout
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Huaijin <haohuaijin@gmail.com>
- [x] ~add sort when to many groups, we can do this, but for now do not
do this~
- [x] remove the unsafe for memory pool
- [x] add approx_distinct improve, already fork, will add in next pr
- [x] fork vortex to add cast fix, and export
`.exclude_schemes([IntDictScheme.id()])`
- [x] ~enable_physical_uncorrelated_scalar_subquery test~
Close#12345
## Problem
OTLP histogram metrics sent over HTTP/JSON (e.g. otel-collector with
`encoding: json`) created streams but stored 0 events, with no
`_count`/`_sum`/`_bucket` streams. Int-valued counters/gauges were
silently recorded as 0. The protobuf path was unaffected, which is why
this only surfaced for collectors explicitly configured with JSON
encoding.
## Root cause
`otlp_json` deserializes the body directly into opentelemetry-proto's
serde-derived types, which only partially implement the proto3 JSON
mapping (64-bit integers encoded as strings, enums by name, zero/empty
fields omitted). `Metric.data` is a flattened `Option` oneof, and serde
converts any inner deserialization error of a flattened `Option` into
`None` instead of failing — so the whole metric silently became `data:
None`, while the stream itself had already been created by the schema
setup. Same mechanism on `NumberDataPoint.value` turned string `asInt`
values into 0.
## Changes
- Upgrade opentelemetry stack 0.31 → 0.32, which fixes string-encoded
u64 handling for plain histograms and summaries upstream.
- Add `otlp_json_compat` normalizer for the gaps still present in 0.32
(each verified empirically, with unit tests):
- string `asInt` values on data points and exemplars
- `aggregationTemporality` as enum name
- `Exemplar`'s value oneof is not flattened in 0.32
- missing `serde(default)` on
`ExponentialHistogramDataPoint`/`Buckets`/`ValueAtQuantile`/`Exemplar`,
which rejected omitted zero/empty fields
- Count and log metrics whose data oneof failed to decode
(`partial_success.rejected_data_points`) instead of silently skipping
them, and stop overwriting specific partial-success error messages with
the generic retention message.
- Adapt to OTLP 0.32 API additions (`KeyValue.key_strindex`,
`AnyValue::StringValueStrindex`).
- Upgrade workspace reqwest 0.12 → 0.13, required because
opentelemetry-http 0.32 only implements `HttpClient` for reqwest 0.13
(`rustls` feature replaces `rustls-tls-native-roots` via the platform
verifier; `json`/`query`/`form` are now explicit features). The
`segment` crate is pinned to reqwest 0.12, so `TELEMETRY_CLIENT` moved
into `telemetry.rs` as a plain `reqwest::Client` that posts the Track
message itself, replicating `segment::HttpClient::send`.
- Remove the unused pyroscope profiling integration (optional feature,
dependencies, agent setup, `ZO_PROF_PYROSCOPE_*` config).
## Verification
- 10 unit tests in `otlp_json_compat` covering the full payload matrix
(collector-style string integers, enum names, omitted zero fields,
exemplars with and without trace context, already-normalized payloads,
malformed input).
- End-to-end: posted the issue's collector-style JSON payload to
`/api/default/v1/metrics` — `_bucket` (3 events, correct cumulative
counts with exemplars attached), `_count`, `_sum` populated for
histogram and exponential histogram, summary quantiles stored, `asInt`
counter stored as 42 instead of 0.
- `cargo clippy -- -D warnings` clean; lib test failures are
pre-existing local-environment issues that reproduce identically on
main.
## Summary
- Adds **239 new tests** covering the vortex columnar file format across
78 scenarios from the test plan
- **`tests/api-testing/tests/vortex/`** — 9 test modules (83 tests):
format conversion, schema evolution, data integrity, FTS, downsampling,
API format awareness, server config, and performance/stress
- **`tests/api-testing/tests/query_agent/test_3_vortex.py`** — 156
tests: the full 155-query SQL correctness suite + scenario 39
(self-join) run post-flush against vortex files
- **`api-testing.yml`** — adds `--ignore=tests/vortex` to the OSS
default-set run; vortex tests require the enterprise binary with
`ZO_FILE_FORMAT=vortex` and are gated to enterprise CI only
## Test plan coverage
| Module | Scenarios |
|--------|-----------|
| `test_format_conversion.py` | 1–13, 7, 8, 9, 10 |
| `test_schema_evolution.py` | 14–25 |
| `test_data_integrity.py` | 55–62 |
| `test_fts.py` | 51–54 |
| `test_downsampling.py` | 47–50 |
| `test_api_format.py` | 71–73 |
| `test_config.py` | 66–67 |
| `test_performance.py` | 74–78 |
| `test_3_vortex.py` | 26–46 (155 queries + JOIN) |
## Enterprise CI
Two dedicated jobs added in the companion `openobserve/o2-enterprise` PR
on the same branch:
- **`api_query_agent_vortex`** — runs `test_3_vortex.py` with
`ZO_FILE_FORMAT=vortex`
- **`api_vortex_format_tests`** — runs the full `tests/vortex/` suite
with 4-phase server lifecycle (parquet→vortex→parquet→vortex+compaction)
OSS CI is unchanged except for the ignore rule — no vortex tests run
against the OSS binary.
## Test checklist
- [ ] `tests/vortex/` collects cleanly: `83 tests collected`
- [ ] `tests/query_agent/test_3_vortex.py` collects cleanly: `156 tests
collected`
- [ ] Enterprise CI `api_query_agent_vortex` job passes
- [ ] Enterprise CI `api_vortex_format_tests` job passes
🤖 Generated with [Claude Code](https://claude.ai/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
## Summary
- Expands the SQL regression catalog from **155 to 200 queries** across
all 12 categories
- Adds **7 queries** extracted from abc.sql (Q156-Q162) — synthesized
from real-world OpenObserve SQL patterns with all data/fields/streams
generated from scratch
- Adds **230 auto-generated queries** (Q163-Q392) from templates to
cover query format diversity
- Both test phases pass: **200 memtable + 200 parquet = 400 total**
## Key changes
- **data_gen.py**: 43 new fields (record_type, site_name, latency_ms,
payload_size, alert_tags, phase_data, db_info, etc.), generates 3920
deterministic records
- **compute_counts.py**: new translations for regexp_like,
str_match_ignore_case, approx_distinct, json_get_str/json_get_json
chaining; empty-aggregate result detection (OO returns 0 rows vs
DuckDB's 1 NULL row)
- **conftest.py**: PID-based stream name suffix for cross-invocation
isolation (prevents data accumulation)
## Test plan
- [x] test_1_memtable.py — 392 passed
- [x] test_2_parquet.py — 392 passed
- [x] All 7 abc.sql queries pass (Q156-Q162)
- [x] No real stream names, field names, or data copied from external
sources
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
- ENT CI runners are slower than local/OSS runners — the
`ingest_query_agent_data` fixture was hitting the 120s timeout before
1550 records became searchable, causing all 310 query_agent tests to
fail at setup
- Bumped `wait_until` timeout to 300s
## Test plan
- [ ] OSS `api_query_agent_tests` passes as before
- [ ] ENT `api_query_agent_tests` no longer hits WaitTimeout on ingest
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Neha P
Bhargav
Hengfei Yang
Ashish Kolheandgithub-actions[bot]
udevandsai nikhil kethe
Prabhat Sharma
Vamsi Karthik R
