starred/openobserve
1
0
Fork 0
mirror of https://github.com/openobserve/openobserve.git synced 2026-11-03 16:10:21 +00:00
Code Issues Packages Projects Releases Wiki Activity
7,712 Commits 641 Branches 243 Tags
610cd70a136f27b4fa26e46cf667fcd94875fafb
Commit Graph
22 Commits
Author SHA1 Message Date
Shrinath Rao 88fdb1166c test(ui): split GeneralTests into Management/Org/Streams + dedup (#14950) 2026-09-29 08:04:33 +00:00
Vamsi Karthik RandShrinath Rao 88e7dabb24 test(alpha1): document how the alpha cloud matrix is decided (#14778) 
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
 2026-09-23 08:24:01 +00:00
Shrinath Rao c9f8d93cc6 ci: comment Playwright shard-selection triage on PRs (#14347) 2026-09-09 18:06:32 +00:00
Shrinath Rao f9afa5e324 ci: rename regression workflow to Playwright Bug-Fix Regression Tests (#14201) 2026-09-04 13:48:08 +00:00
Hengfei Yang 44a95200a7 ci: reduce Rust cache pressure and checkout cost (#14171) 2026-09-03 14:32:30 +00:00
Ashish KolheandShrinath Rao 226b401da8 ci: changed-path smoke selection for Playwright PR runs (#14107) 
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
 2026-09-02 13:14:04 +00:00
Vamsi Karthik R 7b79fdcdbe ci(playwright): stop SLO-Measurement backfill from serializing 21 daily chunks (#13870) 
## What this fixes

The `SLO-Measurement` shard (`slo-timeslice.spec.js` +
`slo-alerts.spec.js`) has been taking **~26 minutes**

## Root cause

Both specs seed **8 days of history** and create SLOs with a **7-day
rolling window**, then block in `beforeAll` until the SLO is actually
measured (`waitForSloMeasured`). The server backfills history in
`ZO_SLO_BACKFILL_CHUNK_SECS` chunks — default **86400s (1 day)** — and
only **one chunk at a time** (`ZO_SCHEDULER_SLO_BACKFILL_CONCURRENCY=1`,
by design, so bulk backfill never starves real-time alerts).

The shard backfills **3 SLOs** across the two specs, each needing 7
daily chunks to cover its 7-day window:

- `slo-timeslice.spec.js` `beforeAll`: 2 SLOs (`sloLt`, `sloGt`) × 7
chunks = 14
- `slo-alerts.spec.js` `beforeAll`: 1 SLO × 7 chunks = 7

**21 chunks total, strictly serial**, at roughly 40s/chunk ≈ **~14
minutes of pure backfill wait**, before any UI assertion runs. This is
the server behaving correctly and exactly as designed for production —
the test is just paying for it the slow way.

## Fix

Neither spec asserts on per-chunk timing or partial coverage, only on
the final measured state, so raising the chunk size to cover the whole
window in one pass is free wall-clock, not a behavior change.

- Threaded `slo_backfill_chunk_secs` through `build-ci-matrix.js`'s
existing per-shard passthrough, same pattern already used for
`ingest_allowed_upto` / `quick_mode_enabled`.
- Set it to `604800` (7 days) for the `SLO-Measurement` shard only — the
server-wide default (`86400`) is untouched everywhere else.
- Documented the new knob in `tests/ui-testing/ci-matrix/README.md`.

## Validation

**Synthetic (API-only), same seed shape as the real specs** (8 days,
1-min interval, deterministic 3h fault window), against a local build:

| | Default (86400s chunk) | Fixed (604800s chunk) |
|---|---|---|
| Time to measured | 246.9s (~4.1 min) for **one** SLO, stepping 1/7 →
2/7 → ... every ~40s | **3.0s** |
| Final SLI | 98.21428571428571% | 98.21428571428571% (**identical**) |

Same math, same result — only the number of scheduler round-trips
changes.

**Real shard, end-to-end**, run exactly as CI invokes it (`npx
playwright test slo-timeslice.spec.js slo-alerts.spec.js --workers=1`)
against a server mirroring this shard's full env including the fix:

- **25/25 tests passed**, 0 regressions
- Total wall-clock (server start → global teardown): **4m Avg**

Local timing won't map 1:1 onto the GitHub Actions runner (different
disk/scheduler characteristics — even the "before" baseline was faster
locally than the ~26min/~5min-per-spec seen in CI), so the authoritative
number is whatever this PR's own CI run reports. But the structural win
— 1 chunk instead of 7 per SLO — is verified correct and regression-free
independent of machine.

## Companion PR

The enterprise repo's `playwright.yml` independently consumes per-shard
matrix fields (it explicitly mirrors OSS's `ingest_allowed_upto`
handling for the same reason), so it needs the equivalent env
passthrough or it'll keep paying the old cost even after this merges.
Companion PR: openobserve/o2-enterprise#2460
 2026-08-19 00:58:33 +00:00
Vamsi Karthik R 99b1a23773 test(slo): E2E + API suite for SLOs (#13836) 
Adds automated test coverage for **SLOs** (#13547, #13577, #13579,
#13704, #13761, #13784), which shipped with none. **149 tests** — 77
Playwright + 72 pytest — all three SLI types, both query languages,
happy / negative / edge paths. Clean branch off `main`.

**UI (Playwright · `tests/ui-testing`)**
- `pages/sloPages/` — 4 page objects (verified `data-test` selectors,
hydration + Monaco-sync waits, OSelect commit-retry, virtualized-list
filtering).
- `playwright-tests/SLO/slo-crud.spec.js` (24) — create
count/time-slice/grouped through the form, detail, edit, SLI-type
change, tags, pause, delete, filters; negative: dismissed delete, empty
search, cancel, unmeasured SLI is absent not zero.
- `playwright-tests/SLO/slo-form-validation.spec.js` (20) — server
errors surfacing in the form, conditional fields per SLI type, count +
time-slice previews, **PromQL count and time-slice created end to end**;
negative: no stream, invalid aggregate, over-long name, target
boundaries.
- `playwright-tests/SLO/slo-alert-sli.spec.js` (8) — **the `alert` SLI
type**: source picker, create, config carries only the alert id, detail
names the source; negative: ineligible source offered-but-disabled,
grouped alert SLI, slice finer than source cadence.
- `playwright-tests/SLO/slo-timeslice.spec.js` (10) — **#13761 P0
regression** (measured SLI, not the preview), PromQL, previews, gaps,
language-flip clearing.
- `playwright-tests/SLO/slo-alerts.spec.js` (15) — burn-rate +
error-budget kinds, presets, edit, freeze door, burndown; negative: no
destination, cancel, blank/invalid name.
- `playwright-tests/utils/slo-seed.js` — backdated seeding, metrics
seeding, eligible-source-alert fixture, ground-truth query, fail-fast
env guard, transient-store retry, full fixture teardown (alerts →
streams → destinations → templates).

**API (pytest · `tests/api-testing`)**
- `tests/slos/test_slo_validation.py` (72) — full REST surface
(create/get/update/enable/delete/groups/slo-eligible), validation on
**both create and update** (name, target, window, slice, config shapes,
comparators, float-threshold round-trip), boundaries (target
0.001/99.999, name 256 vs 257, threshold 0/negative/1e12/NaN, unicode
names), alert-source eligibility (silence-gated, too-infrequent, cron),
PromQL config shapes, auth 401/404.

**CI**
- Two shards: `SLO` (`workers: 3`) and `SLO-Measurement` (`workers: 1`).
Split because the backfill job runs with
`ZO_SCHEDULER_SLO_BACKFILL_CONCURRENCY=1`, so measurement specs sharing
an instance starve each other's waits — `fullyParallel` races spec
*files*, which `mode:'serial'` cannot prevent. The 3-worker pin is
SQLite: at 5, concurrent SLO/alert/destination writes trip `database is
locked` on the single-node meta store.
- New optional per-shard `ingest_allowed_upto` (`240`) alongside the
existing `quick_mode_enabled`: the workflow default of 5 **hours**
silently drops the 8 days of history an SLO window needs, while still
answering `200`. Other shards keep the default. Same precedent as
`api-testing.yml`'s `48`.

**Feature code**: 9 additive `data-test`/`dataTest` attributes in
`SloDetail.vue`, `SloList.vue`, `SloAlertCondition.vue` — no logic
changes, `vue-tsc` clean. The stat tiles, tabs and burn-window inputs
had no stable selectors; `OStatStrip`/`OTab` already forward the
attribute.

**Behaviour**: self-cleaning (`e2e_slo_*`, worker-scoped since
`afterAll` runs per worker).

**Verified in CI** on this branch: `e2e / SLO` **52 passed** (1.8m,
workers pinned to 3), `e2e / SLO-Measurement` **25 passed** (11.3m,
workers pinned to 1) — the measurement shard passing confirms the
per-shard `ingest_allowed_upto` took effect, since those tests are
impossible under the 5-hour default. pytest **72 passed** locally; in
CI's 599-test run one setup delete hit a transient 500 (see findings)
and now retries.

**Findings for the feature owner** (reported, not fixed here):
- A blank "good when" shows `Request failed with status code 422` —
`pruned()` drops the empty key, so the API's real 400 message never
reaches `save()`.
- A rejected alert name shows no banner: `SloAlertForm.submit()` returns
on `nameError` before setting `saveError`.
- `avg(latency_ms); DROP TABLE users--` is accepted at create time
(parsed only at measurement).
- The name-length message says "less than 256 characters" but 256 is
accepted — 257 is the first rejection.
- Duplicate alert names on one SLO are accepted (duplicate SLO names are
a 409), and the panel lists them by name, so they are indistinguishable.
- SLO names accept characters that break the list's own
`slos-slolist-edit-${name}` selectors; alert names correctly reject
them.
- All chart/preview root `data-test` ids are unreachable — a parent's
fallthrough attribute overrides the child root's.
- Both edit forms hydrate asynchronously, so a blank form can flash
before the stored values appear.
- `DELETE /slos/{id}` on a freshly-created SLO returned **500** once
under CI load (not reproducible locally in 12 rapid iterations).
Contention is arguably a retryable 503, and the body carried no
actionable message.
- **A scheduled alert whose source stream is deleted is retried
forever** (`Stream "…" not found in schema`), and the SLO backfill lane
shares the scheduler's worker budget — so accumulated dead alerts
*starve SLO measurement*. Observed: 65 of them held coverage at 1 of 7
chunks indefinitely; deleting them restored 0.143 → 1.0 in 240s.
Harmless in CI (fresh instance per shard) but it degrades any long-lived
environment. The suite now cleans up after itself; the retry-forever
behaviour itself may deserve a look.

**Paired ENT PR: openobserve/o2-enterprise#2434.** ENT's matrix is this
repo's manifest plus an overlay, merged by this repo's
`build-ci-matrix.js`, so both SLO shards reach ENT automatically —
verified: 31 merged shards, both SLO entries carrying
`ingest_allowed_upto=240` and their worker pins, no other shard
affected. What ENT was missing is *consuming* those two fields, which
#2434 adds. Merge the two together: this one alone leaves ENT booting
the SLO shards on the 5-hour ingest window, and #2434 alone is inert.

*(An earlier review reported ENT as using an inline matrix with no SLO
shard — that turned out not to be the case.)*
 2026-08-17 18:50:29 +00:00
Shrinath Rao 22d6ecad2d feat(ci): parametrize SHARD_PREFIX in o2-run-report.sh (#13663) 
## What

Parametrize the shard-matching prefix in
`.github/scripts/o2-run-report.sh`.

Adds a `SHARD_PREFIX` env (default `"e2e / "`) used everywhere the
script identifies e2e shard jobs by name — the run-doc shard tallies
(`shards_total/passed/failed/skipped`) and the per-shard rows written to
`<STREAM>_shards`.

## Why

The ENT **alpha1 cloud e2e** suite names its shard jobs `alpha1-e2e /
<folder>`, which the hardcoded `e2e /` prefix did not match. This
enabling change lets alpha1 (and any future differently-named suite)
report per-shard rows to its own dedicated OpenObserve streams. See the
companion ENT PR `test/alpha1-o2-reporting`.

## Safety

The default `"e2e / "` (trailing space significant — `module` = job name
minus prefix) preserves current OSS/regression behavior **exactly**.
Verified with jq that the default still yields `module="Alerts"` for
`"e2e / Alerts"` (no leading space) and that `"alpha1-e2e / "` yields
`module="Alerts"` for the alpha1 job names.

Cross-repo sibling: **o2-enterprise#`test/alpha1-o2-reporting`** (same
branch slug).
 2026-08-06 10:09:40 +00:00
ktx-akshay 7999fd33f3 test: enhance logs visualization error handling and persistence tests (#13591) 
## Summary

Adds **10 E2E tests** covering the two defects fixed in #13244, plus the
`SELECT *` behaviour change that shipped alongside them. Also introduces
per-shard backend configuration so the `SELECT *` tests can actually run
in CI.

## Test coverage

### `logsVisualizePersistence.spec.js` — 7 tests (quick mode **off**)

| # | Test | Defect | Fix code exercised |
|---|---|---|---|
| 1 | Renders without the false required-fields error | A |
`convertPanelData.ts` `!query?.customQuery` |
| 2 | Restores chart type, query and chart after URL refresh (h-bar) | B
| `setupLogsTab` re-run + `isInitialVisualizeRestore` |
| 3 | Re-renders after reload with the default chart type | B |
permanent No-Data half |
| 4 | Restores a non-default chart type (area) after refresh | B |
`restoreVisualizationFromUrlOnLoad()` |
| 5 | Chart still renders after reload with a VRL function applied | B |
`prepareVisualizeMode()` VRL copy |
| 6 | Never shows required-fields error toggling Visualize → Logs →
Visualize | A | manual-toggle rebuild path |
| 7 | No `SELECT *` toast opening Visualize with no stream in the URL |
B | page-load ordering guard |

**Defect A** — false "Please select required fields to render the chart"
on the custom-query Timechart panel.
**Defect B** — URL refresh produced a `SELECT *` toast plus permanent No
Data.

### `logsVisualizeSelectStar.spec.js` — 3 tests (quick mode **on**,
dedicated shard)

| # | Test | Behaviour verified | Call site |
|---|---|---|---|
| 8 | Table chart on `SELECT *` → toast shown **and** chart type reverts
| Table is blocked | `Index.vue:2265` (watcher) |
| 9 | Histogram chart on `SELECT *` → no toast, renders |
Line/bar/area/scatter allowed | entry path |
| 10 | `SELECT *` line chart survives browser refresh → no toast, state
restored | Restore path doesn't re-emit | `Index.vue:2465` (restore) |

## Infrastructure changes

| File | Change | Why |
|---|---|---|
| `ci_matrix.json` | New `Logs-SelectStar` shard with
`quick_mode_enabled: "true"` | `isSelectStarForTable()` only exists when
the server runs with quick mode on |
| `playwright.yml` | Per-shard `ZO_QUICK_MODE_ENABLED` env on `Start
OpenObserve` | The flag is read once at server start — a shard is the
smallest unit that can vary it |
| `build-ci-matrix.js` | Emit `quick_mode_enabled` (defaults `"false"`)
| The field was being silently stripped by the emit whitelist |
| `logsPage.js` | 13 assertion helpers + fixed chart-type detection |
Detection read `data-selected` off the wrong element and matched retired
CSS classes, so it always returned `null` |

All 24 existing shards are unaffected — they omit the field and default
to `false`.

## Notes on test quality

- Every negative assertion ("no toast", "no error") is sequenced after a
deterministic positive end-state, and gated on a precondition so it
cannot pass vacuously.
- Toast absence is proved with a MutationObserver recorder installed
before the navigation under test, since toasts auto-dismiss.
- The quick-mode precondition is a hard assertion, not `test.skip` — a
misconfigured shard fails loudly instead of silently reporting
"skipped".
 2026-08-04 12:09:26 +00:00
Shrinath Rao d76c0a6bbb ci(playwright): single-source ALL shard matrices via ci-matrix/ manifests (#13606) 
## Why
The OSS and ENT Playwright shard matrices (`run_files` per `testfolder`)
were hand-maintained inline in each `playwright.yml` and had to be
synced on every spec add — error-prone and drift-prone (ENT was even
missing `dashboard-favorites`). This makes the shard list a **single
source of truth** for **every** Playwright workflow.

## What (OSS side)
All matrices are generated at run time from JSON manifests in
`tests/ui-testing/ci-matrix/`:

- **`ci-matrix/ci_matrix.json`** — PR-gate base (24 shards). The only
place shared shards live.
- **`ci-matrix/ci_matrix_regression.json`** — regression base (10
shards); `RegressionSet/<Feature>` mapping now carried in
`actual_folder`.
- **`.github/scripts/build-ci-matrix.js`** — merges base (+ optional ENT
overlay) → GitHub matrix; validates uniqueness / no dup specs / no spec
both active+disabled. Reused by ENT from its OSS checkout.
- **`ci-matrix/README.md`** — manifest index + how to add/move/disable a
spec.
- Workflows `playwright.yml` + `playwright_regression.yml`: new
`generate_matrix` job → `strategy.matrix: fromJSON(...)`; `case` blocks
→ `${{ matrix.actual_folder }}`.

Disabling a spec (JSON has no `//`): move it into that shard's
`disabled: [{file, reason}]` array — never emitted (won't run), record
survives. Every shard ships a `"disabled": []` placeholder.

## Safety (verified)
- **OSS `playwright.yml` and `playwright_regression.yml` matrices are
byte-identical to before** — zero behavior change.
- Companion ENT PR (same branch slug
`test/playwright-matrix-single-source`) derives its matrices from these
manifests + overlays.

## Validate
Add the `e2e` label; confirm `generate_matrix` prints the shard count
and the `e2e /` jobs fan out.
 2026-08-03 17:36:29 +00:00
udev d161965b14 chore: add ci check for schema version uptick (#13339) 
## Summary

- Add a script to detect newly added database migrations.
- Require `DB_SCHEMA_VERSION` to increase when migrations are added.
- Run the check for pull requests and merge queue events with full Git
history.

## Testing

- Not run (not requested)
 2026-07-22 16:07:40 +00:00
Shrinath RaoandClaude Opus 4.8 43263ad45e test: fix regression Merge Reports failing with missing html-report (force reporters) (#12939) 
## Did we introduce this?

**No.** This is a pre-existing fragility in the `Merge Reports and
Upload to TestDino` job of the **Playwright Regression Tests** workflow
— the `unset CI` mechanism has been there since the workflow was created
(#11669, 2026-05). It is unrelated to any test change.

## Symptom

Job fails with exit code 2 **even when all e2e shards pass** (e.g. run
[28410300804](https://github.com/openobserve/openobserve/actions/runs/28410300804/job/84183381446)
— 11/11 shards green, only Merge Reports red):

```
Contents of playwright-results:
-rw-r--r-- 1 runner runner 0 ... .gitkeep
ls: cannot access 'playwright-results/html-report/': No such file or directory
##[error]Process completed with exit code 2.
```

## Root cause

`playwright.config.js` selects reporters by `process.env.CI`:

```js
reporter: process.env.CI
  ? [['blob', { outputDir: 'blob-report' }]]
  : [['html', { outputFolder: 'playwright-results/html-report' }], ['json', { outputFile: 'playwright-results/report.json' }]]
```

The merge step depended on `unset CI` to take the html/json branch. The
runner exports `CI=true`, and `unset CI` does not reliably reach the
`merge-reports` subprocess — so the config re-selects the **`blob`**
reporter, which writes a merged `blob-report/` and **nothing** into
`playwright-results/`. The downstream `ls
playwright-results/html-report/` then exits 2.

### Reproduced locally with this run's blob artifacts
- `CI=true` + `merge-reports --config playwright.config.js` → writes
`blob-report/report.zip`, no `playwright-results/html-report/` → `ls`
exit 2 (matches CI exactly).
- Reporters forced explicitly → always writes `html-report/index.html` +
`report.json`, regardless of `CI`.

## Fix

Stop depending on `unset CI`. Force the reporters and their output paths
explicitly:

```bash
PLAYWRIGHT_HTML_OUTPUT_DIR=playwright-results/html-report \
PLAYWRIGHT_HTML_OPEN=never \
PLAYWRIGHT_JSON_OUTPUT_NAME=playwright-results/report.json \
  npx playwright merge-reports --reporter html,json ./all-blob-reports
```

Plus a guard that fails with a clear message (and lists
`all-blob-reports/`) if the expected outputs are missing — instead of
the cryptic `ls` exit 2.

## Verification (local, against this run's 10 blob artifacts)

Simulating CI (`CI=true GITHUB_ACTIONS=true`):

```
merge exit=0
GUARD: PASS
playwright-results/html-report/index.html   (717 KB)
playwright-results/report.json              (587 KB, stats: expected=75 unexpected=0 flaky=0 skipped=9)
blob-report/   → not created
```

Downstream `jq '.stats'` reads the produced `report.json` correctly.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-30 01:49:58 +00:00
Shrinath RaoandClaude Opus 4.8 1155da8400 ci: per-test flaky/failed → ci_test_results + per-shard API counts → ci_test_counts (#12930) 
## Why
Two CI-observability gaps, both about **per-test/per-shard counts** that
the run-level metrics couldn't show:

1. **Flaky vs failed was conflated.** "Failed Shards by Module" counted
shard-*job* failures — but a shard that **failed then passed on rerun is
flaky, not failed**. A bare count told you nothing and misclassified
recovered flakes as failures.
2. **API had no test counts at all.** API `ci_test_runs` rows carried
only run health (conclusion/duration) — `tests_total` was NULL — so the
Test Counts dashboard had nothing for API.

## What

### Per-test flaky **and** failed → `ci_test_results`
- The **merge job** parses the merged `report.json` for tests with
`status="flaky"` (failed→passed on retry) or `status="unexpected"`
(failed after retries) via the new commented
**`.github/scripts/extract-flaky.js`**, emitting **title + module +
status**.
- **report_to_openobserve** posts **one row per such test** to
**`ci_test_results`**.
- Row schema: `{run_id, repo, suite, workflow, ingest_source, status
("flaky"|"failed"), module, test_title, test_file}` — powers accurate
per-module **flaky** *and* **failed** dashboards, no misclassification.

### Per-shard API test counts → `ci_test_counts` (+ run-doc total)
- Each API pytest job emits `--junitxml` and uploads an
**`api-junit__<category>__<leg>`** artifact (category = the test shard:
integration / query_agent / regression / … ENT also cli / vortex_* /
oss_in_ent / ent_top / ent_rbac).
- **report_to_openobserve** runs
**`.github/scripts/parse-api-junit.py`**, which reads the category per
artifact and **dedups config-leg reruns by test identity** (query_agent
opt true|false, regression flag combos) so the same suite isn't counted
twice.
- The deduped **total** folds into the run doc (`ci_test_runs`,
suite=api) so API shows in the same KPI/pie panels as UI; **per-category
rows** go to the new **`ci_test_counts`** stream for the per-shard
breakdown (the API analog of UI's `ci_test_modules`).

## Notes
- **Forward-only** — historical runs never stored junit / per-test names
(only counts), so no backfill; both streams accumulate from merge.
- All **non-fatal**: any parse/ingest error only warns, never fails the
workflow. `_timestamp` unset → ingest time.
- Pairs with the same-named branch on the other repo (OSS ⇄ ENT).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-29 13:52:50 +00:00
Shrinath RaoandClaude Opus 4.8 0a2cbdb387 ci: emit per-shard rows for UI runs (ci_test_runs_shards) — e2e per-shard dashboards (#12919) 
Gives the **e2e/UI tabs** the same per-shard breakdown that Regression
already has (Slowest Shards / Shard Duration over time / Per-Module
Health).

`o2-run-report.sh` already sees every `e2e /` shard job in `THIS_JOBS`;
this also flattens them to one row per shard in
**`ci_test_runs_shards`** (mirrors `ci_regression_shards`). API runs
have no `e2e /` jobs → no-op. `_timestamp` unset → ingest time (aligned
with the run doc). bash + jq verified locally.

Pairs with the same-branch ENT PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-29 03:06:50 +00:00
Shrinath RaoandClaude Opus 4.8 48d8cf770c fix: broken UI/API metrics ingest (bash brace-default mangled jq --argjson) (#12918) 
## Urgent — regression from #12917
`o2-run-report.sh` used `--argjson shards "${SHARDS:-{}}"`. In bash,
`${SHARDS:-{}}` parses as **“SHARDS or `{`” plus a trailing literal
`}`**, so when SHARDS was set jq received `{...}}` — invalid JSON →
`payload build failed` → **no `ci_test_runs` doc ingested for any UI/API
run since #12917 merged** (~1h of missing UI/API run metrics).

Regression + inventory are unaffected (they don't use this script).

## Fix
`SHARDS` and `BUILD_DUR` are already defaulted (`'{}'` / `null`) just
above, so pass them directly:
```
--argjson build "$BUILD_DUR" --argjson shards "$SHARDS"
```

Verified locally: payload now builds (`build_duration_sec:360,
shards_total:21, shards_failed:1`); `bash -n` clean. Please merge ASAP
to restore UI/API reporting.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-28 08:16:20 +00:00
Shrinath RaoandClaude Opus 4.8 65296d88ab ci: capture build time + shard tallies (UI/API) + per-shard regression rows (#12917) 
Closes the **UI ↔ Regression dashboard parity** gaps by capturing the
few fields the dashboards were missing. One PR (pairs with the
same-branch ENT PR).

## Changes
- **`o2-run-report.sh`** (UI + API runs → `ci_test_runs`): adds
  - `build_duration_sec` (from the `build_binary` job)
- `shards_total / shards_passed / shards_failed / shards_skipped` (from
the `e2e /` shard jobs)
  - Null/0 for suites without them (API) — harmless.
- **`playwright_regression.yml`**: after the run doc, emits **one row
per shard** to a new **`ci_regression_shards`** stream (flattened from
`payload.shards`) → enables per-shard duration / slowest-shard /
per-module panels.

## Why
The Regression tab had per-shard + build-time panels that UI/Combined
couldn't match because the data wasn't captured. Now it is.

## Safety
All additions are non-fatal (the report job is `continue-on-error`;
shard emit is guarded). Verified locally: build/shard jq (build=360,
shards total/passed/failed correct), per-shard row shape (module
extracted), and YAML.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-28 05:56:41 +00:00
Shrinath RaoandClaude Opus 4.8 9bf7164e2b ci: capture UI (Playwright) per-test counts (ci_test_runs) (#12915) 
## What

Capture **UI (Playwright) per-test counts** in the `ci_test_runs`
OpenObserve stream — `tests_total/passed/failed/flaky/skipped` per run.

This was meant to land in #12914 but missed that squash-merge (pushed
just after), so it's its own PR now. Companion: o2-enterprise#2046.

## How
- `playwright.yml` merge job gains a `stats` output (from the merged
`report.json`).
- `report_to_openobserve` reads it and emits the test-count fields via a
small generic **`EXTRA_JSON`** merge hook added to `o2-run-report.sh`.
- Non-fatal as before; no-ops when secrets are unset.

API (pytest) counts are not included (would need `--junitxml` + XML
aggregation — separate change).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-27 17:29:44 +00:00
Shrinath RaoandClaude Opus 4.8 94f6aab74e ci: live retry-aware metrics for playwright, api-testing + regression (#12914) 
## What

Makes the per-run test dashboards **live**. Adds non-fatal
`report_to_openobserve` jobs that emit **one retry-aware run-summary doc
per run** to OpenObserve, following the metrics work merged earlier
(regression/docgen/council). Companion PR in the other repo mirrors
this.

**Scope (this PR):**
1. **`playwright.yml` + `api-testing.yml`** → new report jobs →
`ci_test_runs` (`suite=ui` / `suite=api`).
2. **`playwright_regression.yml`** → retry-aware reporting →
`ci_regression` (OSS: augments the existing job; ENT: adds the job — was
uninstrumented).

## How it works

- **Shared `.github/scripts/o2-run-report.sh`** (used by playwright/api)
derives the run's `conclusion` from job results, wall-clock from the
jobs API, and **sums cost across all attempts**. Regression jobs do the
same inline (they also emit shard/build/test fields).
- Emits: `run_id`, `run_attempt`, `retries`, `was_retried`,
`conclusion`, `suite`, `repo`, `branch`, `author`, `commit_sha`,
`pr_number`, `trigger`, `final_duration_sec`, `total_duration_sec`,
`retry_wasted_sec`.
- **Dedup model:** re-runs append one doc per attempt; dashboards keep
the **latest attempt per `run_id`** via `row_number() OVER (PARTITION BY
run_id ORDER BY run_attempt DESC)`. Retries become an *attribute*, never
duplicate rows; `total_duration_sec` on the kept row carries cumulative
compute.
- **Safe by construction:** `continue-on-error` + the never-fail poster
— can't fail/slow a test run; skips when no tests ran; **no-ops until
`O2_REPORTING_*` secrets are set**.

## Review
AI review addressed (mktemp payload, `GH_TOKEN` guard, non-negative
duration clamp). YAML + jq unit-tested + `actionlint` clean on the added
jobs.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-27 16:13:05 +00:00
Shrinath RaoandClaude Opus 4.8 bfade1f4f3 ci: report regression run metrics to OpenObserve (ci_regression) (#12912) 
## What

Adds a CI→OpenObserve metrics sidecar to the Playwright Regression
workflow. A new terminal job `report_to_openobserve` emits **one summary
JSON document per run** into the `ci_regression` OpenObserve stream, so
we can build reliability/performance dashboards.

This is the OSS half of a cross-repo change — the companion
o2-enterprise PR (`ci/o2-reporting`) instruments the DocGen and E2E
Council engines into `ci_docgen` / `ci_council`.

## How it works (safe by construction)

- **Non-invasive:** derives per-job timings + per-shard pass/fail from
the GitHub jobs API and the merged Playwright stats. The only edit to an
existing job is one additive `stats` output on
`merge_and_upload_reports`.
- **Cannot break CI:** the job is `continue-on-error` and posts through
`.github/scripts/o2-report.sh`, which always `exit 0`s (a missing secret
/ rotated passcode / unreachable host only warns).
- **No-ops until secrets exist** — safe to merge first.

## Metrics captured
Reliability (conclusion, shard pass/fail, test pass/fail/flaky/skipped),
duration & cost (wall-clock, `runner_seconds`, build time, per-shard
durations), throughput (trigger/actor/branch).

## Required secrets (this repo + o2-enterprise)
- `O2_REPORTING_INGEST_BASE` — `https://<host>/api/<org>`
- `O2_REPORTING_AUTH` — `base64("email:passcode")`

Full schema, dashboard SQL, and setup notes:
`.github/scripts/O2-REPORTING.md`.

## Validation
- YAML parse + jq unit tests + `actionlint` clean on the added code.
- Live **HTTP 200** test-ingest into the target instance, both via raw
curl and through the poster script.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-27 05:34:29 +00:00
Shrinath RaoandClaude Opus 4.8 0ee8a667c2 test: E2E Council Refiner — remediate Sentinel findings instead of dead-ending (#12876) 
## Why

The E2E Council blocked **#12857** (no test PR opened) because the
post-heal **Sentinel re-audit found 2 critical quality issues** — even
though **9/10 tests passed**. The tests were **not worthless**; 2 just
*under-claimed* (the body asserted something weaker than the test name
promised):

1. `"should not persist FTS default across page reload"` — re-searches
after reload and asserts the column visible, which passes whether it
*persisted* or *re-resolved*. Never verifies non-persistence.
2. `"should clear FTS marker and re-resolve for new stream on stream
change"` — only checks "table rendered", never that FTS actually
re-resolved.

Blocking was correct (green ≠ meaningful), but a **dead-end block throws
away the other 90%**. The pipeline should **fix the problem and make it
right**, not give up. A no-PR outcome should be a *true last resort*.

## What — The Refiner (Phase 5.5)

A remediation loop **inside `verify_heal`**, after the Sentinel re-audit
and before the quality gate (**reuses the already-booted binary** — no
rebuild):

- **Only fires when needed.** If the re-audit is clean (PASS / 0
critical), the step reads the verdict, sets `ran=false`, and exits —
**no DeepSeek call, no test re-run**. Cheap on the happy path.
- **Strengthen-first.** Feeds the exact Sentinel findings to the
Refiner, which **adds/tightens assertions** so each test verifies what
its name promises → re-runs on the live binary (**green must hold**) →
re-checks assertion-integrity → re-audits. Capped at **2 rounds**.
- **Anti-laundering (the whole game).** Assertions may only be
**ADDED/strengthened — never dropped, inverted in place, or
renamed-to-dodge** a missing assertion. If strengthening goes red
because the **feature isn't wired**, it takes the honest `test.fixme` +
evidence exit (no green-wash).
- **`assert_integrity.py --strengthen`**: permits *added* negative
matchers (assert-absence is legit strengthening) while **still blocking
any dropped assertion** (removed/skipped guards + all static checks
stay). Unit-proven in the commit.
- **Quality gate** now judges the **post-refine** state; blocks only if
the Refiner couldn't keep the spec green honestly.
- **No-PR report** now phrases a block as a true last resort ("the
Refiner attempted to strengthen … but could not resolve every finding
honestly").

### Reuse path (clobber off)
`verify_heal` now also runs on **reuse runs**, and the "already covered"
short-circuit is suppressed on reuse — so **re-running `/e2e` on a
previously-BLOCKED spec re-validates + refines the existing tests into a
PR** instead of stopping at "already covered". No new tests are added on
reuse; the Refiner only strengthens what's there. (Cost: a reuse run
rebuilds even when tests were already perfect — acceptable since re-runs
are explicit; can add a "skip if a green test PR already exists" guard
later.)

## Validation
- `assert_integrity.py` unit-checked: added negative → FAIL without
`--strengthen`, PASS with it; dropped assertion → still FAIL with
`--strengthen`.
- Workflow YAML validated; step order confirmed **Sentinel re-audit →
Refiner → Quality gate**.
- Live end-to-end best validated by re-running `/e2e` on **#12857 with
clobber off** — the Refiner should strengthen the 2 flagged tests and
open the PR.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-25 09:20:31 +00:00
Shrinath RaoandClaude Opus 4.8 900ebe67c6 ci(e2e-council): test-quality gate — stop certifying green-but-worthless specs (#12722) 
## The problem (from the #12719 review)

The pipeline's only hard gate was **"the Healer's authoritative run is
green."** That objective *rewards* making a spec pass by gutting the
assertions that test the feature. In #12719 the result was a suite that:
- dropped the fix-query-card assertion,
- asserted `count >= 0` (a tautology),
- buried real assertions inside `if`s whose `else` just logs, and
- **asserted the feature is absent** (`expectFixQueryCardNotVisible`) —
certifying a half-built feature and set to **turn red when the feature
is finished**.

> A passing test and a valuable test are different things, and we only
measured the first.

This PR makes **"done" mean "the tests would fail if the feature were
broken."**

## Deterministic backbone (no LLM) —
`.github/scripts/assert_integrity.py`

The most important guards don't depend on the model behaving:
- **Diff-aware heal guard** — snapshot the spec's assertions *before*
healing; **reject** any heal that **deleted**, **inverted**
(`toBeVisible` → `not.toBeVisible`), or **skipped** assertions to reach
green.
- **Static anti-patterns** — tautologies (`count >= 0`, `expect(true)`)
and **conditional-only** tests (every `expect` inside an `if` → the test
can pass asserting nothing).
- Conservative by design (only high-confidence patterns) — a gate with
false positives gets disabled by humans. Restored from a **trusted ref**
in verify_heal (never the PR tree's copy).

## Workflow

- New **Assertion-integrity gate** after the authoritative run + a
combined **Quality gate decision** (integrity **+** Sentinel verdict).
`pr_back` now requires `quality_status == pass` → a green-but-worthless
spec **never becomes a PR**.
- The post-heal **Sentinel re-audit now feeds the blocking gate** (was
advisory-only). LLM flakiness fails **open**; the deterministic gate is
the hard teeth.
- **`feature-incomplete`** honest exit: when a test can't pass because
the feature isn't wired, the Healer parks it `test.fixme` + writes
**evidence**; the workflow opens **no PR** and posts the gap (file:line)
on the source PR — the single highest-value output.
- `pr_back` **reconciles `test-summary.json` against the FINAL spec**
(drops phantom rows) so the PR body can't claim tests that don't exist.

## Prompts

- **Healer:** hard anti-weakening rules + the `feature-incomplete`
honest exit.
- **Sentinel:** tautology / conditional-only / negative-only-on-feature
/ name-mismatch as **CRITICAL** checks.
- **Engineer:** no dead page-object scaffolding; ban `waitForTimeout` as
a sync primitive; assertions must be real and match the test name.
- **Architect:** one spec = one area (no Frankenstein specs);
de-duplicate scenarios.

## Considered and deferred: mutation-sanity (extra feature-absent build)

Theoretically the cleanest definition of a valuable test, but **dropped
for now**: it overlaps the cheap static checks, has **no clean
"feature-absent" baseline** under merged/multi-PR input, risks false
positives that erode trust, and doubles the build. The static guards
catch the same #12719 failures at near-zero cost; prevention (Healer
can't invert + honest `feature-incomplete`) beats the expensive
detector.

## Testing

- `assert_integrity.py` unit-tested against tautology / conditional-only
(incl. single-line) / heal-removed / heal-inverted fixtures, with no
false positive on a legit `if` + unconditional `expect`.
- Workflow YAML validated; analyzer compiles.
- **First live run still needed** (per the project's "run the path
before done" rule) — to see the gate block a real weak spec and the
`feature-incomplete` comment fire.

Stacked on #12716 (ENT sister flow); the diff collapses to just the
quality program once #12716 lands.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-21 09:30:03 +00:00