mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-03 16:10:21 +00:00
Shrinath RaoandClaude Opus 4.8
1155da8400
## Why
Two CI-observability gaps, both about **per-test/per-shard counts** that
the run-level metrics couldn't show:
1. **Flaky vs failed was conflated.** "Failed Shards by Module" counted
shard-*job* failures — but a shard that **failed then passed on rerun is
flaky, not failed**. A bare count told you nothing and misclassified
recovered flakes as failures.
2. **API had no test counts at all.** API `ci_test_runs` rows carried
only run health (conclusion/duration) — `tests_total` was NULL — so the
Test Counts dashboard had nothing for API.
## What
### Per-test flaky **and** failed → `ci_test_results`
- The **merge job** parses the merged `report.json` for tests with
`status="flaky"` (failed→passed on retry) or `status="unexpected"`
(failed after retries) via the new commented
**`.github/scripts/extract-flaky.js`**, emitting **title + module +
status**.
- **report_to_openobserve** posts **one row per such test** to
**`ci_test_results`**.
- Row schema: `{run_id, repo, suite, workflow, ingest_source, status
("flaky"|"failed"), module, test_title, test_file}` — powers accurate
per-module **flaky** *and* **failed** dashboards, no misclassification.
### Per-shard API test counts → `ci_test_counts` (+ run-doc total)
- Each API pytest job emits `--junitxml` and uploads an
**`api-junit__<category>__<leg>`** artifact (category = the test shard:
integration / query_agent / regression / … ENT also cli / vortex_* /
oss_in_ent / ent_top / ent_rbac).
- **report_to_openobserve** runs
**`.github/scripts/parse-api-junit.py`**, which reads the category per
artifact and **dedups config-leg reruns by test identity** (query_agent
opt true|false, regression flag combos) so the same suite isn't counted
twice.
- The deduped **total** folds into the run doc (`ci_test_runs`,
suite=api) so API shows in the same KPI/pie panels as UI; **per-category
rows** go to the new **`ci_test_counts`** stream for the per-shard
breakdown (the API analog of UI's `ci_test_modules`).
## Notes
- **Forward-only** — historical runs never stored junit / per-test names
(only counts), so no backfill; both streams accumulate from merge.
- All **non-fatal**: any parse/ingest error only warns, never fails the
workflow. `_timestamp` unset → ingest time.
- Pairs with the same-named branch on the other repo (OSS ⇄ ENT).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
61 lines
2.2 KiB
JavaScript
61 lines
2.2 KiB
JavaScript
#!/usr/bin/env node
|
|
/**
|
|
* Extract flaky + failed tests from a Playwright JSON report for OpenObserve.
|
|
*
|
|
* Usage: node extract-flaky.js <path-to-report.json>
|
|
* Output: a JSON array on stdout, one object per flaky-or-failed test:
|
|
* [{ module, title, file, status }] status = "flaky" | "failed"
|
|
*
|
|
* "flaky" = Playwright status "flaky" (failed, then passed on retry)
|
|
* "failed" = Playwright status "unexpected" (failed even after retries)
|
|
*
|
|
* BOTH are captured on purpose. The consumer stream is `ci_test_results` (NOT
|
|
* `ci_test_flaky`) and every row carries a `status` field so dashboards can tell
|
|
* flaky from failed — a shard that failed then passed on rerun is flaky, not
|
|
* failed, and the per-test verdict here is what keeps that distinction honest.
|
|
*
|
|
* `title` comes from the SPEC (sp.title): in Playwright's JSON schema the spec is
|
|
* the test() call and holds the title; the per-project `test` objects nested under
|
|
* it have NO title field, so t.title would be undefined. Do not change to t.title.
|
|
*
|
|
* Never throws: a missing/malformed report is logged to stderr and prints `[]`
|
|
* with exit 0, so the calling workflow step stays non-fatal.
|
|
*/
|
|
const fs = require("fs");
|
|
|
|
const reportPath = process.argv[2];
|
|
if (!reportPath || !fs.existsSync(reportPath)) {
|
|
console.error(`extract-flaky: report not found at ${reportPath || "<no path>"}`);
|
|
process.stdout.write("[]");
|
|
process.exit(0);
|
|
}
|
|
|
|
let report;
|
|
try {
|
|
report = JSON.parse(fs.readFileSync(reportPath, "utf8"));
|
|
} catch (e) {
|
|
console.error("extract-flaky: report.json parse failed:", e && e.message);
|
|
process.stdout.write("[]");
|
|
process.exit(0);
|
|
}
|
|
|
|
const out = [];
|
|
(function walk(suite) {
|
|
(suite.specs || []).forEach((sp) =>
|
|
(sp.tests || []).forEach((t) => {
|
|
if (t.status === "flaky" || t.status === "unexpected") {
|
|
const file = sp.file || suite.file || "";
|
|
out.push({
|
|
module: file.split("/")[0] || "unknown", // top-level test folder
|
|
title: sp.title,
|
|
file,
|
|
status: t.status === "flaky" ? "flaky" : "failed",
|
|
});
|
|
}
|
|
})
|
|
);
|
|
(suite.suites || []).forEach(walk);
|
|
})({ suites: report.suites || [] });
|
|
|
|
process.stdout.write(JSON.stringify(out));
|