test: fix regression Merge Reports failing with missing html-report (force reporters) (#12939)

## Did we introduce this?

**No.** This is a pre-existing fragility in the `Merge Reports and
Upload to TestDino` job of the **Playwright Regression Tests** workflow
— the `unset CI` mechanism has been there since the workflow was created
(#11669, 2026-05). It is unrelated to any test change.

## Symptom

Job fails with exit code 2 **even when all e2e shards pass** (e.g. run
[28410300804](https://github.com/openobserve/openobserve/actions/runs/28410300804/job/84183381446)
— 11/11 shards green, only Merge Reports red):

```
Contents of playwright-results:
-rw-r--r-- 1 runner runner 0 ... .gitkeep
ls: cannot access 'playwright-results/html-report/': No such file or directory
##[error]Process completed with exit code 2.
```

## Root cause

`playwright.config.js` selects reporters by `process.env.CI`:

```js
reporter: process.env.CI
  ? [['blob', { outputDir: 'blob-report' }]]
  : [['html', { outputFolder: 'playwright-results/html-report' }], ['json', { outputFile: 'playwright-results/report.json' }]]
```

The merge step depended on `unset CI` to take the html/json branch. The
runner exports `CI=true`, and `unset CI` does not reliably reach the
`merge-reports` subprocess — so the config re-selects the **`blob`**
reporter, which writes a merged `blob-report/` and **nothing** into
`playwright-results/`. The downstream `ls
playwright-results/html-report/` then exits 2.

### Reproduced locally with this run's blob artifacts
- `CI=true` + `merge-reports --config playwright.config.js` → writes
`blob-report/report.zip`, no `playwright-results/html-report/` → `ls`
exit 2 (matches CI exactly).
- Reporters forced explicitly → always writes `html-report/index.html` +
`report.json`, regardless of `CI`.

## Fix

Stop depending on `unset CI`. Force the reporters and their output paths
explicitly:

```bash
PLAYWRIGHT_HTML_OUTPUT_DIR=playwright-results/html-report \
PLAYWRIGHT_HTML_OPEN=never \
PLAYWRIGHT_JSON_OUTPUT_NAME=playwright-results/report.json \
  npx playwright merge-reports --reporter html,json ./all-blob-reports
```

Plus a guard that fails with a clear message (and lists
`all-blob-reports/`) if the expected outputs are missing — instead of
the cryptic `ls` exit 2.

## Verification (local, against this run's 10 blob artifacts)

Simulating CI (`CI=true GITHUB_ACTIONS=true`):

```
merge exit=0
GUARD: PASS
playwright-results/html-report/index.html   (717 KB)
playwright-results/report.json              (587 KB, stats: expected=75 unexpected=0 flaky=0 skipped=9)
blob-report/   → not created
```

Downstream `jq '.stats'` reads the produced `report.json` correctly.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Shrinath Rao
2026-06-30 01:49:58 +00:00
committed by GitHub
co-authored by Claude Opus 4.8
parent 01f4a6cc08
commit 43263ad45e
3 changed files with 84 additions and 18 deletions
+15 -4
View File
@@ -34,23 +34,34 @@ gh_get(){ curl -s -H "Authorization: token ${GH_TOKEN}" -H "Accept: application/
# wall-clock of one attempt = max(completed) - min(started) across its jobs (clamped non-negative
# to guard against clock-skew where a completed_at precedes a started_at).
attempt_wall(){ jq '[.jobs[]|select(.completed_at!=null and .started_at!=null)] as $j
| (if ($j|length)>0 then (($j|map(.completed_at|fromdateiso8601)|max)-($j|map(.started_at|fromdateiso8601)|min)) else 0 end)
# $1 = floor epoch (the attempt's run_started_at). On a RE-RUN, GitHub's per-attempt jobs API
# carries over jobs that did NOT re-run with their ORIGINAL (earlier-attempt) timestamps, so a
# naive min(started) spans the idle gap between a failed run and a much-later manual re-run —
# inflating the duration by hours. Flooring the start at run_started_at counts only this
# attempt's actual work (no effect on normal runs, where min(started) >= run_started_at).
attempt_wall(){ jq --argjson floor "${1:-0}" '[.jobs[]|select(.completed_at!=null and .started_at!=null)] as $j
| (if ($j|length)>0 then (($j|map(.completed_at|fromdateiso8601)|max)-([($j|map(.started_at|fromdateiso8601)|min), $floor]|max)) else 0 end)
| if . < 0 then 0 else . end'; }
# Epoch seconds of a run/attempt object's run_started_at (0 if absent).
run_started_epoch(){ jq 'if .run_started_at then (.run_started_at|fromdateiso8601) else 0 end'; }
RUN_JSON=$(gh_get "repos/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}" 2>/dev/null)
[ -n "$RUN_JSON" ] || { echo "::warning::could not fetch run object — skipping"; exit 0; }
ATT="${GITHUB_RUN_ATTEMPT:-1}"
# This attempt's start — the floor for its wall-clock (see attempt_wall).
RUN_STARTED=$(printf '%s' "$RUN_JSON" | run_started_epoch 2>/dev/null || echo 0)
[ -n "$RUN_STARTED" ] || RUN_STARTED=0
THIS_JOBS=$(gh_get "repos/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/attempts/${ATT}/jobs?per_page=100" 2>/dev/null)
FINAL_DUR=$(printf '%s' "$THIS_JOBS" | attempt_wall 2>/dev/null || echo 0)
FINAL_DUR=$(printf '%s' "$THIS_JOBS" | attempt_wall "$RUN_STARTED" 2>/dev/null || echo 0)
# total across attempts: add each prior attempt's wall-clock.
TOTAL_DUR="$FINAL_DUR"
if [ "$ATT" -gt 1 ] 2>/dev/null; then
SUM="$FINAL_DUR"
for n in $(seq 1 $((ATT-1))); do
W=$(gh_get "repos/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/attempts/${n}/jobs?per_page=100" 2>/dev/null | attempt_wall 2>/dev/null || echo 0)
NFLOOR=$(gh_get "repos/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/attempts/${n}" 2>/dev/null | run_started_epoch 2>/dev/null || echo 0)
W=$(gh_get "repos/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/attempts/${n}/jobs?per_page=100" 2>/dev/null | attempt_wall "${NFLOOR:-0}" 2>/dev/null || echo 0)
SUM=$(awk -v a="$SUM" -v b="$W" 'BEGIN{print a+b}')
done
TOTAL_DUR="$SUM"