ci(playwright): stop SLO-Measurement backfill from serializing 21 daily chunks (#13870)

## What this fixes

The `SLO-Measurement` shard (`slo-timeslice.spec.js` +
`slo-alerts.spec.js`) has been taking **~26 minutes**

## Root cause

Both specs seed **8 days of history** and create SLOs with a **7-day
rolling window**, then block in `beforeAll` until the SLO is actually
measured (`waitForSloMeasured`). The server backfills history in
`ZO_SLO_BACKFILL_CHUNK_SECS` chunks — default **86400s (1 day)** — and
only **one chunk at a time** (`ZO_SCHEDULER_SLO_BACKFILL_CONCURRENCY=1`,
by design, so bulk backfill never starves real-time alerts).

The shard backfills **3 SLOs** across the two specs, each needing 7
daily chunks to cover its 7-day window:

- `slo-timeslice.spec.js` `beforeAll`: 2 SLOs (`sloLt`, `sloGt`) × 7
chunks = 14
- `slo-alerts.spec.js` `beforeAll`: 1 SLO × 7 chunks = 7

**21 chunks total, strictly serial**, at roughly 40s/chunk ≈ **~14
minutes of pure backfill wait**, before any UI assertion runs. This is
the server behaving correctly and exactly as designed for production —
the test is just paying for it the slow way.

## Fix

Neither spec asserts on per-chunk timing or partial coverage, only on
the final measured state, so raising the chunk size to cover the whole
window in one pass is free wall-clock, not a behavior change.

- Threaded `slo_backfill_chunk_secs` through `build-ci-matrix.js`'s
existing per-shard passthrough, same pattern already used for
`ingest_allowed_upto` / `quick_mode_enabled`.
- Set it to `604800` (7 days) for the `SLO-Measurement` shard only — the
server-wide default (`86400`) is untouched everywhere else.
- Documented the new knob in `tests/ui-testing/ci-matrix/README.md`.

## Validation

**Synthetic (API-only), same seed shape as the real specs** (8 days,
1-min interval, deterministic 3h fault window), against a local build:

| | Default (86400s chunk) | Fixed (604800s chunk) |
|---|---|---|
| Time to measured | 246.9s (~4.1 min) for **one** SLO, stepping 1/7 →
2/7 → ... every ~40s | **3.0s** |
| Final SLI | 98.21428571428571% | 98.21428571428571% (**identical**) |

Same math, same result — only the number of scheduler round-trips
changes.

**Real shard, end-to-end**, run exactly as CI invokes it (`npx
playwright test slo-timeslice.spec.js slo-alerts.spec.js --workers=1`)
against a server mirroring this shard's full env including the fix:

- **25/25 tests passed**, 0 regressions
- Total wall-clock (server start → global teardown): **4m Avg**

Local timing won't map 1:1 onto the GitHub Actions runner (different
disk/scheduler characteristics — even the "before" baseline was faster
locally than the ~26min/~5min-per-spec seen in CI), so the authoritative
number is whatever this PR's own CI run reports. But the structural win
— 1 chunk instead of 7 per SLO — is verified correct and regression-free
independent of machine.

## Companion PR

The enterprise repo's `playwright.yml` independently consumes per-shard
matrix fields (it explicitly mirrors OSS's `ingest_allowed_upto`
handling for the same reason), so it needs the equivalent env
passthrough or it'll keep paying the old cost even after this merges.
Companion PR: openobserve/o2-enterprise#2460
This commit is contained in:
Vamsi Karthik R
2026-08-19 00:58:33 +00:00
committed by GitHub
parent 87f7960d48
commit 7b79fdcdbe
4 changed files with 30 additions and 1 deletions
+11
View File
@@ -115,6 +115,13 @@ for (const s of include) {
// silently DROPS older rows while still returning a success-shaped response. A
// suite that seeds history — SLOs measure a rolling 7-day window — needs a wider
// one, and scoping it to those shards keeps every other shard on the default.
//
// slo_backfill_chunk_secs follows the same rule for the same class of reason:
// it's how much history one SLO backfill chunk covers, read once at server
// start. The workflow default (86400 = 1 day) makes a 7-day-window test SLO
// backfill in 7 sequential chunks under ZO_SCHEDULER_SLO_BACKFILL_CONCURRENCY=1;
// SLO-Measurement widens it to the full window so its 3 backfilled SLOs don't
// pay 21 chunks serially.
const matrix = include.map((s) => ({
testfolder: s.testfolder,
actual_folder: s.actual_folder,
@@ -126,6 +133,10 @@ const matrix = include.map((s) => ({
s.ingest_allowed_upto === undefined || s.ingest_allowed_upto === null
? ""
: String(s.ingest_allowed_upto),
slo_backfill_chunk_secs:
s.slo_backfill_chunk_secs === undefined || s.slo_backfill_chunk_secs === null
? ""
: String(s.slo_backfill_chunk_secs),
// Per-shard worker count. Empty = use playwright.config.js (5 in CI).
//
// `fullyParallel` runs separate spec FILES concurrently, and