mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-04 00:08:55 +00:00
ci(playwright): stop SLO-Measurement backfill from serializing 21 daily chunks (#13870)
## What this fixes The `SLO-Measurement` shard (`slo-timeslice.spec.js` + `slo-alerts.spec.js`) has been taking **~26 minutes** ## Root cause Both specs seed **8 days of history** and create SLOs with a **7-day rolling window**, then block in `beforeAll` until the SLO is actually measured (`waitForSloMeasured`). The server backfills history in `ZO_SLO_BACKFILL_CHUNK_SECS` chunks — default **86400s (1 day)** — and only **one chunk at a time** (`ZO_SCHEDULER_SLO_BACKFILL_CONCURRENCY=1`, by design, so bulk backfill never starves real-time alerts). The shard backfills **3 SLOs** across the two specs, each needing 7 daily chunks to cover its 7-day window: - `slo-timeslice.spec.js` `beforeAll`: 2 SLOs (`sloLt`, `sloGt`) × 7 chunks = 14 - `slo-alerts.spec.js` `beforeAll`: 1 SLO × 7 chunks = 7 **21 chunks total, strictly serial**, at roughly 40s/chunk ≈ **~14 minutes of pure backfill wait**, before any UI assertion runs. This is the server behaving correctly and exactly as designed for production — the test is just paying for it the slow way. ## Fix Neither spec asserts on per-chunk timing or partial coverage, only on the final measured state, so raising the chunk size to cover the whole window in one pass is free wall-clock, not a behavior change. - Threaded `slo_backfill_chunk_secs` through `build-ci-matrix.js`'s existing per-shard passthrough, same pattern already used for `ingest_allowed_upto` / `quick_mode_enabled`. - Set it to `604800` (7 days) for the `SLO-Measurement` shard only — the server-wide default (`86400`) is untouched everywhere else. - Documented the new knob in `tests/ui-testing/ci-matrix/README.md`. ## Validation **Synthetic (API-only), same seed shape as the real specs** (8 days, 1-min interval, deterministic 3h fault window), against a local build: | | Default (86400s chunk) | Fixed (604800s chunk) | |---|---|---| | Time to measured | 246.9s (~4.1 min) for **one** SLO, stepping 1/7 → 2/7 → ... every ~40s | **3.0s** | | Final SLI | 98.21428571428571% | 98.21428571428571% (**identical**) | Same math, same result — only the number of scheduler round-trips changes. **Real shard, end-to-end**, run exactly as CI invokes it (`npx playwright test slo-timeslice.spec.js slo-alerts.spec.js --workers=1`) against a server mirroring this shard's full env including the fix: - **25/25 tests passed**, 0 regressions - Total wall-clock (server start → global teardown): **4m Avg** Local timing won't map 1:1 onto the GitHub Actions runner (different disk/scheduler characteristics — even the "before" baseline was faster locally than the ~26min/~5min-per-spec seen in CI), so the authoritative number is whatever this PR's own CI run reports. But the structural win — 1 chunk instead of 7 per SLO — is verified correct and regression-free independent of machine. ## Companion PR The enterprise repo's `playwright.yml` independently consumes per-shard matrix fields (it explicitly mirrors OSS's `ingest_allowed_upto` handling for the same reason), so it needs the equivalent env passthrough or it'll keep paying the old cost even after this merges. Companion PR: openobserve/o2-enterprise#2460
This commit is contained in:
@@ -115,6 +115,13 @@ for (const s of include) {
|
||||
// silently DROPS older rows while still returning a success-shaped response. A
|
||||
// suite that seeds history — SLOs measure a rolling 7-day window — needs a wider
|
||||
// one, and scoping it to those shards keeps every other shard on the default.
|
||||
//
|
||||
// slo_backfill_chunk_secs follows the same rule for the same class of reason:
|
||||
// it's how much history one SLO backfill chunk covers, read once at server
|
||||
// start. The workflow default (86400 = 1 day) makes a 7-day-window test SLO
|
||||
// backfill in 7 sequential chunks under ZO_SCHEDULER_SLO_BACKFILL_CONCURRENCY=1;
|
||||
// SLO-Measurement widens it to the full window so its 3 backfilled SLOs don't
|
||||
// pay 21 chunks serially.
|
||||
const matrix = include.map((s) => ({
|
||||
testfolder: s.testfolder,
|
||||
actual_folder: s.actual_folder,
|
||||
@@ -126,6 +133,10 @@ const matrix = include.map((s) => ({
|
||||
s.ingest_allowed_upto === undefined || s.ingest_allowed_upto === null
|
||||
? ""
|
||||
: String(s.ingest_allowed_upto),
|
||||
slo_backfill_chunk_secs:
|
||||
s.slo_backfill_chunk_secs === undefined || s.slo_backfill_chunk_secs === null
|
||||
? ""
|
||||
: String(s.slo_backfill_chunk_secs),
|
||||
// Per-shard worker count. Empty = use playwright.config.js (5 in CI).
|
||||
//
|
||||
// `fullyParallel` runs separate spec FILES concurrently, and
|
||||
|
||||
Reference in New Issue
Block a user