mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-04 00:08:55 +00:00
Vamsi Karthik R
7b79fdcdbe
## What this fixes The `SLO-Measurement` shard (`slo-timeslice.spec.js` + `slo-alerts.spec.js`) has been taking **~26 minutes** ## Root cause Both specs seed **8 days of history** and create SLOs with a **7-day rolling window**, then block in `beforeAll` until the SLO is actually measured (`waitForSloMeasured`). The server backfills history in `ZO_SLO_BACKFILL_CHUNK_SECS` chunks — default **86400s (1 day)** — and only **one chunk at a time** (`ZO_SCHEDULER_SLO_BACKFILL_CONCURRENCY=1`, by design, so bulk backfill never starves real-time alerts). The shard backfills **3 SLOs** across the two specs, each needing 7 daily chunks to cover its 7-day window: - `slo-timeslice.spec.js` `beforeAll`: 2 SLOs (`sloLt`, `sloGt`) × 7 chunks = 14 - `slo-alerts.spec.js` `beforeAll`: 1 SLO × 7 chunks = 7 **21 chunks total, strictly serial**, at roughly 40s/chunk ≈ **~14 minutes of pure backfill wait**, before any UI assertion runs. This is the server behaving correctly and exactly as designed for production — the test is just paying for it the slow way. ## Fix Neither spec asserts on per-chunk timing or partial coverage, only on the final measured state, so raising the chunk size to cover the whole window in one pass is free wall-clock, not a behavior change. - Threaded `slo_backfill_chunk_secs` through `build-ci-matrix.js`'s existing per-shard passthrough, same pattern already used for `ingest_allowed_upto` / `quick_mode_enabled`. - Set it to `604800` (7 days) for the `SLO-Measurement` shard only — the server-wide default (`86400`) is untouched everywhere else. - Documented the new knob in `tests/ui-testing/ci-matrix/README.md`. ## Validation **Synthetic (API-only), same seed shape as the real specs** (8 days, 1-min interval, deterministic 3h fault window), against a local build: | | Default (86400s chunk) | Fixed (604800s chunk) | |---|---|---| | Time to measured | 246.9s (~4.1 min) for **one** SLO, stepping 1/7 → 2/7 → ... every ~40s | **3.0s** | | Final SLI | 98.21428571428571% | 98.21428571428571% (**identical**) | Same math, same result — only the number of scheduler round-trips changes. **Real shard, end-to-end**, run exactly as CI invokes it (`npx playwright test slo-timeslice.spec.js slo-alerts.spec.js --workers=1`) against a server mirroring this shard's full env including the fix: - **25/25 tests passed**, 0 regressions - Total wall-clock (server start → global teardown): **4m Avg** Local timing won't map 1:1 onto the GitHub Actions runner (different disk/scheduler characteristics — even the "before" baseline was faster locally than the ~26min/~5min-per-spec seen in CI), so the authoritative number is whatever this PR's own CI run reports. But the structural win — 1 chunk instead of 7 per SLO — is verified correct and regression-free independent of machine. ## Companion PR The enterprise repo's `playwright.yml` independently consumes per-shard matrix fields (it explicitly mirrors OSS's `ingest_allowed_upto` handling for the same reason), so it needs the equivalent env passthrough or it'll keep paying the old cost even after this merges. Companion PR: openobserve/o2-enterprise#2460