mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-04 16:09:05 +00:00
Shrinath RaoandClaude Opus 4.8
097deb4ee9
## Problem The **Logs-Builder-Basic** e2e shard is frequently flaky — it times out on `unflattened.spec.js` and, because it's a big shard, a single timeout re-runs the whole thing. Example: [run 27521591530](https://github.com/openobserve/openobserve/actions/runs/27521591530/job/81341218030). In that run: - `unflattened.spec.js:205` — hard failure, **timed out (300s) on all 3 retries** - `unflattened.spec.js:100` and `logsQueryBuilder-chart.spec.js:148` — flaky (passed on retry) ## Root cause The failures are **timeouts, not assertion bugs**. The unflattened tests: 1. Slept **~76s of fixed `waitForTimeout`** across the file (pure dead time). 2. Ran a retry loop that **re-ingested 3,848 records up to 5×/test** while waiting for the `_o2_id` field to index after enabling *Store Original Data*. Each test + setup is ~80s on a fast local machine; CI runners are ~3-4× slower **and** run all 4 specs in the shard in parallel against one backend. That pushes individual tests right onto the 300s/test ceiling. The re-ingestion balloon also hammered the shared backend, slowing the other specs (`logsQueryBuilder`, `pagination`) — which is why they flaked as collateral. ## Fix - Add `waitForO2IdQueryable()` to the unflattened page object: **polls the search API until `_o2_id` is actually queryable** instead of sleeping a fixed interval and hoping indexing caught up. - Gate the UI scan on that readiness signal, removing the 15s/5s blind schema-propagation sleeps. - Turn the expensive **re-ingest** retry loop into a cheap **re-query** loop (attempts 5 → 3) — the gate guarantees the data is already in the backend, so a UI miss is only render lag. No change to what the tests actually assert. ## Verification (local, against self-hosted OpenObserve) | Check | Result | |---|---| | Unflattened solo ×2 (cleanup between) | 2 passed each, `_o2_id` on **row 0 / attempt 1** every time, **2.3m** (was 2.8m) | | `logsQueryBuilder-chart:148` isolated ×5 | **5/5 passed** → not independently flaky (collateral) | | **Full 4-spec shard, 4 workers (CI scenario)** | **52 passed, 0 failed, 0 flaky** | The retry loop now never fires — the readiness gate makes the first UI scan succeed every run, killing both the timeout and the cross-spec contention. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>