Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Bumps
[taiki-e/install-action](https://github.com/taiki-e/install-action) from
2.85.10 to 2.86.5.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/taiki-e/install-action/releases">taiki-e/install-action's
releases</a>.</em></p>
<blockquote>
<h2>2.86.5</h2>
<ul>
<li>
<p>Update <code>zola@latest</code> to 0.23.4.</p>
</li>
<li>
<p>Update <code>wasm-tools@latest</code> to 1.257.1.</p>
</li>
<li>
<p>Update <code>protoc@latest</code> to 3.36.0.</p>
</li>
<li>
<p>Update <code>mise@latest</code> to 2026.8.10.</p>
</li>
<li>
<p>Update <code>cargo-dinghy@latest</code> to 0.8.6.</p>
</li>
<li>
<p>Update <code>wasmtime@latest</code> to 48.0.0.</p>
</li>
</ul>
<h2>2.86.4</h2>
<ul>
<li>
<p>Update <code>oxfmt@latest</code> to 1.79.0.</p>
</li>
<li>
<p>Update <code>mise@latest</code> to 2026.8.8.</p>
</li>
<li>
<p>Update <code>martin@latest</code> to 1.14.0.</p>
</li>
<li>
<p>Update <code>dprint@latest</code> to 0.56.0.</p>
</li>
<li>
<p>Update <code>cargo-about@latest</code> to 0.9.2.</p>
</li>
<li>
<p>Update <code>biome@latest</code> to 2.5.9.</p>
</li>
</ul>
<h2>2.86.3</h2>
<ul>
<li>
<p>Update <code>prek@latest</code> to 0.4.14.</p>
</li>
<li>
<p>Update <code>osv-scanner@latest</code> to 2.5.1.</p>
</li>
<li>
<p>Update <code>cargo-llvm-cov@latest</code> to 0.9.0.</p>
</li>
</ul>
<h2>2.86.2</h2>
<ul>
<li>
<p>Update <code>uv@latest</code> to 0.12.5.</p>
</li>
<li>
<p>Update <code>mise@latest</code> to 2026.8.6.</p>
</li>
<li>
<p>Update <code>cargo-tarpaulin@latest</code> to 0.37.2.</p>
</li>
</ul>
<h2>2.86.1</h2>
<ul>
<li>Fix an issue where <code>oxfmt</code> was accidentally installed as
<code>oxfmt-{target}{exe}</code>. (<a
href="https://redirect.github.com/taiki-e/install-action/pull/1969">#1969</a>)</li>
</ul>
<h2>2.86.0</h2>
<ul>
<li>Support <code>oxfmt</code>. (<a
href="https://redirect.github.com/taiki-e/install-action/pull/1967">#1967</a>,
thanks <a
href="https://github.com/rami3l"><code>@rami3l</code></a>)</li>
</ul>
<h2>2.85.14</h2>
<ul>
<li>
<p>Update <code>uv@latest</code> to 0.12.4.</p>
</li>
<li>
<p>Update <code>trivy@latest</code> to 0.74.0.</p>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/taiki-e/install-action/blob/main/CHANGELOG.md">taiki-e/install-action's
changelog</a>.</em></p>
<blockquote>
<h2>[2.86.5] - 2026-08-21</h2>
<ul>
<li>
<p>Update <code>zola@latest</code> to 0.23.4.</p>
</li>
<li>
<p>Update <code>wasm-tools@latest</code> to 1.257.1.</p>
</li>
<li>
<p>Update <code>protoc@latest</code> to 3.36.0.</p>
</li>
<li>
<p>Update <code>mise@latest</code> to 2026.8.10.</p>
</li>
<li>
<p>Update <code>cargo-dinghy@latest</code> to 0.8.6.</p>
</li>
<li>
<p>Update <code>wasmtime@latest</code> to 48.0.0.</p>
</li>
</ul>
<h2>[2.86.4] - 2026-08-20</h2>
<ul>
<li>
<p>Update <code>oxfmt@latest</code> to 1.79.0.</p>
</li>
<li>
<p>Update <code>mise@latest</code> to 2026.8.8.</p>
</li>
<li>
<p>Update <code>martin@latest</code> to 1.14.0.</p>
</li>
<li>
<p>Update <code>dprint@latest</code> to 0.56.0.</p>
</li>
<li>
<p>Update <code>cargo-about@latest</code> to 0.9.2.</p>
</li>
<li>
<p>Update <code>biome@latest</code> to 2.5.9.</p>
</li>
</ul>
<h2>[2.86.3] - 2026-08-18</h2>
<ul>
<li>
<p>Update <code>prek@latest</code> to 0.4.14.</p>
</li>
<li>
<p>Update <code>osv-scanner@latest</code> to 2.5.1.</p>
</li>
<li>
<p>Update <code>cargo-llvm-cov@latest</code> to 0.9.0.</p>
</li>
</ul>
<h2>[2.86.2] - 2026-08-17</h2>
<ul>
<li>
<p>Update <code>uv@latest</code> to 0.12.5.</p>
</li>
<li>
<p>Update <code>mise@latest</code> to 2026.8.6.</p>
</li>
<li>
<p>Update <code>cargo-tarpaulin@latest</code> to 0.37.2.</p>
</li>
</ul>
<h2>[2.86.1] - 2026-08-15</h2>
<ul>
<li>Fix an issue where <code>oxfmt</code> was accidentally installed as
<code>oxfmt-{target}{exe}</code>. (<a
href="https://redirect.github.com/taiki-e/install-action/pull/1969">#1969</a>)</li>
</ul>
<h2>[2.86.0] - 2026-08-15</h2>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/taiki-e/install-action/commit/ba47c86ac325773530516bb756137ac718732518"><code>ba47c86</code></a>
Release 2.86.5</li>
<li><a
href="https://github.com/taiki-e/install-action/commit/5e0b59bc928f672ccd1072a19b6815d0cf5b21cb"><code>5e0b59b</code></a>
Update <code>zola@latest</code> to 0.23.4</li>
<li><a
href="https://github.com/taiki-e/install-action/commit/60dfdd7edfbb113f6254c9b3889a61fb3e20baf4"><code>60dfdd7</code></a>
Update <code>wasm-tools@latest</code> to 1.257.1</li>
<li><a
href="https://github.com/taiki-e/install-action/commit/c262502059d0289897f7720971cb62c8b83217f1"><code>c262502</code></a>
Update <code>protoc@latest</code> to 3.36.0</li>
<li><a
href="https://github.com/taiki-e/install-action/commit/4985cbc05f60164fac55ad0f7e788c0a97fd2aed"><code>4985cbc</code></a>
Update <code>mise@latest</code> to 2026.8.10</li>
<li><a
href="https://github.com/taiki-e/install-action/commit/e782b6cb9189134d57cdae9cdf8ac0ae4c62384c"><code>e782b6c</code></a>
Update <code>cargo-dinghy@latest</code> to 0.8.6</li>
<li><a
href="https://github.com/taiki-e/install-action/commit/e891b49f0d0c25fc74d5bf0f29a720e43c063750"><code>e891b49</code></a>
Update biome manifest</li>
<li><a
href="https://github.com/taiki-e/install-action/commit/394bb5ddf8b2bba382638dadf8c93ff59b9b3fb5"><code>394bb5d</code></a>
Update wasmtime manifest</li>
<li><a
href="https://github.com/taiki-e/install-action/commit/a2a5f6e99e1a31540baa0468acfa302cff0f359f"><code>a2a5f6e</code></a>
Release 2.86.4</li>
<li><a
href="https://github.com/taiki-e/install-action/commit/5afbb1a197d47fef2a5793727b494b64e09ac265"><code>5afbb1a</code></a>
Update wasm-tools manifest</li>
<li>Additional commits viewable in <a
href="https://github.com/taiki-e/install-action/compare/v2.85.10...v2.86.5">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
## What
- **CLAUDE.md** (new, tracked): project-level rules for AI coding
sessions — build/workflow basics, Rust code organization (item order
`mod → use → const/static → types → impls → fns`, pub/private function
grouping, `#[cfg(test)] mod tests` always last, structural thresholds
are ratchet-only), comment discipline (one line, only for non-obvious
constraints), PR conventions, and enterprise-gated code verification
(including keeping `Cargo.toml.openobserve` in the o2-enterprise repo in
sync with this repo's root `Cargo.toml`). Replaces `.claude/.clauderc`,
which is not a file Claude Code reads.
- **Structural clippy ratchets in CI**: the clippy step now also enables
`too_many_lines`, `cognitive_complexity`, and `excessive_nesting`, with
clippy.toml thresholds set just above today's worst offenders (1024
lines / complexity 100 / nesting 10), so the current tree passes clean.
Measured across the workspace with `--all-targets`: max function 1019
lines, max complexity 94, max nesting 10. The thresholds only ratchet
down as code is cleaned up — CI failing on one of these means split the
function, not raise the number.
- **clippy.toml**: add `source-item-ordering = ["module"]` so the opt-in
`clippy::arbitrary_source_item_ordering` lint checks module-level item
order only (its default also forces alphabetical struct fields/enum
variants).
- **.gitignore**: reorganized into commented sections and slimmed from
116 to 75 lines — removed duplicate entries, patterns shadowed by other
patterns, one dead rule, and 14 stale one-off entries with no matching
files; simplified the `.vscode`/`.opencode`/`.claude` blocks.
## Why
Coding sessions kept placing functions in arbitrary positions (above
constants, mixed into the wrong visibility group, tests at the top of
files) and writing multi-line comment essays, and nothing bounded
function size/complexity growth. CLAUDE.md states the conventions once
for every contributor's sessions; enforcement stays with the standard
Rust toolchain (rustfmt, clippy thresholds, CI's `-D warnings`).
## Notes for reviewers
- The enterprise CI clippy step runs its own command without these `-W`
flags, so it is unaffected.
- An earlier revision of this branch added a PostToolUse hook script for
style checking; it was dropped in favor of toolchain + prompt rules
only, and `.claude/settings.json` stays gitignored.
- The gitignore rewrite was verified behavior-preserving at each step:
the untracked-file list and an 86-path rule battery were compared
before/after every change; the only intentional semantic changes are the
removals listed above.
## What this PR adds
end-to-end test suite for the **Mobile RUM SDK** under
`tests/mobile-testing/`, plus a **self-contained CI workflow** that runs
it on real emulators/simulators against an OpenObserve **built from this
PR's source**. Test / fixture / CI only — no product-code/existing test
changes
## How each test works (3 layers)
Every scenario is a real end-to-end journey, so a failure points at the
actual broken layer:
1. **Drive** — a Maestro flow taps through the app like a user
(`maestro/<track>/*.yaml`).
2. **API** — query the `_rumdata` stream and assert the telemetry is
correct (`utils/ooClient.js`, poll-and-retry for async ingestion).
3. **UI** — open the RUM dashboard and assert the sessions
## Coverage (42 tests across 4 platforms)
The **native platforms (android-native, ios-native) are now at parity
with React Native** — each new
native test (masking, network, bg/fg, user-identity, no-phone-home) was
built + validated on a real
emulator/simulator before landing.
| Capability | Layer | RN-Android | Android-native | RN-iOS | iOS-native
|
|---|---|:-:|:-:|:-:|:-:|
| Core telemetry (views/actions/handled-error/crash → `_rumdata`) | API
| ✅ | ✅ | ✅ | ✅ |
| Crash session viewable in dashboard | UI | ✅ | ✅ | ✅ | ✅ |
| Attributes: env / service / version tagging | API | ✅ | ✅ | ✅ | ✅ |
| User identity (`setUser`) | API | ✅ | ✅ | ✅ | ✅ |
| Network: fetch → resource + 4xx/5xx | API | ✅ | ✅ | ✅ | ✅ |
| Session-replay PII masking | UI | ✅ | ✅ | ✅ | ✅ |
| Background / foreground view transitions | API | ✅ | ✅ | — | ✅ |
| No-phone-home (no egress to Datadog hosts) | security | ✅ | ✅ | — | ✅
|
- Shared per-capability factories keep the specs DRY
(`utils/rumChecks.js`, `utils/coreRumSpec.js`).
- Negatives are covered: 4xx capture, PII-must-not-leak (masking), no
phone-home, handled-error ≠ crash.
- `o2-enterprise#2289` dashboard bugs are kept as inline skips that flip
green once fixed.
- Full capability × platform detail + on-device status:
`docs/COVERAGE-MATRIX.md`.
## Fixture apps + build-time config
- Four vendored apps under `apps/`: `o2-rum-tester` (React Native —
drives both RN platforms), `o2-native-android` (Kotlin), `o2-native-ios`
(Swift / xcodegen).
- Each reads its RUM target (endpoint / org / token) and SDK version
from a **build-time-overridable** config. The committed default is the
current pin, so local dev is unchanged; CI regenerates it to point at
the runner's OpenObserve and can pin the latest SDK on release.
## CI — `.github/workflows/mobile-sdk-e2e.yml` (self-contained, no
secrets)
- **Build once per OS from source.** `build_oo_linux` / `build_oo_macos`
compile OpenObserve (binary + embedded web UI) from this PR's code and
cache it. Device jobs download and run it **natively**, and the app
posts real RUM to it (Android reaches the host at `10.0.2.2`, the iOS
simulator via `localhost`). No Docker, no shared cluster, no secrets.
- **Device jobs run in parallel.** `android` (Linux + KVM emulator →
rn-android + android-native) and `ios` (macOS simulator, pinned Xcode
16.4 → rn-ios + ios-native).
- **`merge_reports`** combines both platforms into one browsable HTML
report — even when a device job fails.
- **`decide` picks which platform runs, to keep scarce macOS runners in
check:**
- PR touching Android or shared mobile-RUM code → **Android auto-runs**;
iOS is gated behind the `mobile-e2e` label.
- `mobile-e2e` label → both platforms (full pre-merge gate).
- `repository_dispatch(mobile-sdk-release)` from an SDK repo → runs that
SDK's platform (companion PRs wire this up:
openobserve-react-native-rum#3, openobserve-sdk-android#1,
openobserve-sdk-ios#1).
- `workflow_dispatch` → manual, choose platforms.
*Separate workflow :* it needs macOS **and** Android emulators, builds 4
phone apps, and is dispatch-driven — matching the repo's existing
per-suite split
## Status
Green end-to-end on GitHub runners (Android + iOS device jobs + merge) :
https://github.com/openobserve/openobserve/actions/runs/31733717758/job/94565520891
---------
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
Automated coverage for **email alert destinations and distribution
lists**, and unblocks the email tests that have been skipped since they
were written. **32 cases** — 27 active, 5 skipped regressions tracking
open defects (openobserve/o2-enterprise#2471).
**The unblocking.** The two skipped tests said *"the test environment
does not have email infrastructure set up"*. That over-stated it:
creating, editing, validating and deleting an email destination needs
SMTP switched **on**, and never reads a mailbox. Only delivery
assertions do. The suite is tiered on that:
| Tier | Needs | Cases |
|---|---|---|
| A | SMTP enabled | 24 |
| B | a readable sink | 2 |
| C | an alias that fans out | 1 |
**Tests**
- `Alerts/dl-email-destinations.spec.js` (32) — the DL half (alias
accepted, addressed, fanned out) and the recipient-list half, where our
code actually runs: splitting/trimming/lowercasing, the org-membership
gate, edit round-trips, add/remove/delete, both form paradigms (prebuilt
free-text vs the custom path's org-user picker), template auto-attach,
duplicate names, cancel, type-switching, keyboard, dark mode at 420px.
- `utils/mail-sink.js` — pluggable sink: Mailpit, a Mailinator inbox, or
a clean skip with a stated reason.
- `utils/o2-api.js` — storage is asserted from the API, never the form
field: an input can echo what you typed while the stored value differs.
- `scripts/dl-relay.js` — SMTP relay that expands one alias into
per-member deliveries, so fan-out runs with no external provider. Same
pattern as the in-test HTTP receiver in
`alerts-content-templates.spec.js`.
- `alertDestinationsPage.js` — +17 methods; the spec carries zero raw
locators.
**Replaces, without losing coverage.** The two skipped tests in
`alerts-destinations-prebuilt.spec.js` are removed for a pointer mapping
their intent to its new home: create (`D-01`), edit-add (`D-07`),
edit-remove (`ML-08`), delete (`UI-12`), format validation (`UI-02`).
Three of those did not exist before. That file now has no skips.
**CI.** `ZO_SMTP_ENABLED: true` was already set in `playwright.yml` but
had no host or port, so it fell back to `localhost:25` where nothing
listens — destinations could be **saved** while every **send** failed
silently. The workflow now starts Mailpit and a DL relay as static
binaries before the app (no Docker-in-Docker on these runners) and
points `ZO_SMTP_*` at them.
All three tiers run with no secrets and no egress. Verified in the CI
topology: **27 passed, 5 skipped**, `D-02` confirmed reaching all three
member mailboxes.
**Behaviour.** Self-cleaning via the `auto_dest_dl_*` prefix
`cleanup.spec.js` already sweeps, plus an API `afterAll`. `mode:
'parallel'` — each test owns its fixtures.
**Findings** are filed in openobserve/o2-enterprise#2471, not fixed
here. Two P1s: the *"Show matching rows"* toggle is ignored so log rows
are always embedded, and a user removed from the org keeps receiving
alerts while the destination can no longer be saved. The regression
tests for each are in this PR, skipped until the fix lands.
**Paired ENT PR: openobserve/o2-enterprise#2472.** ENT's workflow sets
`ZO_SMTP_ENABLED: false`, so without it the suite skips all 27 cases
there. The shard matrix needs no ENT change — ENT builds it from this
repo's `ci_matrix.json` plus its overlay, so the new spec is picked up
automatically. Merge the two together: this one alone leaves enterprise
with no email coverage.
## What
Adds `.github/workflows/merge-queue-auto-requeue.yml` — **Auto Merge
Queuer**.
When a PR is kicked **out of the merge queue because its required checks
failed** (`reason == CI_FAILURE`) or **timed out** (`reason ==
CI_TIMEOUT`), the workflow adds it straight back to the queue — **up to
twice**.
## Behavior
- **Trigger:** `pull_request: [dequeued]`, gated at the job `if` to
`CI_FAILURE` / `CI_TIMEOUT` only.
- **Manual removals** (`reason == MANUAL`) and every other reason
(`MERGE_CONFLICT`, `QUEUE_CLEARED`, `BRANCH_PROTECTIONS`, …) are
**ignored** — the job never runs for them.
- **Limit:** attempt count tracked via labels `auto-requeue-1` /
`auto-requeue-2` (max 2 re-adds).
- **Re-queue:** GraphQL `enqueuePullRequest` mutation, authenticated
with `secrets.ORG_ADMIN_TOKEN` (a `GITHUB_TOKEN` enqueue would not
re-trigger the `merge_group` check workflows).
- **One sticky comment, edited in place** (found by a hidden marker):
- 1st kick → "added back (attempt 1 of 2)"
- 2nd kick → same comment edited → "re-added a second time (2 of 2) —
last automatic retry"
- 3rd kick → same comment edited → "🛑 no-go for auto re-add" and it does
**not** re-queue.
All `gh` calls run under the PAT, so the comment, labels and enqueue are
authored by the token owner (not `github-actions[bot]`).
## Notes for reviewers
- Requires repo secret `ORG_ADMIN_TOKEN` with merge-queue write (already
used by 26 workflows in this repo).
- `github.event.reason` on the `dequeued` payload drives the gate; worth
confirming on the first real CI-caused dequeue. Safe failure mode if
absent: the job no-ops.
## What & why
Repackages our existing automated PR reviewer under a branded identity —
**OpenObserve Code Reviewer** — and gives its comment a colorful,
attention-grabbing format. This is **purely aesthetic + repackaging**.
> [!IMPORTANT]
> The review **engine is frozen**. The 5 specialist agents, coordinator
judgment/rubric, risk tiering, XML finding contract, re-review
PATCH-in-place logic, total-outage guard, DeepSeek/opencode wiring,
read-only security model, and OTel tracing are all **unchanged**. Only
*who signs the comment* and *how it looks* change.
## The two things that move
**1. Identity (fail-safe).** The workflow mints a GitHub App
installation token via `actions/create-github-app-token`, **gated on the
`OO_REVIEWER_APP_ID` variable**. If the app isn't configured, it skips
and falls back to `github.token` (posts as `github-actions[bot]`,
exactly as today). So this PR is **safe to merge before the app exists**
and auto-activates once the secrets are provisioned.
**2. Presentation.** The coordinator now emits:
- Decision as a native GitHub **callout box** — 🟢 `[!TIP]` approved · 🔵
`[!NOTE]` approved-with-comments · 🟡 `[!WARNING]` minor · 🔴 `[!CAUTION]`
significant.
- Rebranded heading `## 🔎 OpenObserve Code Review`.
- Per-category glyphs 🔒⚡🧩📝📦 on each finding line.
`sanitizeReviewBody` normalizes the heading (tolerates a missing emoji /
legacy heading, idempotent on re-review) and the deterministic fallback
renderer gets the same cosmetic treatment.
## To activate the brand (one-time, org admin)
Register a GitHub App "OpenObserve Code Reviewer", install on this repo,
then set:
- Variable `OO_REVIEWER_APP_ID` (numeric App ID — this is the gate)
- Secret `OO_REVIEWER_APP_PRIVATE_KEY` (the app's `.pem`)
Remove the variable to instantly roll back to `github-actions[bot]`.
## Verification
- JS parses; both workflows valid YAML.
- Engine files byte-identical with the o2-enterprise counterpart PR.
- 6/6 sanitizer heading cases pass (fresh, no-emoji, legacy, preamble,
idempotent, trailing-words).
- This PR's own review comment will preview the new format.
Paired with the identical o2-enterprise PR (same branch name).
## What
Removes the **Qodo PR-Agent** workflow (`qodo-ai/pr-agent@main`).
## Why
- It auto-runs its code-suggestions tool on every PR via OpenAI; that
call now fails, so it posts **"Failed to generate code suggestions for
PR"** as noise on **every** PR (e.g. o2-enterprise#2475).
- It's **superseded by the OpenObserve Code Reviewer** app, which now
handles automated PR review.
- Keeping it means a failing, redundant OpenAI dependency pinned to a
**mutable `@main`** tag.
Added Jun 2025 (OSS #7041 / ENT #577); effectively unused since the
branded reviewer shipped.
## Scope
Deletes only the PR-Agent workflow file. No other CI touched. Paired
OSS/ENT change.
## What
Heals the two Traces-Regression tests that broke after #13852
(\`fix(ui): clean up SLO detail and traces navigation\`).
Failing scheduled runs:
- ENT:
https://github.com/openobserve/o2-enterprise/actions/runs/32213206401/job/95953026991
— \`traces-ent.spec.js\` Service Graph test
- OSS:
https://github.com/openobserve/openobserve/actions/runs/32213278081 —
\`traces-regression.spec.js\` Service Catalog test
Both failed all 4 retries with \`locator.click: Timeout 45000ms
exceeded\` on the left-rail flyout item.
## Root cause
#13852 rerouted the Traces rail-flyout children through the Traces
page's in-page \`?tab=\` views. That changed **both**:
| | before | after |
|---|---|---|
| flyout \`data-test\` | \`nav-group-item-serviceGraph\` |
\`nav-group-item-traces-service-graph\` |
| | \`nav-group-item-servicesCatalog\` |
\`nav-group-item-traces-services-catalog\` |
| URL | \`/traces/service-graph\` | \`/traces?tab=service-graph\` |
| | \`/traces/services\` | \`/traces?tab=services-catalog\` |
(\`childDataTest = nav-group-item-\${child.name}-\${child.tab}\`; legacy
paths now only exist as redirects.) The stale \`data-test\` made
\`.click()\` wait the full 45s action timeout.
## Fix
\`serviceGraphPage.navigateToServiceGraph()\` and
\`servicesCatalogPage.clickServiceCatalogTab()\`:
- point at the new flyout \`data-test\` ids
- \`waitFor\` the teleported flyout item to be **visible** before
clicking — the flyout opens after ONavGroup's 120ms \`OPEN_DELAY\`, so
the item isn't in the DOM the instant \`hover()\` resolves
- match the new \`/traces?tab=\` URL
Engine/other nav helpers untouched.
## Verification
Run locally against a #13852 frontend (localhost:5080, enterprise
build):
- ✅ Live flyout click →
\`/web/traces?org_identifier=default&tab=service-graph\` and
\`...&tab=services-catalog\`
- ✅ **OSS** \`traces-regression.spec.js\` "Clearing search filter should
not blank out Service Catalog table" — **passes**
- ✅ **ENT** \`traces-ent.spec.js\` Service Graph nav now works — test
reaches the chart assertion (line 54); the only residual local failure
is absent service-graph topology (0 nodes locally; CI seeds it via
ingestion + the daemon)
## Follow-up (not in this PR / not in any CI matrix)
\`Traces/traces-toolbar-tabs.spec.js\` +
\`tracesPage.switchToServiceMaps()\` still assert the pre-#13852 "flyout
→ standalone route" dual-semantics that #13852 removed. Not run by any
CI matrix, so left for a separate change aligned with the new
single-\`?tab=\` semantics.
## What this fixes
The `SLO-Measurement` shard (`slo-timeslice.spec.js` +
`slo-alerts.spec.js`) has been taking **~26 minutes**
## Root cause
Both specs seed **8 days of history** and create SLOs with a **7-day
rolling window**, then block in `beforeAll` until the SLO is actually
measured (`waitForSloMeasured`). The server backfills history in
`ZO_SLO_BACKFILL_CHUNK_SECS` chunks — default **86400s (1 day)** — and
only **one chunk at a time** (`ZO_SCHEDULER_SLO_BACKFILL_CONCURRENCY=1`,
by design, so bulk backfill never starves real-time alerts).
The shard backfills **3 SLOs** across the two specs, each needing 7
daily chunks to cover its 7-day window:
- `slo-timeslice.spec.js` `beforeAll`: 2 SLOs (`sloLt`, `sloGt`) × 7
chunks = 14
- `slo-alerts.spec.js` `beforeAll`: 1 SLO × 7 chunks = 7
**21 chunks total, strictly serial**, at roughly 40s/chunk ≈ **~14
minutes of pure backfill wait**, before any UI assertion runs. This is
the server behaving correctly and exactly as designed for production —
the test is just paying for it the slow way.
## Fix
Neither spec asserts on per-chunk timing or partial coverage, only on
the final measured state, so raising the chunk size to cover the whole
window in one pass is free wall-clock, not a behavior change.
- Threaded `slo_backfill_chunk_secs` through `build-ci-matrix.js`'s
existing per-shard passthrough, same pattern already used for
`ingest_allowed_upto` / `quick_mode_enabled`.
- Set it to `604800` (7 days) for the `SLO-Measurement` shard only — the
server-wide default (`86400`) is untouched everywhere else.
- Documented the new knob in `tests/ui-testing/ci-matrix/README.md`.
## Validation
**Synthetic (API-only), same seed shape as the real specs** (8 days,
1-min interval, deterministic 3h fault window), against a local build:
| | Default (86400s chunk) | Fixed (604800s chunk) |
|---|---|---|
| Time to measured | 246.9s (~4.1 min) for **one** SLO, stepping 1/7 →
2/7 → ... every ~40s | **3.0s** |
| Final SLI | 98.21428571428571% | 98.21428571428571% (**identical**) |
Same math, same result — only the number of scheduler round-trips
changes.
**Real shard, end-to-end**, run exactly as CI invokes it (`npx
playwright test slo-timeslice.spec.js slo-alerts.spec.js --workers=1`)
against a server mirroring this shard's full env including the fix:
- **25/25 tests passed**, 0 regressions
- Total wall-clock (server start → global teardown): **4m Avg**
Local timing won't map 1:1 onto the GitHub Actions runner (different
disk/scheduler characteristics — even the "before" baseline was faster
locally than the ~26min/~5min-per-spec seen in CI), so the authoritative
number is whatever this PR's own CI run reports. But the structural win
— 1 chunk instead of 7 per SLO — is verified correct and regression-free
independent of machine.
## Companion PR
The enterprise repo's `playwright.yml` independently consumes per-shard
matrix fields (it explicitly mirrors OSS's `ingest_allowed_upto`
handling for the same reason), so it needs the equivalent env
passthrough or it'll keep paying the old cost even after this merges.
Companion PR: openobserve/o2-enterprise#2460
Adds automated test coverage for **SLOs** (#13547, #13577, #13579,
#13704, #13761, #13784), which shipped with none. **149 tests** — 77
Playwright + 72 pytest — all three SLI types, both query languages,
happy / negative / edge paths. Clean branch off `main`.
**UI (Playwright · `tests/ui-testing`)**
- `pages/sloPages/` — 4 page objects (verified `data-test` selectors,
hydration + Monaco-sync waits, OSelect commit-retry, virtualized-list
filtering).
- `playwright-tests/SLO/slo-crud.spec.js` (24) — create
count/time-slice/grouped through the form, detail, edit, SLI-type
change, tags, pause, delete, filters; negative: dismissed delete, empty
search, cancel, unmeasured SLI is absent not zero.
- `playwright-tests/SLO/slo-form-validation.spec.js` (20) — server
errors surfacing in the form, conditional fields per SLI type, count +
time-slice previews, **PromQL count and time-slice created end to end**;
negative: no stream, invalid aggregate, over-long name, target
boundaries.
- `playwright-tests/SLO/slo-alert-sli.spec.js` (8) — **the `alert` SLI
type**: source picker, create, config carries only the alert id, detail
names the source; negative: ineligible source offered-but-disabled,
grouped alert SLI, slice finer than source cadence.
- `playwright-tests/SLO/slo-timeslice.spec.js` (10) — **#13761 P0
regression** (measured SLI, not the preview), PromQL, previews, gaps,
language-flip clearing.
- `playwright-tests/SLO/slo-alerts.spec.js` (15) — burn-rate +
error-budget kinds, presets, edit, freeze door, burndown; negative: no
destination, cancel, blank/invalid name.
- `playwright-tests/utils/slo-seed.js` — backdated seeding, metrics
seeding, eligible-source-alert fixture, ground-truth query, fail-fast
env guard, transient-store retry, full fixture teardown (alerts →
streams → destinations → templates).
**API (pytest · `tests/api-testing`)**
- `tests/slos/test_slo_validation.py` (72) — full REST surface
(create/get/update/enable/delete/groups/slo-eligible), validation on
**both create and update** (name, target, window, slice, config shapes,
comparators, float-threshold round-trip), boundaries (target
0.001/99.999, name 256 vs 257, threshold 0/negative/1e12/NaN, unicode
names), alert-source eligibility (silence-gated, too-infrequent, cron),
PromQL config shapes, auth 401/404.
**CI**
- Two shards: `SLO` (`workers: 3`) and `SLO-Measurement` (`workers: 1`).
Split because the backfill job runs with
`ZO_SCHEDULER_SLO_BACKFILL_CONCURRENCY=1`, so measurement specs sharing
an instance starve each other's waits — `fullyParallel` races spec
*files*, which `mode:'serial'` cannot prevent. The 3-worker pin is
SQLite: at 5, concurrent SLO/alert/destination writes trip `database is
locked` on the single-node meta store.
- New optional per-shard `ingest_allowed_upto` (`240`) alongside the
existing `quick_mode_enabled`: the workflow default of 5 **hours**
silently drops the 8 days of history an SLO window needs, while still
answering `200`. Other shards keep the default. Same precedent as
`api-testing.yml`'s `48`.
**Feature code**: 9 additive `data-test`/`dataTest` attributes in
`SloDetail.vue`, `SloList.vue`, `SloAlertCondition.vue` — no logic
changes, `vue-tsc` clean. The stat tiles, tabs and burn-window inputs
had no stable selectors; `OStatStrip`/`OTab` already forward the
attribute.
**Behaviour**: self-cleaning (`e2e_slo_*`, worker-scoped since
`afterAll` runs per worker).
**Verified in CI** on this branch: `e2e / SLO` **52 passed** (1.8m,
workers pinned to 3), `e2e / SLO-Measurement` **25 passed** (11.3m,
workers pinned to 1) — the measurement shard passing confirms the
per-shard `ingest_allowed_upto` took effect, since those tests are
impossible under the 5-hour default. pytest **72 passed** locally; in
CI's 599-test run one setup delete hit a transient 500 (see findings)
and now retries.
**Findings for the feature owner** (reported, not fixed here):
- A blank "good when" shows `Request failed with status code 422` —
`pruned()` drops the empty key, so the API's real 400 message never
reaches `save()`.
- A rejected alert name shows no banner: `SloAlertForm.submit()` returns
on `nameError` before setting `saveError`.
- `avg(latency_ms); DROP TABLE users--` is accepted at create time
(parsed only at measurement).
- The name-length message says "less than 256 characters" but 256 is
accepted — 257 is the first rejection.
- Duplicate alert names on one SLO are accepted (duplicate SLO names are
a 409), and the panel lists them by name, so they are indistinguishable.
- SLO names accept characters that break the list's own
`slos-slolist-edit-${name}` selectors; alert names correctly reject
them.
- All chart/preview root `data-test` ids are unreachable — a parent's
fallthrough attribute overrides the child root's.
- Both edit forms hydrate asynchronously, so a blank form can flash
before the stored values appear.
- `DELETE /slos/{id}` on a freshly-created SLO returned **500** once
under CI load (not reproducible locally in 12 rapid iterations).
Contention is arguably a retryable 503, and the body carried no
actionable message.
- **A scheduled alert whose source stream is deleted is retried
forever** (`Stream "…" not found in schema`), and the SLO backfill lane
shares the scheduler's worker budget — so accumulated dead alerts
*starve SLO measurement*. Observed: 65 of them held coverage at 1 of 7
chunks indefinitely; deleting them restored 0.143 → 1.0 in 240s.
Harmless in CI (fresh instance per shard) but it degrades any long-lived
environment. The suite now cleans up after itself; the retry-forever
behaviour itself may deserve a look.
**Paired ENT PR: openobserve/o2-enterprise#2434.** ENT's matrix is this
repo's manifest plus an overlay, merged by this repo's
`build-ci-matrix.js`, so both SLO shards reach ENT automatically —
verified: 31 merged shards, both SLO entries carrying
`ingest_allowed_upto=240` and their worker pins, no other shard
affected. What ENT was missing is *consuming* those two fields, which
#2434 adds. Merge the two together: this one alone leaves ENT booting
the SLO shards on the 5-hour ingest window, and #2434 alone is inert.
*(An earlier review reported ENT as using an inline matrix with no SLO
shard — that turned out not to be the case.)*
Two regression sentinels for the FST projection rewrite path fixed in
openobserve/openobserve#13808 (currently in the merge queue on
branch-v0.40.0 and awaiting the main-branch counterpart):
Q1001 — CTE with trim(log) alias, match_all in WHERE, column-pruned
outer SELECT. Mirrors the shape of the PR's own unit test
(test_cse_projection_schema_propagation) — CSE inserts a
ProjectionExec between filter and datasource; a regression in the
rewriter's f_down handling would surface here.
Q1002 — five-CTE pipeline (filtered → in_shop → scoped → normalized
→ cleaned) mixing match_all with LIKE, split_part, regexp_replace ID
normalization, and GROUP BY aggregation. Higher structural pressure:
more intermediate ProjectionExec nodes and more chances for a stale
column index to leak through.
Uses only test-fixture fields (log, resource_path, page_url,
http_method); no customer identifiers.
Also adds a third leg to the query-agent CI matrix that runs with
ZO_FEATURE_QUERY_REMOVE_FILTER_WITH_INDEX=false — exercises the
non-index-filter path that the match_all projection rewriter also
touches.
The wide JSON churn across the other query files is compute_counts.py
re-baselining expected results against a fresh BASE_TS
(base_ts_override.json is untracked); tuple-order shuffles are harmless
under results_mode: rowsort.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Problem
The API/DB integration test jobs intermittently fail — often, and
retries don't reliably help. CI reports the failure on the **`Pin
cpython`** step:
```
rye: command not found
##[error]Process completed with exit code 127
```
But that step is a **victim, not the cause**. The real failure is the
preceding **`Setup rye`** step (`eifinger/setup-rye@v4`). It downloads
and extracts the `rye` binary fine, then runs `rye self install`, which
downloads a **python-build-standalone toolchain (3.11.7)** from GitHub
to bootstrap Rye's own venv:
```
Downloading Rye from ".../rye/releases/download/0.27.0/rye-x86_64-linux.gz" ... ✓
Extracting downloaded archive... ✓
Determined RYE_TOOLCHAIN_VERSION: 3.11.7
Installing Rye into /opt/hostedtoolcache/setup-rye-.../0.27.0/x86_64
##[error]The process '.../_temp/...' failed with exit code 1 ← real failure
```
That download flakes and exits 1, so the `rye` binary is never placed on
`PATH`, and the next step (`rye pin cpython@3.11.6`) dies with `command
not found` (exit 127). Retries only pass when the download happens to
succeed.
Example failing run:
https://github.com/openobserve/openobserve/actions/runs/31679986019/job/94383108990
## Fix
Every one of these jobs already provisions CPython 3.11.6 via
`actions/setup-python`. Point **`RYE_TOOLCHAIN`** at that interpreter so
`rye self install` reuses it instead of downloading a second toolchain —
removing the flaky network fetch entirely:
```yaml
- name: Setup Python
uses: actions/setup-python@v7
id: setup-python
with:
python-version: "3.11.6"
- name: Setup rye
uses: eifinger/setup-rye@v4
env:
RYE_TOOLCHAIN: ${{ steps.setup-python.outputs.python-path }}
with:
...
```
The `setup-python` step's official `python-path` output is the absolute
path to the interpreter it provisioned — used directly (no hardcoded
`/bin/python`), and present in every affected job (verified 1:1).
## Scope
- `.github/workflows/api-testing.yml` — 4 `Setup rye` blocks
- `.github/workflows/db-testing.yml` — 1 `Setup rye` block
Additive, low-risk: only redirects Rye's internal bootstrap; `rye pin` /
`rye sync` are untouched.
## Companion PR
Enterprise: openobserve/o2-enterprise#2424 — same fix on
`api-testing.yml` (8 blocks) + `api-testing_env.yml` (2 blocks).
Hengfei Yang
Shrinath Rao
Bhargav
Ashish KolheandShrinath Rao
Harsh Mahajan
dependabot[bot]
HuaijinandHengfei Yang
Vamsi Karthik RandShrinath Rao
Neha PandClaude Opus 4.7