mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-04 16:09:05 +00:00
495f6a0f99b4fd847e41bd8446dd4b5765decc4e
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Shrinath RaoandClaude Opus 4.8
|
0ee8a667c2 |
test: E2E Council Refiner — remediate Sentinel findings instead of dead-ending (#12876)
## Why
The E2E Council blocked **#12857** (no test PR opened) because the
post-heal **Sentinel re-audit found 2 critical quality issues** — even
though **9/10 tests passed**. The tests were **not worthless**; 2 just
*under-claimed* (the body asserted something weaker than the test name
promised):
1. `"should not persist FTS default across page reload"` — re-searches
after reload and asserts the column visible, which passes whether it
*persisted* or *re-resolved*. Never verifies non-persistence.
2. `"should clear FTS marker and re-resolve for new stream on stream
change"` — only checks "table rendered", never that FTS actually
re-resolved.
Blocking was correct (green ≠ meaningful), but a **dead-end block throws
away the other 90%**. The pipeline should **fix the problem and make it
right**, not give up. A no-PR outcome should be a *true last resort*.
## What — The Refiner (Phase 5.5)
A remediation loop **inside `verify_heal`**, after the Sentinel re-audit
and before the quality gate (**reuses the already-booted binary** — no
rebuild):
- **Only fires when needed.** If the re-audit is clean (PASS / 0
critical), the step reads the verdict, sets `ran=false`, and exits —
**no DeepSeek call, no test re-run**. Cheap on the happy path.
- **Strengthen-first.** Feeds the exact Sentinel findings to the
Refiner, which **adds/tightens assertions** so each test verifies what
its name promises → re-runs on the live binary (**green must hold**) →
re-checks assertion-integrity → re-audits. Capped at **2 rounds**.
- **Anti-laundering (the whole game).** Assertions may only be
**ADDED/strengthened — never dropped, inverted in place, or
renamed-to-dodge** a missing assertion. If strengthening goes red
because the **feature isn't wired**, it takes the honest `test.fixme` +
evidence exit (no green-wash).
- **`assert_integrity.py --strengthen`**: permits *added* negative
matchers (assert-absence is legit strengthening) while **still blocking
any dropped assertion** (removed/skipped guards + all static checks
stay). Unit-proven in the commit.
- **Quality gate** now judges the **post-refine** state; blocks only if
the Refiner couldn't keep the spec green honestly.
- **No-PR report** now phrases a block as a true last resort ("the
Refiner attempted to strengthen … but could not resolve every finding
honestly").
### Reuse path (clobber off)
`verify_heal` now also runs on **reuse runs**, and the "already covered"
short-circuit is suppressed on reuse — so **re-running `/e2e` on a
previously-BLOCKED spec re-validates + refines the existing tests into a
PR** instead of stopping at "already covered". No new tests are added on
reuse; the Refiner only strengthens what's there. (Cost: a reuse run
rebuilds even when tests were already perfect — acceptable since re-runs
are explicit; can add a "skip if a green test PR already exists" guard
later.)
## Validation
- `assert_integrity.py` unit-checked: added negative → FAIL without
`--strengthen`, PASS with it; dropped assertion → still FAIL with
`--strengthen`.
- Workflow YAML validated; step order confirmed **Sentinel re-audit →
Refiner → Quality gate**.
- Live end-to-end best validated by re-running `/e2e` on **#12857 with
clobber off** — the Refiner should strengthen the 2 flagged tests and
open the PR.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
Shrinath RaoandClaude Opus 4.8
|
93d5e5b90d |
ci(test): e2e-council enrichment — summary table, auto-review, + CLA bot-identity fix (#12684)
Post-PR "enrichment" follow-ups for the E2E test bot, plus a CLA fix. (Was stacked on #12682; rebased onto `main` after that merged.) ## What this adds **1. Test-case summary table** When the test PR opens, the bot posts a **table of the test cases it added/changed** — in the test PR body *and* the "E2E Council" comment on the source PR: | Test case | Type | What it verifies | |---|---|---| | `should …` | new / append / extend | one-line intent | (Engineer emits `test-summary.json`; `pr_back` renders it. No table when there's nothing to add.) **2. Auto-address the AI review (OFF by default)** New final job `review_address`, gated by repo variable **`E2E_AUTO_REVIEW`** (default off). After the test PR opens it waits ~6 min for the repo's AI review, then a **conservative "Responder" agent**: - **validates** each finding (the bot is noisy — default is to *not* change code), - fixes **only clearly-valid** ones in `tests/ui-testing` (never `.github` / `.opencode` / `playwright.yml` / source), - pushes tests-only fixes (the repo's Playwright CI re-validates the PR), - posts a rationale on the test PR and minimizes the bot comment. One bounded pass. Scoped `GITHUB_TOKEN` only. **To enable:** create repo variable `E2E_AUTO_REVIEW=true`. **3. CLA fix — commit as `github-actions[bot]`** The pipeline previously committed generated tests as `openobserve-bot <bot@openobserve.ai>`, which isn't a GitHub account → CLA assistant couldn't match/sign it and blocked the test PR. Now it commits as the canonical **`github-actions[bot]`** identity (a real bot account GitHub resolves via its noreply email), across generate + verify_heal + review_address. > **Admin action:** allowlist `github-actions[bot]` in CLA assistant, then `recheck`. --- <details> <summary>Design notes</summary> - Responder is deliberately conservative: a wrong "fix" that breaks a passing test is worse than leaving a nitpick. It dismisses stale/incorrect/style findings. - Both auto-features are flag-gated OFF (same posture as `E2E_AUTO_ON_MERGE`) — auto-acting warrants opt-in. - No heavy re-build in `review_address`; the repo's existing Playwright CI re-validates the pushed fix. - CodeQL untrusted-checkout on the new `review_address` checkout (#416) dismissed — same allowlist false positive as prior PRs. </details> 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Shrinath RaoandClaude Opus 4.8
|
7655dfb67b |
ci: e2e-council pre-use hardening (PR-bound trigger, agent-discovery fix, live feedback, exact /e2e) (#12657)
## Summary Follow-up to #12652 that makes the **E2E Council** workflow correct and pleasant for real use, after the first live dry-run triages exposed gaps. v1 scope is unchanged: **OSS-only, manual-first, dry-run** (only Job 1 triage runs; Jobs 2–4 are gated scaffolds). ## Changes **Trigger is strictly PR-bound** - Removed the free-form `oss_branch` input (it let you generate tests detached from a PR, and dispatching from `main` diffed main-against-itself = nothing). - `pr_number` is now **required** for manual dispatch and accepts a **number, `#123`, or a full PR URL** (normalized to an integer). - A test is only ever generated for a PR. No PR ⇒ no run (empty/missing diff fails fast with a clear message). **`/e2e` is an exact match** - Triggers only when the comment body is exactly `/e2e`. `Generate /e2e`, `/e2e please`, etc. do **not** fire anything (the job's `if` is false → no reaction, no comment, no run). **Agent discovery fixed (the important one)** - The first live run logged `agent "e2e-ci-triage" not found. Falling back to default agent` — OpenCode does not auto-discover agents nested under `.opencode/agents/<subdir>/`. It only worked by luck (the default agent read the spec file). - Fixed by registering all six agents in `opencode.jsonc` under the `agent` key with `prompt: "{file:...}"`. **Confirmed working** on the next run (`> e2e-ci-triage · deepseek-v4-pro`, no fallback). **Immediate user feedback on a run** - Posts a 👀 reaction on the `/e2e` comment and an "⏳ triage in progress…" comment within seconds, then **edits that same comment in place** with the decision (no second comment). Runs with `always()` so a failed/empty run still updates it (never stuck on "in progress"). **Form/docs polish** - Decluttered the Run-workflow form (short input labels; full guidance lives in the workflow header). ## Verified live (dry-run triages on this PR) - OpenCode installs (v1.17.7); DeepSeek model `deepseek/deepseek-v4-pro` works. - Secure path confirmed: PR read as a **diff via API**, no untrusted PR-head checkout. - Triage decisions correct + conservative (CI-only change → skip). - Agent discovery fix confirmed (named agent loads as the system prompt). ## Not in scope (still scaffolds) Jobs 2–4 (generate → heal → PR-back). Build those once the triage gate is trusted over a dry-run period. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Shrinath RaoandClaude Opus 4.8
|
efd00ca128 |
ci: add e2e_ci_council_of_agents triage pipeline (dry-run, manual-first) (#12652)
## What this adds A CI-side **E2E test-generation pipeline** (Council of Agents) that runs in GitHub Actions via **OpenCode on DeepSeek-V4-Pro**, fronting the existing Playwright + TestDino CI. **v1 scope: OSS-only, manual-first, dry-run.** On this first cut the workflow does **only triage** — it classifies a change and posts the decision as a PR comment. It does **not** generate tests, edit `playwright.yml`, run anything, or open PRs. ## Files - `.opencode/agents/e2e_ci_council_of_agents/` — 6 CI-adjusted agents (triage, analyst, architect, engineer, sentinel, healer) + README. `mode: primary`, file-based handoff, non-interactive. Engineer also gains `playwright.yml` registration. Scribe excluded; ENT skipped at triage. - `opencode.jsonc` — DeepSeek provider. - `.github/workflows/e2e-council.yml` — Job 1 (triage + dry-run comment) is **live**; Jobs 2–4 (generate / heal / PR-back) are **gated scaffolds** for when we flip the switch. - `.gitignore` — track only the CI agent folder under `.opencode`. ## Triggers - `workflow_dispatch` (pick branch) and `/e2e` comment on a PR. Auto-on-merge is **off**. - Both read the workflow from the **default branch**, so they only activate after this merges. ## Before first run - [ ] Add repo secret **`DEEPSEEK_API_KEY`**. - [ ] Merge to main. - [ ] Trigger via Actions → "E2E Council (Test Generation)" → Run, or `/e2e` on a PR. ## Verify on first run (deferred by design) - OpenCode install command/version. - Whether `--agent e2e-ci-*` resolves agents nested under `.opencode/agents/<subdir>/`. - DeepSeek model id `deepseek/deepseek-v4-pro`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |