mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-03 16:10:21 +00:00
Shrinath RaoandClaude Opus 4.8
0ee8a667c2
## Why
The E2E Council blocked **#12857** (no test PR opened) because the
post-heal **Sentinel re-audit found 2 critical quality issues** — even
though **9/10 tests passed**. The tests were **not worthless**; 2 just
*under-claimed* (the body asserted something weaker than the test name
promised):
1. `"should not persist FTS default across page reload"` — re-searches
after reload and asserts the column visible, which passes whether it
*persisted* or *re-resolved*. Never verifies non-persistence.
2. `"should clear FTS marker and re-resolve for new stream on stream
change"` — only checks "table rendered", never that FTS actually
re-resolved.
Blocking was correct (green ≠ meaningful), but a **dead-end block throws
away the other 90%**. The pipeline should **fix the problem and make it
right**, not give up. A no-PR outcome should be a *true last resort*.
## What — The Refiner (Phase 5.5)
A remediation loop **inside `verify_heal`**, after the Sentinel re-audit
and before the quality gate (**reuses the already-booted binary** — no
rebuild):
- **Only fires when needed.** If the re-audit is clean (PASS / 0
critical), the step reads the verdict, sets `ran=false`, and exits —
**no DeepSeek call, no test re-run**. Cheap on the happy path.
- **Strengthen-first.** Feeds the exact Sentinel findings to the
Refiner, which **adds/tightens assertions** so each test verifies what
its name promises → re-runs on the live binary (**green must hold**) →
re-checks assertion-integrity → re-audits. Capped at **2 rounds**.
- **Anti-laundering (the whole game).** Assertions may only be
**ADDED/strengthened — never dropped, inverted in place, or
renamed-to-dodge** a missing assertion. If strengthening goes red
because the **feature isn't wired**, it takes the honest `test.fixme` +
evidence exit (no green-wash).
- **`assert_integrity.py --strengthen`**: permits *added* negative
matchers (assert-absence is legit strengthening) while **still blocking
any dropped assertion** (removed/skipped guards + all static checks
stay). Unit-proven in the commit.
- **Quality gate** now judges the **post-refine** state; blocks only if
the Refiner couldn't keep the spec green honestly.
- **No-PR report** now phrases a block as a true last resort ("the
Refiner attempted to strengthen … but could not resolve every finding
honestly").
### Reuse path (clobber off)
`verify_heal` now also runs on **reuse runs**, and the "already covered"
short-circuit is suppressed on reuse — so **re-running `/e2e` on a
previously-BLOCKED spec re-validates + refines the existing tests into a
PR** instead of stopping at "already covered". No new tests are added on
reuse; the Refiner only strengthens what's there. (Cost: a reuse run
rebuilds even when tests were already perfect — acceptable since re-runs
are explicit; can add a "skip if a green test PR already exists" guard
later.)
## Validation
- `assert_integrity.py` unit-checked: added negative → FAIL without
`--strengthen`, PASS with it; dropped assertion → still FAIL with
`--strengthen`.
- Workflow YAML validated; step order confirmed **Sentinel re-audit →
Refiner → Quality gate**.
- Live end-to-end best validated by re-running `/e2e` on **#12857 with
clobber off** — the Refiner should strengthen the 2 flagged tests and
open the PR.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
66 lines
2.3 KiB
JSON
66 lines
2.3 KiB
JSON
{
|
|
// OpenCode configuration for the e2e_ci_council_of_agents pipeline.
|
|
// Used by .github/workflows/e2e-council.yml to run the CI test-generation agents on DeepSeek.
|
|
// Docs: https://opencode.ai/docs/ — verify provider shape against current docs at build time.
|
|
"$schema": "https://opencode.ai/config.json",
|
|
|
|
"provider": {
|
|
"deepseek": {
|
|
"npm": "@ai-sdk/openai-compatible",
|
|
"name": "DeepSeek",
|
|
"options": {
|
|
"baseURL": "https://api.deepseek.com",
|
|
"apiKey": "{env:DEEPSEEK_API_KEY_E2E}"
|
|
},
|
|
"models": {
|
|
"deepseek-v4-pro": {
|
|
"name": "DeepSeek-V4-Pro",
|
|
"limit": { "context": 1048576, "output": 262144 }
|
|
}
|
|
}
|
|
}
|
|
},
|
|
|
|
// Default model for runs that don't pass --model explicitly.
|
|
// The workflow passes --model deepseek/deepseek-v4-pro per agent invocation.
|
|
"model": "deepseek/deepseek-v4-pro",
|
|
|
|
// Register the CI agents explicitly so `--agent e2e-ci-*` resolves deterministically.
|
|
// OpenCode does NOT auto-discover agents nested in .opencode/agents/<subdir>/ (the first CI
|
|
// run fell back to the default agent), so we point each name at its prompt file here.
|
|
"agent": {
|
|
"e2e-ci-triage": {
|
|
"mode": "primary",
|
|
"prompt": "{file:./.opencode/agents/e2e_ci_council_of_agents/e2e-ci-triage.md}"
|
|
},
|
|
"e2e-ci-analyst": {
|
|
"mode": "primary",
|
|
"prompt": "{file:./.opencode/agents/e2e_ci_council_of_agents/e2e-ci-analyst.md}"
|
|
},
|
|
"e2e-ci-architect": {
|
|
"mode": "primary",
|
|
"prompt": "{file:./.opencode/agents/e2e_ci_council_of_agents/e2e-ci-architect.md}"
|
|
},
|
|
"e2e-ci-engineer": {
|
|
"mode": "primary",
|
|
"prompt": "{file:./.opencode/agents/e2e_ci_council_of_agents/e2e-ci-engineer.md}"
|
|
},
|
|
"e2e-ci-sentinel": {
|
|
"mode": "primary",
|
|
"prompt": "{file:./.opencode/agents/e2e_ci_council_of_agents/e2e-ci-sentinel.md}"
|
|
},
|
|
"e2e-ci-healer": {
|
|
"mode": "primary",
|
|
"prompt": "{file:./.opencode/agents/e2e_ci_council_of_agents/e2e-ci-healer.md}"
|
|
},
|
|
"e2e-ci-refiner": {
|
|
"mode": "primary",
|
|
"prompt": "{file:./.opencode/agents/e2e_ci_council_of_agents/e2e-ci-refiner.md}"
|
|
},
|
|
"e2e-ci-responder": {
|
|
"mode": "primary",
|
|
"prompt": "{file:./.opencode/agents/e2e_ci_council_of_agents/e2e-ci-responder.md}"
|
|
}
|
|
}
|
|
}
|