From ea039d09403b7cbda40ee59b74db214839f282a7 Mon Sep 17 00:00:00 2001 From: Neha P <144308844+neha00290@users.noreply.github.com> Date: Wed, 1 Jul 2026 10:54:27 +0530 Subject: [PATCH] ci: add test-assist-ci caller + fixture data (#12946) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary Thin OSS caller for the upcoming **Test-Assist** automated bug-verification pipeline — first piece of a 3-PR rollout. Mirrors the existing [`docgen-caller.yml`](.github/workflows/docgen-caller.yml) and [`e2e-council-caller.yml`](.github/workflows/e2e-council-caller.yml) patterns: the heavy engine will live in `o2-enterprise`; this caller only gates the trigger and dispatches via `ORG_ADMIN_TOKEN`. - **`.github/workflows/test-assist-caller.yml`** — caller workflow with two doors (`workflow_dispatch` + `/verify-bug` issue comments by org members). Defaults to `fixture_mode=true` / `write_mode=false` so the engine can't post to real issues until explicitly flipped. - **`tests/test-data/test-assist-ci/`** — fixture issue (#90001) + log seed data + `expected-verdicts.json` contract. Lets the engine soak in `FIXTURE_MODE=true` against synthetic data without touching real `Needs-Testing` issues. - **`tests/ui-testing/.gitignore`** — adds `_ci_tmp/` for the throwaway specs the engine generates per run. ### Differences from docgen / e2e-council callers | Dimension | docgen / e2e-council | test-assist | |---|---|---| | Operates on | PRs | **Issues** (`.issue.pull_request` filter dropped) | | Comment trigger | `/docgen`, `/e2e` | `/verify-bug` | | Inputs | pr_number, dry_run | issue_number, **fixture_mode**, **write_mode** | | Default safety | dry_run=true on comment | write_mode=false on every door | ### Rollout sequencing 1. **This PR** — OSS caller + fixtures + gitignore (no engine yet → caller will 404 on dispatch; intentional) 2. **Follow-up ENT PR** — `o2-enterprise/.opencode/agents/test_assist_ci/` (6 opencode sub-agents + README) 3. **Follow-up ENT PR** — `o2-enterprise/.github/workflows/test-assist.yml` (the engine itself) ## Test plan - [ ] CI: workflow YAML parses (verified locally via `python3 -c "import yaml; yaml.safe_load(open(...))"` — passes) - [ ] After ENT engine PR merges: trigger via `workflow_dispatch` with default inputs (`issue_number=90001`, `fixture_mode=true`, `write_mode=false`) → expect engine ack + fixture verdict in job summary, **no** GitHub writes - [ ] After soak: trigger `workflow_dispatch` with `fixture_mode=false` on a hand-picked `Needs-Testing` issue, `write_mode=false` → verdict in summary only - [ ] After verdict accuracy is calibrated: org member comments `/verify-bug` on a real issue, `write_mode=true` flipped via repo vars → comment + labels post 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) --- .github/workflows/test-assist-caller.yml | 140 ++++++++++++++++++ tests/test-data/test-assist-ci/README.md | 47 ++++++ .../test-assist-ci/expected-verdicts.json | 7 + .../fixture-issues/bug-001-fixed.json | 10 ++ .../test-assist-ci/seed/logs_seed.json | 7 + tests/test-data/test-assist-ci/seed/seed.sh | 31 ++++ tests/ui-testing/.gitignore | 5 +- 7 files changed, 246 insertions(+), 1 deletion(-) create mode 100644 .github/workflows/test-assist-caller.yml create mode 100644 tests/test-data/test-assist-ci/README.md create mode 100644 tests/test-data/test-assist-ci/expected-verdicts.json create mode 100644 tests/test-data/test-assist-ci/fixture-issues/bug-001-fixed.json create mode 100644 tests/test-data/test-assist-ci/seed/logs_seed.json create mode 100755 tests/test-data/test-assist-ci/seed/seed.sh diff --git a/.github/workflows/test-assist-caller.yml b/.github/workflows/test-assist-caller.yml new file mode 100644 index 0000000000..0659fb31fb --- /dev/null +++ b/.github/workflows/test-assist-caller.yml @@ -0,0 +1,140 @@ +name: Test-Assist Caller (openobserve) + +# Thin caller: on an org member's `/verify-bug` comment on an OSS ISSUE (or manual dispatch), +# DISPATCH the Test-Assist engine that lives in the PRIVATE o2-enterprise repo, authenticated +# with the org-admin PAT. Mirrors docgen-caller.yml / e2e-council-caller.yml. +# +# Differs from docgen/e2e callers: this one operates on ISSUES, not PRs. The `.issue.pull_request` +# filter is intentionally absent — `/verify-bug` is a comment on a bug issue (a PR is not a bug). +# +# WHY DISPATCH (not `workflow_call`/`uses:`): cross-repo reusable-workflow calls require the +# private repo to grant an Actions access policy. A PAT with `Actions: R/W` on o2-enterprise can +# trigger its `workflow_dispatch` with no such org/repo setting. The engine then runs IN +# o2-enterprise CI (boots OpenObserve, runs the bug-verification agent) and posts results back +# to the OSS issue using the same PAT. +# +# This caller only: (1) gates the command to org members, (2) resolves issue/fixture-mode/actor, +# (3) dispatches the engine via the PAT, (4) posts an immediate ack on the OSS issue. +# +# REQUIRED — secrets.ORG_ADMIN_TOKEN (fine-grained PAT) permissions: +# o2-enterprise: Actions: R/W (dispatch) + Issues/Pull requests: R/W (engine comments) +# openobserve: Issues/Pull requests: R/W (engine comments back on the source issue) +# This PAT also posts THIS caller's own "started" ack as the human (not github-actions[bot]). +# KILL SWITCH: the engine honors the TEST_ASSIST_DISABLED variable. + +on: + issue_comment: + types: [created] + workflow_dispatch: + inputs: + issue_number: + description: "OSS issue number to verify (default 90001 = fixture #1)" + required: true + default: "90001" + fixture_mode: + description: "Read fixture issues from tests/test-data/ instead of GitHub" + type: boolean + required: true + default: true + write_mode: + description: "Allow the engine to post labels/comments to real issues (otherwise dry-run)" + type: boolean + required: true + default: false + +permissions: + contents: read + issues: write # post the ack comment on the OSS issue (this run is in OSS context) + +# No caller-level concurrency: issue_comment fires the caller for EVERY comment on an issue (the +# /verify-bug filter is only in the job `if:`), and a run enters the concurrency group BEFORE its +# `if:` is evaluated — cancel-in-progress would let an unrelated comment cancel an in-flight +# /verify-bug DISPATCH. The ENGINE's concurrency (keyed on source_repo+issue) owns dedup. + +jobs: + dispatch-test-assist: + # Two doors: + # - workflow_dispatch (manual): GitHub already restricts dispatch to users with repo write. + # - issue_comment: exact `/verify-bug` on an ISSUE (NOT a PR) by an org member/collaborator. + # The `.issue.pull_request == null` filter excludes /verify-bug typed on a PR — test-assist + # verifies bug ISSUES, and a PR number would confuse the engine downstream. The + # org-member gate is the trust boundary for the PAT — a stranger's comment fails this + # `if:`, so the dispatch step (and therefore the PAT) never runs. + if: > + github.event_name == 'workflow_dispatch' || + (github.event_name == 'issue_comment' && + github.event.issue.pull_request == null && + github.event.comment.body == '/verify-bug' && + contains(fromJSON('["OWNER","MEMBER","COLLABORATOR"]'), github.event.comment.author_association)) + runs-on: ubuntu-latest + timeout-minutes: 10 # dispatch is a quick API call; bound it as defense-in-depth + steps: + - name: Resolve context (untrusted event data via env, never spliced into commands) + id: ctx + env: + IN_ISSUE: ${{ inputs.issue_number || github.event.issue.number }} + IS_DISPATCH: ${{ github.event_name == 'workflow_dispatch' }} + DISPATCH_FIXTURE: ${{ inputs.fixture_mode }} + DISPATCH_WRITE: ${{ inputs.write_mode }} + COMMENT_LOGIN: ${{ github.event.comment.user.login }} + DISPATCH_ACTOR: ${{ github.actor }} + run: | + ISSUE="$IN_ISSUE" + [[ "$ISSUE" =~ ^[0-9]+$ ]] || { echo "::error::invalid issue number: $ISSUE"; exit 1; } + if [ "$IS_DISPATCH" = "true" ]; then + FIXTURE="$DISPATCH_FIXTURE"; WRITE="$DISPATCH_WRITE"; ACTOR="$DISPATCH_ACTOR" + else + # Comment door defaults: fixture OFF (use the real issue), write OFF (dry-run). + # WRITE flip happens via manual dispatch until we trust verdict accuracy. + FIXTURE="false"; WRITE="false"; ACTOR="$COMMENT_LOGIN" + fi + ACTOR=$(printf '%s' "$ACTOR" | tr -cd 'A-Za-z0-9-') # GitHub login charset; defensive + { echo "issue=$ISSUE"; echo "fixture=$FIXTURE"; echo "write=$WRITE"; echo "actor=$ACTOR"; } >> "$GITHUB_OUTPUT" + + - name: Dispatch the Test-Assist engine (o2-enterprise) via the org-admin PAT + env: + GH_TOKEN: ${{ secrets.ORG_ADMIN_TOKEN }} + ISSUE: ${{ steps.ctx.outputs.issue }} + FIXTURE: ${{ steps.ctx.outputs.fixture }} + WRITE: ${{ steps.ctx.outputs.write }} + ACTOR: ${{ steps.ctx.outputs.actor }} + run: | + gh workflow run test-assist.yml --repo openobserve/o2-enterprise \ + -f source_repo=openobserve \ + -f issue_number="$ISSUE" \ + -f fixture_mode="$FIXTURE" \ + -f write_mode="$WRITE" \ + -f actor="$ACTOR" + echo "Dispatched Test-Assist engine for openobserve issue #$ISSUE (fixture=$FIXTURE, write=$WRITE, by @$ACTOR)." + + - name: Acknowledge on the source issue + # success() => only ack if the dispatch actually succeeded (don't mislead the commenter if + # the dispatch 403'd on a missing PAT permission, etc.). Only on a human /verify-bug comment. + if: github.event_name == 'issue_comment' && success() + env: + GH_TOKEN: ${{ secrets.ORG_ADMIN_TOKEN }} # PAT only — post as the human, NEVER github-actions[bot] + ISSUE: ${{ steps.ctx.outputs.issue }} + run: | + gh issue comment "$ISSUE" --repo openobserve/openobserve --body \ + "### 🤖 Test-Assist — started + + Running bug verification in o2-enterprise CI now — I'll post the verdict back here when ready. ([engine runs](https://github.com/openobserve/o2-enterprise/actions/workflows/test-assist.yml))" + + - name: Notify dispatch failure on the source issue + # If the dispatch itself failed (expired/insufficient PAT, missing engine workflow, network), + # the /verify-bug commenter would otherwise get NO feedback. Fall back to the run token so a + # broken PAT can't silence the error — same pattern as e2e-council-caller's failure notice. + if: github.event_name == 'issue_comment' && failure() + env: + PAT: ${{ secrets.ORG_ADMIN_TOKEN }} + BOT: ${{ github.token }} + ISSUE: ${{ steps.ctx.outputs.issue }} + RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + run: | + [ -n "$ISSUE" ] || exit 0 + BODY="### ❌ Test-Assist — could not start + + Dispatching the Test-Assist engine failed (likely an expired/insufficient \`ORG_ADMIN_TOKEN\`, a missing engine workflow on o2-enterprise main, or a transient error). See the [caller run logs]($RUN_URL) and retry \`/verify-bug\` once resolved." + GH_TOKEN="$PAT" gh issue comment "$ISSUE" --repo openobserve/openobserve --body "$BODY" \ + || GH_TOKEN="$BOT" gh issue comment "$ISSUE" --repo openobserve/openobserve --body "$BODY" \ + || true diff --git a/tests/test-data/test-assist-ci/README.md b/tests/test-data/test-assist-ci/README.md new file mode 100644 index 0000000000..9b511319e2 --- /dev/null +++ b/tests/test-data/test-assist-ci/README.md @@ -0,0 +1,47 @@ +# test-assist-ci fixtures + +Throwaway scaffolding for bringing up the test-assist CI agent. Will be replaced once real +`Needs-Testing` issues flow through the pipeline. + +## Layout + +| Path | Purpose | +|------|---------| +| `seed/logs_seed.json` | Tiny log batch ingested before the agent runs, so Logs/Streams pages are non-empty. | +| `seed/seed.sh` | Idempotent script the engine calls in fixture mode. POSTs `logs_seed.json` to `ZO_BASE_URL`. Uses `set -euo pipefail` + header-based auth (no creds in `ps`). | +| `fixture-issues/` | Synthetic GitHub issue payloads — same JSON shape as `gh issue view`. Lets Scout run without real issues. | +| `expected-verdicts.json` | Verdict each fixture issue should produce. Pipeline regression contract. | + +## How the agent consumes this + +1. Phase 0.5 (new) → `seed/seed.sh` runs against `$ZO_BASE_URL` (only when `FIXTURE_MODE=true`). +2. Phase 1 (Scout) → if `FIXTURE_MODE=true`, reads `fixture-issues/*.json` instead of calling GitHub. +3. End of run → engine diffs final verdicts against `expected-verdicts.json` and fails the job on mismatch. + +## Required env (consumed by `seed.sh`) + +| Variable | Default | Notes | +|----------|---------|-------| +| `ZO_BASE_URL` | — (required) | The OpenObserve instance the engine booted. | +| `ZO_ROOT_USER_EMAIL` | — (required) | Root user for basic auth. | +| `ZO_ROOT_USER_PASSWORD` | — (required) | Root user password. | +| `ORGNAME` | `default` | OpenObserve org to ingest into. | +| `CI_SEED_STREAM` | `ci_seed_stream` | Stream name the seed creates; the fixture spec references this exact name. Override only when running multiple seeders against one instance. | + +## `expected-verdicts.json` schema + +A JSON object keyed by issue number. Each entry: + +```jsonc +{ + "90001": { + "fixture": "bug-001-fixed.json", // filename under fixture-issues/ + "expected_status": "FIXED", // one of FIXED | STILL_PRESENT | INCONCLUSIVE + "expected_confidence": "HIGH" // one of HIGH | LOW + } +} +``` + +The engine's regression-mode reads this file after Reporter finishes, compares each fixture +issue's actual verdict against `expected_status` + `expected_confidence`, and fails the job +on mismatch. Real (non-fixture) issues never appear here and are not regression-checked. diff --git a/tests/test-data/test-assist-ci/expected-verdicts.json b/tests/test-data/test-assist-ci/expected-verdicts.json new file mode 100644 index 0000000000..619cad4feb --- /dev/null +++ b/tests/test-data/test-assist-ci/expected-verdicts.json @@ -0,0 +1,7 @@ +{ + "90001": { + "fixture": "bug-001-fixed.json", + "expected_status": "FIXED", + "expected_confidence": "HIGH" + } +} diff --git a/tests/test-data/test-assist-ci/fixture-issues/bug-001-fixed.json b/tests/test-data/test-assist-ci/fixture-issues/bug-001-fixed.json new file mode 100644 index 0000000000..206d04581a --- /dev/null +++ b/tests/test-data/test-assist-ci/fixture-issues/bug-001-fixed.json @@ -0,0 +1,10 @@ +{ + "number": 90001, + "title": "[fixture] Logs page should show ingested ci_seed_stream entries", + "body": "## Reproduction steps\n\n1. Log in to OpenObserve\n2. Navigate to Logs\n3. Select the `ci_seed_stream` stream from the stream picker\n4. Run a query over the last 15 minutes\n\n## Expected\n\nThe results table shows at least one log row from `ci_seed_stream`.\n\n## Actual\n\n(Synthetic — used to validate the test-assist CI plumbing. Expected verdict: FIXED.)", + "labels": [ + {"name": "Needs-Testing"}, + {"name": "fixture"} + ], + "comments": [] +} diff --git a/tests/test-data/test-assist-ci/seed/logs_seed.json b/tests/test-data/test-assist-ci/seed/logs_seed.json new file mode 100644 index 0000000000..dfa93a69dc --- /dev/null +++ b/tests/test-data/test-assist-ci/seed/logs_seed.json @@ -0,0 +1,7 @@ +[ + {"level": "info", "service": "ci-seed", "message": "user login ok", "code": 200}, + {"level": "warn", "service": "ci-seed", "message": "slow query detected", "code": 200}, + {"level": "error", "service": "ci-seed", "message": "upstream connect failed", "code": 502}, + {"level": "info", "service": "ci-seed", "message": "cache miss", "code": 200}, + {"level": "error", "service": "ci-seed", "message": "auth token expired", "code": 401} +] diff --git a/tests/test-data/test-assist-ci/seed/seed.sh b/tests/test-data/test-assist-ci/seed/seed.sh new file mode 100755 index 0000000000..d536c81ec5 --- /dev/null +++ b/tests/test-data/test-assist-ci/seed/seed.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +# Idempotent seeding for test-assist CI. Throwaway — replace once real Needs-Testing flow is up. +# +# pipefail is intentional — without it, `curl … | tee …` would mask a non-zero curl exit (set -e +# does NOT propagate through pipes). Auth is passed via Authorization header (NOT `curl -u`) so +# the password never appears in `ps` output on shared runners. +set -euo pipefail + +: "${ZO_BASE_URL:?ZO_BASE_URL not set}" +: "${ZO_ROOT_USER_EMAIL:?ZO_ROOT_USER_EMAIL not set}" +: "${ZO_ROOT_USER_PASSWORD:?ZO_ROOT_USER_PASSWORD not set}" +ORG="${ORGNAME:-default}" +STREAM="${CI_SEED_STREAM:-ci_seed_stream}" + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +echo "Seeding $STREAM on $ZO_BASE_URL (org=$ORG)" + +# Build basic-auth header in-process; the resulting variable is passed via -H, not as a CLI arg +# visible in process listings. +AUTH_B64=$(printf '%s' "${ZO_ROOT_USER_EMAIL}:${ZO_ROOT_USER_PASSWORD}" | base64) + +curl -sS --fail-with-body \ + -H "Authorization: Basic ${AUTH_B64}" \ + -H 'Content-Type: application/json' \ + --data-binary "@${HERE}/logs_seed.json" \ + "${ZO_BASE_URL%/}/api/${ORG}/${STREAM}/_json" \ + | tee /tmp/test-assist-ci-seed.log + +echo +echo "Seed complete." diff --git a/tests/ui-testing/.gitignore b/tests/ui-testing/.gitignore index bc094ea9fa..d5473e43d7 100644 --- a/tests/ui-testing/.gitignore +++ b/tests/ui-testing/.gitignore @@ -17,4 +17,7 @@ coverage/ # Test output logs test-output.log -*.log \ No newline at end of file +*.log + +# test-assist-ci throwaway specs (generated per run, deleted after) +playwright-tests/_ci_tmp/