ci: add test-assist-ci caller + fixture data (#12946)

## Summary Thin OSS caller for the upcoming **Test-Assist** automated bug-verification pipeline — first piece of a 3-PR rollout. Mirrors the existing [`docgen-caller.yml`](.github/workflows/docgen-caller.yml) and [`e2e-council-caller.yml`](.github/workflows/e2e-council-caller.yml) patterns: the heavy engine will live in `o2-enterprise`; this caller only gates the trigger and dispatches via `ORG_ADMIN_TOKEN`. - **`.github/workflows/test-assist-caller.yml`** — caller workflow with two doors (`workflow_dispatch` + `/verify-bug` issue comments by org members). Defaults to `fixture_mode=true` / `write_mode=false` so the engine can't post to real issues until explicitly flipped. - **`tests/test-data/test-assist-ci/`** — fixture issue (#90001) + log seed data + `expected-verdicts.json` contract. Lets the engine soak in `FIXTURE_MODE=true` against synthetic data without touching real `Needs-Testing` issues. - **`tests/ui-testing/.gitignore`** — adds `_ci_tmp/` for the throwaway specs the engine generates per run. ### Differences from docgen / e2e-council callers | Dimension | docgen / e2e-council | test-assist | |---|---|---| | Operates on | PRs | **Issues** (`.issue.pull_request` filter dropped) | | Comment trigger | `/docgen`, `/e2e` | `/verify-bug` | | Inputs | pr_number, dry_run | issue_number, **fixture_mode**, **write_mode** | | Default safety | dry_run=true on comment | write_mode=false on every door | ### Rollout sequencing 1. **This PR** — OSS caller + fixtures + gitignore (no engine yet → caller will 404 on dispatch; intentional) 2. **Follow-up ENT PR** — `o2-enterprise/.opencode/agents/test_assist_ci/` (6 opencode sub-agents + README) 3. **Follow-up ENT PR** — `o2-enterprise/.github/workflows/test-assist.yml` (the engine itself) ## Test plan - [ ] CI: workflow YAML parses (verified locally via `python3 -c "import yaml; yaml.safe_load(open(...))"` — passes) - [ ] After ENT engine PR merges: trigger via `workflow_dispatch` with default inputs (`issue_number=90001`, `fixture_mode=true`, `write_mode=false`) → expect engine ack + fixture verdict in job summary, **no** GitHub writes - [ ] After soak: trigger `workflow_dispatch` with `fixture_mode=false` on a hand-picked `Needs-Testing` issue, `write_mode=false` → verdict in summary only - [ ] After verdict accuracy is calibrated: org member comments `/verify-bug` on a real issue, `write_mode=true` flipped via repo vars → comment + labels post 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-11-04 00:08:55 +00:00 · 2026-07-01 05:24:27 +00:00
parent ca252815cb
commit ea039d0940
7 changed files with 246 additions and 1 deletions
@@ -0,0 +1,140 @@
+name: Test-Assist Caller (openobserve)
+
+# Thin caller: on an org member's `/verify-bug` comment on an OSS ISSUE (or manual dispatch),
+# DISPATCH the Test-Assist engine that lives in the PRIVATE o2-enterprise repo, authenticated
+# with the org-admin PAT. Mirrors docgen-caller.yml / e2e-council-caller.yml.
+#
+# Differs from docgen/e2e callers: this one operates on ISSUES, not PRs. The `.issue.pull_request`
+# filter is intentionally absent — `/verify-bug` is a comment on a bug issue (a PR is not a bug).
+#
+# WHY DISPATCH (not `workflow_call`/`uses:`): cross-repo reusable-workflow calls require the
+# private repo to grant an Actions access policy. A PAT with `Actions: R/W` on o2-enterprise can
+# trigger its `workflow_dispatch` with no such org/repo setting. The engine then runs IN
+# o2-enterprise CI (boots OpenObserve, runs the bug-verification agent) and posts results back
+# to the OSS issue using the same PAT.
+#
+# This caller only: (1) gates the command to org members, (2) resolves issue/fixture-mode/actor,
+# (3) dispatches the engine via the PAT, (4) posts an immediate ack on the OSS issue.
+#
+# REQUIRED — secrets.ORG_ADMIN_TOKEN (fine-grained PAT) permissions:
+#   o2-enterprise:  Actions: R/W (dispatch) + Issues/Pull requests: R/W (engine comments)
+#   openobserve:    Issues/Pull requests: R/W (engine comments back on the source issue)
+#   This PAT also posts THIS caller's own "started" ack as the human (not github-actions[bot]).
+# KILL SWITCH: the engine honors the TEST_ASSIST_DISABLED variable.
+
+on:
+  issue_comment:
+    types: [created]
+  workflow_dispatch:
+    inputs:
+      issue_number:
+        description: "OSS issue number to verify (default 90001 = fixture #1)"
+        required: true
+        default: "90001"
+      fixture_mode:
+        description: "Read fixture issues from tests/test-data/ instead of GitHub"
+        type: boolean
+        required: true
+        default: true
+      write_mode:
+        description: "Allow the engine to post labels/comments to real issues (otherwise dry-run)"
+        type: boolean
+        required: true
+        default: false
+
+permissions:
+  contents: read
+  issues: write   # post the ack comment on the OSS issue (this run is in OSS context)
+
+# No caller-level concurrency: issue_comment fires the caller for EVERY comment on an issue (the
+# /verify-bug filter is only in the job `if:`), and a run enters the concurrency group BEFORE its
+# `if:` is evaluated — cancel-in-progress would let an unrelated comment cancel an in-flight
+# /verify-bug DISPATCH. The ENGINE's concurrency (keyed on source_repo+issue) owns dedup.
+
+jobs:
+  dispatch-test-assist:
+    # Two doors:
+    #   - workflow_dispatch (manual): GitHub already restricts dispatch to users with repo write.
+    #   - issue_comment: exact `/verify-bug` on an ISSUE (NOT a PR) by an org member/collaborator.
+    #     The `.issue.pull_request == null` filter excludes /verify-bug typed on a PR — test-assist
+    #     verifies bug ISSUES, and a PR number would confuse the engine downstream. The
+    #     org-member gate is the trust boundary for the PAT — a stranger's comment fails this
+    #     `if:`, so the dispatch step (and therefore the PAT) never runs.
+    if: >
+      github.event_name == 'workflow_dispatch' ||
+      (github.event_name == 'issue_comment' &&
+       github.event.issue.pull_request == null &&
+       github.event.comment.body == '/verify-bug' &&
+       contains(fromJSON('["OWNER","MEMBER","COLLABORATOR"]'), github.event.comment.author_association))
+    runs-on: ubuntu-latest
+    timeout-minutes: 10   # dispatch is a quick API call; bound it as defense-in-depth
+    steps:
+      - name: Resolve context (untrusted event data via env, never spliced into commands)
+        id: ctx
+        env:
+          IN_ISSUE: ${{ inputs.issue_number || github.event.issue.number }}
+          IS_DISPATCH: ${{ github.event_name == 'workflow_dispatch' }}
+          DISPATCH_FIXTURE: ${{ inputs.fixture_mode }}
+          DISPATCH_WRITE: ${{ inputs.write_mode }}
+          COMMENT_LOGIN: ${{ github.event.comment.user.login }}
+          DISPATCH_ACTOR: ${{ github.actor }}
+        run: |
+          ISSUE="$IN_ISSUE"
+          [[ "$ISSUE" =~ ^[0-9]+$ ]] || { echo "::error::invalid issue number: $ISSUE"; exit 1; }
+          if [ "$IS_DISPATCH" = "true" ]; then
+            FIXTURE="$DISPATCH_FIXTURE"; WRITE="$DISPATCH_WRITE"; ACTOR="$DISPATCH_ACTOR"
+          else
+            # Comment door defaults: fixture OFF (use the real issue), write OFF (dry-run).
+            # WRITE flip happens via manual dispatch until we trust verdict accuracy.
+            FIXTURE="false"; WRITE="false"; ACTOR="$COMMENT_LOGIN"
+          fi
+          ACTOR=$(printf '%s' "$ACTOR" | tr -cd 'A-Za-z0-9-')   # GitHub login charset; defensive
+          { echo "issue=$ISSUE"; echo "fixture=$FIXTURE"; echo "write=$WRITE"; echo "actor=$ACTOR"; } >> "$GITHUB_OUTPUT"
+
+      - name: Dispatch the Test-Assist engine (o2-enterprise) via the org-admin PAT
+        env:
+          GH_TOKEN: ${{ secrets.ORG_ADMIN_TOKEN }}
+          ISSUE: ${{ steps.ctx.outputs.issue }}
+          FIXTURE: ${{ steps.ctx.outputs.fixture }}
+          WRITE: ${{ steps.ctx.outputs.write }}
+          ACTOR: ${{ steps.ctx.outputs.actor }}
+        run: |
+          gh workflow run test-assist.yml --repo openobserve/o2-enterprise \
+            -f source_repo=openobserve \
+            -f issue_number="$ISSUE" \
+            -f fixture_mode="$FIXTURE" \
+            -f write_mode="$WRITE" \
+            -f actor="$ACTOR"
+          echo "Dispatched Test-Assist engine for openobserve issue #$ISSUE (fixture=$FIXTURE, write=$WRITE, by @$ACTOR)."
+
+      - name: Acknowledge on the source issue
+        # success() => only ack if the dispatch actually succeeded (don't mislead the commenter if
+        # the dispatch 403'd on a missing PAT permission, etc.). Only on a human /verify-bug comment.
+        if: github.event_name == 'issue_comment' && success()
+        env:
+          GH_TOKEN: ${{ secrets.ORG_ADMIN_TOKEN }}   # PAT only — post as the human, NEVER github-actions[bot]
+          ISSUE: ${{ steps.ctx.outputs.issue }}
+        run: |
+          gh issue comment "$ISSUE" --repo openobserve/openobserve --body \
+          "### 🤖 Test-Assist — started
+
+          Running bug verification in o2-enterprise CI now — I'll post the verdict back here when ready. ([engine runs](https://github.com/openobserve/o2-enterprise/actions/workflows/test-assist.yml))"
+
+      - name: Notify dispatch failure on the source issue
+        # If the dispatch itself failed (expired/insufficient PAT, missing engine workflow, network),
+        # the /verify-bug commenter would otherwise get NO feedback. Fall back to the run token so a
+        # broken PAT can't silence the error — same pattern as e2e-council-caller's failure notice.
+        if: github.event_name == 'issue_comment' && failure()
+        env:
+          PAT: ${{ secrets.ORG_ADMIN_TOKEN }}
+          BOT: ${{ github.token }}
+          ISSUE: ${{ steps.ctx.outputs.issue }}
+          RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+        run: |
+          [ -n "$ISSUE" ] || exit 0
+          BODY="### ❌ Test-Assist — could not start
+
+          Dispatching the Test-Assist engine failed (likely an expired/insufficient \`ORG_ADMIN_TOKEN\`, a missing engine workflow on o2-enterprise main, or a transient error). See the [caller run logs]($RUN_URL) and retry \`/verify-bug\` once resolved."
+          GH_TOKEN="$PAT" gh issue comment "$ISSUE" --repo openobserve/openobserve --body "$BODY" \
+            || GH_TOKEN="$BOT" gh issue comment "$ISSUE" --repo openobserve/openobserve --body "$BODY" \
+            || true
@@ -0,0 +1,47 @@
+# test-assist-ci fixtures
+
+Throwaway scaffolding for bringing up the test-assist CI agent. Will be replaced once real
+`Needs-Testing` issues flow through the pipeline.
+
+## Layout
+
+| Path | Purpose |
+|------|---------|
+| `seed/logs_seed.json` | Tiny log batch ingested before the agent runs, so Logs/Streams pages are non-empty. |
+| `seed/seed.sh` | Idempotent script the engine calls in fixture mode. POSTs `logs_seed.json` to `ZO_BASE_URL`. Uses `set -euo pipefail` + header-based auth (no creds in `ps`). |
+| `fixture-issues/` | Synthetic GitHub issue payloads — same JSON shape as `gh issue view`. Lets Scout run without real issues. |
+| `expected-verdicts.json` | Verdict each fixture issue should produce. Pipeline regression contract. |
+
+## How the agent consumes this
+
+1. Phase 0.5 (new) → `seed/seed.sh` runs against `$ZO_BASE_URL` (only when `FIXTURE_MODE=true`).
+2. Phase 1 (Scout) → if `FIXTURE_MODE=true`, reads `fixture-issues/*.json` instead of calling GitHub.
+3. End of run → engine diffs final verdicts against `expected-verdicts.json` and fails the job on mismatch.
+
+## Required env (consumed by `seed.sh`)
+
+| Variable | Default | Notes |
+|----------|---------|-------|
+| `ZO_BASE_URL` | — (required) | The OpenObserve instance the engine booted. |
+| `ZO_ROOT_USER_EMAIL` | — (required) | Root user for basic auth. |
+| `ZO_ROOT_USER_PASSWORD` | — (required) | Root user password. |
+| `ORGNAME` | `default` | OpenObserve org to ingest into. |
+| `CI_SEED_STREAM` | `ci_seed_stream` | Stream name the seed creates; the fixture spec references this exact name. Override only when running multiple seeders against one instance. |
+
+## `expected-verdicts.json` schema
+
+A JSON object keyed by issue number. Each entry:
+
+```jsonc
+{
+  "90001": {
+    "fixture": "bug-001-fixed.json",        // filename under fixture-issues/
+    "expected_status": "FIXED",             // one of FIXED | STILL_PRESENT | INCONCLUSIVE
+    "expected_confidence": "HIGH"           // one of HIGH | LOW
+  }
+}
+```
+
+The engine's regression-mode reads this file after Reporter finishes, compares each fixture
+issue's actual verdict against `expected_status` + `expected_confidence`, and fails the job
+on mismatch. Real (non-fixture) issues never appear here and are not regression-checked.
@@ -0,0 +1,7 @@
+{
+  "90001": {
+    "fixture": "bug-001-fixed.json",
+    "expected_status": "FIXED",
+    "expected_confidence": "HIGH"
+  }
+}
@@ -0,0 +1,10 @@
+{
+  "number": 90001,
+  "title": "[fixture] Logs page should show ingested ci_seed_stream entries",
+  "body": "## Reproduction steps\n\n1. Log in to OpenObserve\n2. Navigate to Logs\n3. Select the `ci_seed_stream` stream from the stream picker\n4. Run a query over the last 15 minutes\n\n## Expected\n\nThe results table shows at least one log row from `ci_seed_stream`.\n\n## Actual\n\n(Synthetic — used to validate the test-assist CI plumbing. Expected verdict: FIXED.)",
+  "labels": [
+    {"name": "Needs-Testing"},
+    {"name": "fixture"}
+  ],
+  "comments": []
+}
@@ -0,0 +1,7 @@
+[
+  {"level": "info",  "service": "ci-seed", "message": "user login ok",          "code": 200},
+  {"level": "warn",  "service": "ci-seed", "message": "slow query detected",    "code": 200},
+  {"level": "error", "service": "ci-seed", "message": "upstream connect failed", "code": 502},
+  {"level": "info",  "service": "ci-seed", "message": "cache miss",             "code": 200},
+  {"level": "error", "service": "ci-seed", "message": "auth token expired",     "code": 401}
+]
@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+# Idempotent seeding for test-assist CI. Throwaway — replace once real Needs-Testing flow is up.
+#
+# pipefail is intentional — without it, `curl … | tee …` would mask a non-zero curl exit (set -e
+# does NOT propagate through pipes). Auth is passed via Authorization header (NOT `curl -u`) so
+# the password never appears in `ps` output on shared runners.
+set -euo pipefail
+
+: "${ZO_BASE_URL:?ZO_BASE_URL not set}"
+: "${ZO_ROOT_USER_EMAIL:?ZO_ROOT_USER_EMAIL not set}"
+: "${ZO_ROOT_USER_PASSWORD:?ZO_ROOT_USER_PASSWORD not set}"
+ORG="${ORGNAME:-default}"
+STREAM="${CI_SEED_STREAM:-ci_seed_stream}"
+
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+
+echo "Seeding $STREAM on $ZO_BASE_URL (org=$ORG)"
+
+# Build basic-auth header in-process; the resulting variable is passed via -H, not as a CLI arg
+# visible in process listings.
+AUTH_B64=$(printf '%s' "${ZO_ROOT_USER_EMAIL}:${ZO_ROOT_USER_PASSWORD}" | base64)
+
+curl -sS --fail-with-body \
+  -H "Authorization: Basic ${AUTH_B64}" \
+  -H 'Content-Type: application/json' \
+  --data-binary "@${HERE}/logs_seed.json" \
+  "${ZO_BASE_URL%/}/api/${ORG}/${STREAM}/_json" \
+  | tee /tmp/test-assist-ci-seed.log
+
+echo
+echo "Seed complete."
@@ -17,4 +17,7 @@ coverage/

 # Test output logs
 test-output.log
-*.log
+*.log
+
+# test-assist-ci throwaway specs (generated per run, deleted after)
+playwright-tests/_ci_tmp/