starred/openobserve
1
0
Fork 0
mirror of https://github.com/openobserve/openobserve.git synced 2026-11-05 16:10:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
74df120055e523fa00a3aef378b8a0e2142ff2bb
openobserve/tests/test-data/test-assist-ci
T
History
Neha PandClaude Opus 4.7 0be7644721 test: common data seed for real-issue verification (#13029) 
Yesterday's real-bug proof-of-concept showed the pipeline plumbing
works, but every real bug came back INCONCLUSIVE because the Tester (via
Playwright MCP) opened pages on an **empty OpenObserve instance**. There
was nothing to observe — no streams, no alerts, no pipelines, no traces.

This PR adds a common data seed the engine runs before every dispatch.
**Pairs with
[o2-enterprise#XXXX](https://github.com/openobserve/o2-enterprise/pull/XXXX)
— both must merge together.**

## Files added

| File | Contents |
|------|----------|
| `seed/seed-common.sh` | Main orchestrator. POSTs everything via HTTP
Basic auth (matches OSS Playwright's `data-ingestion.js` — header-based,
no creds on CLI args). |
| `seed/traces_seed.json` | 5-span OTLP payload (synthetic — OSS doesn't
ship a static traces JSON; `trace-ingestion.js` generates
programmatically) |
| `seed/metrics_seed.json` | 3-metric OTLP payload (HTTP counter +
latency gauge + memory gauge; synthetic for the same reason) |
| `seed/alerts_seed.json` | 3 alerts (one scheduled, one real-time, one
disabled) |
| `seed/pipeline_seed.json` | 1 pipeline (default → ci_pipeline_out) |

## What runs (in order)

1. **Fixture logs** → `ci_seed_stream` (kept for fixture #90001 backward
compat via existing `logs_seed.json`)
2. **Bulk logs** (~3.8k entries from existing
`tests/test-data/logs_data.json`) → 4 streams: `default`,
`e2e_automate`, `web_logs`, `k8s_logs`. **Same file** OSS Playwright's
`globalSetup` uses.
3. **OTLP traces** → `default` trace stream
4. **OTLP metrics** → `default` metrics stream
5. **3 alert definitions** (POST /api/{org}/{stream}/alerts)
6. **1 pipeline** (POST /api/{org}/pipelines)

Idempotent — all ingestion is best-effort; single-endpoint failures log
the HTTP status code but don't abort the seed. Unpopulated areas still
yield INCONCLUSIVE downstream (better than the whole engine failing at
seed).

## Coverage for the 10 Needs-Testing bugs we're firing next

| Bug | Data type needed | Seed step |
|-----|------------------|-----------|
| #12452 SQL editor overflow | logs | 2 |
| #11700 Streams spacing | streams (auto-created via ingest) | 2 |
| #5042 Include search cmd+enter | logs | 2 |
| #12603 Traces page issues | traces | 3 |
| #12820 Alerts UX | alerts | 5 |
| #6750 Multi-window custom time | alerts | 5 |
| #10530 Pipeline cron alignment | pipeline | 6 |
| #12647 Pipelines popup overflow | pipeline | 6 |
| #9252 Pipeline destination verbiage | pipeline | 6 |
| #12803 Language translation | any page (no seed needed) | — |

## Validated end-to-end

Fixture #90002 dispatched on the paired branches ([run
28657529004](https://github.com/openobserve/o2-enterprise/actions/runs/28657529004))
— all 26 pipeline steps completed successfully, including the new Seed
step showing 6 sub-seeds with their HTTP status codes.

## What's NOT in this PR

- The old `seed/seed.sh` stays for one merge cycle (deprecated in the
README) — will delete once two clean real-bug batches confirm
seed-common works
- No `Test-Assist-CI-Verified` label changes
- No qa-reports table updates

## Test plan

- [x] Fixture #90002 with `source_ref=feat/test-assist-common-seed` —
all 26 steps green
- [ ] After merge (this + ENT PR): dispatch on any real bug — expect
seed step to run before Scout, and Tester to find non-empty pages

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-07-03 15:37:05 +00:00
..
fixture-issues
test(test-assist-ci): vision-preprocessor fixture + caller filter fix (#12973)
2026-07-01 14:46:20 +00:00
seed
test: common data seed for real-issue verification (#13029)
2026-07-03 15:37:05 +00:00
expected-verdicts.json
test(test-assist-ci): vision-preprocessor fixture + caller filter fix (#12973)
2026-07-01 14:46:20 +00:00
README.md
test: common data seed for real-issue verification (#13029)
2026-07-03 15:37:05 +00:00

README.md

test-assist-ci fixtures

Throwaway scaffolding for bringing up the test-assist CI agent. Will be replaced once real Needs-Testing issues flow through the pipeline.

Layout

Path Purpose
seed/seed-common.sh Main seed — runs for BOTH fixture AND real-issue dispatches. Ingests logs (4 streams), OTLP traces, OTLP metrics, 3 alerts, 1 pipeline. This is what unblocks real-bug verification: without seed, Tester finds an empty app and can't observe reported behaviours.
seed/logs_seed.json Tiny 5-entry log batch → ci_seed_stream (kept for fixture #90001 backward compat). Read by seed-common.sh.
seed/traces_seed.json Small OTLP resource-spans payload (5 spans across a checkout flow).
seed/metrics_seed.json Small OTLP resource-metrics payload (HTTP counters + latency gauge + memory gauge).
seed/alerts_seed.json 3 sample alerts (scheduled + real-time + disabled) — populates the Alerts page.
seed/pipeline_seed.json 1 basic pipeline (default → ci_pipeline_out) — populates the Pipelines page.
seed/seed.sh ⚠️ Deprecated — the fixture-only seed. Superseded by seed-common.sh; delete after two clean batches confirm the common seed works.
fixture-issues/ Synthetic GitHub issue payloads — same JSON shape as gh issue view. Lets Scout run without real issues.
expected-verdicts.json Verdict each fixture issue should produce. Pipeline regression contract.

How the agent consumes this

  1. Phase 0.5 (new) → seed/seed.sh runs against $ZO_BASE_URL (only when FIXTURE_MODE=true).
  2. Phase 1 (Scout) → if FIXTURE_MODE=true, reads fixture-issues/*.json instead of calling GitHub.
  3. End of run → engine diffs final verdicts against expected-verdicts.json and fails the job on mismatch.

Required env (consumed by seed.sh)

Variable Default Notes
ZO_BASE_URL — (required) The OpenObserve instance the engine booted.
ZO_ROOT_USER_EMAIL — (required) Root user for basic auth.
ZO_ROOT_USER_PASSWORD — (required) Root user password.
ORGNAME default OpenObserve org to ingest into.
CI_SEED_STREAM ci_seed_stream Stream name the seed creates; the fixture spec references this exact name. Override only when running multiple seeders against one instance.

expected-verdicts.json schema

A JSON object keyed by issue number. Each entry:

{
  "90001": {
    "fixture": "bug-001-fixed.json",        // filename under fixture-issues/
    "expected_status": "FIXED",             // one of FIXED | STILL_PRESENT | INCONCLUSIVE
    "expected_confidence": "HIGH"           // one of HIGH | LOW
  }
}

The engine's regression-mode reads this file after Reporter finishes, compares each fixture issue's actual verdict against expected_status + expected_confidence, and fails the job on mismatch. Real (non-fixture) issues never appear here and are not regression-checked.

Reference in New Issue View Git Blame Copy Permalink