mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-06 00:09:22 +00:00
Neha PandClaude Opus 4.7
eb7cce9b96
## Summary
155-query SQL regression test suite covering 12 categories. Adding a new
query requires zero test code changes — drop a JSON entry into the right
category file and run `compute_counts.py`.
**Sqllogictest-style comparison**: DuckDB pre-computes full expected
result sets (not just row counts). The test runner extracts actual
values from OO hits, sorts both sides, and compares cell-by-cell with 5%
relative float tolerance. This catches value-level regressions that
row-count-only checks miss.
Two-phase execution: **memtable** (pre-flush) then **parquet**
(post-flush + Tantivy). **310/310 pass**.
## Architecture
```
tests/
├── api-testing/tests/query_agent/
│ ├── conftest.py # Shared fixtures, run_query(), tolerance comparison
│ ├── test_1_memtable.py # Phase 1: all queries pre-flush (FTS skipped)
│ └── test_2_parquet.py # Phase 2: post-flush (module-scoped fixture triggers flush)
└── test-data/query-agent/
├── data_gen.py # Deterministic data generator (single source of truth)
├── compute_counts.py # DuckDB oracle: computes expected result sets
├── base_ts_override.json # Locks BASE_TS so compute and tests share same timestamps
└── queries/ # 12 JSON files, 155 queries with expected results
```
### How it works
1. **Deterministic data** — `data_gen.py` generates 10 records per query
(1550 total) with a warehouse/supply-chain IoT schema. Each query Q{N}
gets isolated timestamps. `BASE_TS` is locked via
`base_ts_override.json` so compute and tests share identical timestamps
(prevents histogram bucket drift on minute boundaries).
2. **Unique stream per session** — `STREAM =
f"query_agent_test_{int(time.time())}"` prevents accumulated-data bleed
from prior runs.
3. **DuckDB oracle** — `compute_counts.py` loads the same dataset into
DuckDB, creates a pre-filtered VIEW per query (so time filters work for
CTEs, subqueries, UNIONs — no fragile SQL injection), translates
OO-specific functions, and writes `expected.results` blocks into the
query JSONs.
4. **Histogram translation** — `histogram(_timestamp, 'N unit')` is
translated to integer arithmetic matching OO's `date_bin` with origin at
2001-01-01T00:00:00 UTC. Verified against DataFusion CLI — 100/100
records match across all interval types (15s, 30s, 1m, 2m, 3m, 4m, 5m).
ISO timestamp post-processing converts epoch-µs bucket values to
`YYYY-MM-DDTHH:MM:SS` strings matching OO's serialization format.
5. **Sqllogictest comparison** — `run_query()` checks for
`expected.results`. When present, it extracts column values from hits as
strings, sorts both actual and expected (`results_mode: "rowsort"`), and
compares cell-by-cell with 5% relative float tolerance (handles
cross-engine floating-point drift in AVG/STDDEV/etc.).
6. **Two-phase testing** — `test_1_memtable.py` runs all queries
pre-flush (FTS sqllogictest skipped — Tantivy indexes don't exist yet).
`test_2_parquet.py` has a module-scoped `post_flush` fixture that
triggers `/node/flush`, re-polls until data is searchable, then runs all
queries with full assertions.
### Comparison modes
| Mode | Count | Mechanism |
|------|-------|-----------|
| **Sqllogictest** | 140 | Cell-by-cell comparison with 5% float
tolerance |
| **Skip sqllogictest** | 9 | DuckDB results for documentation, fall
back to legacy assertions |
| **Legacy** | 6 | 5 auto-interval histograms + Q122 (STRING_AGG
DISTINCT ORDER BY) |
**Skip sqllogictest queries** — known OO-DuckDB divergence:
| Query | Reason |
|-------|--------|
| Q037 | ROW_NUMBER tie-breaking differs between engines |
| Q058 | Self-join pairings non-deterministic across engines |
| Q072 | FULL OUTER JOIN histogram — NULL handling differs |
| Q080 | FTS `match_all` + histogram — Tantivy vs SQL LIKE |
| Q111 | ROW_NUMBER ties in pagination |
| Q117 | STRING_AGG ordering differs between engines |
| Q121 | STRING_AGG concatenation order |
| Q127 | UNION across 5 overlapping groups — boundary records compete
for LIMIT spots |
| Q152 | PERCENT_RANK tie-breaking differs between engines |
### Key design decisions
| Decision | Why |
|----------|-----|
| Full result-set comparison | Catches value-level regressions (e.g.,
ROUND producing wrong decimal) |
| `rowsort` mode | DuckDB and OO order ties differently — sorting both
sides eliminates spurious failures |
| 5% float tolerance | Cross-engine AVG/STDDEV/PERCENTILE differ at
least-significant digits |
| `''` = `'None'` normalization | OO serializes NULL as empty string;
DuckDB uses `"None"` |
| DuckDB VIEW for time filtering | WHERE injection into CTEs/subqueries
is fragile — VIEW filters at scan level |
| Dynamic columns excluded | `_timestamp` and `_time` differ between
compute time and test time |
| OO-dropped LAG columns excluded | `prev_tput`, `hours_2_back`,
`delta_from_prev`, `temp_diff` |
| Histogram → DuckDB integer math | DuckDB lacks `date_bin`; integer
formula verified via DataFusion CLI |
| BASE_TS override file | Prevents histogram timestamp drift when
compute and test run at different minutes |
| 1s time window margins | Minimizes cross-query data bleed while
capturing all 10 records |
| `pytest.fail()` on fixture failure | Stops all tests instead of
running 310 against an empty stream |
### Categories covered
| Category | Queries | Description |
|---|---|---|
| aggregation | Q001-Q020 | GROUP BY, COUNT, AVG, SUM, MIN, MAX, FILTER,
HAVING |
| basic_select | Q021-Q035 | WHERE, IN, BETWEEN, LIKE, ORDER BY,
DISTINCT, CASE |
| combined | Q036-Q045 | JOINs, multi-condition filters, combined
aggregations |
| cte_subquery | Q046-Q062 | CTEs, subqueries, derived tables, EXISTS |
| date_time | Q063-Q073 | Date parsing, formatting, extraction, interval
arithmetic, histogram |
| full_text_search | Q074-Q082 | match_all, match against fields |
| histogram | Q083-Q093 | histogram() bucketing, aggregation over
buckets |
| math_functions | Q094-Q102 | ROUND, SQRT, ABS, POWER, arithmetic
expressions |
| pagination | Q103-Q112 | LIMIT, OFFSET, combination with filters and
sorts |
| string_functions | Q113-Q122 | SUBSTR, CONCAT, UPPER, LOWER, TRIM,
LIKE patterns |
| union | Q123-Q134 | UNION ALL, UNION (dedup), CTE + UNION combinations
|
| window | Q135-Q155 | ROW_NUMBER, RANK, DENSE_RANK, LAG, LEAD, SUM/AVG
OVER |
## Test plan
- [x] All 155 queries pass in both phases (310/310)
- [x] 140 queries with sqllogictest cell-by-cell comparison
- [x] 17 histogram queries converted from legacy to sqllogictest
- [x] 9 queries with skip_sqllogictest for known OO-DuckDB divergence
- [x] 6 queries in legacy mode (auto-interval histograms + Q122)
- [x] BASE_TS locked via override file — no timestamp drift
- [x] Dedicated CI job (`api_query_agent_tests`) runs in parallel
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
70 lines
3.1 KiB
Python
70 lines
3.1 KiB
Python
"""Shared deterministic data generator for query agent tests.
|
|
|
|
Used by both conftest.py (test fixture) and compute_counts.py (DuckDB oracle).
|
|
"""
|
|
|
|
import json
|
|
import random
|
|
from datetime import datetime, timedelta, timezone
|
|
from pathlib import Path
|
|
|
|
random.seed(42)
|
|
|
|
# Check for a BASE_TS override saved by compute_counts.py so that the
|
|
# compute oracle and the test harness share the same BASE_TS. Without
|
|
# this, a minute boundary crossing between compute and test shifts
|
|
# all histogram bucket timestamps.
|
|
_OVERRIDE_FILE = Path(__file__).parent / "base_ts_override.json"
|
|
if _OVERRIDE_FILE.exists():
|
|
BASE_TS = json.loads(_OVERRIDE_FILE.read_text())["BASE_TS"]
|
|
else:
|
|
_now = datetime.now(timezone.utc).replace(second=0, microsecond=0)
|
|
BASE_TS = int((_now - timedelta(hours=4)).timestamp() * 1_000_000)
|
|
|
|
FIELD_POOL = {
|
|
"pallet_id": ["PL-001", "PL-002", "PL-003", "PL-004", "PL-005", "PL-006", "PL-007", "PL-008"],
|
|
"load_factor": [12.5, 33.0, 55.2, 72.8, 88.1, 94.6, 45.3, 67.9, 21.4, 99.0],
|
|
"charge_remaining":[10.0, 25.0, 42.5, 58.0, 75.3, 88.7, 95.0, 5.0, 33.3, 66.6],
|
|
"throughput_rate": [150.0, 320.0, 480.0, 610.0, 725.0, 840.0, 290.0, 555.0, 190.0, 900.0],
|
|
"sorter_model": ["SORT-X1", "SORT-A7", "SORT-M3", "SORT-Z9", "SORT-X1", "SORT-A7", "SORT-M3", "SORT-Q5"],
|
|
"conveyor_lane": ["LANE-A", "LANE-B", "LANE-C", "LANE-D", "LANE-A", "LANE-B", "LANE-C", "LANE-D"],
|
|
"facility_zone": ["ZONE-1", "ZONE-2", "ZONE-3", "ZONE-4", "ZONE-1", "ZONE-2", "ZONE-3", "ZONE-4"],
|
|
"control_center": ["CC-ALPHA", "CC-BETA", "CC-GAMMA", "CC-DELTA", "CC-ALPHA", "CC-BETA", "CC-GAMMA", "CC-DELTA"],
|
|
"item_count": [50, 120, 200, 340, 88, 155, 410, 275, 60, 500],
|
|
"defect_limit": [0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 0.8, 1.2, 2.8, 3.5],
|
|
"runtime_hours": [24, 72, 168, 360, 720, 48, 120, 500, 90, 1000],
|
|
"operation_mode": ["auto", "manual", "hybrid", "auto", "manual", "hybrid", "auto", "manual"],
|
|
"package_size": [100, 250, 500, 750, 1000, 200, 350, 600, 850, 1200],
|
|
"scan_attempts": [1, 2, 3, 1, 2, 4, 1, 3, 2, 5],
|
|
"conveyor_segment":[1, 2, 3, 4, 5, 1, 2, 3, 4, 5],
|
|
"cell_temp": [18.5, 22.0, 26.3, 30.1, 35.7, 19.2, 24.8, 28.9, 32.5, 40.0],
|
|
"build_signature": ["BS-2026-A1", "BS-2026-B2", "BS-2026-C3", "BS-2025-D4", "BS-2026-A1",
|
|
"BS-2026-B2", "BS-2026-C3", "BS-2025-D4"],
|
|
}
|
|
|
|
STREAM_VALUES = ["stdout", "stdout", "stdout", "stderr"]
|
|
|
|
|
|
def make_record(ts, idx, qid):
|
|
"""Build a single deterministic data record."""
|
|
r = {
|
|
"_timestamp": ts,
|
|
"log": f"{qid} warehouse event record {idx}",
|
|
"stream": STREAM_VALUES[idx % len(STREAM_VALUES)],
|
|
}
|
|
for field, pool in FIELD_POOL.items():
|
|
r[field] = pool[idx % len(pool)]
|
|
return r
|
|
|
|
|
|
def build_dataset(num_queries=155):
|
|
"""Generate deterministic records for queries Q001-Q{num_queries}."""
|
|
records = []
|
|
for qi in range(1, num_queries + 1):
|
|
qid = f"Q{qi:03d}"
|
|
base = BASE_TS + (qi - 1) * 60_000_000
|
|
for i in range(10):
|
|
ts = base + i * 18_000_000
|
|
records.append(make_record(ts, i, qid))
|
|
return records
|