starred/openobserve
1
0
Fork 0
mirror of https://github.com/openobserve/openobserve.git synced 2026-11-05 16:10:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
6,266 Commits 627 Branches 244 Tags
f2ccdadcf6ad6efc19fbe4d508bafac394e91c07
Commit Graph
8 Commits
Author SHA1 Message Date
Neha PandClaude Opus 4.7 494025f83b test: add ~100 query-agent queries covering re_match, production patterns, deep nesting, edge cases (#12735) 
## Summary
- Expands the query agent test suite from **574 → 675 queries** across
13 category files
- **20 re_match queries** — testing `\d`, `[abc]`, `^anchor`, `a|b`,
coercion, boolean mode, COALESCE+CASE patterns found 224x in production
- **19 production-inspired queries** —
histogram+re_match+CASE+percentage, COALESCE chains,
re_match+re_not_match combined
- **19 deep-nesting queries** — INTERSECT, EXCEPT, subquery-in-HAVING,
double-NOT-EXISTS, 2-level correlated subqueries
- **15 edge-case queries** — empty results, division-by-zero via NULLIF,
STDDEV of single row, NULL grouping, BETWEEN edge
- **10 window-depth queries** — multiple window functions, WINDOW clause
reuse, NTH_VALUE, NTILE, CUME_DIST
- **5 histogram+re_match queries** — covering production pattern of
histogram + filtered metric extraction
- **5 FTS combo queries** — match_all +
histogram/re_match/window/aggregation
- **5 aggregation edge queries** — multiple FILTER clauses, STRING_AGG
with ORDER BY, HAVING complexity
- Expected values regenerated by DuckDB oracle

## Test plan
- [ ] CI passes (memtable + parquet phases)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
 2026-06-17 12:51:13 +00:00
Neha PandClaude Opus 4.7 1484f920a9 test: expand query agent suite to 570 queries with production SQL patterns (#12534) 
## Summary

- Added **46 new queries** (Q511-Q556) to the query agent regression
suite, bringing the total from 524 to **570 unique queries**
- Mined from real production SQL patterns (june7.sql) and adapted to use
existing test data fields
- Fixed **15 pre-existing duplicate IDs** (Q226-Q240 existed in both
aggregation.json and basic_select.json — renumbered to Q557-Q571)

## New query distribution

| Category | New | Description |
|---|---|---|
| string_functions | +10 | spath() + str_match_ignore_case, split_part()
+ chr(), multi-COALESCE chains, CAST AS VARCHAR |
| histogram | +7 | 30-second/min intervals, COUNT FILTER, SUM(CASE
WHEN), NULL breakdowns |
| date_time | +6 | arrow_cast(), date_trunc(), to_timestamp_micros(),
timestamp arithmetic, LAG for gaps |
| combined | +6 | count(distinct) in CASE WHEN, re_match + re_not_match
combo, FILTER in aggregation, HAVING percentages |
| full_text_search | +5 | re_match + re_not_match, regexp_like,
match_all(), str_match_ignore_case with COALESCE |
| cte_subquery | +4 | CTE with spath, CTE with COALESCE fallbacks,
correlated subquery |
| aggregation | +2 | IS NOT NULL exploration patterns, multi-column
GROUP BY |
| window | +2 | NTILE with COALESCE PARTITION, LAG with delta
calculation |
| basic_select | +2 | Complex COALESCE + CAST chains, NULLS LAST
ordering |
| cross_stream | +1 | LEFT JOIN on dual keys across streams |
| union | +1 | UNION ALL with different metric types |

## New OO-specific functions covered

Four functions previously had **zero test coverage**:
- `chr()` — 1 query (string splitting by character code)
- `arrow_cast()` — 1 query (DataFusion timestamp arithmetic)
- `date_trunc()` — 1 query (minute-level timestamp bucketing)
- `to_timestamp_micros()` — 1 query (microsecond-to-timestamp
conversion)

Plus expanded coverage of rare functions: `spath()` (+3), `split_part()`
(+3).

## Test modes

- **38/46** new queries have full sqllogictest comparison (DuckDB
computed expected results)
- **8/46** use legacy assertions for OO-specific functions that DuckDB
cannot replicate (spath, arrow_cast, date_trunc, to_timestamp_micros)
- Overall suite: **533** sqllogictest, **37** legacy

## Changes

| File | Change |
|---|---|
| `data_gen.py` | `NUM_QUERIES` 520 → 575 |
| `gen_q511_q556.py` | New: query generation script |
| 13 category JSON files | Updated with new queries + dedup fix |

## Verification

- `compute_counts.py` ran successfully on all 570 queries
- 0 duplicate IDs across all 13 category files
- Only gap: Q025 (pre-existing)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
 2026-06-10 06:24:40 +00:00
Neha PandClaude Sonnet 4.6 146d5e7612 feat(tests): vortex file format test suite (scenarios 1–83) (#12472) 
## Summary

- Adds **239 new tests** covering the vortex columnar file format across
78 scenarios from the test plan
- **`tests/api-testing/tests/vortex/`** — 9 test modules (83 tests):
format conversion, schema evolution, data integrity, FTS, downsampling,
API format awareness, server config, and performance/stress
- **`tests/api-testing/tests/query_agent/test_3_vortex.py`** — 156
tests: the full 155-query SQL correctness suite + scenario 39
(self-join) run post-flush against vortex files
- **`api-testing.yml`** — adds `--ignore=tests/vortex` to the OSS
default-set run; vortex tests require the enterprise binary with
`ZO_FILE_FORMAT=vortex` and are gated to enterprise CI only

## Test plan coverage

| Module | Scenarios |
|--------|-----------|
| `test_format_conversion.py` | 1–13, 7, 8, 9, 10 |
| `test_schema_evolution.py` | 14–25 |
| `test_data_integrity.py` | 55–62 |
| `test_fts.py` | 51–54 |
| `test_downsampling.py` | 47–50 |
| `test_api_format.py` | 71–73 |
| `test_config.py` | 66–67 |
| `test_performance.py` | 74–78 |
| `test_3_vortex.py` | 26–46 (155 queries + JOIN) |

## Enterprise CI

Two dedicated jobs added in the companion `openobserve/o2-enterprise` PR
on the same branch:
- **`api_query_agent_vortex`** — runs `test_3_vortex.py` with
`ZO_FILE_FORMAT=vortex`
- **`api_vortex_format_tests`** — runs the full `tests/vortex/` suite
with 4-phase server lifecycle (parquet→vortex→parquet→vortex+compaction)

OSS CI is unchanged except for the ignore rule — no vortex tests run
against the OSS binary.

## Test checklist

- [ ] `tests/vortex/` collects cleanly: `83 tests collected`
- [ ] `tests/query_agent/test_3_vortex.py` collects cleanly: `156 tests
collected`
- [ ] Enterprise CI `api_query_agent_vortex` job passes
- [ ] Enterprise CI `api_vortex_format_tests` job passes

🤖 Generated with [Claude Code](https://claude.ai/claude-code)

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-08 13:57:17 +00:00
Neha PandClaude Opus 4.7 008398163b test: add more queries to query regression (#12487) 
## Summary
- Adds **100 new structural-pattern queries** (Q411-Q510) across all 13
categories, bringing the suite from 425 to **525 queries**
- Adds **cross-stream JOIN/UNION support**: `stream_offset` in
data_gen.py, `STREAM2` in conftest, dual-table DuckDB oracle
- 15 cross-stream queries (Q401-Q415) covering INNER/LEFT/RIGHT JOIN,
EXISTS/NOT EXISTS, UNION ALL across streams
- Bumps data_gen.py from 510 to 520 query capacity

## Changes
- **Infrastructure**:
[conftest.py](tests/api-testing/tests/query_agent/conftest.py),
[compute_counts.py](tests/test-data/query-agent/compute_counts.py),
[data_gen.py](tests/test-data/query-agent/data_gen.py)
- **New file**:
[cross_stream.json](tests/test-data/query-agent/queries/cross_stream.json)
— 15 queries
- **12 categories**: 85 queries across aggregation, basic_select,
combined, cte_subquery, date_time, full_text_search, histogram,
math_functions, pagination, string_functions, union, window

## Test plan
- [x] `compute_counts.py` generates expected results for all 525 queries
- [ ] Run `pytest tests/query_agent/ -v` against target environment (565
tests expected)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
 2026-06-08 04:38:40 +00:00
Neha PandClaude Opus 4.7 0334dda9a3 test: add data variety and NULL stress to query agent data generation (#12485) 
## Summary

Two high-value improvements to the query agent regression test data:

1. **Data variety** — Per-field rotation offsets (`qi*13 + i*7`) replace
the old uniform `idx % len(pool)` approach. Previously all fields picked
from the same pool position, creating only 5 fixed row shapes per query.
Now field-value combinations vary independently per field.

2. **NULL stress** — 8 fields (`bot_flag`, `info_tag`,
`upstream_error_code`, `page_url`, `endpoint_path`, `resource_path`,
`variant_tag`, `threat_flag`) get `None` for 40% of records per query.
This exercises `IS NULL` / `IS NOT NULL` / `COALESCE` paths that were
previously untested.

## Changes
- `data_gen.py`: Rewrote `make_record()` with per-field rotation and
NULL injection
- 24 queries: Adjusted WHERE/HAVING filters to match new data
distribution
- All 12 query JSONs: Regenerated expected results via
`compute_counts.py`

## Impact
- 400 queries all have results (18 pre-existing legacy/skip queries
unchanged)
- No change to record count (still 5/query, 2000 total) — test time
unaffected (~7 min)

## Test plan
- [x] `compute_counts.py` regenerates all expected results
- [ ] Run `pytest tests/query_agent/ -v --tb=short` against dev
environment

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
 2026-06-07 08:11:47 +00:00
Neha PandClaude Opus 4.7 5eb63395f1 test: broaden coverage with structural query templates (#12482) 
## Summary
- Adds 200 new SQL queries (Q201-Q400) across all 12 categories,
extracted as structural templates from production `june6.sql` query
patterns
- All queries use the existing synthetic `FIELD_POOL` schema — zero real
stream/field names
- Bumps `data_gen.py` from 200 → 400 queries (4000 records total, 10 per
query)
- Introduces new structural patterns previously missing from the suite

## New patterns
- **CONCAT + COALESCE chains**: Multi-field string construction for
display messages
- **CASE WHEN inside histogram COUNT**: Platform mapping + conditional
aggregation
- **Multi-field LIKE / NOT LIKE OR chains**: Text search across 4-8
fields
- **NOT IN + IS NOT NULL**: Exclusion filters with null safety
- **Long IN lists**: 5+ values in WHERE clauses
- **Percentage calculations**: `100.0 * COUNT(CASE WHEN ...) / COUNT(*)`
patterns
- **approx_percentile_cont in subqueries**: P95/P99 in nested SELECTs
- **Histogram with breakdown**: `GROUP BY _time, breakdown` pattern
- **char_length CASE fallbacks**: `CASE WHEN char_length(f) > 0 THEN f
ELSE fallback END`
- **CTE with multiple subquery references**: Self-join and multi-CTE
patterns

## Distribution
| Category | New | Range |
|----------|-----|-------|
| aggregation | 25 | Q201-Q225 |
| basic_select | 25 | Q226-Q250 |
| combined | 20 | Q251-Q270 |
| cte_subquery | 25 | Q271-Q295 |
| date_time | 10 | Q296-Q305 |
| full_text_search | 20 | Q306-Q325 |
| histogram | 25 | Q326-Q350 |
| math_functions | 8 | Q351-Q358 |
| pagination | 7 | Q359-Q365 |
| string_functions | 15 | Q366-Q380 |
| union | 12 | Q381-Q392 |
| window | 8 | Q393-Q400 |

## Test plan
- [x] `compute_counts.py` generates DuckDB expected results for all 400
queries
- [ ] CI: 800 tests (400 × 2 phases) pass

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
 2026-06-07 02:31:06 +00:00
Neha PandClaude Opus 4.7 4d48bbb21b test: expand query agent regression suite (#12479) 
## Summary
- Expands the SQL regression catalog from **155 to 200 queries** across
all 12 categories
- Adds **7 queries** extracted from abc.sql (Q156-Q162) — synthesized
from real-world OpenObserve SQL patterns with all data/fields/streams
generated from scratch
- Adds **230 auto-generated queries** (Q163-Q392) from templates to
cover query format diversity
- Both test phases pass: **200 memtable + 200 parquet = 400 total**

## Key changes
- **data_gen.py**: 43 new fields (record_type, site_name, latency_ms,
payload_size, alert_tags, phase_data, db_info, etc.), generates 3920
deterministic records
- **compute_counts.py**: new translations for regexp_like,
str_match_ignore_case, approx_distinct, json_get_str/json_get_json
chaining; empty-aggregate result detection (OO returns 0 rows vs
DuckDB's 1 NULL row)
- **conftest.py**: PID-based stream name suffix for cross-invocation
isolation (prevents data accumulation)

## Test plan
- [x] test_1_memtable.py — 392 passed
- [x] test_2_parquet.py — 392 passed
- [x] All 7 abc.sql queries pass (Q156-Q162)
- [x] No real stream names, field names, or data copied from external
sources

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
 2026-06-06 04:59:20 +00:00
Neha PandClaude Opus 4.7 eb7cce9b96 test: query agent SQL regression suite — 155 queries, sqllogictest-style, two-phase (#12384) 
## Summary

155-query SQL regression test suite covering 12 categories. Adding a new
query requires zero test code changes — drop a JSON entry into the right
category file and run `compute_counts.py`.

**Sqllogictest-style comparison**: DuckDB pre-computes full expected
result sets (not just row counts). The test runner extracts actual
values from OO hits, sorts both sides, and compares cell-by-cell with 5%
relative float tolerance. This catches value-level regressions that
row-count-only checks miss.

Two-phase execution: **memtable** (pre-flush) then **parquet**
(post-flush + Tantivy). **310/310 pass**.

## Architecture

```
tests/
├── api-testing/tests/query_agent/
│   ├── conftest.py            # Shared fixtures, run_query(), tolerance comparison
│   ├── test_1_memtable.py     # Phase 1: all queries pre-flush (FTS skipped)
│   └── test_2_parquet.py      # Phase 2: post-flush (module-scoped fixture triggers flush)
└── test-data/query-agent/
    ├── data_gen.py             # Deterministic data generator (single source of truth)
    ├── compute_counts.py       # DuckDB oracle: computes expected result sets
    ├── base_ts_override.json   # Locks BASE_TS so compute and tests share same timestamps
    └── queries/                # 12 JSON files, 155 queries with expected results
```

### How it works

1. **Deterministic data** — `data_gen.py` generates 10 records per query
(1550 total) with a warehouse/supply-chain IoT schema. Each query Q{N}
gets isolated timestamps. `BASE_TS` is locked via
`base_ts_override.json` so compute and tests share identical timestamps
(prevents histogram bucket drift on minute boundaries).

2. **Unique stream per session** — `STREAM =
f"query_agent_test_{int(time.time())}"` prevents accumulated-data bleed
from prior runs.

3. **DuckDB oracle** — `compute_counts.py` loads the same dataset into
DuckDB, creates a pre-filtered VIEW per query (so time filters work for
CTEs, subqueries, UNIONs — no fragile SQL injection), translates
OO-specific functions, and writes `expected.results` blocks into the
query JSONs.

4. **Histogram translation** — `histogram(_timestamp, 'N unit')` is
translated to integer arithmetic matching OO's `date_bin` with origin at
2001-01-01T00:00:00 UTC. Verified against DataFusion CLI — 100/100
records match across all interval types (15s, 30s, 1m, 2m, 3m, 4m, 5m).
ISO timestamp post-processing converts epoch-µs bucket values to
`YYYY-MM-DDTHH:MM:SS` strings matching OO's serialization format.

5. **Sqllogictest comparison** — `run_query()` checks for
`expected.results`. When present, it extracts column values from hits as
strings, sorts both actual and expected (`results_mode: "rowsort"`), and
compares cell-by-cell with 5% relative float tolerance (handles
cross-engine floating-point drift in AVG/STDDEV/etc.).

6. **Two-phase testing** — `test_1_memtable.py` runs all queries
pre-flush (FTS sqllogictest skipped — Tantivy indexes don't exist yet).
`test_2_parquet.py` has a module-scoped `post_flush` fixture that
triggers `/node/flush`, re-polls until data is searchable, then runs all
queries with full assertions.

### Comparison modes

| Mode | Count | Mechanism |
|------|-------|-----------|
| **Sqllogictest** | 140 | Cell-by-cell comparison with 5% float
tolerance |
| **Skip sqllogictest** | 9 | DuckDB results for documentation, fall
back to legacy assertions |
| **Legacy** | 6 | 5 auto-interval histograms + Q122 (STRING_AGG
DISTINCT ORDER BY) |

**Skip sqllogictest queries** — known OO-DuckDB divergence:

| Query | Reason |
|-------|--------|
| Q037 | ROW_NUMBER tie-breaking differs between engines |
| Q058 | Self-join pairings non-deterministic across engines |
| Q072 | FULL OUTER JOIN histogram — NULL handling differs |
| Q080 | FTS `match_all` + histogram — Tantivy vs SQL LIKE |
| Q111 | ROW_NUMBER ties in pagination |
| Q117 | STRING_AGG ordering differs between engines |
| Q121 | STRING_AGG concatenation order |
| Q127 | UNION across 5 overlapping groups — boundary records compete
for LIMIT spots |
| Q152 | PERCENT_RANK tie-breaking differs between engines |

### Key design decisions

| Decision | Why |
|----------|-----|
| Full result-set comparison | Catches value-level regressions (e.g.,
ROUND producing wrong decimal) |
| `rowsort` mode | DuckDB and OO order ties differently — sorting both
sides eliminates spurious failures |
| 5% float tolerance | Cross-engine AVG/STDDEV/PERCENTILE differ at
least-significant digits |
| `''` = `'None'` normalization | OO serializes NULL as empty string;
DuckDB uses `"None"` |
| DuckDB VIEW for time filtering | WHERE injection into CTEs/subqueries
is fragile — VIEW filters at scan level |
| Dynamic columns excluded | `_timestamp` and `_time` differ between
compute time and test time |
| OO-dropped LAG columns excluded | `prev_tput`, `hours_2_back`,
`delta_from_prev`, `temp_diff` |
| Histogram → DuckDB integer math | DuckDB lacks `date_bin`; integer
formula verified via DataFusion CLI |
| BASE_TS override file | Prevents histogram timestamp drift when
compute and test run at different minutes |
| 1s time window margins | Minimizes cross-query data bleed while
capturing all 10 records |
| `pytest.fail()` on fixture failure | Stops all tests instead of
running 310 against an empty stream |

### Categories covered

| Category | Queries | Description |
|---|---|---|
| aggregation | Q001-Q020 | GROUP BY, COUNT, AVG, SUM, MIN, MAX, FILTER,
HAVING |
| basic_select | Q021-Q035 | WHERE, IN, BETWEEN, LIKE, ORDER BY,
DISTINCT, CASE |
| combined | Q036-Q045 | JOINs, multi-condition filters, combined
aggregations |
| cte_subquery | Q046-Q062 | CTEs, subqueries, derived tables, EXISTS |
| date_time | Q063-Q073 | Date parsing, formatting, extraction, interval
arithmetic, histogram |
| full_text_search | Q074-Q082 | match_all, match against fields |
| histogram | Q083-Q093 | histogram() bucketing, aggregation over
buckets |
| math_functions | Q094-Q102 | ROUND, SQRT, ABS, POWER, arithmetic
expressions |
| pagination | Q103-Q112 | LIMIT, OFFSET, combination with filters and
sorts |
| string_functions | Q113-Q122 | SUBSTR, CONCAT, UPPER, LOWER, TRIM,
LIKE patterns |
| union | Q123-Q134 | UNION ALL, UNION (dedup), CTE + UNION combinations
|
| window | Q135-Q155 | ROW_NUMBER, RANK, DENSE_RANK, LAG, LEAD, SUM/AVG
OVER |

## Test plan

- [x] All 155 queries pass in both phases (310/310)
- [x] 140 queries with sqllogictest cell-by-cell comparison
- [x] 17 histogram queries converted from legacy to sqllogictest
- [x] 9 queries with skip_sqllogictest for known OO-DuckDB divergence
- [x] 6 queries in legacy mode (auto-interval histograms + Q122)
- [x] BASE_TS locked via override file — no timestamp drift
- [x] Dedicated CI job (`api_query_agent_tests`) runs in parallel

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
 2026-06-03 09:30:02 +00:00