mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-04 16:09:05 +00:00
Neha PandClaude Opus 4.7
8a4f7ea23b
## Summary Adds Q1003–Q1072 (70 queries) to `tests/test-data/query-agent/queries/full_text_search.json`. All exercise the `match_all` + FST projection rewrite path fixed in #13808, under diverse plan shapes. Structural patterns mirror real production customer queries (fan-out UNION ALL, multi-CTE, CSE-triggering scalar wrappers on the FTS field) but use **only test-fixture fields and values** — no customer identifiers. ## Shape coverage | Family | Range | Count | What it exercises | |---|---|---|---| | A. Subquery + `match_all` + GROUP BY | Q1003–Q1012 | 10 | Simple inline-view aggregations | | B. CTE with fan-out UNION ALL, `match_all` per branch | Q1013–Q1027 | 15 | Dominant customer pattern (multi-branch UNION + per-branch FTS + LIKE/regex + outer aggregate) | | C. CTE + scalar wrapper on `log` (trim/upper/split_part/regexp_replace/substring/concat/length) + `match_all` | Q1028–Q1037 | 10 | **CSE-trigger sentinels** — the wrappers cause DataFusion CSE to insert the intermediate `ProjectionExec` that #13808 fixes | | D. Deep multi-CTE + `match_all` + regex/LIKE/CASE combos | Q1038–Q1047 | 10 | Deep plan trees, many projection candidates | | E. Edge cases (DISTINCT, HAVING, CASE, IN-list) | Q1048–Q1052 | 5 | Corner combinations | | F. `match_all` inside JOINs (self-join, LEFT JOIN, cross-stream) | Q1053–Q1058 | 6 | JOIN builders inject their own projections | | G. `match_all` + window functions (ROW_NUMBER, RANK, DENSE_RANK, SUM/COUNT OVER) | Q1059–Q1063 | 5 | Window planning inserts projections above filter | | H. Negated `match_all` (NOT, AND NOT, UNION of pos/neg) | Q1064–Q1066 | 3 | Rewriter's negation code path | | I. Multiple `match_all` in same query (AND, OR, nested) | Q1067–Q1069 | 3 | Two FST fields injected simultaneously | | J. `histogram` + `match_all` time-series | Q1070–Q1072 | 3 | Common customer pattern; histogram optimizer + FST rewriter together | ## Verification - All 70 queries have DuckDB oracle results auto-populated via `compute_counts.py`. - Two initial cross-stream JOIN queries (Q1055, Q1058) were rewritten to disambiguate the FTS field between streams — now they scope `match_all` inside a per-stream CTE before joining. - Audit for customer-token leakage (pharma, tomcat, o2_test, usgm, storenumber, tagname, cash, credit_card, etc.) returns **zero hits** across Q1003–Q1072. ## Implementation notes - Added generator script [tests/test-data/query-agent/gen_q1003_q1052.py](tests/test-data/query-agent/gen_q1003_q1052.py) alongside existing `gen_q201_q400.py` / `gen_q511_q556.py` convention. Deterministic regeneration. - Wide JSON churn across the other 12 category files is `compute_counts.py` re-baselining expected results against a fresh `BASE_TS` (`base_ts_override.json` is untracked); tuple-order shuffles are harmless under `results_mode: rowsort`. - Coverage is most meaningful under the `remove_filter_with_index_false` CI leg (added in #13810), where `match_all` actually reaches `RewriteMatchPhysical`. Under the default flag (`true`), the index optimizer removes `match_all` before the rewriter runs, masking the bug path. ## Test plan - [ ] `api_query_agent / single_node_opt_true` — green - [ ] `api_query_agent / single_node_opt_false` — green - [ ] `api_query_agent / remove_filter_with_index_false` — green (this is the leg that actually validates the rewriter under all 70 shapes) - [ ] Q1003–Q1072 all pass on both memtable and parquet phases 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>