mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-04 08:09:27 +00:00
Neha PandClaude Sonnet 4.6
afd3758972
## What Extends `tests/api-testing/tests/vortex/test_schema_evolution.py` with **34 new tests** across 10 new classes covering all realistic field drift scenarios for petabyte-scale schema evolution. Grows the schema evolution suite from 13 scenarios (14–25) to **46 scenarios (14–59)**. ## The scenario (shared across all groups) Think of a post office logbook. In January every letter has *From, To, Weight*. In February the form changes — Weight is dropped, Stamp Color is added. In March they go back plus add more fields. ``` Batch 1 [fa, fb, fc] — initial deployment Batch 2 [fa, fb, fd] — update: fc dropped, fd introduced Batch 3 [fa, fb, fc, fd, fe] — redeployment: fc back, fd kept, fe added ``` Each batch is flushed to a separate on-disk file before the next is ingested, forcing the query planner to reconcile three distinct schemas — the exact code path touched by `perf: Batch schema evolution checks` (v0.80.0). ## Test classes and scenarios added | Class | Scenarios | What it checks | |---|---|---| | `TestSchemaDivergenceConvergence` | 26–28 | Nullability per batch, cross-batch aggregation, field coexistence isolation | | `TestNullHandling` | 29–31 | IS NULL / IS NOT NULL exhaustive for absent fields | | `TestAggregationsOnPartialFields` | 32–36 | COUNT, SUM, MIN/MAX, COUNT DISTINCT on partial fields | | `TestGroupByOrderByEvolved` | 37–40 | GROUP BY / ORDER BY when grouped field is absent in some batches | | `TestTypeChangeOnReappearance` | 41–44 | Field returns as different type; OO type-preservation behavior documented | | `TestMultipleDivergenceCycles` | 45–46 | Field disappears and reappears multiple times | | `TestComplexQueriesEvolved` | 47–51 | SELECT *, CASE WHEN, subquery, HAVING on evolved schemas | | `TestTimeRangeEvolved` | 52–55 | Time-range filters correctly isolate individual batch windows | | `TestFTSEvolved` | 56–58 | Full-text search across batches with absent fields | | `TestHighFieldCountDivergence` | 59 | Wide schema: 20-field batch diverging to 10-field batch | ## Key findings documented in tests - **OO string-null behavior**: absent string fields are stored as `''` not SQL NULL — correct check is `IS NOT NULL AND != ''`; numeric fields use `IS NOT NULL` only - **OO type preservation**: when a field returns as a different numeric type, OO preserves the original type on disk; test_42 documents this behavior explicitly ## Code review fixes included - Added missing `fb` assertion in test_26 - Removed incorrect `!= ''` check on numeric `latency_ms` field (tests 27, 32–35) - Added comment in test_28 explaining the OO string-null quirk - Updated module docstring to list all 9 scenario groups - Added docstrings to all `_ingest` / `_ingest_three_batches` helper methods ## Why The v0.80.0 commit changed how OpenObserve batches schema reconciliation across multiple on-disk files. A bug in that path can: - Silently return fewer records than expected (data loss with no error) - Return wrong nullability for fields that reappear after being absent - Mis-count aggregations over fields that span only some of the files None of the 13 existing tests covered the diverge-then-converge pattern. This PR closes that gap across all query types. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>