mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-04 08:09:27 +00:00
6dfcb8a44a2cc4e976b9c772ac3b9d54b846933f
174
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  
|
fd85566742 |
feat: workflows version 1 (#13156)
This PR introduces workflows design at: https://github.com/openobserve/designs/blob/main/infrastructure/workflows/design.md --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
 Neha PandClaude Opus 4.7
|
f202a0c94f |
test: repro vortex utf8→i64 cast pushdown bug (o2-enterprise#2179) (#13210)
## Summary - Adds `TestVortexUtf8Int64CastPushdown` in [tests/api-testing/tests/query_agent/test_3_vortex.py](tests/api-testing/tests/query_agent/test_3_vortex.py) — regression coverage for the bug fixed by [o2-enterprise#2179](https://github.com/openobserve/o2-enterprise/pull/2179). - The fixture ingests a batch as **i64** and then a batch as **utf8** on the same field into a fresh stream, flushes to force vortex file generation, then runs `SELECT * FROM {stream} WHERE value > 0`. - The bug: DataFusion coerces the predicate to `CAST(value AS Int64) > 0`; vortex used to accept the cast for pushdown but had no `utf8→i64` cast kernel, so execution failed with `No CastReduce to cast constant array from utf8? to i64?` / `No CastKernel …`. - Assertion: status 200 + empty `partial_error` — the query must not 5xx or partially fail. ## Where this runs - Runs in the enterprise `api_query_agent_vortex` job (server booted with `ZO_FILE_FORMAT=vortex`). - Not enabled in OSS `api-testing.yml` (test file is enterprise-only per the file header). ## Test plan - [ ] `api_query_agent_vortex` on the paired ent PR passes - [ ] Ent PR (same branch name) needed to fire ent's `api-testing.yml` — the OSS `ent-build-status-check` only runs `cargo check/test`, not the Python vortex suite 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Neha PandClaude Opus 4.7
|
9af9a47fea |
test: add error path, explain analyze, and concurrency stress tests for query agent (#12809)
## Summary - **Phase 4 — Error Paths** (`test_4_error_paths.py`): 28 cases across 8 categories that send intentionally invalid SQL and assert 4xx not 5xx. Self-contained, runs in ~24s without data dependency. - **Phase 5 — Concurrency Stress** (`test_5_concurrency.py`): 26 queries (2 per category) run simultaneously at 5/10/15 worker levels via ThreadPoolExecutor. Each thread gets its own client. Catches deadlocks and thread-safety bugs. - **EXPLAIN ANALYZE** (`test_explain_analyze.py`): Verifies EXPLAIN/EXPLAIN ANALYZE return 200 with plan output and no `partial_error`. Also validates EXPLAIN ANALYZE on cross-schema dot-notation queries returns 4xx (not 500 crash). - **Fix**: `load_all_queries()` now skips JSON files without a `"queries"` key. ## Test plan - [x] Error path tests pass on pentest (28/28, all return 400) - [ ] CI run: `rye run pytest tests/query_agent/test_4_error_paths.py -v` - [ ] CI run: `rye run pytest tests/query_agent/test_5_concurrency.py -v` - [ ] CI run: `rye run pytest tests/query_agent/test_explain_analyze.py -v` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
 BhargavandClaude Opus 4.8
|
2a3416ee35 |
fix: Rum automation (#13064)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Neha PandClaude Sonnet 4.6
|
afd3758972 |
test: add schema evolution stress tests — scenarios 26–59 (diverge-then-converge + 9 scenario groups) (#12888)
## What Extends `tests/api-testing/tests/vortex/test_schema_evolution.py` with **34 new tests** across 10 new classes covering all realistic field drift scenarios for petabyte-scale schema evolution. Grows the schema evolution suite from 13 scenarios (14–25) to **46 scenarios (14–59)**. ## The scenario (shared across all groups) Think of a post office logbook. In January every letter has *From, To, Weight*. In February the form changes — Weight is dropped, Stamp Color is added. In March they go back plus add more fields. ``` Batch 1 [fa, fb, fc] — initial deployment Batch 2 [fa, fb, fd] — update: fc dropped, fd introduced Batch 3 [fa, fb, fc, fd, fe] — redeployment: fc back, fd kept, fe added ``` Each batch is flushed to a separate on-disk file before the next is ingested, forcing the query planner to reconcile three distinct schemas — the exact code path touched by `perf: Batch schema evolution checks` (v0.80.0). ## Test classes and scenarios added | Class | Scenarios | What it checks | |---|---|---| | `TestSchemaDivergenceConvergence` | 26–28 | Nullability per batch, cross-batch aggregation, field coexistence isolation | | `TestNullHandling` | 29–31 | IS NULL / IS NOT NULL exhaustive for absent fields | | `TestAggregationsOnPartialFields` | 32–36 | COUNT, SUM, MIN/MAX, COUNT DISTINCT on partial fields | | `TestGroupByOrderByEvolved` | 37–40 | GROUP BY / ORDER BY when grouped field is absent in some batches | | `TestTypeChangeOnReappearance` | 41–44 | Field returns as different type; OO type-preservation behavior documented | | `TestMultipleDivergenceCycles` | 45–46 | Field disappears and reappears multiple times | | `TestComplexQueriesEvolved` | 47–51 | SELECT *, CASE WHEN, subquery, HAVING on evolved schemas | | `TestTimeRangeEvolved` | 52–55 | Time-range filters correctly isolate individual batch windows | | `TestFTSEvolved` | 56–58 | Full-text search across batches with absent fields | | `TestHighFieldCountDivergence` | 59 | Wide schema: 20-field batch diverging to 10-field batch | ## Key findings documented in tests - **OO string-null behavior**: absent string fields are stored as `''` not SQL NULL — correct check is `IS NOT NULL AND != ''`; numeric fields use `IS NOT NULL` only - **OO type preservation**: when a field returns as a different numeric type, OO preserves the original type on disk; test_42 documents this behavior explicitly ## Code review fixes included - Added missing `fb` assertion in test_26 - Removed incorrect `!= ''` check on numeric `latency_ms` field (tests 27, 32–35) - Added comment in test_28 explaining the OO string-null quirk - Updated module docstring to list all 9 scenario groups - Added docstrings to all `_ingest` / `_ingest_three_batches` helper methods ## Why The v0.80.0 commit changed how OpenObserve batches schema reconciliation across multiple on-disk files. A bug in that path can: - Silently return fewer records than expected (data loss with no error) - Return wrong nullability for fields that reappear after being absent - Mis-count aggregations over fields that span only some of the files None of the 13 existing tests covered the diverge-then-converge pattern. This PR closes that gap across all query types. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Neha PandClaude Opus 4.7
|
7d3e67903e |
ci: add single_node_opt matrix mode to query agent tests (#12756)
## Summary - Adds a matrix strategy to the query agent SQL regression suite jobs in both `api-testing.yml` and `query-agent.yml` - Each job now runs with `ZO_FEATURE_SINGLE_NODE_OPTIMIZE_ENABLED=false` and `ZO_FEATURE_SINGLE_NODE_OPTIMIZE_ENABLED=true` - Verifies both DataFusion physical optimizer paths produce identical query results - Matrix legs run in parallel with `fail-fast: false` - All artifact/log names include the matrix label to avoid collisions ## Test plan - [ ] PR CI shows two `api_query_agent` matrix legs (single_node_opt_false and single_node_opt_true), both pass - [ ] `/query-agent` PR comment triggers both modes and posts per-mode results comments 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
 Hengfei Yang
|
ad743e9017 |
fix: add ttv inspect (#12743)
```
./target/debug/openobserve ttv-inspect -f data/openobserve/stream/files/default/index/stream_0_logs/2026/06/16/01/747246720234867916913ad.ttv
file : data/openobserve/stream/files/default/index/stream_0_logs/2026/06/16/01/747246720234867916913ad.ttv
total_bytes : 21163306
blob_count : 6
── file properties ──────────────────────────
row_group_size : 131072
── blobs ────────────────────────────────────
# name type offset length comp
0 meta.json o2-ttv-v1 4 1650 none
1 c057f4e04d9840849c7cb2311344be18.term o2-ttv-v1 1654 7466195 none
2 c057f4e04d9840849c7cb2311344be18.fast o2-ttv-v1 7467849 9752452 none
3 c057f4e04d9840849c7cb2311344be18.idx o2-ttv-v1 17220301 3898370 none
4 c057f4e04d9840849c7cb2311344be18.pos o2-ttv-v1 21118671 123 none
5 footer_cache o2-ttv-footer-v1 21118794 43493 none
── tantivy index (meta.json) ────────────────
segments : 1
total_docs : 480000 (deleted: 0)
opstamp : 0
- c057f4e0-4d98-4084-9c7c-b2311344be18: max_doc=480000 deleted=0
schema fields : 5
ip text [indexed,fast, tokenizer=raw]
request_id text [indexed,fast, tokenizer=raw]
trace_id text [indexed,fast, tokenizer=raw]
server_name text [indexed,fast, tokenizer=raw]
_timestamp i64 [fast]
```
|
||
|
4f3c0fcba3 |
fix(tests): verify match_all FTS result correctness and enforce Tantivy readiness (#12717)
## Summary
Fixes a test gap where `match_all` Tantivy queries could return wrong
results after a DataFusion upgrade without being caught by existing
assertions.
### Problem
After a DataFusion upgrade merge, the DataFusion access plan was no
longer used for `match_all` queries, causing Tantivy to return wrong
results. The existing tests only compared row counts against a DuckDB
oracle that emulates `match_all` with `LIKE '%term%'` — so wrong rows
that still matched the LIKE pattern passed undetected.
Additionally, `node/flush` does not generate Tantivy indexes
immediately. The default `ZO_MAX_FILE_RETENTION_TIME=600s` means
WAL→parquet→Tantivy conversion can lag, so FTS queries may fall back to
a non-Tantivy path during tests.
### Changes
1. **Content verification**
([conftest.py](tests/api-testing/tests/query_agent/conftest.py)): Every
hit from a `match_all` query that returns the `log` field is now checked
— each search term word must be present in `log`. A hard assertion fails
if Tantivy returns rows that don't contain the search terms.
2. **Tantivy readiness gate**
([test_2_parquet.py](tests/api-testing/tests/query_agent/test_2_parquet.py)):
After `node/flush`, the test now polls with a `match_all('warehouse')
LIMIT 1` query (up to 120s timeout) to confirm Tantivy indexes are built
before running FTS assertions. Previously only COUNT(*) was checked.
3. **Multi-word match_all emulation**
([compute_counts.py](tests/test-data/query-agent/compute_counts.py)):
`match_all('ACK batch')` now translates to `(log LIKE '%ACK%' AND log
LIKE '%batch%')` — per-token AND conditions matching Tantivy's
token-level search semantics, instead of the old phrase match `log LIKE
'%ACK batch%'`.
4. **Regenerated expected results** across all query JSON files.
## Test plan
- [ ] Run query agent test suite against an environment with Tantivy
enabled
- [ ] Verify `match_all` content assertions catch intentionally
corrupted results
- [ ] Confirm post-flush Tantivy readiness poll works within the timeout
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Huaijin <haohuaijin@gmail.com>
|
||
|
Huaijin
|
0831badd02 |
perf: upgrade datafusion to 54 (#12665)
- [x] ~add sort when to many groups, we can do this, but for now do not do this~ - [x] remove the unsafe for memory pool - [x] add approx_distinct improve, already fork, will add in next pr - [x] fork vortex to add cast fix, and export `.exclude_schemes([IntDictScheme.id()])` - [x] ~enable_physical_uncorrelated_scalar_subquery test~ |
||
 udev
|
6761ce2dbf |
fix: refactor online evals (#12654)
move online evals to enterprise repo |
||
|
Hengfei Yang
|
9c82beeeed |
fix: OTLP/JSON metrics silently dropping histograms and int values (#12615)
Close #12345 ## Problem OTLP histogram metrics sent over HTTP/JSON (e.g. otel-collector with `encoding: json`) created streams but stored 0 events, with no `_count`/`_sum`/`_bucket` streams. Int-valued counters/gauges were silently recorded as 0. The protobuf path was unaffected, which is why this only surfaced for collectors explicitly configured with JSON encoding. ## Root cause `otlp_json` deserializes the body directly into opentelemetry-proto's serde-derived types, which only partially implement the proto3 JSON mapping (64-bit integers encoded as strings, enums by name, zero/empty fields omitted). `Metric.data` is a flattened `Option` oneof, and serde converts any inner deserialization error of a flattened `Option` into `None` instead of failing — so the whole metric silently became `data: None`, while the stream itself had already been created by the schema setup. Same mechanism on `NumberDataPoint.value` turned string `asInt` values into 0. ## Changes - Upgrade opentelemetry stack 0.31 → 0.32, which fixes string-encoded u64 handling for plain histograms and summaries upstream. - Add `otlp_json_compat` normalizer for the gaps still present in 0.32 (each verified empirically, with unit tests): - string `asInt` values on data points and exemplars - `aggregationTemporality` as enum name - `Exemplar`'s value oneof is not flattened in 0.32 - missing `serde(default)` on `ExponentialHistogramDataPoint`/`Buckets`/`ValueAtQuantile`/`Exemplar`, which rejected omitted zero/empty fields - Count and log metrics whose data oneof failed to decode (`partial_success.rejected_data_points`) instead of silently skipping them, and stop overwriting specific partial-success error messages with the generic retention message. - Adapt to OTLP 0.32 API additions (`KeyValue.key_strindex`, `AnyValue::StringValueStrindex`). - Upgrade workspace reqwest 0.12 → 0.13, required because opentelemetry-http 0.32 only implements `HttpClient` for reqwest 0.13 (`rustls` feature replaces `rustls-tls-native-roots` via the platform verifier; `json`/`query`/`form` are now explicit features). The `segment` crate is pinned to reqwest 0.12, so `TELEMETRY_CLIENT` moved into `telemetry.rs` as a plain `reqwest::Client` that posts the Track message itself, replicating `segment::HttpClient::send`. - Remove the unused pyroscope profiling integration (optional feature, dependencies, agent setup, `ZO_PROF_PYROSCOPE_*` config). ## Verification - 10 unit tests in `otlp_json_compat` covering the full payload matrix (collector-style string integers, enum names, omitted zero fields, exemplars with and without trace context, already-normalized payloads, malformed input). - End-to-end: posted the issue's collector-style JSON payload to `/api/default/v1/metrics` — `_bucket` (3 events, correct cumulative counts with exemplars attached), `_count`, `_sum` populated for histogram and exponential histogram, summary quantiles stored, `asInt` counter stored as 42 instead of 0. - `cargo clippy -- -D warnings` clean; lib test failures are pre-existing local-environment issues that reproduce identically on main. |
||
|
Sai Nikhilandudev
|
57e57a56c9 |
fix: import and export functionality for score and score configs in llm evals (#12517)
Co-authored-by: udev <uddhav@openobserve.ai> |
||
|
Neha PandClaude Sonnet 4.6
|
146d5e7612 |
feat(tests): vortex file format test suite (scenarios 1–83) (#12472)
## Summary - Adds **239 new tests** covering the vortex columnar file format across 78 scenarios from the test plan - **`tests/api-testing/tests/vortex/`** — 9 test modules (83 tests): format conversion, schema evolution, data integrity, FTS, downsampling, API format awareness, server config, and performance/stress - **`tests/api-testing/tests/query_agent/test_3_vortex.py`** — 156 tests: the full 155-query SQL correctness suite + scenario 39 (self-join) run post-flush against vortex files - **`api-testing.yml`** — adds `--ignore=tests/vortex` to the OSS default-set run; vortex tests require the enterprise binary with `ZO_FILE_FORMAT=vortex` and are gated to enterprise CI only ## Test plan coverage | Module | Scenarios | |--------|-----------| | `test_format_conversion.py` | 1–13, 7, 8, 9, 10 | | `test_schema_evolution.py` | 14–25 | | `test_data_integrity.py` | 55–62 | | `test_fts.py` | 51–54 | | `test_downsampling.py` | 47–50 | | `test_api_format.py` | 71–73 | | `test_config.py` | 66–67 | | `test_performance.py` | 74–78 | | `test_3_vortex.py` | 26–46 (155 queries + JOIN) | ## Enterprise CI Two dedicated jobs added in the companion `openobserve/o2-enterprise` PR on the same branch: - **`api_query_agent_vortex`** — runs `test_3_vortex.py` with `ZO_FILE_FORMAT=vortex` - **`api_vortex_format_tests`** — runs the full `tests/vortex/` suite with 4-phase server lifecycle (parquet→vortex→parquet→vortex+compaction) OSS CI is unchanged except for the ignore rule — no vortex tests run against the OSS binary. ## Test checklist - [ ] `tests/vortex/` collects cleanly: `83 tests collected` - [ ] `tests/query_agent/test_3_vortex.py` collects cleanly: `156 tests collected` - [ ] Enterprise CI `api_query_agent_vortex` job passes - [ ] Enterprise CI `api_vortex_format_tests` job passes 🤖 Generated with [Claude Code](https://claude.ai/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Neha PandClaude Opus 4.7
|
008398163b |
test: add more queries to query regression (#12487)
## Summary - Adds **100 new structural-pattern queries** (Q411-Q510) across all 13 categories, bringing the suite from 425 to **525 queries** - Adds **cross-stream JOIN/UNION support**: `stream_offset` in data_gen.py, `STREAM2` in conftest, dual-table DuckDB oracle - 15 cross-stream queries (Q401-Q415) covering INNER/LEFT/RIGHT JOIN, EXISTS/NOT EXISTS, UNION ALL across streams - Bumps data_gen.py from 510 to 520 query capacity ## Changes - **Infrastructure**: [conftest.py](tests/api-testing/tests/query_agent/conftest.py), [compute_counts.py](tests/test-data/query-agent/compute_counts.py), [data_gen.py](tests/test-data/query-agent/data_gen.py) - **New file**: [cross_stream.json](tests/test-data/query-agent/queries/cross_stream.json) — 15 queries - **12 categories**: 85 queries across aggregation, basic_select, combined, cte_subquery, date_time, full_text_search, histogram, math_functions, pagination, string_functions, union, window ## Test plan - [x] `compute_counts.py` generates expected results for all 525 queries - [ ] Run `pytest tests/query_agent/ -v` against target environment (565 tests expected) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Neha PandClaude Opus 4.7
|
4d48bbb21b |
test: expand query agent regression suite (#12479)
## Summary - Expands the SQL regression catalog from **155 to 200 queries** across all 12 categories - Adds **7 queries** extracted from abc.sql (Q156-Q162) — synthesized from real-world OpenObserve SQL patterns with all data/fields/streams generated from scratch - Adds **230 auto-generated queries** (Q163-Q392) from templates to cover query format diversity - Both test phases pass: **200 memtable + 200 parquet = 400 total** ## Key changes - **data_gen.py**: 43 new fields (record_type, site_name, latency_ms, payload_size, alert_tags, phase_data, db_info, etc.), generates 3920 deterministic records - **compute_counts.py**: new translations for regexp_like, str_match_ignore_case, approx_distinct, json_get_str/json_get_json chaining; empty-aggregate result detection (OO returns 0 rows vs DuckDB's 1 NULL row) - **conftest.py**: PID-based stream name suffix for cross-invocation isolation (prevents data accumulation) ## Test plan - [x] test_1_memtable.py — 392 passed - [x] test_2_parquet.py — 392 passed - [x] All 7 abc.sql queries pass (Q156-Q162) - [x] No real stream names, field names, or data copied from external sources 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
0aab1ae261 |
feat: online evaluations (#12352)
- **feat(online-eval): add PipelineKind::Evaluation and dual realtime pipeline support** - **feat(online-eval): add provider, score_config, scorer, and online_eval_jobs DB entities** - **feat(online-eval): add config CRUD services and API handlers** - **feat(online-eval): reconcile jobs into hidden evaluation pipelines** - **feat(online-eval): add scorer preparation and execution backends** - **feat(online-eval): write scores and evaluator traces to system streams** - **refactor(online-eval): remove legacy eval templates backend** - **test(online-eval): add API integration coverage for eval resources** --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> |
||
|
Neha PandClaude Sonnet 4.6
|
73ce75c883 |
fix(tests): bump query_agent ingest timeout 120s → 300s (#12444)
## Summary - ENT CI runners are slower than local/OSS runners — the `ingest_query_agent_data` fixture was hitting the 120s timeout before 1550 records became searchable, causing all 310 query_agent tests to fail at setup - Bumped `wait_until` timeout to 300s ## Test plan - [ ] OSS `api_query_agent_tests` passes as before - [ ] ENT `api_query_agent_tests` no longer hits WaitTimeout on ingest 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Neha PandClaude Opus 4.7
|
1196045636 |
test: add org-level ingestion token API and Playwright tests (#12439)
PR #11691 introduces org-level ingestion tokens with o2oi_ prefix (37 chars). This commit adds comprehensive automated test coverage: ## API Tests (35 tests in test_ingestion_tokens.py) ### Token CRUD (9 tests) - Create token returns o2oi_ prefix and 37-char length - Create token persists name, created_by, enabled=true, is_default=false - Create token with name only (no description) — description is empty - Create token with hyphens and underscores allowed - Duplicate name returns 400 with "already exists" message - Empty name returns 400 with "cannot be empty" message - Whitespace-only name returns 400 with "cannot be empty" message - Name too long (>256 chars) returns 400 with non-empty error message - Parametrized: special chars (spaces, @, ., /) rejected with "alphanumeric" message ### List Tokens (3 tests) - GET returns data array - Created token appears in list - Token values are unmasked in list response ### Enable/Disable (5 tests) - Disable token via PATCH {enabled: false} - Enable disabled token via PATCH {enabled: true} - Double disable is a no-op (idempotent) - Toggle nonexistent token returns 404 - DELETE returns 405 (not supported) ### Ingestion Auth — Org Token (6 tests) - Ingest with org token (org_id:o2oi_token) succeeds - Ingest with disabled org token fails (401) - Ingest with nonexistent o2oi_ token fails (401) - Ingest to wrong org with valid token fails (401) - Org token blocked on search endpoint (401) - Org token blocked on dashboards endpoint (401) ### Ingestion Auth — Backward Compat (6 tests) - User password ingestion still works (pre-PR behavior) - Non-o2oi_ service account token ingestion works (falls through to user lookup) - User password on search succeeds (200) — mirrors org token blocking tests - User password on dashboards succeeds (200) — mirrors org token blocking tests - Non-o2oi_ SA token NOT blocked by o2oi_ prefix check on search (≠401) - Non-o2oi_ SA token NOT blocked by o2oi_ prefix check on dashboards (≠401) ### Multi-token and Security (6 tests) - Create 5 tokens, all visible in list - Disable one token, others remain enabled - SQL injection in token name rejected (special chars → 400) - Empty name, whitespace name, too-long name (with message assertions) - Special chars parametrized: spaces, @, ., / ## Playwright UI Tests (7 tests in ingestionTokens.spec.js) - Ingestion Tokens tab renders with table and columns (P0) - Create token via UI dialog with name and description (P0) - Create token with empty name — button disabled, enabled on fill, disabled on clear - Revealed token dialog — copy token with clipboard verification - Toggle token enable/disable from table actions (P0) - Manage Tokens button on ingestion page navigates to IAM - Ingestion page token selector dropdown shows available tokens ## Page Object (ingestionTokensPage.js) - data-test selectors for IAM nav, token list, create dialog, toggle, toast, revealed dialog - Registered in centralized PageManager --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> |
||
 Shrinath Rao
|
a647e6aee9 |
test: add retry_on_db_lock helper to support/wait.py (#12434)
## Summary
- Adds `retry_on_db_lock()` to `tests/api-testing/support/wait.py`
- Companion to the enterprise-side fix (openobserve/o2-enterprise#TBD)
that applies the helper across all RBAC entity base tests
## Root cause
The `ent_rbac` CI shard runs with `-n 2` (two pytest-xdist workers).
SQLite's single-writer model means parallel workers racing on write
operations produce transient `500 {"message": "database is locked"}`
responses (SQLite error codes 5 / 517). These surface as flaky assertion
failures across many PRs.
## Fix
`retry_on_db_lock(fn)` wraps an HTTP call and retries up to 3 times with
exponential backoff (0.5 → 1 → 2 s) only when the response is `500` with
`"database is locked"` in the body. Parallelism is preserved; the brief
retry window absorbs the time the other writer needs to commit and
release the lock.
## Test plan
- [ ] OSS CI passes with this branch
- [ ] Enterprise PR with the same branch name picks up the new utility
and applies it in all RBAC entity base tests
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
Neha PandClaude Opus 4.7
|
eb7cce9b96 |
test: query agent SQL regression suite — 155 queries, sqllogictest-style, two-phase (#12384)
## Summary
155-query SQL regression test suite covering 12 categories. Adding a new
query requires zero test code changes — drop a JSON entry into the right
category file and run `compute_counts.py`.
**Sqllogictest-style comparison**: DuckDB pre-computes full expected
result sets (not just row counts). The test runner extracts actual
values from OO hits, sorts both sides, and compares cell-by-cell with 5%
relative float tolerance. This catches value-level regressions that
row-count-only checks miss.
Two-phase execution: **memtable** (pre-flush) then **parquet**
(post-flush + Tantivy). **310/310 pass**.
## Architecture
```
tests/
├── api-testing/tests/query_agent/
│ ├── conftest.py # Shared fixtures, run_query(), tolerance comparison
│ ├── test_1_memtable.py # Phase 1: all queries pre-flush (FTS skipped)
│ └── test_2_parquet.py # Phase 2: post-flush (module-scoped fixture triggers flush)
└── test-data/query-agent/
├── data_gen.py # Deterministic data generator (single source of truth)
├── compute_counts.py # DuckDB oracle: computes expected result sets
├── base_ts_override.json # Locks BASE_TS so compute and tests share same timestamps
└── queries/ # 12 JSON files, 155 queries with expected results
```
### How it works
1. **Deterministic data** — `data_gen.py` generates 10 records per query
(1550 total) with a warehouse/supply-chain IoT schema. Each query Q{N}
gets isolated timestamps. `BASE_TS` is locked via
`base_ts_override.json` so compute and tests share identical timestamps
(prevents histogram bucket drift on minute boundaries).
2. **Unique stream per session** — `STREAM =
f"query_agent_test_{int(time.time())}"` prevents accumulated-data bleed
from prior runs.
3. **DuckDB oracle** — `compute_counts.py` loads the same dataset into
DuckDB, creates a pre-filtered VIEW per query (so time filters work for
CTEs, subqueries, UNIONs — no fragile SQL injection), translates
OO-specific functions, and writes `expected.results` blocks into the
query JSONs.
4. **Histogram translation** — `histogram(_timestamp, 'N unit')` is
translated to integer arithmetic matching OO's `date_bin` with origin at
2001-01-01T00:00:00 UTC. Verified against DataFusion CLI — 100/100
records match across all interval types (15s, 30s, 1m, 2m, 3m, 4m, 5m).
ISO timestamp post-processing converts epoch-µs bucket values to
`YYYY-MM-DDTHH:MM:SS` strings matching OO's serialization format.
5. **Sqllogictest comparison** — `run_query()` checks for
`expected.results`. When present, it extracts column values from hits as
strings, sorts both actual and expected (`results_mode: "rowsort"`), and
compares cell-by-cell with 5% relative float tolerance (handles
cross-engine floating-point drift in AVG/STDDEV/etc.).
6. **Two-phase testing** — `test_1_memtable.py` runs all queries
pre-flush (FTS sqllogictest skipped — Tantivy indexes don't exist yet).
`test_2_parquet.py` has a module-scoped `post_flush` fixture that
triggers `/node/flush`, re-polls until data is searchable, then runs all
queries with full assertions.
### Comparison modes
| Mode | Count | Mechanism |
|------|-------|-----------|
| **Sqllogictest** | 140 | Cell-by-cell comparison with 5% float
tolerance |
| **Skip sqllogictest** | 9 | DuckDB results for documentation, fall
back to legacy assertions |
| **Legacy** | 6 | 5 auto-interval histograms + Q122 (STRING_AGG
DISTINCT ORDER BY) |
**Skip sqllogictest queries** — known OO-DuckDB divergence:
| Query | Reason |
|-------|--------|
| Q037 | ROW_NUMBER tie-breaking differs between engines |
| Q058 | Self-join pairings non-deterministic across engines |
| Q072 | FULL OUTER JOIN histogram — NULL handling differs |
| Q080 | FTS `match_all` + histogram — Tantivy vs SQL LIKE |
| Q111 | ROW_NUMBER ties in pagination |
| Q117 | STRING_AGG ordering differs between engines |
| Q121 | STRING_AGG concatenation order |
| Q127 | UNION across 5 overlapping groups — boundary records compete
for LIMIT spots |
| Q152 | PERCENT_RANK tie-breaking differs between engines |
### Key design decisions
| Decision | Why |
|----------|-----|
| Full result-set comparison | Catches value-level regressions (e.g.,
ROUND producing wrong decimal) |
| `rowsort` mode | DuckDB and OO order ties differently — sorting both
sides eliminates spurious failures |
| 5% float tolerance | Cross-engine AVG/STDDEV/PERCENTILE differ at
least-significant digits |
| `''` = `'None'` normalization | OO serializes NULL as empty string;
DuckDB uses `"None"` |
| DuckDB VIEW for time filtering | WHERE injection into CTEs/subqueries
is fragile — VIEW filters at scan level |
| Dynamic columns excluded | `_timestamp` and `_time` differ between
compute time and test time |
| OO-dropped LAG columns excluded | `prev_tput`, `hours_2_back`,
`delta_from_prev`, `temp_diff` |
| Histogram → DuckDB integer math | DuckDB lacks `date_bin`; integer
formula verified via DataFusion CLI |
| BASE_TS override file | Prevents histogram timestamp drift when
compute and test run at different minutes |
| 1s time window margins | Minimizes cross-query data bleed while
capturing all 10 records |
| `pytest.fail()` on fixture failure | Stops all tests instead of
running 310 against an empty stream |
### Categories covered
| Category | Queries | Description |
|---|---|---|
| aggregation | Q001-Q020 | GROUP BY, COUNT, AVG, SUM, MIN, MAX, FILTER,
HAVING |
| basic_select | Q021-Q035 | WHERE, IN, BETWEEN, LIKE, ORDER BY,
DISTINCT, CASE |
| combined | Q036-Q045 | JOINs, multi-condition filters, combined
aggregations |
| cte_subquery | Q046-Q062 | CTEs, subqueries, derived tables, EXISTS |
| date_time | Q063-Q073 | Date parsing, formatting, extraction, interval
arithmetic, histogram |
| full_text_search | Q074-Q082 | match_all, match against fields |
| histogram | Q083-Q093 | histogram() bucketing, aggregation over
buckets |
| math_functions | Q094-Q102 | ROUND, SQRT, ABS, POWER, arithmetic
expressions |
| pagination | Q103-Q112 | LIMIT, OFFSET, combination with filters and
sorts |
| string_functions | Q113-Q122 | SUBSTR, CONCAT, UPPER, LOWER, TRIM,
LIKE patterns |
| union | Q123-Q134 | UNION ALL, UNION (dedup), CTE + UNION combinations
|
| window | Q135-Q155 | ROW_NUMBER, RANK, DENSE_RANK, LAG, LEAD, SUM/AVG
OVER |
## Test plan
- [x] All 155 queries pass in both phases (310/310)
- [x] 140 queries with sqllogictest cell-by-cell comparison
- [x] 17 histogram queries converted from legacy to sqllogictest
- [x] 9 queries with skip_sqllogictest for known OO-DuckDB divergence
- [x] 6 queries in legacy mode (auto-interval histograms + Q122)
- [x] BASE_TS locked via override file — no timestamp drift
- [x] Dedicated CI job (`api_query_agent_tests`) runs in parallel
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
Hengfei YangandClaude Opus 4.8
|
ad9fef2242 |
fix: fuzzy tests (#12405)
- [x] Bulk ingestion panic - [x] Create function panic - [x] Dashboards/Alerts/Incidents panic when list data - [x] Match http response error code - [x] Skip some endpoint that fuzz can't generate data --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Shrinath RaoandClaude Opus 4.7
|
b0118090e1 |
test: api-testing follow-ups — pytest-xdist dep + standardize test user display name (#12393)
## Summary
Companion PR to o2-enterprise PR 1841 (api-testing follow-ups). All
ENT-side test/CI work lands there; this OSS PR carries the two changes
that must live on the OSS side because they touch shared code.
Both PRs live on the same-named branch (`test/api-test-followups`) so
ENT CI's cp-merge picks up this branch and sees both changes together.
## What's in this PR
### 1 — Add `pytest-xdist` dependency
Adds `pytest-xdist>=3.0` to `tests/api-testing/pyproject.toml`. Needed
by the ENT api-testing CI's `api_integration / ent_rbac` shard to run
RBAC tests in parallel (`-n 4`).
CI-verified speedup: **~70 s → 21.7 s (~69% reduction)** on the ENT RBAC
shard's 280+ tests, with zero failures.
Foundation only — no test code in this repo uses xdist yet. Available
for any future shard that wants it.
### 2 — Standardize automated-test user display name to "Automation
TestUser"
Two helper files in `tests/api-testing/helpers/workflow/pages/` had a
personal name (`"Shyam P"` / `"Shyam Panjiyar"`) hardcoded in their user
/ service-account create payloads. Those helpers are consumed by the
workflow chain (`helpers/workflow/create_objects.py` +
`edit_objects.py`), which doesn't reliably clean up its created users —
so the names persisted in the IAM users page and looked confusingly like
real teammates' accounts.
Files changed:
- `tests/api-testing/helpers/workflow/pages/user_page.py`
- `tests/api-testing/helpers/workflow/pages/serviceaccount_page.py`
Both now use:
```
first_name: "Automation"
last_name: "TestUser"
```
Convention going forward: any account named `Automation TestUser` in the
IAM users page is automatically identifiable as test-created and safe to
clean up. Email addresses still use random suffixes
(`d4m21_{random_int}`) so uniqueness isn't affected — only the display
name standardizes.
## Test plan
- [ ] CI green on this branch
- [ ] Companion ENT PR 1841 (which depends on the `pytest-xdist` dep
added here) stays green
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Shrinath RaoandClaude Opus 4.7
|
61aa772d57 |
test: revamp api-testing suite (framework + 16 file rewrites + CI matrix) (#12372)
## Summary
Comprehensive rework of the `tests/api-testing/` suite. 30+ commits
implementing all 6 phases of the [revamp
plan](../../tests/ui-testing/MD_Files/api%20tests%20revamp/API_TESTS_REVAMP.md).
**Before → After:**
| | Baseline | Final | Δ |
|---|---|---|---|
| passed | 421 | **443** | **+22 ✓** |
| failed | 3 | **2** | -1 (only same pre-existing) |
| skipped | 53 | 39 | -14 (unskipped dead tests now run) |
| xfailed | 1 | 2 | +1 (new visible backend-bug signal) |
| runtime | 588s | **387s** | **-201s (34% faster)** |
**Zero tests that passed in baseline are now failing.** The 2 remaining
failures are the exact pre-existing ones
(`test_enrichment_table_join.py` backend panic + `test_workflow.py` SMTP
env-miss).
## What's in the diff
- **New `support/` framework** — `OpenObserveClient`, endpoint wrappers
(`alerts`, `dashboards`, `streams`, `search`, etc.), `wait_until()`,
payload factories, pytest fixtures, SSE parser. 14 framework selfcheck
tests.
- **16 of 16 OLD-tier test files rewritten** — using the new framework,
unique-suffix names, body validation, `wait_until` instead of
`time.sleep`, parametrize where appropriate.
- **29 files moved into 12 feature folders** mirroring the UI suite
layout
(`tests/{alerts,dashboards,ingest,orgs,pipelines,regression,rum,search,streams,users,workflow}/`).
- **−6,500 LOC net** (copy-paste + dead code removal; new framework adds
back ~1,500).
- **Ruff config** in pyproject.toml: T201 ban on `print()`, PT pytest
rules, B bugbear, UP pyupgrade.
## CI workflow changes
- New job `api_regression_tests`: `strategy.matrix` over the 4 env-flag
combinations (UTF8_VIEW × JOIN_MATCH_ONE) — now runs in parallel instead
of as 4 sequential server reboots in the main job.
- New job `api_fuzz_tests`: schemathesis OpenAPI-driven fuzz against
`/api-doc/openapi.json` (214 endpoints tested). Informational
(`continue-on-error: true`). Each run surfaces **7–10 distinct 5xx
panics** (e.g., empty `sql` body returning 500 instead of 400, traces
endpoints crashing on unescaped path params that get interpolated into
SQL). Filed to backend team for separate triage.
- Server `o2_*.log` files uploaded as artifacts on failure (was:
silently lost).
- `api_tests_summary` now requires BOTH integration + regression to
pass.
### CI infrastructure fixes
A short hardening pass landed after the initial CI run surfaced several
issues:
- **pytest-sugar count inflation** — dropped `--force-sugar` from both
pytest invocations. The flag double-registered sugar on top of the
auto-loaded entry point, inflating the CI test counter ~3x (display
showed 1364 when collection was 486). Counter is now honest.
- **schemathesis 4.x flag migration** — `--base-url` → `-u`,
`--hypothesis-max-examples` → `-n`, `--hypothesis-deadline` →
`--request-timeout`, `--data-generation-method` → `-m`, dropped
`--auth-type basic`. The fuzz job was failing immediately on flag-parse
before this.
- **Regression matrix binary path** — used `${{ github.workspace
}}/target/debug/openobserve` so the absolute path resolves correctly
under `working-directory: tests/api-testing/` (was: every matrix job
failed with "No such file or directory").
- **`permissions: contents: read`** added to all 4 workflow jobs —
closes CodeQL alert #369.
- **Removed redundant server boot** in `api_integration_tests` (leftover
from main's structure after the regression matrix split; was silently
failing to bind to :5080 and getting masked by the first server).
## Backend bugs surfaced (for backend team to track)
The rework uncovered and visibly flagged these as `xfail`/documented
test comments:
1. `DELETE /api/organizations/{id}` returns 404 unconditionally even for
just-created orgs — endpoint appears unimplemented.
2. `test_dashboard_missing_required_fields` triggers backend panic
(returns 500 instead of 400/422).
3. JS `getEnrichmentTableData()` restricted to `_meta` org AND not
exposed in the `/functions/test` endpoint runtime — `ReferenceError`
even with explicit `enrichment_tables: [...]`.
4. Bulk API silently sanitizes invalid `_index` values (`****` → `_`)
without flagging — user data lands in unexpected index.
5. `AVG` of a single value drifts by ~1e-14 (likely float reduction
order; documented tolerance in `test_advanced_sql.py`).
6. Org name validation rejects hyphens (only
`A-Z/a-z/0-9/spaces/underscores`) — not in spec.
Plus 7–10 additional 5xx panics surfaced per fuzz run (input validation
gaps — separate triage list in fuzz job logs).
## Test bugs uncovered + fixed
- `test_pipeline.py`: had two function definitions named
`test_pipeline_creation_and_action_with_schedule` — Python's last-wins
meant the first test's body never executed for years.
- `test_alertsV2.py`: `assert alert_disabled == "False"` was indented
inside an `else:` branch that never executed because the response is
always a dict; AND `enabled` is a bool, not the string `"False"`. Two
layers of broken.
- `test_bulk.py`: variable typo (`valid_bulk_payload` reused on the POST
call) made the invalid-index test silently duplicate the valid case.
- `test_organisations.py`: `test_create_organization` used pre-pytest-4
`yield`-style fixtures — silently ignored by pytest 8.
- `test_js_result_array.py`: 8 tests skipped with `"Temporarily disabled
- failing test"` — the actual cause was that JS functions are restricted
to `_meta` org. One-line fix unskipped all 8.
- `test_fuzzy.py`: 4 tests `@pytest.mark.skip`-ed because they used a
1-minute time window against older static test data → 0 hits → flake →
skip. Wider window unskips them.
- 547 LOC of zombie commented-out test blocks deleted (15 commented `#
def test_` definitions, including one orphan case where the `def` was
commented but the indented body wasn't).
## Folder layout
```
tests/api-testing/
├── support/ # NEW framework
│ ├── client.py # OpenObserveClient
│ ├── endpoints/ # 7 per-resource wrappers
│ ├── factories.py # unique_name, time_window, *_payload
│ ├── wait.py # wait_until + WaitTimeout
│ ├── sse.py # read_sse_response
│ └── fixtures.py # client, temp_user_email, temp_stream_name, ...
├── helpers/ # Non-test shared code (was scattered in tests/)
│ ├── sourcemap_helpers.py
│ └── workflow/ # workflow chain + pages/ live here now
├── fixtures/
│ └── rum/ # Standalone RUM ingest scripts (was in tests/)
└── tests/
├── alerts/ # 3 files
├── dashboards/ # 4 files
├── ingest/ # 4 files
├── orgs/ # 2 files
├── pipelines/ # 10 files
├── regression/ # 2 files (env-flag matrix targets)
├── rum/ # 3 files
├── search/ # 8 files
├── streams/ # 1 file
├── users/ # 1 file
├── workflow/ # 3 files
├── conftest.py
└── test_framework_selfcheck.py
```
## Test plan
- [x] `rye run pytest --collect-only` — 486 tests discovered
- [x] Full default-set run (`--ignore=tests/regression`) — 443 pass / 2
fail / 39 skip / 2 xfail in 387s
- [x] `rye run pytest -m framework` — 14/14 selfcheck tests pass
- [x] `rye run ruff check support/ tests/test_framework_selfcheck.py` —
clean
- [x] CI run on this PR — `api_integration_tests` + all 4
`api_regression_tests` matrix jobs green
- [x] CI run — `api_fuzz_tests` runs cleanly (informational; surfaces
7–10 unique 5xx panics per run)
## Note on test count
The `test_streaming.py` rewrite folded 7 "tests" (`test_ingest_data`,
`test_ingest_join`, `test_disable_streaming`,
`test_update_max_query_range`, `test_enable_streaming`,
`test_delete_stream`, `test_delete_stream_join`) into pytest fixtures.
Each only checked `status_code == 200` of a setup/teardown operation —
they were imperative steps masquerading as tests. The fixtures still
perform the same operations and assert their success; they're just no
longer counted as discrete pytest items. **Functional coverage is
identical.**
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Shrinath RaoandClaude Sonnet 4.6
|
3e172523d5 |
test: sync regression workflow runners and TestDino with playwright.yml (#12370)
## Summary - **Runners**: `merge_and_upload_reports` and `playwright_summary` now use `ubicloud-standard-4` (was `-8`), matching `playwright.yml` - **TestDino rerun optimization**: added `check_rerun` step + full `last-failed` logic (same as `playwright.yml`) — on re-run-failed, only the previously failed test in each shard is re-executed - **Upload step**: added failures-only filter (`node -e` filterSuite), `startTime` ISO 8601 conversion, and `--verbose`; now uploads `report-failures-only.json` instead of the full report ## Test plan - [ ] Trigger a regression run and confirm it completes with the new runner sizes - [ ] Intentionally fail a test, re-run failed jobs, and verify TestDino optimization runs only the failed test - [ ] Confirm Upload to TestDino step produces a filtered report 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Shrinath RaoandClaude Sonnet 4.6
|
cc1f09c5c1 |
test: fix selector drift across ENT specs, re-enable enterprise tests, and improve CI (#12358)
## Summary Broad test stabilization pass covering Quasar component selector drift in ENT UI specs, re-enabling previously disabled enterprise tests, and CI infrastructure fixes. ### UI test fixes (selector drift) Quasar's O2 component library wraps native elements — `OInput`, `OTextarea`, `ODropdown`, `OToast`, `OToggleGroup`, `OToggleGroupItem`, `ODrawer`, `ODialog`, `OFile` — in ways that break naive attribute/role selectors. This PR updates affected specs to use the correct locators. | Spec | Fixes | |------|-------| | `service-graph.spec.js` | `OToggleGroupItem` selector, `ODrawer` close btn, `beforeAll` timeout | | `alerts-ui-operations.spec.js` | Remove `waitForResponse` overhead from enterprise block | | `logoManagement.spec.js` | `OInput` / `OFile` / `ODialog` / `OToast` locators | | `jobSearch.spec.js` | `OInput` fill targets wrapper div, `OTable` index-based rows | | `searchPatterns.spec.js` | `OToggleGroup` same-element selector, pattern details dialog | | `pipelineImport.spec.js` | `OInput` / `ODropdown` / Monaco / `OToast` selector drift | | `regexPatternManagement.spec.js` | Correct test-data import file path | | `cipherKeys.spec.js` | `OInput`/`OTextarea` `-field` suffix, `OToast` selector, 403 skip guard, all-toasts polling | ### Enterprise tests re-enabled - Removed `/* ... */` block comment disabling `"Alerts & Incidents Page Navigation"` describe block (`@enterprise`) in `alerts-ui-operations.spec.js` - 3 tests restored: page navigation, element visibility, round-trip nav back to Alerts - Requires `O2_INCIDENTS_ENABLED=true` (set in ENT CI) ### CI improvements - **playwright.yml**: Install `ffmpeg` via local Playwright binary (required for `video: retain-on-failure` recording) - **api-testing.yml**: Reduce runner from `ubicloud-standard-8` → `ubicloud-standard-4` - **test_stream_settings_dedup.py**: Fix field-filtering logic and assertion count in `test_multiple_distinct_fields_no_duplicates` ## Paired ENT PR Full ENT-only test restoration (SDR, RUM, cipherKeys, etc.): https://github.com/openobserve/o2-enterprise/pull/1835 ## Test plan - [ ] ENT CI: `"Alerts & Incidents Page Navigation"` enterprise block passes - [ ] Playwright CI: video artifacts generated on failure (ffmpeg install works) - [ ] API tests: `test_multiple_distinct_fields_no_duplicates` passes 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
|
1afabd35ce |
feat: bloom filter pruning layer above tantivy (#12121)
## Summary Design at: https://github.com/openobserve/designs/blob/main/infrastructure/bloomfilter/DESIGN.md A bloom-filter pruning layer that sits **between `file_list` and tantivy** in the search path. It drops candidate files before any `.ttv` is opened, for **positive equality / IN** lookups on high-cardinality fields (`trace_id`, `request_id`, …). It is built **on top of** tantivy: the `.bf` is constructed by iterating the tantivy term dictionary, so a field must be a tantivy secondary index (`index_fields`) **and** opted into `bloom_filter_fields`. Only the intersection is built — a bloom-only field produces nothing. To make that intersection non-empty by construction, enabling a field for bloom now folds it into `index_fields` automatically (see Config). Values are not inspected; the operator chooses which fields are worth it (see guidance below). Design notes live in the module-level docs under `src/infra/src/bloom/` (`mod.rs`, `writer.rs`, `reader.rs`, `sbbf.rs`). ## Design - **Transposed (block-major) `.bf` layout.** All files in a `.bf` share `num_blocks = B`, so `block_index = fastmap(hash(value), B)` is identical across files and the blocks a query needs form **one contiguous row**. The search side reads **one `M × 32`-byte row per `.bf`** to answer membership for all `M` files — prune cost is `O(groups)`, not `O(files)`. (`infra::bloom`, own ~120-LOC SBBF over gxhash64.) - **Sub-grouping.** A hour's new files are sorted by record count and split into chunks of ≤ `ZO_BLOOM_FILTER_MAX_FILES_PER_BF` (default 256), one `.bf` each. Bounds compactor build memory (`M × B × 32`) and `.bf` object size; query reads become `ceil(files / 256)` rows. - **Footer-only reader + batched fetch.** Footer (≈ `M × 12` B) comes from `BLOOM_FOOTER_CACHE` or one 16 KB suffix GET (with a precise re-read when a large footer overflows the probe); rows are pulled with one `cache_storage::get_ranges` call. `cache_storage` was reworked to do real batched cache reads (memory: 1 lookup + N slices; disk: 1 `File::open` + N `pread`). - **B sizing.** `num_blocks_for(max_records_in_chunk, ZO_BLOOM_FILTER_FPP)` (default fpp `0.01`) — record count is a safe NDV upper bound, so the SBBF never saturates for high-cardinality fields. - **Versioned, with orphan cleanup.** A file's `bloom_ver` is write-once — a row is never re-stamped, so dumped rows keep pointing at the `.bf` they were built into. But `bloom_ver` is **not** append-only at the bucket level: a merge produces new output files with a fresh `bloom_ver` and deletes the small inputs, so the old inputs' `.bf` can become orphaned. After each merge round the compactor retires any `.bf` whose `bloom_ver` is no longer referenced by a live `file_list` row in the bucket (enqueued through the existing `file_list_deleted` path). `bloom_ver` is preserved through every file_list INSERT/SELECT path (incl. the SQLite read-cache) and the dump parquet. - **Observability.** Prometheus metrics for prune (`bloom_prune_keep_ratio`, `bloom_prune_duration_seconds`); `bloom-inspect` CLI dumps a `.bf` footer (fields, B, row bytes, file names); footer-cache hit/miss via the shared `BytesCache` tagged metrics; `Local::get_ranges` now reports `STORAGE_READ_*` (previously bypassed). ## When it helps (measured, S3, no disk cache) Same stream `benchtest`, 170 files / 14.64 GB index, single matching row. The only variable is whether the field's values are time-correlated: | Field | Type | tantivy only | bloom + tantivy | Outcome | |---|---|---|---|---| | `trace_id` (16-byte random) | random | **2584 ms** (opens all 170 `.ttv`) | 24 ms prune + 65 ms = **89 ms** | **~29× faster** | | `request_id` (`…-7e74-…`, UUIDv7) | time-ordered | **154 ms** (sparse index range-prunes) | 42 ms + 154 ms = **196 ms** | ~1.3× **slower** | The whole 170-file group was answered by **one row read of 5440 bytes** (`170 × 32`) + a cached footer. **Guidance — enable only when all three hold (advisory; not enforced):** 1. **High cardinality** field. 2. **Random values** (not time-ordered) — so tantivy can't range-prune and must scan every file. Time-ordered ids (UUIDv7, snowflake) are already pruned by tantivy's sparse index in memory → bloom is pure overhead there. 3. **Many files per hour** — hourly volume ≳ 10 × `ZO_COMPACT_MAX_FILE_SIZE` (≳10 `.ttv` files), so there's something to prune. Bloom turns N file opens into one row read; at 1–2 files/hour it's net overhead. The 29× result above had 170 files/hour all answered by one `.bf`. ❌ Low-cardinality / non-indexed / range / `LIKE` / regex / `!=` / OR: no effect. ## Config - `ZO_BLOOM_FILTER_ENABLED` (default true) — global gate. - Per stream: a field must be in **both** `index_fields` and `bloom_filter_fields`. Adding a field to `bloom_filter_fields` auto-adds it to `index_fields` (bloom is built on the secondary index), and the default index fields include the default bloom fields. - `ZO_BLOOM_FILTER_FPP` (default 0.01) — target false-positive probability; sizes the SBBF block count (smaller fpp = fewer false survivors, larger `.bf`). Out-of-range falls back to 0.01. - `ZO_BLOOM_FILTER_MAX_FILES_PER_BF` (default 256) — files per `.bf`; trades query reads vs build memory. - `ZO_BLOOM_FILTER_DEFAULT_FIELDS` (default empty) — fields bloomed for every stream. - `ZO_BLOOM_FOOTER_CACHE_MAX_SIZE` (default 1% mem, clamped 32–256 MB). Removed in this PR: `ZO_BLOOM_FILTER_DISABLED_ON_SEARCH`, `ZO_BLOOM_FILTER_NDV_RATIO` (B is now sized from record count + fpp, not an ndv ratio). ## Test plan - [x] `cargo build`, `cargo clippy -p infra -p openobserve --lib -- -D warnings`, `cargo fmt --check` clean - [x] Unit tests: `cargo test -p infra --lib bloom`, `cargo test -p openobserve --lib bloom` - [x] E2E on S3 (results above): random field prunes 170→1 and is ~29× faster; transposed read = 1 request / 5440 B per group - [x] Reviewer: confirm gating expectations (operator opts fields in; no auto value detection) - [x] Test: orphan bloom: 2 * 256MB -> merge to 512MB -> merge to 1024MB, the first .bf should be deleted - [x] Test: orphan bloom: 6 * 256MB -> merge to 1GB + 512MB -> 512 MB merge to 1024MB, the first .bf include 2 files(1GB + 512MB), after merge again, the first still have 2 files but one of them already deleted. we should keep the first .bf, because still have files in use. ## Notes - `.bf` format is `VERSION 0x01` (transposed). Not compatible with any pre-merge experimental `.bf`; rebuild by wiping the `bloom/` dir and resetting `bloom_ver=0`. - Hash is gxhash64 (project-wide `config::utils::hash`), not the Parquet spec's xxhash64 — `.bf` is internal, never read by external tools. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
|
e9266c4510 |
fix: restore retry guard in enrichment table URL validation (#12218)
Fixes https://github.com/openobserve/o2-enterprise/issues/1796 (item 1: "Reload Existing URLs" returns error). ## Summary Commit [`85d261b5d0`](https://github.com/openobserve/openobserve/commit/85d261b5d0e4b68dbc79ca18b844fc53e2368ee9) (vulnerability scan fixes, Apr 14 2026) removed the `!retry` guard from enrichment table URL validation. This broke the **reload** update mode because the frontend sends an empty URL when retrying/reprocessing existing URLs — the backend already has them stored and just needs to re-trigger processing. ## Root Cause Before the breaking commit, URL validation was guarded by `!retry`: ```rust if !retry && let Err(err_msg) = validate_enrichment_url(&request_body.url) { return MetaHttpResponse::bad_request(err_msg); } ``` The security fix removed the guard under the assumption that skipping validation on retry was an SSRF bypass vector. However, the retry flow sends `url=""` intentionally — the existing URLs are in the database. ## Fix Re-adds the `!retry` guard around URL validation. In retry mode, URL validation is skipped entirely (the URLs come from DB). All other modes (create, append, replace, replace_failed) still require a non-empty, valid URL with full SSRF protection. ## Test Plan - [ ] Create enrichment table from URL - [ ] Edit table → Reload existing URLs → Should succeed (no "URL cannot be empty") - [ ] Edit table → Add new URL → Should still validate URL format - [ ] Edit table → Replace all URLs → Should still validate URL format - [ ] Edit table → Replace failed URL → Should still validate URL format 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com> |
||
|
Subhra264
|
6ec195ac81 |
fix: revamp rbac (#11610)
Revamps the complex rbac logic to make it simpler. Also, - Adds RBAC for anomaly apis - Fixes https://github.com/openobserve/openobserve/issues/11710 |
||
 Ashish Kolhe
|
3161d13785 | fix: refactor ssrf (#11734) | ||
|
Neha PandClaude
|
031bad113f |
test: add API tests for stream settings deduplication (#11516)
## Summary - Add API tests verifying that stream settings fields (index_fields, bloom_filter_fields, full_text_search_keys) are deduplicated when the same field is added multiple times - Tests cover: single field dedup, multiple fields, repeated adds (4x), remove+re-add, empty payloads, and partial updates - All tests use existing schema fields from the seed data stream (`stream_pytest_data`) ## Test plan - [x] All 10 tests pass against testing environment - Tests run via `cd tests/api-testing && rye run pytest tests/test_stream_settings_dedup.py -v` 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
udev
|
5665a2059f | ci: fix flaky tests (#11504) | ||
 ktx-kirtan
|
8bdd333a0f |
Revert "feat: dashboard validation sync from BE" (#11437)
Reverts openobserve/openobserve#11034 |
||
   
|
78b7b7b622 |
feat: dashboard validation sync from BE (#11034)
Design at: https://github.com/openobserve/designs/blob/main/dashboards/dashboard-validation-sync --------- Co-authored-by: ktx-vaidehi <vaidehi.akhani@kiara.tech> Co-authored-by: udev <uddhav@openobserve.ai> Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Abhay Padamani <undefined@users.noreply.github.com> |
||
 ByteBaker
|
d25fbf97c1 |
fix: do not show service account token post creation (#11110)
So far, service account token could be viewed and copied even after an SA was created. This was bad design and posed risk of impersonation. The changes prevent this from happening by returning only the redacted token in the API response. Full token with create API itself, and removing GET API to get the service account details (which returned raw token). The same affects token rotation as well. |
||
|
|
b1fe77871a |
test: add E2E tests for RUM sourcemap feature (#11014)
## Summary - Add comprehensive E2E test coverage for RUM sourcemap functionality (26 total tests) - API tests: 20 pytest tests (upload/list/delete/resolve operations + RBAC) - UI tests: 6 Playwright tests for error tracking UI - Helper scripts for RUM data ingestion ## Implementation Details ### API Tests (pytest) - **test_sourcemap_api.py**: 15 tests covering: - Sourcemap upload (multipart/form-data) - List sourcemaps - Delete sourcemaps - Stacktrace resolution with sourcemaps - **test_sourcemap_rbac.py**: 5 tests for role-based access control - **ingest_rum_errors.py**: Helper to inject synthetic RUM error data for UI tests - **ingest_rum_performance.py**: Helper to enable RUM feature in UI ### UI Tests (Playwright) - **sourcemap-ui.spec.js**: 6 tests validating: - Error tracking page loads - Query execution - Error list display - Error detail navigation - Metadata display - Stack trace display - **rumPage.js**: Updated page object with API response interception - Workaround for Vue 3 @click handlers not responding to Playwright clicks - Navigate via URL using error IDs extracted from API responses - Tests run in **parallel mode** with proper assertions (no graceful skips) ## Test Execution ```bash # API tests (pytest) cd tests/api-testing pytest tests/test_sourcemap_api.py -v # 15 tests pytest tests/test_sourcemap_rbac.py -v # 5 tests # UI tests (Playwright) cd tests/ui-testing npx playwright test playwright-tests/RUM/sourcemap-ui.spec.js --workers=3 # 6 tests ``` ## Technical Notes - Vue 3 production builds don't respond to Playwright's synthetic click events on @click handlers - Solution: Intercept API responses to extract error IDs, then navigate directly via URL - All tests use hard assertions - tests fail if data doesn't exist (no graceful skips) --------- Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com> Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Shrinath Rao <undefined@users.noreply.github.com> |
||
 ktx-akshay
|
6c5500eed3 | test: add API tests for dashboard multi-window VRL functionality (#10864) | ||
|
Subhra264
|
3760b55cf9 |
fix: multi-window dashboard should support vrl functions (#10860)
Currently dashboard with Multi-window set up does not really run the result array vrl functions. This pr adds support for vrl function when `/_search_multi_stream` is used. When `per_query_response` is true, and the vrl function is result array vrl function, the function is applied on the whole combined response (including both the time windows). The input of the result array vrl function becomes a 2d array where each child array represent a time window. Otherwise, if `per_query_response` is false, or the vrl function is not result array vrl function, then the function is applied individually on every result of every time-window queries. |
||
|
Neha PandClaude
|
33608a09b2 |
test: add max_query_range enforcement tests (#10907)
Add comprehensive pytest tests for max_query_range stream setting that silently truncates queries exceeding the configured time limit. - TestLogsMaxQueryRange: 4 tests verifying truncation for logs streams - TestMetricsMaxQueryRange: 1 test for metrics stream truncation - TestTracesMaxQueryRange: 1 test for traces stream truncation - TestMaxQueryRangeNegativeCases: 7 edge case tests (zero value, negative values, inverted time range, non-existent stream, boundary conditions) Tests inject backdated data (3h, 2h, 90min, 45min, 30min, 15min, 5min, now) and verify that with max_query_range=1h, only data within 1 hour is returned. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Neha PandClaude
|
4f931b2d9d |
test: add Alert History API tests with trigger setup fixture (#10812)
Adds comprehensive tests for GET /api/{org_id}/alerts/history:
- Contract validation for response schema
- Sort parameter tests (13 sort_by fields, asc/desc)
- Error path tests (invalid params, unauthorized)
- Pagination edge cases
Includes setup fixture that creates and triggers an alert to ensure the
triggers stream exists in fresh CI environments.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
Neha PandClaude
|
fef55301ab |
test: add time_shift dashboard and multi_time_range alert methods (#10782)
- Add create_dashboard_with_time_shift() to dashboard_page.py - Add create_scheduled_sql_alert_multi_time_range() to alertV2_page.py Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Neha PandClaude
|
f7b63ec883 |
test: add URL-based enrichment table methods to page object (#10770)
- create_enrichment_table_from_url(): Create table from public CSV URL
- get_enrichment_table_url_status(): Check async job status
- wait_for_url_enrichment_job(): Wait for job completion
API: POST /api/{org}/enrichment_tables/{table_name}/url Returns 202 (job
runs async)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
Neha PandClaude
|
4ff58d0510 |
test: add VRL alert test cases (#10759)
- Add 3 new VRL test functions in test_alertsV2.py: - test_new_alert_create_with_vrl_no_trigger - test_new_alert_create_with_vrl_trigger - test_update_alert_vrl_function - Fix base64 encoding in alertV2_page.py VRL methods - Add optional webhook_url param to destination_page.py --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Neha PandClaude
|
da65a0ec8d |
test: add broadcast joins tests (#10557)
Add comprehensive API tests for logs-to-logs joins with asymmetric data sizes: - 29 test cases covering INNER/LEFT/RIGHT/FULL joins, aggregations, filters - Subquery tests with IN/NOT IN, substr, length, CONCAT functions - Tests run with both ZO_FEATURE_JOIN_MATCH_ONE_ENABLED=true and false - Test data: logs_small.json (10 records) and logs_large.json (50 records) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
ByteBakerandgithub-actions[bot]
|
cf98c39c81 |
fix: semantic groups, service graph, dedup, dashboard cleanup (#10502)
## Summary
- **Semantic Groups**: add JSON export button, fix broken JSON import
(`handleJsonUpdate` was a no-op), fix "Applied 0 changes" toast bug
- **Service Graph**: remove unused `availableStreams` from
`build_topology` return and API response, skip `_`-prefixed orgs in
processor
- **Dedup Config**: change `time_window_minutes` from `Option<i64>` to
`Option<i32>`, add `deserialize_optional_i32` for defense-in-depth,
sanitize empty/NaN values on frontend before sending
- **Dashboard**: remove dead `/export` endpoint (identical to GET,
unused by frontend)
## Test plan
- [x] Semantic groups: export downloads JSON, import with file upload
works, import shows correct change count
- [x] Service graph: topology loads, stream dropdown works (populated by
`loadTraceStreams`)
- [x] Dedup: clearing time window field and saving no longer returns 422
- [x] Dashboard: GET `/{dashboard_id}` still works, `/export` route
removed
- [x] `cargo check` clean, `cargo clippy` clean
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
|
||
|
Hengfei Yang
|
f00fe6b1ec | fix: cache with query size (#10384) | ||
|
Ashish Kolheandgithub-actions[bot]
|
c34988a3ea |
fix: insights traces changes (#10304)
### **PR Type** Enhancement ___ ### **Description** - Introduce `timeStart`/`timeEnd` in filters - Enable baseline-only and comparison modes - Conditional single vs comparison query logic - Always show Insights button and update translations ___ ### Diagram Walkthrough ```mermaid flowchart LR start["Open analysis"] start -- "Has time filters?" --> comp["Comparison mode"] start -- "No time filters" --> base["Baseline-only mode"] comp -- "Query both ranges" --> merge["Merge & score"] base -- "Use baseline only" --> sort["Sort by baseline percent"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>3 files</summary><table> <tr> <td><strong>useLatencyInsightsAnalysis.ts</strong><dd><code>Support time-based filters and baseline-only mode</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-343474bc1f120f5ecebfc595bf0d3151b2db096d9226369c496b7a0cfaa33899">+87/-54</a> </td> </tr> <tr> <td><strong>useLatencyInsightsDashboard.ts</strong><dd><code>Conditional single/comparison query generation</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-960dec4a653d7c079157499a2473d0e87bd8b61463e1c3f633e7682ba763a4b1">+58/-7</a> </td> </tr> <tr> <td><strong>TracesMetricsDashboard.vue</strong><dd><code>Always show Insights button and update labels</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-7b6f29a2737d9501b18dbd8622eba4965e4bc2b92115627ae7d2e70cc3382087">+68/-54</a> </td> </tr> </table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>11 files</summary><table> <tr> <td><strong>de.json</strong><dd><code>Add `volumeInsights.insightsButtonLabel` translation</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-78f24ecb2dbcfc38fdf3ec22b2559a5b40d236ea689d1c55f5d3e87168310264">+3/-1</a> </td> </tr> <tr> <td><strong>en.json</strong><dd><code>Add insights translation keys to English</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-892aa3d75ab6e129ef9116aa21bec1b9a15ba69ce5366d2a19128078c690d824">+2/-0</a> </td> </tr> <tr> <td><strong>es.json</strong><dd><code>Add insights translation keys to Spanish</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-ac2a2167b8b973e3f776f9fe3b43bef4576470d8ad33f3d7cfaff6edff0e5778">+3/-1</a> </td> </tr> <tr> <td><strong>fr.json</strong><dd><code>Add insights translation keys to French</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-db80e762b88ddf183820c050cdc6b215e6b02b365b5e73f5152af46b0d26f1a2">+3/-1</a> </td> </tr> <tr> <td><strong>it.json</strong><dd><code>Add insights translation keys to Italian</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-2de78074a5f887afddce60d694ead9322e6ab223eb70251c13819f3028bdc70d">+3/-1</a> </td> </tr> <tr> <td><strong>ja.json</strong><dd><code>Add insights translation keys to Japanese</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-092e49c5f9fbc1b539ab9c2866d108d0ce1d314022b4c294eb4611b285e7bbd3">+3/-1</a> </td> </tr> <tr> <td><strong>ko.json</strong><dd><code>Add insights translation keys to Korean</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-50e27ca1b4825a5026f3a251bfcf79c97cab07cfade2ebd512d332453d707c48">+3/-1</a> </td> </tr> <tr> <td><strong>nl.json</strong><dd><code>Add insights translation keys to Dutch</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-a1706021687ae5d8a1c006c49eab3946174fecd895f9b70a89c2e76d97250906">+3/-1</a> </td> </tr> <tr> <td><strong>pt.json</strong><dd><code>Add insights translation keys to Portuguese</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-38e804f64cb5e0e451849fc84e9b46e53c954a4a4263d51c18665ad45f4b4815">+3/-1</a> </td> </tr> <tr> <td><strong>tr.json</strong><dd><code>Add insights translation keys to Turkish</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-6f0f131e9e2591480c4c5d1eaffc72d02dc31cbd8d3f90d8f012f08cabac8818">+3/-1</a> </td> </tr> <tr> <td><strong>zh.json</strong><dd><code>Add insights translation keys to Chinese</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-8818055db58f83230aeea4ebfd4cec31d924e8c47b6f91298e356959ab25339e">+3/-1</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
|
Subhra264
|
7ebb4c9940 |
fix: backfill icons and edit timerange (#10209)
- Use consistent styling for backfill UI. - The timerange picker on edit page should display the backfill timerange by default. |
||
|
Neha PandClaude
|
59f86c491d |
test: add API tests for pipeline backfill functionality (#10177)
### **User description**
Add comprehensive API tests for backfill core functionality:
- Basic backfill job creation and status retrieval
- Backfill execution with data verification
- List all backfill jobs for organization
- Pause and resume backfill jobs
- Delete backfill jobs
- Invalid time range validation
- Non-existent pipeline handling
- Progress tracking verification
- Non-existent job 404 handling
- Realtime pipeline rejection (only scheduled pipelines allowed)
___
### **PR Type**
Tests, Enhancement
___
### **Description**
- Add comprehensive backfill pipeline API tests
- Creation, execution, listing, pause/resume, delete, validation
- Provide test utilities for ingestion and pipelines
- Configure CI env var for backdated ingestion
___
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>test_backfill.py</strong><dd><code>Add pipeline
backfill API tests</code>
</dd></summary>
<hr>
tests/api-testing/tests/test_backfill.py
<ul><li>Add a suite of 10+ backfill API tests<br> <li> Implement helpers
for log ingestion and pipelines<br> <li> Test job lifecycle: create,
status, execute, pause/resume, delete<br> <li> Validate time range,
non-existent resources, progress tracking</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/10177/files#diff-644fee0c3348626de59a5b38627f499f0c57afcffcc1a203533ff5fefa5df6dd">+706/-0</a>
</td>
</tr>
</table></td></tr><tr><td><strong>Configuration
changes</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>api-testing.yml</strong><dd><code>Allow backdated
ingestion in CI</code>
</dd></summary>
<hr>
.github/workflows/api-testing.yml
<ul><li>Add ZO_INGEST_ALLOWED_UPTO env var<br> to allow backdated data
ingestion</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/10177/files#diff-bf87f46624aaa9c4f1d9faa033baf531a543f2d4ac188cd64e9ab3febfee8f9e">+1/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
Neha PandClaude
|
93d4914b52 |
test: add API tests for cross-stream type JOIN with enrichment tables (#10173)
### **User description** Tests the feature from PR #5452 - joining logs with enrichment tables using both enrichment_tables.{table} and enrich.{table} prefixes. Test cases: - JOIN with enrichment_tables prefix - JOIN with enrich prefix (shorter alias) - LEFT JOIN preserves all rows - JOIN with GROUP BY aggregation - JOIN with WHERE clause filtering - Verify enriched columns appear in results - Both prefixes return identical results 🤖 Generated with [Claude Code](https://claude.com/claude-code) ___ ### **PR Type** Tests ___ ### **Description** - Add API tests for enrichment table cross-stream JOIN - Test both `enrichment_tables` and `enrich` prefixes - Include LEFT JOIN, aggregation, and WHERE filter scenarios - Validate enriched column presence and prefix equivalence ___ <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table> <tr> <td> <details> <summary><strong>test_enrichment_table_join.py</strong><dd><code>Tests for enrichment table JOIN suite</code> </dd></summary> <hr> tests/api-testing/tests/test_enrichment_table_join.py <ul><li>New test suite <code>TestEnrichmentTableJoin</code> with 7 methods<br> <li> Fixture setup/teardown for enrichment table creation<br> <li> Tests cover prefix variants, LEFT JOIN, aggregation, filtering<br> <li> Utility helpers for multipart upload and SQL search</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/10173/files#diff-59300e59ac91f497cbad0e9df7ee3c47ade382b0f37382da4f25861eac3e2628">+338/-0</a> </td> </tr> </table></td></tr></tr></tbody></table> </details> ___ Co-authored-by: Claude <noreply@anthropic.com> |
||
|
d904f2eb26 |
fix: histogram interval validation for multi day factors of 24hrs (#10167)
### Problem
When using `histogram(_timestamp, '7 days')` in SQL queries, the
interval was incorrectly adjusted to 86400 seconds (1 day) instead of
the correct 604800 seconds (7 days).
### Root Cause
The `validate_and_adjust_histogram_interval` function only checked if
intervals were **factors** of 24 hours (values that divide evenly into
24 hours), but didn't check if they were **multiples** of 24 hours (2
days, 7 days, 30 days, etc.).
### Fix
Added a check for intervals that are multiples of 24 hours **before**
checking for factors:
```rust
// Check if the interval is a multiple of 24 hours (like 2 days, 7 days, etc.)
if interval_seconds % TWENTY_FOUR_HOURS_SECONDS == 0 {
return interval_seconds;
}
// Check if the interval can divide 24 hours evenly (is a factor of 24h)
if TWENTY_FOUR_HOURS_SECONDS % interval_seconds == 0 {
return interval_seconds;
}
```
___
### **PR Type**
Bug fix, Enhancement, Tests
___
### **Description**
- Validate interval multiples of 24h first
- Factor-of-24h intervals returned unchanged
- Round other intervals to nearest factor
- Add tests for multiples and edge cases
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["Interval seconds"]
B{"Multiple of 24h?"}
E{"Factor of 24h?"}
F["Round up to nearest factor"]
C["Return adjusted interval"]
A -- "input" --> B
B -- "yes" --> C
B -- "no" --> E
E -- "yes" --> C
E -- "no" --> F
F --> C
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>histogram_interval.rs</strong><dd><code>Enhance
histogram interval validation logic</code>
</dd></summary>
<hr>
src/service/search/sql/visitor/histogram_interval.rs
<ul><li>Added doc comments describing new validation logic<br> <li>
Inserted multiple-of-24h check before factor check<br> <li> Retained
factor check and added rounding fallback<br> <li> Extended tests for
multiples and edge scenarios</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/10167/files#diff-0442907b4d8a3b5c4bd9553fd5475c3044fe74a2853cda8e4017bf295d714b4b">+60/-3</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
---------
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
|