mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-04 16:09:05 +00:00
352bf6f485fdb75e1614e8a962257c4e43e6c3c8
129
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Ashish Kolheandgithub-actions[bot]
|
c34988a3ea |
fix: insights traces changes (#10304)
### **PR Type** Enhancement ___ ### **Description** - Introduce `timeStart`/`timeEnd` in filters - Enable baseline-only and comparison modes - Conditional single vs comparison query logic - Always show Insights button and update translations ___ ### Diagram Walkthrough ```mermaid flowchart LR start["Open analysis"] start -- "Has time filters?" --> comp["Comparison mode"] start -- "No time filters" --> base["Baseline-only mode"] comp -- "Query both ranges" --> merge["Merge & score"] base -- "Use baseline only" --> sort["Sort by baseline percent"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>3 files</summary><table> <tr> <td><strong>useLatencyInsightsAnalysis.ts</strong><dd><code>Support time-based filters and baseline-only mode</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-343474bc1f120f5ecebfc595bf0d3151b2db096d9226369c496b7a0cfaa33899">+87/-54</a> </td> </tr> <tr> <td><strong>useLatencyInsightsDashboard.ts</strong><dd><code>Conditional single/comparison query generation</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-960dec4a653d7c079157499a2473d0e87bd8b61463e1c3f633e7682ba763a4b1">+58/-7</a> </td> </tr> <tr> <td><strong>TracesMetricsDashboard.vue</strong><dd><code>Always show Insights button and update labels</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-7b6f29a2737d9501b18dbd8622eba4965e4bc2b92115627ae7d2e70cc3382087">+68/-54</a> </td> </tr> </table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>11 files</summary><table> <tr> <td><strong>de.json</strong><dd><code>Add `volumeInsights.insightsButtonLabel` translation</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-78f24ecb2dbcfc38fdf3ec22b2559a5b40d236ea689d1c55f5d3e87168310264">+3/-1</a> </td> </tr> <tr> <td><strong>en.json</strong><dd><code>Add insights translation keys to English</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-892aa3d75ab6e129ef9116aa21bec1b9a15ba69ce5366d2a19128078c690d824">+2/-0</a> </td> </tr> <tr> <td><strong>es.json</strong><dd><code>Add insights translation keys to Spanish</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-ac2a2167b8b973e3f776f9fe3b43bef4576470d8ad33f3d7cfaff6edff0e5778">+3/-1</a> </td> </tr> <tr> <td><strong>fr.json</strong><dd><code>Add insights translation keys to French</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-db80e762b88ddf183820c050cdc6b215e6b02b365b5e73f5152af46b0d26f1a2">+3/-1</a> </td> </tr> <tr> <td><strong>it.json</strong><dd><code>Add insights translation keys to Italian</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-2de78074a5f887afddce60d694ead9322e6ab223eb70251c13819f3028bdc70d">+3/-1</a> </td> </tr> <tr> <td><strong>ja.json</strong><dd><code>Add insights translation keys to Japanese</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-092e49c5f9fbc1b539ab9c2866d108d0ce1d314022b4c294eb4611b285e7bbd3">+3/-1</a> </td> </tr> <tr> <td><strong>ko.json</strong><dd><code>Add insights translation keys to Korean</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-50e27ca1b4825a5026f3a251bfcf79c97cab07cfade2ebd512d332453d707c48">+3/-1</a> </td> </tr> <tr> <td><strong>nl.json</strong><dd><code>Add insights translation keys to Dutch</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-a1706021687ae5d8a1c006c49eab3946174fecd895f9b70a89c2e76d97250906">+3/-1</a> </td> </tr> <tr> <td><strong>pt.json</strong><dd><code>Add insights translation keys to Portuguese</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-38e804f64cb5e0e451849fc84e9b46e53c954a4a4263d51c18665ad45f4b4815">+3/-1</a> </td> </tr> <tr> <td><strong>tr.json</strong><dd><code>Add insights translation keys to Turkish</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-6f0f131e9e2591480c4c5d1eaffc72d02dc31cbd8d3f90d8f012f08cabac8818">+3/-1</a> </td> </tr> <tr> <td><strong>zh.json</strong><dd><code>Add insights translation keys to Chinese</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/10304/files#diff-8818055db58f83230aeea4ebfd4cec31d924e8c47b6f91298e356959ab25339e">+3/-1</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
 Subhra264
|
7ebb4c9940 |
fix: backfill icons and edit timerange (#10209)
- Use consistent styling for backfill UI. - The timerange picker on edit page should display the backfill timerange by default. |
||
  Neha PandClaude
|
59f86c491d |
test: add API tests for pipeline backfill functionality (#10177)
### **User description**
Add comprehensive API tests for backfill core functionality:
- Basic backfill job creation and status retrieval
- Backfill execution with data verification
- List all backfill jobs for organization
- Pause and resume backfill jobs
- Delete backfill jobs
- Invalid time range validation
- Non-existent pipeline handling
- Progress tracking verification
- Non-existent job 404 handling
- Realtime pipeline rejection (only scheduled pipelines allowed)
___
### **PR Type**
Tests, Enhancement
___
### **Description**
- Add comprehensive backfill pipeline API tests
- Creation, execution, listing, pause/resume, delete, validation
- Provide test utilities for ingestion and pipelines
- Configure CI env var for backdated ingestion
___
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>test_backfill.py</strong><dd><code>Add pipeline
backfill API tests</code>
</dd></summary>
<hr>
tests/api-testing/tests/test_backfill.py
<ul><li>Add a suite of 10+ backfill API tests<br> <li> Implement helpers
for log ingestion and pipelines<br> <li> Test job lifecycle: create,
status, execute, pause/resume, delete<br> <li> Validate time range,
non-existent resources, progress tracking</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/10177/files#diff-644fee0c3348626de59a5b38627f499f0c57afcffcc1a203533ff5fefa5df6dd">+706/-0</a>
</td>
</tr>
</table></td></tr><tr><td><strong>Configuration
changes</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>api-testing.yml</strong><dd><code>Allow backdated
ingestion in CI</code>
</dd></summary>
<hr>
.github/workflows/api-testing.yml
<ul><li>Add ZO_INGEST_ALLOWED_UPTO env var<br> to allow backdated data
ingestion</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/10177/files#diff-bf87f46624aaa9c4f1d9faa033baf531a543f2d4ac188cd64e9ab3febfee8f9e">+1/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
Neha PandClaude
|
93d4914b52 |
test: add API tests for cross-stream type JOIN with enrichment tables (#10173)
### **User description** Tests the feature from PR #5452 - joining logs with enrichment tables using both enrichment_tables.{table} and enrich.{table} prefixes. Test cases: - JOIN with enrichment_tables prefix - JOIN with enrich prefix (shorter alias) - LEFT JOIN preserves all rows - JOIN with GROUP BY aggregation - JOIN with WHERE clause filtering - Verify enriched columns appear in results - Both prefixes return identical results 🤖 Generated with [Claude Code](https://claude.com/claude-code) ___ ### **PR Type** Tests ___ ### **Description** - Add API tests for enrichment table cross-stream JOIN - Test both `enrichment_tables` and `enrich` prefixes - Include LEFT JOIN, aggregation, and WHERE filter scenarios - Validate enriched column presence and prefix equivalence ___ <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table> <tr> <td> <details> <summary><strong>test_enrichment_table_join.py</strong><dd><code>Tests for enrichment table JOIN suite</code> </dd></summary> <hr> tests/api-testing/tests/test_enrichment_table_join.py <ul><li>New test suite <code>TestEnrichmentTableJoin</code> with 7 methods<br> <li> Fixture setup/teardown for enrichment table creation<br> <li> Tests cover prefix variants, LEFT JOIN, aggregation, filtering<br> <li> Utility helpers for multipart upload and SQL search</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/10173/files#diff-59300e59ac91f497cbad0e9df7ee3c47ade382b0f37382da4f25861eac3e2628">+338/-0</a> </td> </tr> </table></td></tr></tr></tbody></table> </details> ___ Co-authored-by: Claude <noreply@anthropic.com> |
||
 
|
d904f2eb26 |
fix: histogram interval validation for multi day factors of 24hrs (#10167)
### Problem
When using `histogram(_timestamp, '7 days')` in SQL queries, the
interval was incorrectly adjusted to 86400 seconds (1 day) instead of
the correct 604800 seconds (7 days).
### Root Cause
The `validate_and_adjust_histogram_interval` function only checked if
intervals were **factors** of 24 hours (values that divide evenly into
24 hours), but didn't check if they were **multiples** of 24 hours (2
days, 7 days, 30 days, etc.).
### Fix
Added a check for intervals that are multiples of 24 hours **before**
checking for factors:
```rust
// Check if the interval is a multiple of 24 hours (like 2 days, 7 days, etc.)
if interval_seconds % TWENTY_FOUR_HOURS_SECONDS == 0 {
return interval_seconds;
}
// Check if the interval can divide 24 hours evenly (is a factor of 24h)
if TWENTY_FOUR_HOURS_SECONDS % interval_seconds == 0 {
return interval_seconds;
}
```
___
### **PR Type**
Bug fix, Enhancement, Tests
___
### **Description**
- Validate interval multiples of 24h first
- Factor-of-24h intervals returned unchanged
- Round other intervals to nearest factor
- Add tests for multiples and edge cases
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["Interval seconds"]
B{"Multiple of 24h?"}
E{"Factor of 24h?"}
F["Round up to nearest factor"]
C["Return adjusted interval"]
A -- "input" --> B
B -- "yes" --> C
B -- "no" --> E
E -- "yes" --> C
E -- "no" --> F
F --> C
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>histogram_interval.rs</strong><dd><code>Enhance
histogram interval validation logic</code>
</dd></summary>
<hr>
src/service/search/sql/visitor/histogram_interval.rs
<ul><li>Added doc comments describing new validation logic<br> <li>
Inserted multiple-of-24h check before factor check<br> <li> Retained
factor check and added rounding fallback<br> <li> Extended tests for
multiples and edge scenarios</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/10167/files#diff-0442907b4d8a3b5c4bd9553fd5475c3044fe74a2853cda8e4017bf295d714b4b">+60/-3</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
---------
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
|
||
|
Neha PandClaude
|
87a507f83a |
test: add API tests for bulk enable, functions, kvstore, multi-stream search, passcode (#10155)
### **User description** - Add test_alert_bulk_enable.py: bulk enable/disable alerts (5 tests) - Add test_function_test_endpoint.py: VRL function testing (7 tests) - Add test_kvstore.py: KV store CRUD operations (8 tests) - Add test_multi_stream_search.py: multi-stream SSE search (6 tests) - Add test_org_passcode.py: organization passcode API (4 tests) 🤖 Generated with [Claude Code](https://claude.com/claude-code) ___ ### **PR Type** Tests ___ ### **Description** - Bulk alert enable/disable tests for existing alerts - VRL/JS function test endpoint happy and error flows - KV store create, read, list, update, delete tests - Multi-stream SSE search tests for various scenarios - Organization passcode get, update, persistence tests ___ <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table> <tr> <td> <details> <summary><strong>test_alert_bulk_enable.py</strong><dd><code>Bulk alert enable/disable tests</code> </dd></summary> <hr> tests/api-testing/tests/test_alert_bulk_enable.py <ul><li>Fixture setup for session and existing alerts retrieval<br> <li> Tests bulk disable and enable single alerts<br> <li> Tests bulk disable for multiple alerts<br> <li> Idempotency and response structure verification</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/10155/files#diff-bb438bbffe5aad82142e50aa7765cb47c711463ffc0fc1ce72b221cbcadafe9a">+241/-0</a> </td> </tr> <tr> <td> <details> <summary><strong>test_function_test_endpoint.py</strong><dd><code>Function test endpoint tests</code> </dd></summary> <hr> tests/api-testing/tests/test_function_test_endpoint.py <ul><li>Fixture setup for VRL/JS function test endpoint<br> <li> Tests basic function and value modification<br> <li> Multi-event and conditional logic tests<br> <li> Invalid syntax and response structure verification</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/10155/files#diff-8f8ba09e2a96c246c77ae4512443842d0e8accb855bac0f949971941b9956b6c">+230/-0</a> </td> </tr> <tr> <td> <details> <summary><strong>test_kvstore.py</strong><dd><code>KV store CRUD API tests</code> </dd></summary> <hr> tests/api-testing/tests/test_kvstore.py <ul><li>Added KV store CRUD tests (create, read, list)<br> <li> Tests prefix filtering and update behavior<br> <li> Delete operations, 404 and idempotency tests</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/10155/files#diff-b353df992b5db6b02190e9232e077967b8c30d702f24bb1903c8d5c8c32fd14e">+226/-0</a> </td> </tr> <tr> <td> <details> <summary><strong>test_multi_stream_search.py</strong><dd><code>Multi-stream search SSE tests</code> </dd></summary> <hr> tests/api-testing/tests/test_multi_stream_search.py <ul><li>Fixture setup and SSE response parser added<br> <li> Tests single and multi-stream search queries<br> <li> SSE format and metadata structure verification<br> <li> Size limit and invalid stream handling tests</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/10155/files#diff-c20d33fef67f42eb651e7a0d5d97d053c1d0d58307c13fb57ea0a19ffeedb329">+223/-0</a> </td> </tr> <tr> <td> <details> <summary><strong>test_org_passcode.py</strong><dd><code>Organization passcode API tests</code> </dd></summary> <hr> tests/api-testing/tests/test_org_passcode.py <ul><li>Tests organization passcode get and update endpoints<br> <li> Verifies persistence and response structure</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/10155/files#diff-819841cc5f7f2475248a5edd31f9d44c79159fa6fb97a9c2db9ea70139904379">+126/-0</a> </td> </tr> </table></td></tr></tr></tbody></table> </details> ___ --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
 Hengfei Yang
|
91e2f06895 | fix: pipeline destinations incorrectly saved as alert destinations (#10079) | ||
 priyabrata-stackandClaude Sonnet 4.5
|
dacc8f6365 |
chore: remove documentation files and test artifacts from PR
Removed temporary documentation and test result files: - METRICS_CLEANUP_SUMMARY.md - METRICS_TESTS_AUDIT_REPORT.md - POM_COMPLIANCE_FIXES_SUMMARY.md - PR_STRATEGY.md - PR_SUMMARY.md - TRACE_TESTS_FALSE_POSITIVE_ANALYSIS.md - test-results/.last-run.json - tests/test-results/.last-run.json - tests/test-logs/test-run-2025-12-15T17-25-37.log - tests/api-testing/RUM_TEST_README.md - tests/api-testing/rum-node-test.js - tests/api-testing/rum-test.html - tests/ui-testing/METRICS_INGESTION_README.md These files were used for development/documentation purposes and should not be included in the PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> |
||
|
priyabrata-stackandClaude Sonnet 4.5
|
0cf0e8937c |
fix(tests): implement POM compliance for metrics tests (72% complete)
Major refactoring to fix Page Object Model violations in metrics test suite. This addresses critical code quality issues identified in the Sentinel audit. ## Changes ### Page Objects Enhanced - **metricsPage.js**: Added 350+ lines of new methods - Data visualization verification methods - Config sidebar operations - Stream selection, date ranges, notifications - Chart types, inputs, and generic UI helpers ### Test Files Fixed (5/8 - 62.5%) - ✅ **metrics-advanced.spec.js**: Fixed 13 POM violations - ✅ **metrics-aggregations.spec.js**: Fixed 8 POM violations - ✅ **metrics-config-tabs.spec.js**: Fixed 31/33 POM violations - ✅ **metrics-queries.spec.js**: Fixed 28 POM violations - ✅ **metrics-visualizations.spec.js**: Fixed 44 POM violations ### Additional Fixes - **apiCleanup.js**: Fixed error handling consistency - Unified error response format (message vs error field) - Added explicit code: 200 for successful responses ## Statistics - **Violations Fixed**: 132/184 (72%) - **Helper Functions Refactored**: 3 (eliminated 200+ lines of duplication) - **All fixed files**: ✓ Syntax validated ## Remaining Work - metrics-config.spec.js: 28 violations - metrics.spec.js: 30 violations - See POM_COMPLIANCE_FIXES_SUMMARY.md for completion guide 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> |
||
|
Hengfei Yang
|
8b69f8f151 | fix: Axum migration (#10040) | ||
|
Hengfei Yang
|
fdcb375256 | fix: unit tests (#10032) | ||
|
Hengfei Yang
|
1fc4270f54 |
fix: payload limit (#10021)
1. We only need `ZO_PAYLOAD_LIMIT` now, removed `ZO_JSON_LIMIT` 2. Config axum log level `RUST_LOG = "info,config::axum=warn"` |
||
|
Shrinath Rao
|
8bf9eac8a2 |
test: Functions test suite rework - 67% reduction with full validation coverage (#9957)
## Summary Rework of the Functions test suite to reduce test count while maintaining 100% validation coverage. | Metric | Before | After | Change | |--------|--------|-------|--------| | **UI Tests** | 36 | 12 | **-67%** | | **API Tests (Python)** | 0 | 16 | +16 new | | **Validation coverage** | 100% | 100% | Preserved | | **CI Integration** | ❌ | ✅ | Added | **Cleanup performed:** - 3 placeholder tests removed (only had logging, no assertions) - 1 duplicate test removed - 4 cross-file duplicates deduplicated ## What Changed ### Test File Consolidation | Original Spec Files | Final Spec File | Original | Final | Reduction | |---------------------|-----------------|----------|-------|-----------| | `meta-org-js-restriction.spec.js` + `javascript-functions.spec.js` | `js-transform-type.spec.js` | 22 | 6 | 73% | | `result-array-functions.spec.js` | `row-expansion.spec.js` | 14 | 6 | 57% | | **Total** | **2 files** | **36** | **12** | **67%** | ### New Python API Tests Added 16 comprehensive API tests in `tests/api-testing/tests/test_functions_meta_org_restriction.py`: | Category | Count | Tests | |----------|-------|-------| | JS Restriction (blocked in default org) | 5 | create/test/update blocked | | VRL Functions (work in default org) | 4 | create/test/update works | | Auto-Detection | 2 | trans_type auto-detection | | Error Messages | 2 | content and consistency | | Edge Cases | 3 | empty/invalid/case-sensitivity | ### Critical Bugs Fixed 1. **Missing `_meta` org navigation** - JavaScript functions are ONLY allowed in `_meta` organization. Tests were navigating to default org. 2. **Python API tests using wrong field names** - Create/Update use `transType` (camelCase), Test endpoint uses `trans_type` (snake_case). 3. **Missing pipeline cleanup pattern** - Added `realfile*` pattern that was causing 409 errors. ## Files Modified ### UI Tests (Final) - `tests/ui-testing/playwright-tests/Functions/js-transform-type.spec.js` (6 tests) - `tests/ui-testing/playwright-tests/Functions/row-expansion.spec.js` (6 tests) ### UI Tests (Removed/Consolidated) - `meta-org-js-restriction.spec.js` → merged into `js-transform-type.spec.js` - `javascript-functions.spec.js` → merged into `js-transform-type.spec.js` - `result-array-functions.spec.js` → renamed to `row-expansion.spec.js` ### Cleanup Infrastructure - `tests/ui-testing/playwright-tests/cleanup.spec.js` - Consolidated function cleanup, added realfile* pattern - `tests/ui-testing/pages/apiCleanup.js` - Added org-aware cleanup methods ### API Tests (New) - `tests/api-testing/tests/test_functions_meta_org_restriction.py` - 16 new tests ### CI Integration - `.github/workflows/playwright.yml` - Added Functions test matrix entry ## Final Test List (12 UI Tests) ### js-transform-type.spec.js (6 tests) | # | Test | Priority | |---|------|----------| | 1 | P0: JavaScript function creation flow in _meta org | @smoke, @P0 | | 2 | P0: JS function creation and validation flow | @smoke, @P0 | | 3 | P1: JS function execution and error handling | @P1 | | 4 | P1: JS function error cases | @P1 | | 5 | P0: Default org - JS hidden, VRL works | @smoke, @P0 | | 6 | P1: Radio visibility across orgs | @P1 | ### row-expansion.spec.js (6 tests) | # | Test | Priority | |---|------|----------| | 1 | P0: VRL row expansion creation and validation flow | @smoke, @P0 | | 2 | P2: VRL row expansion edge cases | @P2 | | 3 | P0: JS row expansion creation and validation flow | @smoke, @P0 | | 4 | P1: JS row expansion edge cases and transformations | @P1 | | 5 | P1: Compare VRL and JS row expansion output | @P1 | | 6 | P1: Pipeline integration and error handling | @P1 | ## Test plan - [x] All 12 UI tests passing locally - [x] All 16 Python API tests passing locally - [x] Sentinel audit passed (POM compliant, no raw selectors) - [x] Hard assertions applied (no soft assertions) - [ ] CI pipeline runs successfully |
||
|
Ashish KolheandClaude Opus 4.5
|
ce09cd0a4d |
feat: add js for claim parser (#9873)
### **User description** Design at : security/sso-claim-parser/configuration.md ___ ### **PR Type** Enhancement, Tests ___ ### **Description** - Introduce rquickjs-based JS runtime for functions - Add compile_js_function and apply_js_fn utilities - Extend pipeline to support JS via CompiledFunctionRuntime - Update web UI: select JS/VRL transforms, show JS errors ___ ### Diagram Walkthrough ```mermaid flowchart LR A["FunctionNode"] -->|JS| B["compile_js_function"] A -->|VRL| C["compile_vrl_function"] B --> D["apply_js_fn"] C --> E["apply_vrl_fn"] D --> F["send to children"] E --> F ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>13 files</summary><table> <tr> <td><strong>function.ts</strong><dd><code>Add optional `trans_type` to TestFunctionPayload</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-c6766f6c671adce05f9eef5d4a57a32371fa7da35628ef5c142fa503d191cb16">+16/-0</a> </td> </tr> <tr> <td><strong>js.rs</strong><dd><code>Introduce JS runtime, compile and apply functions</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-88e490bed5fa6c9c8e0aa2738b95c5284bb97899298f54b9cd587713e3ef0109">+756/-0</a> </td> </tr> <tr> <td><strong>batch_execution.rs</strong><dd><code>Extend pipeline execution to handle JS functions</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-3f6824a6eddaf1603e5cb0f1a4311ac6a45e1ffb967266ccc1f0bba944625e69">+465/-96</a></td> </tr> <tr> <td><strong>functions.rs</strong><dd><code>Validate and test-run JS transforms, restrict to _meta</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-ed5611a9aa6b5c033e70d7631ba72fafad3cc842672077a7fcf965d11a4fef23">+319/-35</a></td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Forbid JavaScript functions in pipelines globally</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-c186c796a7bdfd9d1301b1429df11456430f1d0d4758783810b9e2e44384e3b2">+39/-1</a> </td> </tr> <tr> <td><strong>jwt.rs</strong><dd><code>Add JS claim parser execution function</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-232cab74f4cbf135af424744272e17732fbda8ac005d33c0acb4597e65479d02">+25/-2</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Wrap JS compile/apply in ingestion API</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-7c7b5bca85f0ced652965cae66540c96f56dfcca4f4c16b77bf7b45aac203d1b">+24/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Include `trans_type` in test_function API payload</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-a5e1b3644cef9feb689dfe66382e44e5434ecadbff3469127346a333d4733102">+8/-3</a> </td> </tr> <tr> <td><strong>AddFunction.vue</strong><dd><code>UI: add JS/VRL transform type selector</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-ebfd6c538f3cd3ecb9ab17550fd76e2e89e7a1abccc83a7505077e4af3eee110">+42/-25</a> </td> </tr> <tr> <td><strong>FunctionsToolbar.vue</strong><dd><code>Toolbar: transform type radio buttons</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-8283d2ff9c4a66b4b59eac306672aca2d54655a462e2ddb88e14e039d79dcdaf">+51/-3</a> </td> </tr> <tr> <td><strong>DomainManagement.vue</strong><dd><code>Track claim-parser function changes for save button</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-5e47cd01fcb63f7ef27dc64d2ecf66c39e17097e29944951c3ae1bfe715bb97c">+17/-9</a> </td> </tr> <tr> <td><strong>TestFunction.vue</strong><dd><code>Support JS error messaging in test UI</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-b9af9a2d1bf04d23422c26a13e6a763c6953c8713357a2f64ed8073ce6c926a3">+15/-5</a> </td> </tr> <tr> <td><strong>PipelineEditor.vue</strong><dd><code>Filter out JS functions in pipeline editor UI</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-68c21f647336067eb79737ce9aca2d49632f2fa1c0878bcfdae55607e3473a15">+12/-4</a> </td> </tr> </table></details></td></tr><tr><td><strong>Configuration changes</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>function.rs</strong><dd><code>Extend `Transform` model for JS support</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-27c9f3d93723397a0e234bc45c9f543b43eff5fd518d76e735f78d95a58f3f1c">+16/-6</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Register new `js` utils module</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-2aa59ce4464e7dbabdb9367fc8bb8c1b489c914b8690577d4916870463ab12db">+1/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Bug fix</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>errors.rs</strong><dd><code>Rename `MemoryTableOverflowError` display message</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-bf45932466265b732f13d0cd9b9cffbaefbe37c5861ace9315449eeca9250713">+1/-1</a> </td> </tr> </table></details></td></tr><tr><td><strong>Tests</strong></td><td><details><summary>3 files</summary><table> <tr> <td><strong>result-array-functions.spec.js</strong><dd><code>Add E2E tests for `#ResultArray#` functions in UI</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-e9911625ad2bab76f2b760f0ad254791331993a6a874b96f6a11fc6d88038b10">+470/-0</a> </td> </tr> <tr> <td><strong>cleanup.spec.js</strong><dd><code>Clean up JS function test data in UI tests</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-c214275a512fa285a7e4f2ab61dfcb42d71a0436b0e4241f94d7b45693587983">+31/-0</a> </td> </tr> <tr> <td><strong>page-manager.js</strong><dd><code>Register `FunctionsPage` in `PageManager`</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-565eab8b3c767ae50c9d05e591621d14f66b81af84d6b5ea6efe136c1c3933df">+6/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Dependencies</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>Cargo.toml</strong><dd><code>Add `rquickjs` dependency for JS runtime</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542">+2/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>18 files</summary><table> <tr> <td><strong>test_functions_meta_org_restriction.py</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-99ef1207d6ae6d7eeaed37d70c8cf0c4770ba78f6f51fbdc067d35492d2fe9ea">+401/-0</a> </td> </tr> <tr> <td><strong>test_js_enrichment.py</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-c5547d46bec60b888d947d1e9b7c5445aee72cf8e837182f7138e0fc903eedbb">+356/-0</a> </td> </tr> <tr> <td><strong>test_js_result_array.py</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-9ed1363a1f58c94e018bb2b0aaa9d08fa71f89e7e55a6ff6be4a82bbeed8985c">+390/-0</a> </td> </tr> <tr> <td><strong>functionsPage.js</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-7bab9cc2eb8f36d3ee5d0aab453c353ec6d6dfecbc401cf401322ff57faa8440">+326/-0</a> </td> </tr> <tr> <td><strong>javascript-functions.spec.js</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-bdae6529346a7222bdec19f75f490cb9d0f3578a260ba835225efcd27db3f64e">+314/-0</a> </td> </tr> <tr> <td><strong>meta-org-js-restriction.spec.js</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-16cc2373f6f1955f8344a59d5f2edcd6ce121c4fdee1d432b1f0ec8e27f25881">+483/-0</a> </td> </tr> <tr> <td><strong>FunctionList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-6ff9af50a1597504b7f0dca91e61cf30c800b4a17b4be2ed2c3a4f7a7b2d7068">+1/-0</a> </td> </tr> <tr> <td><strong>de.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-78f24ecb2dbcfc38fdf3ec22b2559a5b40d236ea689d1c55f5d3e87168310264">+30/-18</a> </td> </tr> <tr> <td><strong>en.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-892aa3d75ab6e129ef9116aa21bec1b9a15ba69ce5366d2a19128078c690d824">+15/-7</a> </td> </tr> <tr> <td><strong>es.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-ac2a2167b8b973e3f776f9fe3b43bef4576470d8ad33f3d7cfaff6edff0e5778">+30/-18</a> </td> </tr> <tr> <td><strong>fr.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-db80e762b88ddf183820c050cdc6b215e6b02b365b5e73f5152af46b0d26f1a2">+30/-18</a> </td> </tr> <tr> <td><strong>it.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-2de78074a5f887afddce60d694ead9322e6ab223eb70251c13819f3028bdc70d">+30/-18</a> </td> </tr> <tr> <td><strong>ja.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-092e49c5f9fbc1b539ab9c2866d108d0ce1d314022b4c294eb4611b285e7bbd3">+30/-18</a> </td> </tr> <tr> <td><strong>ko.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-50e27ca1b4825a5026f3a251bfcf79c97cab07cfade2ebd512d332453d707c48">+30/-18</a> </td> </tr> <tr> <td><strong>nl.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-a1706021687ae5d8a1c006c49eab3946174fecd895f9b70a89c2e76d97250906">+30/-18</a> </td> </tr> <tr> <td><strong>pt.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-38e804f64cb5e0e451849fc84e9b46e53c954a4a4263d51c18665ad45f4b4815">+30/-18</a> </td> </tr> <tr> <td><strong>tr.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-6f0f131e9e2591480c4c5d1eaffc72d02dc31cbd8d3f90d8f012f08cabac8818">+30/-18</a> </td> </tr> <tr> <td><strong>zh.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-8818055db58f83230aeea4ebfd4cec31d924e8c47b6f91298e356959ab25339e">+254/-242</a></td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
  Bhargavandomkark06
|
271ff27fac |
fix: remove traces error and rate value filters (#9617)
### **PR Type**
Bug fix
___
### **Description**
- Merge Rate and Errors filter chip displays
- Remove redundant Errors start-value span
- Always display Duration time range when set
___
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Bug
fix</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>TracesMetricsDashboard.vue</strong><dd><code>Unify
filter chip display logic</code>
</dd></summary>
<hr>
web/src/plugins/traces/metrics/TracesMetricsDashboard.vue
<ul><li>Merged Rate and Errors chips into single time-range
indicator<br> <li> Removed specific Errors value display span<br> <li>
Changed Duration chip to always use time-range v-if</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9617/files#diff-7b6f29a2737d9501b18dbd8622eba4965e4bc2b92115627ae7d2e70cc3382087">+2/-5</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
---------
Co-authored-by: omkark06 <omkar@zinclabs.io>
|
||
|
BhargavandClaude
|
e7703e86ec |
fix: upgrader two eslint packages (#9765)
### **PR Type**
Enhancement
___
### **Description**
- Upgrade eslint-plugin-cypress to ^5.2.0
- Upgrade eslint-plugin-vue to ^10.6.2
___
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Dependencies</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>package.json</strong><dd><code>ESLint plugin versions
bump</code>
</dd></summary>
<hr>
web/package.json
<ul><li>Bump <code>eslint-plugin-cypress</code> from ^4.1.0 to
^5.2.0<br> <li> Bump <code>eslint-plugin-vue</code> from ^9.32.0 to
^10.6.2</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9765/files#diff-b861012a5dd72b8a9f3281b7cf09f5a779c98569d040b1bbc1db50f1b15e7cce">+2/-2</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
Bhargav
|
50402cf46a |
fix: batch 3 package upgrade (#9755)
### **PR Type**
Enhancement
___
### **Description**
- Adjust Vitest config imports and coverage settings
- Refine coverage: add V8 provider and include patterns
- Bump Vitest, Vitest UI, and canvas mock versions
- Upgrade dotenv and remove unused type definitions
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["web/vitest.config.ts"] --> B["Import types corrected"]
A --> C["Coverage provider & include updated"]
D["web/package.json"] --> E["Bump Vitest & mocks versions"]
D --> F["Upgrade dotenv & remove types"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Configuration
changes</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>vitest.config.ts</strong><dd><code>Adjust Vitest config
imports and coverage</code>
</dd></summary>
<hr>
web/vitest.config.ts
<ul><li>Replace type import from <code>vitest/config</code> to
<code>vite</code><br> <li> Remove <code>VitestUserConfig</code> cast for
mergeConfig<br> <li> Add coverage provider <code>v8</code><br> <li>
Update coverage include pattern to
<code>src/**/*.{js,ts,vue}</code></ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9755/files#diff-c3fc010b682e68f2ed0b7b1485b8aa513443a66d500903b8dff1d65908d02aae">+4/-3</a>
</td>
</tr>
</table></td></tr><tr><td><strong>Dependencies</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>package.json</strong><dd><code>Upgrade devDependencies
versions</code>
</dd></summary>
<hr>
web/package.json
<ul><li>Bump <code>@vitest/coverage-v8</code> and
<code>@vitest/ui</code> to 4.x<br> <li> Upgrade <code>vitest</code> and
<code>vitest-canvas-mock</code> versions<br> <li> Update
<code>dotenv</code> to <code>^17.2.3</code><br> <li> Remove unused
<code>@types/dompurify</code></ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9755/files#diff-b861012a5dd72b8a9f3281b7cf09f5a779c98569d040b1bbc1db50f1b15e7cce">+5/-6</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
|
ac60e181dd |
feat: dashboard join support for auto sql (#9343)
# Nested Functions and Joins - User Guide ## Overview This guide covers two powerful features that enable sophisticated data analysis in OpenObserve: - **Nested Functions**: Combine multiple functions together to create complex calculations - **Joins**: Merge data from multiple streams (data sources) based on common fields These features work in the **Auto SQL** mode of dashboard panels, allowing you to write advanced queries without deep SQL knowledge. --- ## What's New ### Why These Features? **Before:** - Limited function composition - you could only use basic, single-level functions - No way to combine data from multiple streams in one query - Complex analytics required exporting data and manual processing **Now:** - Compose unlimited nested functions with full DataFusion support - Join multiple streams directly in the dashboard panel editor - Create sophisticated analytics queries with a visual interface ### Key Benefits ✓ Calculate complex metrics (error rates, percentiles, conditional aggregations) ✓ Build advanced dashboards without custom code ✓ Support for all DataFusion aggregate and scalar functions --- ## Nested Functions ### What Are Nested Functions? Nested functions let you combine multiple functions together - using the output of one function as input to another. Think of it like a cooking recipe where each step builds on the previous one. **Simple Example:** - Basic: `AVG(response_time)` - calculates average response time - Nested: `ROUND(AVG(response_time), 2)` - calculates average AND rounds it to 2 decimal places ### Supported Functions OpenObserve supports **all DataFusion functions**, giving you powerful analytics capabilities: **Aggregate Functions** (work across multiple rows): - `count`, `sum`, `avg`, `min`, `max` - basic aggregations - `median`, `stddev`, `variance` - statistical functions - `percentile_cont` - calculate percentiles (e.g., p95, p99) - `string_agg` - combine text values **Mathematical Functions**: - `round`, `ceil`, `floor` - rounding operations - `abs`, `power`, `sqrt` - mathematical operations - `log`, `log10`, `log2` - logarithmic functions - `sin`, `cos`, `tan` - trigonometric functions **String Functions**: - `concat`, `upper`, `lower`, `trim` - `substring`, `length`, `replace` **Conditional Functions**: - `CASE WHEN ... THEN ... ELSE ... END` - conditional logic - `COALESCE` - return first non-null value 📖 **Full Reference**: [DataFusion Aggregate Functions](https://datafusion.apache.org/user-guide/sql/aggregate_functions.html) | [DataFusion Scalar Functions](https://datafusion.apache.org/user-guide/sql/scalar_functions.html) --- ### Using Nested Functions (Step-by-Step) #### Step 1: Create or Edit a Dashboard Panel Open your dashboard and create a new panel or edit an existing one. Make sure you're in **Auto SQL** mode (not Custom SQL). #### Step 2: Select a Function In the Y-Axis or column configuration, click **Select Function** to choose your base function. <img width="795" alt="Nested Function - Adding Layers" src="https://github.com/user-attachments/assets/0b6b134e-59b8-48b1-a4ae-52fb4800d772" /> In this example, we're using the `Concat` function to combine multiple fields (namespace and pod name). #### Step 3: Add Nested Layers Click the function icon to add another function layer. You can nest functions multiple times. <img width="793" alt="Nested Function - Select Base Function" src="https://github.com/user-attachments/assets/4bd9e03e-1104-4e82-b2ec-a34e6bb64870" /> Here we're building a nested function: - Inner layer: `Avg(severity)` - calculates average severity - Outer layer: `Ceil()` - rounds up the average to the nearest integer #### Step 4: Build Complex Expressions For advanced analytics, you can combine nested functions with conditional logic using `CASE WHEN` statements. <img width="1340" alt="Complex Nested Function with CASE WHEN" src="https://github.com/user-attachments/assets/4c3e3623-bfad-44c4-a760-e10cc94d4a95" /> This example shows a sophisticated calculation: ```sql ROUND(100.0 * SUM(CASE WHEN COALESCE(severity, 0) >= 4 THEN 1 ELSE 0 END) / COUNT(_timestamp), 2) ``` **What this does:** 1. `COALESCE(severity, 0)` - replaces null severity with 0 2. `CASE WHEN severity >= 4 THEN 1 ELSE 0` - returns 1 for high severity, 0 otherwise 3. `SUM(...)` - counts high severity events 4. Divide by `COUNT(_timestamp)` - calculates percentage 5. Multiply by `100.0` - converts to percentage 6. `ROUND(..., 2)` - rounds to 2 decimal places **Result**: Percentage of high-severity events, rounded to 2 decimals #### Step 5: View the Generated SQL Switch to the **Raw** tab to see the complete SQL query that was generated from your nested function configuration. <img width="1388" alt="View Generated SQL" src="https://github.com/user-attachments/assets/d7f6111a-26c6-4dd0-9e9f-e26092fe6368" /> You can review the auto-generated SQL in the Query section at the bottom of the panel editor. The chart visualization updates in real-time as you build your nested functions. --- ## Joins ### What Are Joins? Joins allow you to combine data from two or more different streams (data sources) based on common fields between them. **Example:** **employee_data_join** stream: ```json [ {"employee_id": 1, "name": "Alice", "department_id": 1}, {"employee_id": 2, "name": "Bob", "department_id": 2}, {"employee_id": 3, "name": "Charlie", "department_id": null}, {"employee_id": 4, "name": "David", "department_id": 1}, {"employee_id": 5, "name": "Eve", "department_id": 3} ] ``` **dept_data_join** stream: ```json [ {"department_id": 1, "department_name": "HR"}, {"department_id": 2, "department_name": "Engineering"}, {"department_id": 3, "department_name": "Sales"}, {"department_id": 4, "department_name": "Marketing"} ] ``` A join lets you combine these streams to see the department name alongside each employee, matched by `department_id`. ### Types of Joins OpenObserve supports three types of joins: | Join Type | Description | When to Use | |-----------|-------------|-------------| | **INNER JOIN** | Returns only records with matches in both streams | When you need data that exists in both sources | | **LEFT JOIN** | Returns all records from left stream + matching records from right | When you want all data from your primary source, with supplemental info where available | | **RIGHT JOIN** | Returns all records from right stream + matching records from left | When you want all data from your secondary source, with supplemental info where available | > **Note:** UNION and CROSS joins are not currently supported. --- ### Using Joins (Step-by-Step) #### Step 1: Access the Joins Section In your dashboard panel editor, locate the **Joins** section. Click the **+** button to add a join. <img width="1725" alt="Access Joins Interface" src="https://github.com/user-attachments/assets/35db4f28-c526-43a2-bdd1-470e188f53d7" /> In this example: - **Primary stream**: `employee_data_join` (contains employee names) - We'll join it with `dept_data_join` (contains department information) - The interface shows available fields from both streams in the left panel #### Step 2: Configure the Join Click on the **Joins** dropdown and select the stream you want to join. Then configure: 1. **Joining Stream**: Select the stream to join (e.g., `employee_data_join`) 2. **Join Type**: Choose from LEFT, INNER, or RIGHT using the visual toggle 3. **On Stream**: Select the target stream (e.g., `dept_data_join`) <img width="1725" alt="Configure Join Type" src="https://github.com/user-attachments/assets/35db4f28-c526-43a2-bdd1-470e188f53d7" /> The interface displays: **"Performing a Left Join between employee_data_join and dept_data_join"** Notice the visual toggle showing three join types: - **Left** (selected in this example) - **Inner** - **Right** #### Step 3: Define the Joining Clause Set up the join condition by specifying which fields to match between the two streams: **Clause 1:** - **Select Field** (Left): Choose a field from the first stream (e.g., `department_id`) - **Select Operator**: Choose comparison operator (usually `=`) - **Select Field** (Right): Choose the matching field from the second stream (e.g., `department_id`) You can add multiple clauses using the **+** button for complex join conditions (e.g., joining on multiple fields or adding additional filters). #### Step 4: View the Results After configuring the join, the data table will show combined results from both streams. <img width="1724" alt="Join Results" src="https://github.com/user-attachments/assets/94ca9f33-6cfa-4ad1-9b84-bee9db34f912" /> In this example: - **Name** column from `employee_data_join` (Alice, Bob, Charlie, David, Eve) - **Department Name** column from `dept_data_join` (HR, Engineering, HR, Sales) - Data successfully joined on `department_id` #### Step 5: Review the Generated SQL Check the **Query** section at the bottom to see the generated SQL: ```sql SELECT employee_data_join.name as "x_axis_1", stream_0.department_name as "x_axis_2" FROM "employee_data_join" LEFT JOIN "dept_data_join" AS stream_0 ON employee_data_join.department_id = stream_0.department_id ``` The SQL query shows exactly how OpenObserve is combining your data streams. --- --- ## Examples ### Example 1: String Concatenation with Nested Functions **Use Case**: Combine multiple fields into a single display value ```sql Concat(k8s_namespace_name, '-', k8s_pod_name) ``` This joins namespace and pod names with a hyphen separator. ### Example 2: Statistical Analysis with Nested Math Functions **Use Case**: Calculate rounded average of a metric ```sql Ceil(Avg(severity)) ``` This calculates the average severity level and rounds up to the nearest whole number. ### Example 3: Conditional Percentage Calculation **Use Case**: Calculate percentage of high-severity events ```sql ROUND(100.0 * SUM(CASE WHEN COALESCE(severity, 0) >= 4 THEN 1 ELSE 0 END) / COUNT(_timestamp), 2) ``` **Breakdown:** - Handles null values with `COALESCE` - Counts events where severity >= 4 - Calculates percentage of total events - Rounds to 2 decimal places ### Example 4: Employee-Department Join **Use Case**: Show employee names with their department information ```sql SELECT employee_data_join.name as "Name", dept_data_join.department_name as "Department Name" FROM employee_data_join LEFT JOIN dept_data_join ON employee_data_join.department_id = dept_data_join.department_id ``` This displays all employees with their department names, showing null for employees without a department assignment. ### Example 5: Complex Analytics - Joins + Nested Functions **Use Case**: Calculate error rate per service with deployment information ```sql SELECT logs.service_name, deployments.version, COUNT(*) as total_requests, ROUND((COUNT(CASE WHEN logs.status_code >= 500 THEN 1 END) * 100.0 / COUNT(*)), 2) as error_rate_percent FROM service_logs as logs LEFT JOIN deployments ON logs.service_name = deployments.service_name GROUP BY logs.service_name, deployments.version ORDER BY error_rate_percent DESC ``` This combines: - Join between service logs and deployment data - Nested functions for conditional counting and percentage calculation - Grouping and sorting for analysis --- ## Tips and Best Practices ### Nested Functions Best Practices ✅ **Start Simple**: Build your function one layer at a time - Begin with inner function: `AVG(response_time)` - Add outer layers: `ROUND(AVG(response_time), 2)` ✅ **Test Each Layer**: Verify results before adding complexity - Test the inner function first - Add one layer at a time and check output ✅ **Handle Null Values**: Use `COALESCE` to avoid unexpected nulls ```sql COALESCE(AVG(response_time), 0) ``` ✅ **Use Clear Aliases**: Name your calculated fields descriptively ```sql ROUND(AVG(response_time), 2) as avg_response_ms ``` ✅ **Check for Division by Zero**: Protect against zero denominators ```sql CASE WHEN COUNT(*) > 0 THEN (SUM(errors) * 100.0 / COUNT(*)) ELSE 0 END as error_rate ``` ### Joins Best Practices ✅ **Choose the Right Join Type**: - **INNER JOIN**: Both streams must have matching data - **LEFT JOIN**: Keep all records from primary stream - **RIGHT JOIN**: Keep all records from secondary stream ✅ **Verify Field Compatibility**: - Match data types (don't join string to number) - Ensure field names are correct - Check that values actually match between streams ✅ **Consider Performance**: - Join on indexed fields when possible - Test with limited data first --- ## FAQ ### Q: Can I nest more than 2 functions? **A**: Yes! You can nest as many functions as needed. Example: ```sql ROUND(CEIL(AVG(COALESCE(response_time, 0))), 2) ``` ### Q: What's the difference between LEFT and RIGHT joins? **A**: - **LEFT JOIN**: Keeps all records from the first (left) stream - **RIGHT JOIN**: Keeps all records from the second (right) stream - Choose based on which dataset you want to preserve completely ### Q: Can I join more than 2 streams? **A**: Yes! Click the **+** button in the Joins section to add additional joins. You can chain multiple joins together. ### Q: Do nested functions impact performance? **A**: Complex nested functions may take longer to compute, especially on large datasets. Test with small time ranges first and optimize your expressions. ### Q: Can I use Custom SQL instead of Auto SQL? **A**: Yes! If you prefer writing raw SQL, switch to **Custom SQL** mode. However, Auto SQL provides a visual interface that's easier for building queries. --- ## Additional Resources 📚 **DataFusion Documentation**: - [Aggregate Functions](https://datafusion.apache.org/user-guide/sql/aggregate_functions.html) - COUNT, SUM, AVG, etc. - [Scalar Functions](https://datafusion.apache.org/user-guide/sql/scalar_functions.html) - ROUND, CONCAT, CASE, etc. 🔗 **OpenObserve Resources**: - [GitHub Repository](https://github.com/openobserve/openobserve) - [PR #9343 - Feature Implementation](https://github.com/openobserve/openobserve/pull/9343) - [OpenObserve Documentation](https://openobserve.ai/docs/) 💬 **Get Help**: - [GitHub Issues](https://github.com/openobserve/openobserve/issues) - Report bugs or request features - [Community Discussions](https://github.com/openobserve/openobserve/discussions) - Ask questions and share ideas --- Design at: https://github.com/openobserve/designs/tree/main/joins --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Shrinath Rao <shnath@openobserve.ai> |
||
|
Neha PandClaude
|
52b2759bfa |
test: add INTERSECT and EXCEPT query tests for PR #9544 (#9601)
Added 4 new test cases to verify the join optimization fix: - INTERSECT query: Common user_ids in both streams - EXCEPT query: Users without orders - INTERSECT with WHERE clause: Active users with orders - EXCEPT with multiple columns: Users details without orders Also improved test_joinqueries.py by adding a 3-second sleep after data ingestion to allow proper indexing before querying. Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Neha PandClaude
|
be9aceb881 |
fix: add unique names for alert test cleanup (#9591)
- Add 2 second wait after alert deletion before deleting destination to allow backend cleanup to propagate - Use timestamp-based unique names for template, destination, and alert to avoid conflicts between parallel test runs - Fixes 409 error: "Destination is currently used by alert" --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
priyabrata-stack
|
2a03b2a57d | add pipeline null condition test | ||
|
Shrinath Rao
|
ffc265ceba | test: added auto rerun step to e2e yml (#9466) | ||
 udev
|
aff58b33a4 | fix: use chrono apis to validate time (#9422) | ||
|
Neha P
|
8a158884c8 |
test: match_all_camelcase_tests (#9345)
- Verify full camel case token search: DbException should be found when searching for complete camel case identifiers - Verify atomic token search: UserAccountService should be searchable and return matching results for full service class names - Verify XML acronym tokenization: XMLHttpRequest should be properly tokenized and discoverable through match_all queries - Verify camel case with numbers: OAuth2TokenHandlershould handle mixed alphanumeric camel case tokens correctly - Verify multi-word service search:UserManagementService should support complex multi-word camel case service names |
||
|
Shrinath Rao
|
a4dd576df4 | test: unskipped 2 pipeline-core tests (#9085) | ||
|
Subhra264andShrinath Rao
|
683bf89c7b |
test: Add tests for alerts and pipelines history (#9064)
Co-authored-by: Shrinath Rao <shnath@openobserve.ai> |
||
|
udev
|
a5b9facdc8 |
fix: migration issue (#9067)
fixes migration issue where older format pipeline would fail to load if the pipeline contains a `condition` node. |
||
|
Shrinath Rao
|
6b11b81d06 | test: updated testDino workflow order (#9059) | ||
|
udev
|
2854a33791 |
feat: pipeline or operator (#8911)
with this PR we add OR operator support in pipeline conditions along with the ability to nest conditions, creating complex conditions quickly. existing conditions will be treated as legacy conditions and still be honoured in the new evaluation logic. |
||
  
|
85a8fc1648 |
feat: license check (#8522)
### **User description** This adds license check job ___ ### **PR Type** Enhancement, Bug fix ___ ### **Description** - Add enterprise license HTTP APIs and DB watch - Enforce quota checks on search and ingestion - Always enable usage reporting; simplify modes - Expose license expiry in config and UI ___ ### Diagram Walkthrough ```mermaid flowchart LR Backend["Backend: License APIs + quota checks"] DB["DB: License watch + dist lock"] Config["Config: usage/reporting & expiry"] UI["UI: Settings License page + banner"] Search["Search/PromQL: quota enforcement"] Backend -- "watch updates" --> DB Backend -- "expose expiry" --> Config UI -- "GET/POST /api/license" --> Backend Search -- "check_search_allowed" --> Backend ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>21 files</summary><table> <tr> <td><strong>config.rs</strong><dd><code>Validate usage mode and publish interval bounds</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+16/-4</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add license GET/POST endpoints with validation</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-d569000673e415b222a20a6f4f50a93c278c189a7d6e43231d5ab8807e93021f">+75/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Register license routes for enterprise builds</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-a1cfd40da1ab6cd4db0181fd8664965034298047b3b7c3320c5385897a7c4aed">+2/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Enforce search quota in PromQL endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-0ad5729a1fbce00ff53076728c51822400c13af9918e1d8554f41be56d060fa0">+24/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add enterprise search quota checks in endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-96630a0dbb6dd7b2b0248fa325c7c7041ce31340c32826675a4a70772e9d29f6">+51/-1</a> </td> </tr> <tr> <td><strong>search_stream.rs</strong><dd><code>Enforce quota in streaming search endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-642de43ff6b19ae41087a9f1efc02d8b0b873809e0c92cbddfac1a30ef3f1184">+21/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Expose license expiry; always enable usage flag</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-ca09c9c9b020049bdbc3382c466bfcca80daac92e98760d9a9a76172abf34049">+8/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Wire license endpoints into router</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+3/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add license check locking and startup watchers</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-f2e4c29b8eef22189ec704a832633c4bd135a24444f054864c98064bd0963681">+57/-0</a> </td> </tr> <tr> <td><strong>license.rs</strong><dd><code>Implement license update broadcast and watch</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-43bb2d7d70d03657337dfabb078079de71fae8a7f58962a10ac68c03594e0f81">+70/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Export license DB module for enterprise</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-b7e3bf7f0ac23dd084c0c817a310a4d5e7a64ff532746d182986ef940fa03fd0">+2/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Enforce ingestion quota for enterprise</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-7c7b5bca85f0ced652965cae66540c96f56dfcca4f4c16b77bf7b45aac203d1b">+10/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Provide search quota check helper</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-c3b85ea662a885fbd39797968928b291cb59a58f53a03f7b7dfd7ba8b6a7985a">+18/-0</a> </td> </tr> <tr> <td><strong>ingestion.rs</strong><dd><code>Simplify reporting mode; always requeue on error</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-3edd5f4ed1eed0f765b7b450ef0fbf587632e6b22302006a5e397049e297057f">+19/-50</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Always enable usage queue and publishing</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-c6255010f7a87d6acdb3ad04954fc9f2c27350ae319c53a1857b120aa06408a3">+2/-25</a> </td> </tr> <tr> <td><strong>useManagementRoutes.ts</strong><dd><code>Add license management route</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-b71a1d39b1fab0749f0668d5f16382536fbda6e2f0c92c56a972fb5d8b5835f5">+8/-0</a> </td> </tr> <tr> <td><strong>license_server.ts</strong><dd><code>Add client for license APIs</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-3ac3fe8faca5f5a6aebd9d5e8885837f1511834eb7c78d0830af8f00c70523cb">+16/-0</a> </td> </tr> <tr> <td><strong>License.vue</strong><dd><code>New license management UI with usage</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-614c7b48f6f2ab1e8041d20c7583b6864285eacf9db032c27508b76f6de8467f">+469/-0</a> </td> </tr> <tr> <td><strong>index.vue</strong><dd><code>Add License tab and routing guards</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-699c63979f851c48ad440a078e93282fd797a18fc065e5331b2e41122ddb8b69">+27/-0</a> </td> </tr> <tr> <td><strong>LicensePeriod.vue</strong><dd><code>License expiry banner component</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-14f02914aaa78ef3e82717173fa95455b7f1092faff2d0dbd51e243663850ad4">+111/-0</a> </td> </tr> <tr> <td><strong>HomeView.vue</strong><dd><code>Show license expiry banner and navigate</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-9af441af191d3254ef5f9db603dca75486de7638cda41a3c050e7b3c33d68280">+9/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Bug fix</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>schema.rs</strong><dd><code>Remove usage_enabled dependency in org creation</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-4696a44abde2df4ac48fed716f930bfdcc01759bdfcfd8d802821c6dd857bff9">+1/-3</a> </td> </tr> </table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>en.json</strong><dd><code>Add i18n key for License tab label</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-892aa3d75ab6e129ef9116aa21bec1b9a15ba69ce5366d2a19128078c690d824">+2/-1</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> |
||
 Simha GandSubhradeep Chakraborty
|
24ac8ee0ff |
fix: Add RBAC to the Alerts and Pipelines history endpoints and add tests (#8930)
# PR Type Feature/Fix # Description The pipelines history and alerts history endpoints are now covered by RBAC. Added basic e2e api test for the same endpoints. --------- Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com> |
||
 Abhay PadamaniandKirtan Thakkar
|
098ee9d7d2 |
fix: dashboard panel re-position (#9016)
- #9020 --------- Co-authored-by: Kirtan Thakkar <kirtan@kiara.tech> |
||
|
Shrinath Rao
|
2e89ae259a | test: skip invalid tests (#9007) | ||
|
Shrinath Rao
|
44de3f8dca | test: fixed failing api tests (#9004) | ||
  
|
db7f947089 |
fix: UI design implementation (#8841)
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> Co-authored-by: omkark06 <omkar@zinclabs.io> Co-authored-by: ktx-vaidehi <vaidehi.akhani@kiara.tech> Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech> Co-authored-by: ktx-akshay <akshay.patil@kiara.tech> |
||
|
ktx-akshay
|
e8f93ca802 | test: add test cases for various filter operators in dashboard (#8918) | ||
 ByteBaker
|
77ab0cff5c |
fix: resolve flaky integration tests
- Ensure memtable isolation to prevent cross-test data contamination - Fix dashboard API version compatibility (v5 -> version-agnostic dashboard_id) - Fix indentation bug causing function/dashboard/stream deletions to run inside alert loop - Add 3-second wait after dashboard deletion for propagation |
||
|
ktx-akshay
|
766bd14db1 | fix: update dashboard tests for schema version v6 (#8909) | ||
|
ktx-akshay
|
3af7349795 | test: add pytest cases for dashboard functionality (#8885) | ||
|
Neha PandSubhradeep Chakraborty
|
56024fe950 |
fix: dynamic stream name should work if it starts with { and ends with } (#8882)
### **PR Type**
Tests, Enhancement
___
### **Description**
- Add comprehensive dynamic template tests
- Introduce reusable validation utilities
- Parameterize edge and nested cases
- Standardize waits and identifiers
___
### Diagram Walkthrough
```mermaid
flowchart LR
Consts["Add constants (waits, limits)"]
Utils["Add utils: time window, query builder, validator, enabler"]
DynamicTest["Enhance dynamic template test"]
ComplexTests["Add complex, special-char, edge, nested tests"]
Perf["Add performance and cleanup tests"]
Consts -- "used by" --> Utils
Utils -- "reused in" --> DynamicTest
Utils -- "reused in" --> ComplexTests
Utils -- "reused in" --> Perf
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>test_pipeline_dynamic.py</strong><dd><code>Expand
pipeline dynamic template tests and utilities</code>
</dd></summary>
<hr>
tests/api-testing/tests/test_pipeline_dynamic.py
<ul><li>Add constants and safety helpers.<br> <li> Add reusable
validation and enable helpers.<br> <li> Expand tests: complex, special
chars, edge, nested.<br> <li> Add performance and cleanup tests.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8882/files#diff-0bbefbbf7a34e3414a0aadd1963ffe6a2af8b5df4d619076207e361b2a42bcb1">+774/-49</a></td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
---------
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
|
||
|
|
0a6d9a86b6 |
fix: pipelines should be able to create streams with dynamic names (#8874)
### **PR Type**
Tests
___
### **Description**
- Add pytest for dynamic namespace routing
- Create and enable pipelines via API
- Ingest logs post-creation and verify routing
- Optional cleanup to delete created pipelines
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["Parametrized test inputs"] -- "build payload" --> B["Create pipeline via API"]
B -- "enable pipeline" --> C["Ingest logs to source"]
C -- "wait and verify" --> D["Search destination stream count"]
C -- "verify source matches" --> E["Search source for condition"]
D -- "report counts" --> F["Keep pipeline for monitoring"]
G["Cleanup test"] -- "find by name" --> H["Delete pipeline via API"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>test_pipeline_dynamic.py</strong><dd><code>Add tests
for pipeline dynamic namespace routing</code>
</dd></summary>
<hr>
tests/api-testing/tests/test_pipeline_dynamic.py
<ul><li>Add parametrized test creating dynamic routing pipelines<br>
<li> Enable pipeline, ingest logs, verify counts via SQL search<br> <li>
Include optional cleanup test to delete created pipelines<br> <li> Use
unique node IDs and names; wait for processing</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8874/files#diff-0bbefbbf7a34e3414a0aadd1963ffe6a2af8b5df4d619076207e361b2a42bcb1">+243/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
---------
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
|
||
|
Neha P
|
c5ea42ec2a |
test: UTF8 queries pytests (#8818)
### **PR Type**
Tests
___
### **Description**
- Add regression tests for substr subqueries
- Validate env flags causing Utf8View panic
- Add alternative query pattern coverage
- Verify data content and string functions
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["Env flags set via monkeypatch"] -- "enable JOIN_MATCH_ONE, UTF8_VIEW" --> B["Execute queries against stream_pytest_data"]
B -- "substr + IN subquery" --> C["Assert 200, validate hits"]
B -- "alternative IN/JOIN/CONCAT/length" --> D["Cross-validate no Utf8View error"]
B -- "data verification" --> E["Inspect container names/prefixes"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>test_subquery_join_match.py</strong><dd><code>Add
regression and coverage tests for substr subqueries</code>
</dd></summary>
<hr>
tests/api-testing/tests/test_subquery_join_match.py
<ul><li>Add end-to-end regression tests reproducing Utf8View panic<br>
<li> Set env vars via monkeypatch for failing conditions<br> <li> Cover
substr IN, JOIN, CONCAT, and length() variants<br> <li> Add data
verification and structural assertions on hits</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8818/files#diff-facac795b55980509f22b59fec9090910a8eb2c8d442936114c4b11f11008152">+564/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
|
ByteBaker
|
934aaadbfc | fix: flaky integration tests | ||
|
ktx-akshay
|
b88b7573b0 |
test: add pytest for dashboard (#8698)
### **PR Type**
Tests
___
### **Description**
- Add end-to-end dashboard API tests
- Cover CRUD flows and edge cases
- Validate SSE search stream responses
- Include auth, pagination, and folders
___
### Diagram Walkthrough
```mermaid
flowchart LR
list["List dashboards"] -- "200 with/without auth" --> validateList["Validate structure"]
create["Create dashboard"] -- "returns v1.dashboardId" --> read["Get by ID"]
read -- "assert fields" --> delete["Delete dashboard"]
updateNE["Update nonexistent"] -- "expect 400/404" --> done1["Handled"]
missingReq["Missing required fields"] -- "expect 400/422" --> done2["Handled"]
dupTitle["Duplicate title create"] -- "allowed" --> cleanup["Cleanup created dashboards"]
pagination["List with limit"] -- "verify dashboards key" --> cleanup2["Cleanup"]
sse["Search stream (SSE)"] -- "parse events/data" --> validateSSE["Validate results/hits"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>test_dashboards.py</strong><dd><code>End-to-end pytest
for dashboard APIs</code>
</dd></summary>
<hr>
tests/api-testing/tests/test_dashboards.py
<ul><li>Add pytest suite for dashboard APIs<br> <li> Test list, create,
get, delete flows<br> <li> Validate SSE search stream parsing<br> <li>
Cover pagination, auth, errors, duplicate titles</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8698/files#diff-936c6e8532b2f4ad8ebd0d5d2bd43638e278ce0880498ccf0c446f9e01a20901">+404/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
|
Hengfei Yang
|
ad79d30c41 | fix: optimize cache metadata (#8671) | ||
|
Neha P
|
891d3f5d1a |
test: pytest tests sql (#8637)
### **PR Type**
Tests
___
### **Description**
- Add time-series window functions test
- Add advanced regex aggregation test
- Add complex nested aggregation test
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["Add E2E test: array case-sensitive"] -- "validates substring match in arrays" --> T["tests/test_advanced_sql.py"]
B["Add E2E test: window functions"] -- "histogram + LAG growth" --> T
C["Add E2E test: regex + metrics"] -- "regexp_match, health status" --> T
D["Add E2E test: complex CTEs"] -- "env metrics, ranks" --> T
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>test_advanced_sql.py</strong><dd><code>Add multiple
advanced E2E SQL tests</code>
</dd></summary>
<hr>
tests/api-testing/tests/test_advanced_sql.py
<ul><li>Add case-sensitive array extraction E2E test.<br> <li> Add
time-series window functions E2E test.<br> <li> Add advanced regex with
JSON-path metrics test.<br> <li> Add complex nested aggregation with
CTEs and ranking test.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8637/files#diff-190cc55e8b961580682e09403c997fda2e8288e5f210dec5ab41c9465c68538f">+451/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
|
Neha PandShrinath Rao
|
3565d4a516 |
test: added advanced sql queries (#8573)
### **PR Type**
Tests
___
### **Description**
SQL Query Test Coverage
- COUNT, SUM, AVG, MIN, MAX
- approx_percentile_cont (P50, P75, P95, P99)
- histogram (time bucketing)
- str_match, str_match_ignore_case, COALESCE
- regexp_match, array_extract (regex groups)
- CASE WHEN (pattern matching)
- HAVING, LIKE, field comparisons
- GROUP BY (multi-field, time buckets)
- Kubernetes fields (kubernetes_container_name, etc.)
- Add advanced SQL E2E tests
- Validate fields and value constraints
- Cover percentiles and aggregations
- Exercise grouping, regex, and filters
___
### Diagram Walkthrough
```mermaid
flowchart LR
tests["New E2E SQL tests"] --> errorRate["Error rate analysis"]
tests --> percentiles["Time-bucket percentiles"]
tests --> volume["Log volume by bucket"]
tests --> source["Source grouping"]
tests --> hostPerf["Host performance stats"]
tests --> compPct["Component percentiles"]
tests --> apiLogs["API log analysis"]
tests --> metadata["Regex-based metadata extraction"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>test_advanced_sql.py</strong><dd><code>Add
comprehensive E2E advanced SQL query tests</code>
</dd></summary>
<hr>
tests/api-testing/tests/test_advanced_sql.py
<ul><li>Add eight E2E tests posting SQL queries<br> <li> Assert HTTP 200
and presence of expected fields<br> <li> Validate rates, percentile
ordering, and counts<br> <li> Use time windows, grouping, regex
extraction</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8573/files#diff-190cc55e8b961580682e09403c997fda2e8288e5f210dec5ab41c9465c68538f">+514/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
---------
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
|
||
|
Neha PandShrinath Rao
|
000009f86d |
test: str_match_coalesce_fix (#8403)
### **PR Type**
Tests
___
### **Description**
- Add e2e test for str_match_ignore_case with coalesce
- Validate all hits contain expected match
- Ensure 200 response and correct schema
- Log query performance and totals
___
### Diagram Walkthrough
```mermaid
flowchart LR
T["New e2e test: str_match_ignore_case + coalesce"] -- "executes query" --> API["/api/{org}/_search?type=logs"]
API -- "returns hits" --> V["Validate coalesce and case-insensitive match"]
V -- "assertions + logs" --> R["Report results and timing"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>test_search.py</strong><dd><code>Add coalesce-aware
case-insensitive match test</code>
</dd></summary>
<hr>
tests/api-testing/tests/test_search.py
<ul><li>Add end-to-end test for str_match_ignore_case with coalesce<br>
<li> Post SQL search over recent timeframe<br> <li> Assert 200, hits
presence, and per-hit validation<br> <li> Print diagnostics and timing
information</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8403/files#diff-f3776a5bc02e67c163ed707f86b2192f47b73c641dde04b83523b95a74518141">+76/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
|
||
|
Neha P
|
ed11cd1af3 |
test: add pytest for queries and assert hits (#8241)
### **PR Type**
Tests
___
### **Description**
- Add end-to-end SQL query validation tests
- Verify WHERE, match_all, ORDER BY semantics
- Enforce LIMIT and result consistency checks
- Validate CTE filtering and fields presence
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["New E2E tests"] -- "WHERE filter" --> B["Validate field equality"]
A -- "match_all" --> C["Check term presence"]
A -- "ORDER BY" --> D["Assert timestamp DESC"]
A -- "LIMIT" --> E["Enforce max hits"]
A -- "CTE" --> F["Validate filtered fields"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>test_search.py</strong><dd><code>Add comprehensive E2E
SQL query tests</code>
</dd></summary>
<hr>
tests/api-testing/tests/test_search.py
<ul><li>Add test for WHERE condition equality<br> <li> Add test for
match_all term presence<br> <li> Add test for ORDER BY timestamp
DESC<br> <li> Add tests for LIMIT and CTE behavior</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8241/files#diff-f3776a5bc02e67c163ed707f86b2192f47b73c641dde04b83523b95a74518141">+252/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
|
Ashish Kolhe
|
65f5ab44e3 |
fix: failing streaming tests (#8037)
### **PR Type**
Tests, Bug fix
___
### **Description**
Fix streaming tests' total aggregation logic
Track latest total instead of summing
Update debug logs to reflect final total
Adjust combined metadata to use final total
___
### Diagram Walkthrough
```mermaid
flowchart LR
Parse["Parse streaming metadata"] -- "extract results.total" --> Track["Track latest total"]
Track -- "set final_total" --> Combine["Combine first metadata with final_total"]
Hits["Collect hits events"] -- "aggregate all hits" --> Combine
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>test_streaming.py</strong><dd><code>Use latest total
for streaming metadata</code>
</dd></summary>
<hr>
tests/api-testing/tests/test_streaming.py
<ul><li>Replace summed <code>total_sum</code> with
<code>final_total</code><br> <li> Update debug prints to reference final
total<br> <li> Set combined metadata <code>results.total</code> to
latest total<br> <li> Clarify comments for streaming semantics</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8037/files#diff-f15e9b733130f72112380fda658264545435fbf956b08869b0ad5a9435d222f7">+8/-8</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|