mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-04 08:09:27 +00:00
ce09cd0a4dada4e4d45677f0b68c5ecfdb753aaf
116
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Ashish KolheandClaude Opus 4.5
|
ce09cd0a4d |
feat: add js for claim parser (#9873)
### **User description** Design at : security/sso-claim-parser/configuration.md ___ ### **PR Type** Enhancement, Tests ___ ### **Description** - Introduce rquickjs-based JS runtime for functions - Add compile_js_function and apply_js_fn utilities - Extend pipeline to support JS via CompiledFunctionRuntime - Update web UI: select JS/VRL transforms, show JS errors ___ ### Diagram Walkthrough ```mermaid flowchart LR A["FunctionNode"] -->|JS| B["compile_js_function"] A -->|VRL| C["compile_vrl_function"] B --> D["apply_js_fn"] C --> E["apply_vrl_fn"] D --> F["send to children"] E --> F ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>13 files</summary><table> <tr> <td><strong>function.ts</strong><dd><code>Add optional `trans_type` to TestFunctionPayload</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-c6766f6c671adce05f9eef5d4a57a32371fa7da35628ef5c142fa503d191cb16">+16/-0</a> </td> </tr> <tr> <td><strong>js.rs</strong><dd><code>Introduce JS runtime, compile and apply functions</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-88e490bed5fa6c9c8e0aa2738b95c5284bb97899298f54b9cd587713e3ef0109">+756/-0</a> </td> </tr> <tr> <td><strong>batch_execution.rs</strong><dd><code>Extend pipeline execution to handle JS functions</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-3f6824a6eddaf1603e5cb0f1a4311ac6a45e1ffb967266ccc1f0bba944625e69">+465/-96</a></td> </tr> <tr> <td><strong>functions.rs</strong><dd><code>Validate and test-run JS transforms, restrict to _meta</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-ed5611a9aa6b5c033e70d7631ba72fafad3cc842672077a7fcf965d11a4fef23">+319/-35</a></td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Forbid JavaScript functions in pipelines globally</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-c186c796a7bdfd9d1301b1429df11456430f1d0d4758783810b9e2e44384e3b2">+39/-1</a> </td> </tr> <tr> <td><strong>jwt.rs</strong><dd><code>Add JS claim parser execution function</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-232cab74f4cbf135af424744272e17732fbda8ac005d33c0acb4597e65479d02">+25/-2</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Wrap JS compile/apply in ingestion API</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-7c7b5bca85f0ced652965cae66540c96f56dfcca4f4c16b77bf7b45aac203d1b">+24/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Include `trans_type` in test_function API payload</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-a5e1b3644cef9feb689dfe66382e44e5434ecadbff3469127346a333d4733102">+8/-3</a> </td> </tr> <tr> <td><strong>AddFunction.vue</strong><dd><code>UI: add JS/VRL transform type selector</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-ebfd6c538f3cd3ecb9ab17550fd76e2e89e7a1abccc83a7505077e4af3eee110">+42/-25</a> </td> </tr> <tr> <td><strong>FunctionsToolbar.vue</strong><dd><code>Toolbar: transform type radio buttons</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-8283d2ff9c4a66b4b59eac306672aca2d54655a462e2ddb88e14e039d79dcdaf">+51/-3</a> </td> </tr> <tr> <td><strong>DomainManagement.vue</strong><dd><code>Track claim-parser function changes for save button</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-5e47cd01fcb63f7ef27dc64d2ecf66c39e17097e29944951c3ae1bfe715bb97c">+17/-9</a> </td> </tr> <tr> <td><strong>TestFunction.vue</strong><dd><code>Support JS error messaging in test UI</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-b9af9a2d1bf04d23422c26a13e6a763c6953c8713357a2f64ed8073ce6c926a3">+15/-5</a> </td> </tr> <tr> <td><strong>PipelineEditor.vue</strong><dd><code>Filter out JS functions in pipeline editor UI</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-68c21f647336067eb79737ce9aca2d49632f2fa1c0878bcfdae55607e3473a15">+12/-4</a> </td> </tr> </table></details></td></tr><tr><td><strong>Configuration changes</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>function.rs</strong><dd><code>Extend `Transform` model for JS support</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-27c9f3d93723397a0e234bc45c9f543b43eff5fd518d76e735f78d95a58f3f1c">+16/-6</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Register new `js` utils module</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-2aa59ce4464e7dbabdb9367fc8bb8c1b489c914b8690577d4916870463ab12db">+1/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Bug fix</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>errors.rs</strong><dd><code>Rename `MemoryTableOverflowError` display message</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-bf45932466265b732f13d0cd9b9cffbaefbe37c5861ace9315449eeca9250713">+1/-1</a> </td> </tr> </table></details></td></tr><tr><td><strong>Tests</strong></td><td><details><summary>3 files</summary><table> <tr> <td><strong>result-array-functions.spec.js</strong><dd><code>Add E2E tests for `#ResultArray#` functions in UI</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-e9911625ad2bab76f2b760f0ad254791331993a6a874b96f6a11fc6d88038b10">+470/-0</a> </td> </tr> <tr> <td><strong>cleanup.spec.js</strong><dd><code>Clean up JS function test data in UI tests</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-c214275a512fa285a7e4f2ab61dfcb42d71a0436b0e4241f94d7b45693587983">+31/-0</a> </td> </tr> <tr> <td><strong>page-manager.js</strong><dd><code>Register `FunctionsPage` in `PageManager`</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-565eab8b3c767ae50c9d05e591621d14f66b81af84d6b5ea6efe136c1c3933df">+6/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Dependencies</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>Cargo.toml</strong><dd><code>Add `rquickjs` dependency for JS runtime</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542">+2/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>18 files</summary><table> <tr> <td><strong>test_functions_meta_org_restriction.py</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-99ef1207d6ae6d7eeaed37d70c8cf0c4770ba78f6f51fbdc067d35492d2fe9ea">+401/-0</a> </td> </tr> <tr> <td><strong>test_js_enrichment.py</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-c5547d46bec60b888d947d1e9b7c5445aee72cf8e837182f7138e0fc903eedbb">+356/-0</a> </td> </tr> <tr> <td><strong>test_js_result_array.py</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-9ed1363a1f58c94e018bb2b0aaa9d08fa71f89e7e55a6ff6be4a82bbeed8985c">+390/-0</a> </td> </tr> <tr> <td><strong>functionsPage.js</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-7bab9cc2eb8f36d3ee5d0aab453c353ec6d6dfecbc401cf401322ff57faa8440">+326/-0</a> </td> </tr> <tr> <td><strong>javascript-functions.spec.js</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-bdae6529346a7222bdec19f75f490cb9d0f3578a260ba835225efcd27db3f64e">+314/-0</a> </td> </tr> <tr> <td><strong>meta-org-js-restriction.spec.js</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-16cc2373f6f1955f8344a59d5f2edcd6ce121c4fdee1d432b1f0ec8e27f25881">+483/-0</a> </td> </tr> <tr> <td><strong>FunctionList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-6ff9af50a1597504b7f0dca91e61cf30c800b4a17b4be2ed2c3a4f7a7b2d7068">+1/-0</a> </td> </tr> <tr> <td><strong>de.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-78f24ecb2dbcfc38fdf3ec22b2559a5b40d236ea689d1c55f5d3e87168310264">+30/-18</a> </td> </tr> <tr> <td><strong>en.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-892aa3d75ab6e129ef9116aa21bec1b9a15ba69ce5366d2a19128078c690d824">+15/-7</a> </td> </tr> <tr> <td><strong>es.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-ac2a2167b8b973e3f776f9fe3b43bef4576470d8ad33f3d7cfaff6edff0e5778">+30/-18</a> </td> </tr> <tr> <td><strong>fr.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-db80e762b88ddf183820c050cdc6b215e6b02b365b5e73f5152af46b0d26f1a2">+30/-18</a> </td> </tr> <tr> <td><strong>it.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-2de78074a5f887afddce60d694ead9322e6ab223eb70251c13819f3028bdc70d">+30/-18</a> </td> </tr> <tr> <td><strong>ja.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-092e49c5f9fbc1b539ab9c2866d108d0ce1d314022b4c294eb4611b285e7bbd3">+30/-18</a> </td> </tr> <tr> <td><strong>ko.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-50e27ca1b4825a5026f3a251bfcf79c97cab07cfade2ebd512d332453d707c48">+30/-18</a> </td> </tr> <tr> <td><strong>nl.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-a1706021687ae5d8a1c006c49eab3946174fecd895f9b70a89c2e76d97250906">+30/-18</a> </td> </tr> <tr> <td><strong>pt.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-38e804f64cb5e0e451849fc84e9b46e53c954a4a4263d51c18665ad45f4b4815">+30/-18</a> </td> </tr> <tr> <td><strong>tr.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-6f0f131e9e2591480c4c5d1eaffc72d02dc31cbd8d3f90d8f012f08cabac8818">+30/-18</a> </td> </tr> <tr> <td><strong>zh.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-8818055db58f83230aeea4ebfd4cec31d924e8c47b6f91298e356959ab25339e">+254/-242</a></td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> |
||
  Bhargavandomkark06
|
271ff27fac |
fix: remove traces error and rate value filters (#9617)
### **PR Type**
Bug fix
___
### **Description**
- Merge Rate and Errors filter chip displays
- Remove redundant Errors start-value span
- Always display Duration time range when set
___
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Bug
fix</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>TracesMetricsDashboard.vue</strong><dd><code>Unify
filter chip display logic</code>
</dd></summary>
<hr>
web/src/plugins/traces/metrics/TracesMetricsDashboard.vue
<ul><li>Merged Rate and Errors chips into single time-range
indicator<br> <li> Removed specific Errors value display span<br> <li>
Changed Duration chip to always use time-range v-if</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9617/files#diff-7b6f29a2737d9501b18dbd8622eba4965e4bc2b92115627ae7d2e70cc3382087">+2/-5</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
---------
Co-authored-by: omkark06 <omkar@zinclabs.io>
|
||
|
BhargavandClaude
|
e7703e86ec |
fix: upgrader two eslint packages (#9765)
### **PR Type**
Enhancement
___
### **Description**
- Upgrade eslint-plugin-cypress to ^5.2.0
- Upgrade eslint-plugin-vue to ^10.6.2
___
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Dependencies</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>package.json</strong><dd><code>ESLint plugin versions
bump</code>
</dd></summary>
<hr>
web/package.json
<ul><li>Bump <code>eslint-plugin-cypress</code> from ^4.1.0 to
^5.2.0<br> <li> Bump <code>eslint-plugin-vue</code> from ^9.32.0 to
^10.6.2</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9765/files#diff-b861012a5dd72b8a9f3281b7cf09f5a779c98569d040b1bbc1db50f1b15e7cce">+2/-2</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
Bhargav
|
50402cf46a |
fix: batch 3 package upgrade (#9755)
### **PR Type**
Enhancement
___
### **Description**
- Adjust Vitest config imports and coverage settings
- Refine coverage: add V8 provider and include patterns
- Bump Vitest, Vitest UI, and canvas mock versions
- Upgrade dotenv and remove unused type definitions
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["web/vitest.config.ts"] --> B["Import types corrected"]
A --> C["Coverage provider & include updated"]
D["web/package.json"] --> E["Bump Vitest & mocks versions"]
D --> F["Upgrade dotenv & remove types"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Configuration
changes</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>vitest.config.ts</strong><dd><code>Adjust Vitest config
imports and coverage</code>
</dd></summary>
<hr>
web/vitest.config.ts
<ul><li>Replace type import from <code>vitest/config</code> to
<code>vite</code><br> <li> Remove <code>VitestUserConfig</code> cast for
mergeConfig<br> <li> Add coverage provider <code>v8</code><br> <li>
Update coverage include pattern to
<code>src/**/*.{js,ts,vue}</code></ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9755/files#diff-c3fc010b682e68f2ed0b7b1485b8aa513443a66d500903b8dff1d65908d02aae">+4/-3</a>
</td>
</tr>
</table></td></tr><tr><td><strong>Dependencies</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>package.json</strong><dd><code>Upgrade devDependencies
versions</code>
</dd></summary>
<hr>
web/package.json
<ul><li>Bump <code>@vitest/coverage-v8</code> and
<code>@vitest/ui</code> to 4.x<br> <li> Upgrade <code>vitest</code> and
<code>vitest-canvas-mock</code> versions<br> <li> Update
<code>dotenv</code> to <code>^17.2.3</code><br> <li> Remove unused
<code>@types/dompurify</code></ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9755/files#diff-b861012a5dd72b8a9f3281b7cf09f5a779c98569d040b1bbc1db50f1b15e7cce">+5/-6</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
  
|
ac60e181dd |
feat: dashboard join support for auto sql (#9343)
# Nested Functions and Joins - User Guide ## Overview This guide covers two powerful features that enable sophisticated data analysis in OpenObserve: - **Nested Functions**: Combine multiple functions together to create complex calculations - **Joins**: Merge data from multiple streams (data sources) based on common fields These features work in the **Auto SQL** mode of dashboard panels, allowing you to write advanced queries without deep SQL knowledge. --- ## What's New ### Why These Features? **Before:** - Limited function composition - you could only use basic, single-level functions - No way to combine data from multiple streams in one query - Complex analytics required exporting data and manual processing **Now:** - Compose unlimited nested functions with full DataFusion support - Join multiple streams directly in the dashboard panel editor - Create sophisticated analytics queries with a visual interface ### Key Benefits ✓ Calculate complex metrics (error rates, percentiles, conditional aggregations) ✓ Build advanced dashboards without custom code ✓ Support for all DataFusion aggregate and scalar functions --- ## Nested Functions ### What Are Nested Functions? Nested functions let you combine multiple functions together - using the output of one function as input to another. Think of it like a cooking recipe where each step builds on the previous one. **Simple Example:** - Basic: `AVG(response_time)` - calculates average response time - Nested: `ROUND(AVG(response_time), 2)` - calculates average AND rounds it to 2 decimal places ### Supported Functions OpenObserve supports **all DataFusion functions**, giving you powerful analytics capabilities: **Aggregate Functions** (work across multiple rows): - `count`, `sum`, `avg`, `min`, `max` - basic aggregations - `median`, `stddev`, `variance` - statistical functions - `percentile_cont` - calculate percentiles (e.g., p95, p99) - `string_agg` - combine text values **Mathematical Functions**: - `round`, `ceil`, `floor` - rounding operations - `abs`, `power`, `sqrt` - mathematical operations - `log`, `log10`, `log2` - logarithmic functions - `sin`, `cos`, `tan` - trigonometric functions **String Functions**: - `concat`, `upper`, `lower`, `trim` - `substring`, `length`, `replace` **Conditional Functions**: - `CASE WHEN ... THEN ... ELSE ... END` - conditional logic - `COALESCE` - return first non-null value 📖 **Full Reference**: [DataFusion Aggregate Functions](https://datafusion.apache.org/user-guide/sql/aggregate_functions.html) | [DataFusion Scalar Functions](https://datafusion.apache.org/user-guide/sql/scalar_functions.html) --- ### Using Nested Functions (Step-by-Step) #### Step 1: Create or Edit a Dashboard Panel Open your dashboard and create a new panel or edit an existing one. Make sure you're in **Auto SQL** mode (not Custom SQL). #### Step 2: Select a Function In the Y-Axis or column configuration, click **Select Function** to choose your base function. <img width="795" alt="Nested Function - Adding Layers" src="https://github.com/user-attachments/assets/0b6b134e-59b8-48b1-a4ae-52fb4800d772" /> In this example, we're using the `Concat` function to combine multiple fields (namespace and pod name). #### Step 3: Add Nested Layers Click the function icon to add another function layer. You can nest functions multiple times. <img width="793" alt="Nested Function - Select Base Function" src="https://github.com/user-attachments/assets/4bd9e03e-1104-4e82-b2ec-a34e6bb64870" /> Here we're building a nested function: - Inner layer: `Avg(severity)` - calculates average severity - Outer layer: `Ceil()` - rounds up the average to the nearest integer #### Step 4: Build Complex Expressions For advanced analytics, you can combine nested functions with conditional logic using `CASE WHEN` statements. <img width="1340" alt="Complex Nested Function with CASE WHEN" src="https://github.com/user-attachments/assets/4c3e3623-bfad-44c4-a760-e10cc94d4a95" /> This example shows a sophisticated calculation: ```sql ROUND(100.0 * SUM(CASE WHEN COALESCE(severity, 0) >= 4 THEN 1 ELSE 0 END) / COUNT(_timestamp), 2) ``` **What this does:** 1. `COALESCE(severity, 0)` - replaces null severity with 0 2. `CASE WHEN severity >= 4 THEN 1 ELSE 0` - returns 1 for high severity, 0 otherwise 3. `SUM(...)` - counts high severity events 4. Divide by `COUNT(_timestamp)` - calculates percentage 5. Multiply by `100.0` - converts to percentage 6. `ROUND(..., 2)` - rounds to 2 decimal places **Result**: Percentage of high-severity events, rounded to 2 decimals #### Step 5: View the Generated SQL Switch to the **Raw** tab to see the complete SQL query that was generated from your nested function configuration. <img width="1388" alt="View Generated SQL" src="https://github.com/user-attachments/assets/d7f6111a-26c6-4dd0-9e9f-e26092fe6368" /> You can review the auto-generated SQL in the Query section at the bottom of the panel editor. The chart visualization updates in real-time as you build your nested functions. --- ## Joins ### What Are Joins? Joins allow you to combine data from two or more different streams (data sources) based on common fields between them. **Example:** **employee_data_join** stream: ```json [ {"employee_id": 1, "name": "Alice", "department_id": 1}, {"employee_id": 2, "name": "Bob", "department_id": 2}, {"employee_id": 3, "name": "Charlie", "department_id": null}, {"employee_id": 4, "name": "David", "department_id": 1}, {"employee_id": 5, "name": "Eve", "department_id": 3} ] ``` **dept_data_join** stream: ```json [ {"department_id": 1, "department_name": "HR"}, {"department_id": 2, "department_name": "Engineering"}, {"department_id": 3, "department_name": "Sales"}, {"department_id": 4, "department_name": "Marketing"} ] ``` A join lets you combine these streams to see the department name alongside each employee, matched by `department_id`. ### Types of Joins OpenObserve supports three types of joins: | Join Type | Description | When to Use | |-----------|-------------|-------------| | **INNER JOIN** | Returns only records with matches in both streams | When you need data that exists in both sources | | **LEFT JOIN** | Returns all records from left stream + matching records from right | When you want all data from your primary source, with supplemental info where available | | **RIGHT JOIN** | Returns all records from right stream + matching records from left | When you want all data from your secondary source, with supplemental info where available | > **Note:** UNION and CROSS joins are not currently supported. --- ### Using Joins (Step-by-Step) #### Step 1: Access the Joins Section In your dashboard panel editor, locate the **Joins** section. Click the **+** button to add a join. <img width="1725" alt="Access Joins Interface" src="https://github.com/user-attachments/assets/35db4f28-c526-43a2-bdd1-470e188f53d7" /> In this example: - **Primary stream**: `employee_data_join` (contains employee names) - We'll join it with `dept_data_join` (contains department information) - The interface shows available fields from both streams in the left panel #### Step 2: Configure the Join Click on the **Joins** dropdown and select the stream you want to join. Then configure: 1. **Joining Stream**: Select the stream to join (e.g., `employee_data_join`) 2. **Join Type**: Choose from LEFT, INNER, or RIGHT using the visual toggle 3. **On Stream**: Select the target stream (e.g., `dept_data_join`) <img width="1725" alt="Configure Join Type" src="https://github.com/user-attachments/assets/35db4f28-c526-43a2-bdd1-470e188f53d7" /> The interface displays: **"Performing a Left Join between employee_data_join and dept_data_join"** Notice the visual toggle showing three join types: - **Left** (selected in this example) - **Inner** - **Right** #### Step 3: Define the Joining Clause Set up the join condition by specifying which fields to match between the two streams: **Clause 1:** - **Select Field** (Left): Choose a field from the first stream (e.g., `department_id`) - **Select Operator**: Choose comparison operator (usually `=`) - **Select Field** (Right): Choose the matching field from the second stream (e.g., `department_id`) You can add multiple clauses using the **+** button for complex join conditions (e.g., joining on multiple fields or adding additional filters). #### Step 4: View the Results After configuring the join, the data table will show combined results from both streams. <img width="1724" alt="Join Results" src="https://github.com/user-attachments/assets/94ca9f33-6cfa-4ad1-9b84-bee9db34f912" /> In this example: - **Name** column from `employee_data_join` (Alice, Bob, Charlie, David, Eve) - **Department Name** column from `dept_data_join` (HR, Engineering, HR, Sales) - Data successfully joined on `department_id` #### Step 5: Review the Generated SQL Check the **Query** section at the bottom to see the generated SQL: ```sql SELECT employee_data_join.name as "x_axis_1", stream_0.department_name as "x_axis_2" FROM "employee_data_join" LEFT JOIN "dept_data_join" AS stream_0 ON employee_data_join.department_id = stream_0.department_id ``` The SQL query shows exactly how OpenObserve is combining your data streams. --- --- ## Examples ### Example 1: String Concatenation with Nested Functions **Use Case**: Combine multiple fields into a single display value ```sql Concat(k8s_namespace_name, '-', k8s_pod_name) ``` This joins namespace and pod names with a hyphen separator. ### Example 2: Statistical Analysis with Nested Math Functions **Use Case**: Calculate rounded average of a metric ```sql Ceil(Avg(severity)) ``` This calculates the average severity level and rounds up to the nearest whole number. ### Example 3: Conditional Percentage Calculation **Use Case**: Calculate percentage of high-severity events ```sql ROUND(100.0 * SUM(CASE WHEN COALESCE(severity, 0) >= 4 THEN 1 ELSE 0 END) / COUNT(_timestamp), 2) ``` **Breakdown:** - Handles null values with `COALESCE` - Counts events where severity >= 4 - Calculates percentage of total events - Rounds to 2 decimal places ### Example 4: Employee-Department Join **Use Case**: Show employee names with their department information ```sql SELECT employee_data_join.name as "Name", dept_data_join.department_name as "Department Name" FROM employee_data_join LEFT JOIN dept_data_join ON employee_data_join.department_id = dept_data_join.department_id ``` This displays all employees with their department names, showing null for employees without a department assignment. ### Example 5: Complex Analytics - Joins + Nested Functions **Use Case**: Calculate error rate per service with deployment information ```sql SELECT logs.service_name, deployments.version, COUNT(*) as total_requests, ROUND((COUNT(CASE WHEN logs.status_code >= 500 THEN 1 END) * 100.0 / COUNT(*)), 2) as error_rate_percent FROM service_logs as logs LEFT JOIN deployments ON logs.service_name = deployments.service_name GROUP BY logs.service_name, deployments.version ORDER BY error_rate_percent DESC ``` This combines: - Join between service logs and deployment data - Nested functions for conditional counting and percentage calculation - Grouping and sorting for analysis --- ## Tips and Best Practices ### Nested Functions Best Practices ✅ **Start Simple**: Build your function one layer at a time - Begin with inner function: `AVG(response_time)` - Add outer layers: `ROUND(AVG(response_time), 2)` ✅ **Test Each Layer**: Verify results before adding complexity - Test the inner function first - Add one layer at a time and check output ✅ **Handle Null Values**: Use `COALESCE` to avoid unexpected nulls ```sql COALESCE(AVG(response_time), 0) ``` ✅ **Use Clear Aliases**: Name your calculated fields descriptively ```sql ROUND(AVG(response_time), 2) as avg_response_ms ``` ✅ **Check for Division by Zero**: Protect against zero denominators ```sql CASE WHEN COUNT(*) > 0 THEN (SUM(errors) * 100.0 / COUNT(*)) ELSE 0 END as error_rate ``` ### Joins Best Practices ✅ **Choose the Right Join Type**: - **INNER JOIN**: Both streams must have matching data - **LEFT JOIN**: Keep all records from primary stream - **RIGHT JOIN**: Keep all records from secondary stream ✅ **Verify Field Compatibility**: - Match data types (don't join string to number) - Ensure field names are correct - Check that values actually match between streams ✅ **Consider Performance**: - Join on indexed fields when possible - Test with limited data first --- ## FAQ ### Q: Can I nest more than 2 functions? **A**: Yes! You can nest as many functions as needed. Example: ```sql ROUND(CEIL(AVG(COALESCE(response_time, 0))), 2) ``` ### Q: What's the difference between LEFT and RIGHT joins? **A**: - **LEFT JOIN**: Keeps all records from the first (left) stream - **RIGHT JOIN**: Keeps all records from the second (right) stream - Choose based on which dataset you want to preserve completely ### Q: Can I join more than 2 streams? **A**: Yes! Click the **+** button in the Joins section to add additional joins. You can chain multiple joins together. ### Q: Do nested functions impact performance? **A**: Complex nested functions may take longer to compute, especially on large datasets. Test with small time ranges first and optimize your expressions. ### Q: Can I use Custom SQL instead of Auto SQL? **A**: Yes! If you prefer writing raw SQL, switch to **Custom SQL** mode. However, Auto SQL provides a visual interface that's easier for building queries. --- ## Additional Resources 📚 **DataFusion Documentation**: - [Aggregate Functions](https://datafusion.apache.org/user-guide/sql/aggregate_functions.html) - COUNT, SUM, AVG, etc. - [Scalar Functions](https://datafusion.apache.org/user-guide/sql/scalar_functions.html) - ROUND, CONCAT, CASE, etc. 🔗 **OpenObserve Resources**: - [GitHub Repository](https://github.com/openobserve/openobserve) - [PR #9343 - Feature Implementation](https://github.com/openobserve/openobserve/pull/9343) - [OpenObserve Documentation](https://openobserve.ai/docs/) 💬 **Get Help**: - [GitHub Issues](https://github.com/openobserve/openobserve/issues) - Report bugs or request features - [Community Discussions](https://github.com/openobserve/openobserve/discussions) - Ask questions and share ideas --- Design at: https://github.com/openobserve/designs/tree/main/joins --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Shrinath Rao <shnath@openobserve.ai> |
||
 Neha PandClaude
|
52b2759bfa |
test: add INTERSECT and EXCEPT query tests for PR #9544 (#9601)
Added 4 new test cases to verify the join optimization fix: - INTERSECT query: Common user_ids in both streams - EXCEPT query: Users without orders - INTERSECT with WHERE clause: Active users with orders - EXCEPT with multiple columns: Users details without orders Also improved test_joinqueries.py by adding a 3-second sleep after data ingestion to allow proper indexing before querying. Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Neha PandClaude
|
be9aceb881 |
fix: add unique names for alert test cleanup (#9591)
- Add 2 second wait after alert deletion before deleting destination to allow backend cleanup to propagate - Use timestamp-based unique names for template, destination, and alert to avoid conflicts between parallel test runs - Fixes 409 error: "Destination is currently used by alert" --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
 priyabrata-stack
|
2a03b2a57d | add pipeline null condition test | ||
|
Shrinath Rao
|
ffc265ceba | test: added auto rerun step to e2e yml (#9466) | ||
 udev
|
aff58b33a4 | fix: use chrono apis to validate time (#9422) | ||
|
Neha P
|
8a158884c8 |
test: match_all_camelcase_tests (#9345)
- Verify full camel case token search: DbException should be found when searching for complete camel case identifiers - Verify atomic token search: UserAccountService should be searchable and return matching results for full service class names - Verify XML acronym tokenization: XMLHttpRequest should be properly tokenized and discoverable through match_all queries - Verify camel case with numbers: OAuth2TokenHandlershould handle mixed alphanumeric camel case tokens correctly - Verify multi-word service search:UserManagementService should support complex multi-word camel case service names |
||
|
Shrinath Rao
|
a4dd576df4 | test: unskipped 2 pipeline-core tests (#9085) | ||
 Subhra264andShrinath Rao
|
683bf89c7b |
test: Add tests for alerts and pipelines history (#9064)
Co-authored-by: Shrinath Rao <shnath@openobserve.ai> |
||
|
udev
|
a5b9facdc8 |
fix: migration issue (#9067)
fixes migration issue where older format pipeline would fail to load if the pipeline contains a `condition` node. |
||
|
Shrinath Rao
|
6b11b81d06 | test: updated testDino workflow order (#9059) | ||
|
udev
|
2854a33791 |
feat: pipeline or operator (#8911)
with this PR we add OR operator support in pipeline conditions along with the ability to nest conditions, creating complex conditions quickly. existing conditions will be treated as legacy conditions and still be honoured in the new evaluation logic. |
||
  
|
85a8fc1648 |
feat: license check (#8522)
### **User description** This adds license check job ___ ### **PR Type** Enhancement, Bug fix ___ ### **Description** - Add enterprise license HTTP APIs and DB watch - Enforce quota checks on search and ingestion - Always enable usage reporting; simplify modes - Expose license expiry in config and UI ___ ### Diagram Walkthrough ```mermaid flowchart LR Backend["Backend: License APIs + quota checks"] DB["DB: License watch + dist lock"] Config["Config: usage/reporting & expiry"] UI["UI: Settings License page + banner"] Search["Search/PromQL: quota enforcement"] Backend -- "watch updates" --> DB Backend -- "expose expiry" --> Config UI -- "GET/POST /api/license" --> Backend Search -- "check_search_allowed" --> Backend ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>21 files</summary><table> <tr> <td><strong>config.rs</strong><dd><code>Validate usage mode and publish interval bounds</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+16/-4</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add license GET/POST endpoints with validation</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-d569000673e415b222a20a6f4f50a93c278c189a7d6e43231d5ab8807e93021f">+75/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Register license routes for enterprise builds</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-a1cfd40da1ab6cd4db0181fd8664965034298047b3b7c3320c5385897a7c4aed">+2/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Enforce search quota in PromQL endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-0ad5729a1fbce00ff53076728c51822400c13af9918e1d8554f41be56d060fa0">+24/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add enterprise search quota checks in endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-96630a0dbb6dd7b2b0248fa325c7c7041ce31340c32826675a4a70772e9d29f6">+51/-1</a> </td> </tr> <tr> <td><strong>search_stream.rs</strong><dd><code>Enforce quota in streaming search endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-642de43ff6b19ae41087a9f1efc02d8b0b873809e0c92cbddfac1a30ef3f1184">+21/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Expose license expiry; always enable usage flag</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-ca09c9c9b020049bdbc3382c466bfcca80daac92e98760d9a9a76172abf34049">+8/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Wire license endpoints into router</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+3/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add license check locking and startup watchers</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-f2e4c29b8eef22189ec704a832633c4bd135a24444f054864c98064bd0963681">+57/-0</a> </td> </tr> <tr> <td><strong>license.rs</strong><dd><code>Implement license update broadcast and watch</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-43bb2d7d70d03657337dfabb078079de71fae8a7f58962a10ac68c03594e0f81">+70/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Export license DB module for enterprise</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-b7e3bf7f0ac23dd084c0c817a310a4d5e7a64ff532746d182986ef940fa03fd0">+2/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Enforce ingestion quota for enterprise</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-7c7b5bca85f0ced652965cae66540c96f56dfcca4f4c16b77bf7b45aac203d1b">+10/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Provide search quota check helper</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-c3b85ea662a885fbd39797968928b291cb59a58f53a03f7b7dfd7ba8b6a7985a">+18/-0</a> </td> </tr> <tr> <td><strong>ingestion.rs</strong><dd><code>Simplify reporting mode; always requeue on error</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-3edd5f4ed1eed0f765b7b450ef0fbf587632e6b22302006a5e397049e297057f">+19/-50</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Always enable usage queue and publishing</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-c6255010f7a87d6acdb3ad04954fc9f2c27350ae319c53a1857b120aa06408a3">+2/-25</a> </td> </tr> <tr> <td><strong>useManagementRoutes.ts</strong><dd><code>Add license management route</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-b71a1d39b1fab0749f0668d5f16382536fbda6e2f0c92c56a972fb5d8b5835f5">+8/-0</a> </td> </tr> <tr> <td><strong>license_server.ts</strong><dd><code>Add client for license APIs</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-3ac3fe8faca5f5a6aebd9d5e8885837f1511834eb7c78d0830af8f00c70523cb">+16/-0</a> </td> </tr> <tr> <td><strong>License.vue</strong><dd><code>New license management UI with usage</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-614c7b48f6f2ab1e8041d20c7583b6864285eacf9db032c27508b76f6de8467f">+469/-0</a> </td> </tr> <tr> <td><strong>index.vue</strong><dd><code>Add License tab and routing guards</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-699c63979f851c48ad440a078e93282fd797a18fc065e5331b2e41122ddb8b69">+27/-0</a> </td> </tr> <tr> <td><strong>LicensePeriod.vue</strong><dd><code>License expiry banner component</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-14f02914aaa78ef3e82717173fa95455b7f1092faff2d0dbd51e243663850ad4">+111/-0</a> </td> </tr> <tr> <td><strong>HomeView.vue</strong><dd><code>Show license expiry banner and navigate</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-9af441af191d3254ef5f9db603dca75486de7638cda41a3c050e7b3c33d68280">+9/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Bug fix</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>schema.rs</strong><dd><code>Remove usage_enabled dependency in org creation</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-4696a44abde2df4ac48fed716f930bfdcc01759bdfcfd8d802821c6dd857bff9">+1/-3</a> </td> </tr> </table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>en.json</strong><dd><code>Add i18n key for License tab label</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-892aa3d75ab6e129ef9116aa21bec1b9a15ba69ce5366d2a19128078c690d824">+2/-1</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> |
||
 Simha GandSubhradeep Chakraborty
|
24ac8ee0ff |
fix: Add RBAC to the Alerts and Pipelines history endpoints and add tests (#8930)
# PR Type Feature/Fix # Description The pipelines history and alerts history endpoints are now covered by RBAC. Added basic e2e api test for the same endpoints. --------- Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com> |
||
 Abhay PadamaniandKirtan Thakkar
|
098ee9d7d2 |
fix: dashboard panel re-position (#9016)
- #9020 --------- Co-authored-by: Kirtan Thakkar <kirtan@kiara.tech> |
||
|
Shrinath Rao
|
2e89ae259a | test: skip invalid tests (#9007) | ||
|
Shrinath Rao
|
44de3f8dca | test: fixed failing api tests (#9004) | ||
  
|
db7f947089 |
fix: UI design implementation (#8841)
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> Co-authored-by: omkark06 <omkar@zinclabs.io> Co-authored-by: ktx-vaidehi <vaidehi.akhani@kiara.tech> Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech> Co-authored-by: ktx-akshay <akshay.patil@kiara.tech> |
||
|
ktx-akshay
|
e8f93ca802 | test: add test cases for various filter operators in dashboard (#8918) | ||
 ByteBaker
|
77ab0cff5c |
fix: resolve flaky integration tests
- Ensure memtable isolation to prevent cross-test data contamination - Fix dashboard API version compatibility (v5 -> version-agnostic dashboard_id) - Fix indentation bug causing function/dashboard/stream deletions to run inside alert loop - Add 3-second wait after dashboard deletion for propagation |
||
|
ktx-akshay
|
766bd14db1 | fix: update dashboard tests for schema version v6 (#8909) | ||
|
ktx-akshay
|
3af7349795 | test: add pytest cases for dashboard functionality (#8885) | ||
|
Neha PandSubhradeep Chakraborty
|
56024fe950 |
fix: dynamic stream name should work if it starts with { and ends with } (#8882)
### **PR Type**
Tests, Enhancement
___
### **Description**
- Add comprehensive dynamic template tests
- Introduce reusable validation utilities
- Parameterize edge and nested cases
- Standardize waits and identifiers
___
### Diagram Walkthrough
```mermaid
flowchart LR
Consts["Add constants (waits, limits)"]
Utils["Add utils: time window, query builder, validator, enabler"]
DynamicTest["Enhance dynamic template test"]
ComplexTests["Add complex, special-char, edge, nested tests"]
Perf["Add performance and cleanup tests"]
Consts -- "used by" --> Utils
Utils -- "reused in" --> DynamicTest
Utils -- "reused in" --> ComplexTests
Utils -- "reused in" --> Perf
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>test_pipeline_dynamic.py</strong><dd><code>Expand
pipeline dynamic template tests and utilities</code>
</dd></summary>
<hr>
tests/api-testing/tests/test_pipeline_dynamic.py
<ul><li>Add constants and safety helpers.<br> <li> Add reusable
validation and enable helpers.<br> <li> Expand tests: complex, special
chars, edge, nested.<br> <li> Add performance and cleanup tests.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8882/files#diff-0bbefbbf7a34e3414a0aadd1963ffe6a2af8b5df4d619076207e361b2a42bcb1">+774/-49</a></td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
---------
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
|
||
|
0a6d9a86b6 |
fix: pipelines should be able to create streams with dynamic names (#8874)
### **PR Type**
Tests
___
### **Description**
- Add pytest for dynamic namespace routing
- Create and enable pipelines via API
- Ingest logs post-creation and verify routing
- Optional cleanup to delete created pipelines
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["Parametrized test inputs"] -- "build payload" --> B["Create pipeline via API"]
B -- "enable pipeline" --> C["Ingest logs to source"]
C -- "wait and verify" --> D["Search destination stream count"]
C -- "verify source matches" --> E["Search source for condition"]
D -- "report counts" --> F["Keep pipeline for monitoring"]
G["Cleanup test"] -- "find by name" --> H["Delete pipeline via API"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>test_pipeline_dynamic.py</strong><dd><code>Add tests
for pipeline dynamic namespace routing</code>
</dd></summary>
<hr>
tests/api-testing/tests/test_pipeline_dynamic.py
<ul><li>Add parametrized test creating dynamic routing pipelines<br>
<li> Enable pipeline, ingest logs, verify counts via SQL search<br> <li>
Include optional cleanup test to delete created pipelines<br> <li> Use
unique node IDs and names; wait for processing</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8874/files#diff-0bbefbbf7a34e3414a0aadd1963ffe6a2af8b5df4d619076207e361b2a42bcb1">+243/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
---------
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
|
||
|
Neha P
|
c5ea42ec2a |
test: UTF8 queries pytests (#8818)
### **PR Type**
Tests
___
### **Description**
- Add regression tests for substr subqueries
- Validate env flags causing Utf8View panic
- Add alternative query pattern coverage
- Verify data content and string functions
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["Env flags set via monkeypatch"] -- "enable JOIN_MATCH_ONE, UTF8_VIEW" --> B["Execute queries against stream_pytest_data"]
B -- "substr + IN subquery" --> C["Assert 200, validate hits"]
B -- "alternative IN/JOIN/CONCAT/length" --> D["Cross-validate no Utf8View error"]
B -- "data verification" --> E["Inspect container names/prefixes"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>test_subquery_join_match.py</strong><dd><code>Add
regression and coverage tests for substr subqueries</code>
</dd></summary>
<hr>
tests/api-testing/tests/test_subquery_join_match.py
<ul><li>Add end-to-end regression tests reproducing Utf8View panic<br>
<li> Set env vars via monkeypatch for failing conditions<br> <li> Cover
substr IN, JOIN, CONCAT, and length() variants<br> <li> Add data
verification and structural assertions on hits</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8818/files#diff-facac795b55980509f22b59fec9090910a8eb2c8d442936114c4b11f11008152">+564/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
|
ByteBaker
|
934aaadbfc | fix: flaky integration tests | ||
|
ktx-akshay
|
b88b7573b0 |
test: add pytest for dashboard (#8698)
### **PR Type**
Tests
___
### **Description**
- Add end-to-end dashboard API tests
- Cover CRUD flows and edge cases
- Validate SSE search stream responses
- Include auth, pagination, and folders
___
### Diagram Walkthrough
```mermaid
flowchart LR
list["List dashboards"] -- "200 with/without auth" --> validateList["Validate structure"]
create["Create dashboard"] -- "returns v1.dashboardId" --> read["Get by ID"]
read -- "assert fields" --> delete["Delete dashboard"]
updateNE["Update nonexistent"] -- "expect 400/404" --> done1["Handled"]
missingReq["Missing required fields"] -- "expect 400/422" --> done2["Handled"]
dupTitle["Duplicate title create"] -- "allowed" --> cleanup["Cleanup created dashboards"]
pagination["List with limit"] -- "verify dashboards key" --> cleanup2["Cleanup"]
sse["Search stream (SSE)"] -- "parse events/data" --> validateSSE["Validate results/hits"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>test_dashboards.py</strong><dd><code>End-to-end pytest
for dashboard APIs</code>
</dd></summary>
<hr>
tests/api-testing/tests/test_dashboards.py
<ul><li>Add pytest suite for dashboard APIs<br> <li> Test list, create,
get, delete flows<br> <li> Validate SSE search stream parsing<br> <li>
Cover pagination, auth, errors, duplicate titles</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8698/files#diff-936c6e8532b2f4ad8ebd0d5d2bd43638e278ce0880498ccf0c446f9e01a20901">+404/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
|
Hengfei Yang
|
ad79d30c41 | fix: optimize cache metadata (#8671) | ||
|
Neha P
|
891d3f5d1a |
test: pytest tests sql (#8637)
### **PR Type**
Tests
___
### **Description**
- Add time-series window functions test
- Add advanced regex aggregation test
- Add complex nested aggregation test
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["Add E2E test: array case-sensitive"] -- "validates substring match in arrays" --> T["tests/test_advanced_sql.py"]
B["Add E2E test: window functions"] -- "histogram + LAG growth" --> T
C["Add E2E test: regex + metrics"] -- "regexp_match, health status" --> T
D["Add E2E test: complex CTEs"] -- "env metrics, ranks" --> T
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>test_advanced_sql.py</strong><dd><code>Add multiple
advanced E2E SQL tests</code>
</dd></summary>
<hr>
tests/api-testing/tests/test_advanced_sql.py
<ul><li>Add case-sensitive array extraction E2E test.<br> <li> Add
time-series window functions E2E test.<br> <li> Add advanced regex with
JSON-path metrics test.<br> <li> Add complex nested aggregation with
CTEs and ranking test.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8637/files#diff-190cc55e8b961580682e09403c997fda2e8288e5f210dec5ab41c9465c68538f">+451/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
|
Neha PandShrinath Rao
|
3565d4a516 |
test: added advanced sql queries (#8573)
### **PR Type**
Tests
___
### **Description**
SQL Query Test Coverage
- COUNT, SUM, AVG, MIN, MAX
- approx_percentile_cont (P50, P75, P95, P99)
- histogram (time bucketing)
- str_match, str_match_ignore_case, COALESCE
- regexp_match, array_extract (regex groups)
- CASE WHEN (pattern matching)
- HAVING, LIKE, field comparisons
- GROUP BY (multi-field, time buckets)
- Kubernetes fields (kubernetes_container_name, etc.)
- Add advanced SQL E2E tests
- Validate fields and value constraints
- Cover percentiles and aggregations
- Exercise grouping, regex, and filters
___
### Diagram Walkthrough
```mermaid
flowchart LR
tests["New E2E SQL tests"] --> errorRate["Error rate analysis"]
tests --> percentiles["Time-bucket percentiles"]
tests --> volume["Log volume by bucket"]
tests --> source["Source grouping"]
tests --> hostPerf["Host performance stats"]
tests --> compPct["Component percentiles"]
tests --> apiLogs["API log analysis"]
tests --> metadata["Regex-based metadata extraction"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>test_advanced_sql.py</strong><dd><code>Add
comprehensive E2E advanced SQL query tests</code>
</dd></summary>
<hr>
tests/api-testing/tests/test_advanced_sql.py
<ul><li>Add eight E2E tests posting SQL queries<br> <li> Assert HTTP 200
and presence of expected fields<br> <li> Validate rates, percentile
ordering, and counts<br> <li> Use time windows, grouping, regex
extraction</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8573/files#diff-190cc55e8b961580682e09403c997fda2e8288e5f210dec5ab41c9465c68538f">+514/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
---------
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
|
||
|
Neha PandShrinath Rao
|
000009f86d |
test: str_match_coalesce_fix (#8403)
### **PR Type**
Tests
___
### **Description**
- Add e2e test for str_match_ignore_case with coalesce
- Validate all hits contain expected match
- Ensure 200 response and correct schema
- Log query performance and totals
___
### Diagram Walkthrough
```mermaid
flowchart LR
T["New e2e test: str_match_ignore_case + coalesce"] -- "executes query" --> API["/api/{org}/_search?type=logs"]
API -- "returns hits" --> V["Validate coalesce and case-insensitive match"]
V -- "assertions + logs" --> R["Report results and timing"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>test_search.py</strong><dd><code>Add coalesce-aware
case-insensitive match test</code>
</dd></summary>
<hr>
tests/api-testing/tests/test_search.py
<ul><li>Add end-to-end test for str_match_ignore_case with coalesce<br>
<li> Post SQL search over recent timeframe<br> <li> Assert 200, hits
presence, and per-hit validation<br> <li> Print diagnostics and timing
information</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8403/files#diff-f3776a5bc02e67c163ed707f86b2192f47b73c641dde04b83523b95a74518141">+76/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
|
||
|
Neha P
|
ed11cd1af3 |
test: add pytest for queries and assert hits (#8241)
### **PR Type**
Tests
___
### **Description**
- Add end-to-end SQL query validation tests
- Verify WHERE, match_all, ORDER BY semantics
- Enforce LIMIT and result consistency checks
- Validate CTE filtering and fields presence
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["New E2E tests"] -- "WHERE filter" --> B["Validate field equality"]
A -- "match_all" --> C["Check term presence"]
A -- "ORDER BY" --> D["Assert timestamp DESC"]
A -- "LIMIT" --> E["Enforce max hits"]
A -- "CTE" --> F["Validate filtered fields"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>test_search.py</strong><dd><code>Add comprehensive E2E
SQL query tests</code>
</dd></summary>
<hr>
tests/api-testing/tests/test_search.py
<ul><li>Add test for WHERE condition equality<br> <li> Add test for
match_all term presence<br> <li> Add test for ORDER BY timestamp
DESC<br> <li> Add tests for LIMIT and CTE behavior</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8241/files#diff-f3776a5bc02e67c163ed707f86b2192f47b73c641dde04b83523b95a74518141">+252/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
|
Ashish Kolhe
|
65f5ab44e3 |
fix: failing streaming tests (#8037)
### **PR Type**
Tests, Bug fix
___
### **Description**
Fix streaming tests' total aggregation logic
Track latest total instead of summing
Update debug logs to reflect final total
Adjust combined metadata to use final total
___
### Diagram Walkthrough
```mermaid
flowchart LR
Parse["Parse streaming metadata"] -- "extract results.total" --> Track["Track latest total"]
Track -- "set final_total" --> Combine["Combine first metadata with final_total"]
Hits["Collect hits events"] -- "aggregate all hits" --> Combine
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>test_streaming.py</strong><dd><code>Use latest total
for streaming metadata</code>
</dd></summary>
<hr>
tests/api-testing/tests/test_streaming.py
<ul><li>Replace summed <code>total_sum</code> with
<code>final_total</code><br> <li> Update debug prints to reference final
total<br> <li> Set combined metadata <code>results.total</code> to
latest total<br> <li> Clarify comments for streaming semantics</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8037/files#diff-f15e9b733130f72112380fda658264545435fbf956b08869b0ad5a9435d222f7">+8/-8</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
|
Neha PandShrinath Rao
|
8a43a378f6 |
test: fix api stream tests (#7923)
### **PR Type**
Tests
___
### **Description**
Re-enable streaming SQL E2E tests
Add values streaming endpoint tests
Cover cache and non-cache paths
Validate response schema and totals
___
### Diagram Walkthrough
```mermaid
flowchart LR
SQLTests["SQL streaming tests"] -- "parametrized inputs" --> StreamNoCache["POST _search_stream use_cache=false"]
SQLTests -- "cache verification" --> StreamCache["POST _search_stream use_cache=true"]
ValuesTests["Values streaming tests"] -- "no cache" --> ValuesNoCache["POST _values_stream use_cache=false"]
ValuesTests -- "with cache" --> ValuesCache["POST _values_stream use_cache=true"]
StreamNoCache --> AssertSQL["Assert 200, totals match"]
StreamCache --> AssertSQLCache["Assert 200, totals match"]
ValuesNoCache --> AssertValues["Assert 200, structure, limits"]
ValuesCache --> AssertValuesCache["Assert 200, structure, limits"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>test_streaming.py</strong><dd><code>Re-enable SQL and
add values streaming tests</code>
</dd></summary>
<hr>
tests/api-testing/tests/test_streaming.py
<ul><li>Uncomment and restore <code>test_streaming_sql</code> with
parametrization<br> <li> Add <code>test_values_streaming_endpoint</code>
for non-cache path<br> <li> Add
<code>test_values_streaming_endpoint_cache</code> for cache path<br>
<li> Validate status, totals, and response schema</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7923/files#diff-f15e9b733130f72112380fda658264545435fbf956b08869b0ad5a9435d222f7">+176/-176</a></td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
|
||
 Loakesh Indiranandsai nikhil kethe
|
e46f6d5b82 |
feat: custom histogram interval & aligned partitions (#7909)
- `is_ui_histogram` query param for search api, the backend will convert the sql to histogram query and respond accordingly - `search_type` for search partition query, and if search type is `ui` only then we generate partitions with mini partition that is aligned with histogram interval ### **PR Type** Enhancement, Bug fix, Tests, Documentation ___ ### **Description** - Add UI histogram query conversion and flag - Align partitions to histogram with mini - Stream empty metadata+hints consistently - Expose histogram eligibility and interval ___ ### Diagram Walkthrough ```mermaid flowchart LR FE["Web UI logs & services"] -- "is_ui_histogram, searchType" --> HTTP["HTTP search endpoints"] HTTP -- "convert to histogram SQL\n(resolve streams, eligibility)" --> SQL["Histogram SQL converter"] HTTP -- "pass search_type, flags" --> SVC["Search service"] SVC -- "is_eligible, gen interval" --> PART["PartitionGenerator"] PART -- "aligned partitions\n(+mini optional)" --> STREAM["Streaming execution"] STREAM -- "SearchResponse + metadata\n(empty hits emitted)" --> FE SVC -- "Response: histogram_interval, is_histogram_eligible, converted_query" --> FE ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>16 files</summary><table> <tr> <td><strong>partition.rs</strong><dd><code>Histogram-aligned partitions with optional mini support</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-348fa23375d0cea3e5fb039cc6b381d79aa1d5121e6b9792acc048d3fd02e37c">+798/-269</a></td> </tr> <tr> <td><strong>execution.rs</strong><dd><code>Always emit responses; accumulate and metadata tweaks</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-75d432351865224f0b00d7cc819554c2ad5fa1f777e9c645d91a3fd5c1113bc0">+148/-152</a></td> </tr> <tr> <td><strong>search_stream.rs</strong><dd><code>HTTP2 stream: UI histogram conversion and interval handling</code></dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-642de43ff6b19ae41087a9f1efc02d8b0b873809e0c92cbddfac1a30ef3f1184">+149/-69</a></td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Partition API: histogram eligibility, interval, UI paths</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-c3b85ea662a885fbd39797968928b291cb59a58f53a03f7b7dfd7ba8b6a7985a">+71/-5</a> </td> </tr> <tr> <td><strong>histogram.rs</strong><dd><code>Convert original SQL to histogram-safe query</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-a5fc4cd11b3ec569b4cedf278ef0b14f540504abfbb23e29eb896d5b2a6cc2b2">+140/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>HTTP search: UI histogram conversion and response enrichment</code></dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-96630a0dbb6dd7b2b0248fa325c7c7041ce31340c32826675a4a70772e9d29f6">+38/-3</a> </td> </tr> <tr> <td><strong>search.rs</strong><dd><code>Response/request fields for histogram meta and streaming chunks</code></dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-373e83a31e5bbc82fba05d7ae274b0dbd624ed748272232cf7eb8b8c3b2f3afc">+45/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>New error: SearchHistogramNotAvailable</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-90c725d0e77a6e7d4bcc130cf5783848a5cf0efdb304b351cd63cacf8775fc36">+7/-0</a> </td> </tr> <tr> <td><strong>multi_streams.rs</strong><dd><code>Multi-stream partition search accepts search_type</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-c3233cdf1e68cb2455f91457719b9c2f477e735ee16546fdfe111e76768bd054">+16/-4</a> </td> </tr> <tr> <td><strong>sql.rs</strong><dd><code>Detect histogram-eligible queries and subqueries</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-a0a09c66a20d878ec7683ae222c919fbe092b8001871ade7f455b89573ce0911">+27/-0</a> </td> </tr> <tr> <td><strong>http.rs</strong><dd><code>Add query flag parser for is_ui_histogram</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-3b50a69c1ec2f63f8eebb5f9880e17826463a586768aed3e3e15242833609d6b">+8/-0</a> </td> </tr> <tr> <td><strong>Index.vue</strong><dd><code>UI: histogram streaming flow, timeouts cleanup</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-0323c3da69b369843c5d72bf4df5b00ca6b5147192a213baf331659ecf92af62">+45/-12</a> </td> </tr> <tr> <td><strong>IndexList.vue</strong><dd><code>Pass searchType=ui to partition API</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-08f058f883b659e713e606991a7ea37ae1ff016b7cccef970489ab5840dc3d09">+1/-0</a> </td> </tr> <tr> <td><strong>search.ts</strong><dd><code>API: support is_ui_histogram, searchType plumbed</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-885ab872dbc6e7b20264df815e3ef4727f0abc86ce90d2fb47e4aff572814c75">+7/-2</a> </td> </tr> <tr> <td><strong>useStreamingSearch.ts</strong><dd><code>Include is_ui_histogram in streaming URL</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-8fce3e25a161cd458e734bd9b1f98aadefe3482da7b6529a5b1f4fa20ec115e2">+1/-0</a> </td> </tr> <tr> <td><strong>usePanelDataLoader.ts</strong><dd><code>Dashboard partition calls pass searchType</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-5a42e9493a7bcd722b88e61fd38883c4f61001bc9cafa399f65f8eafda3aace9">+2/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Tests</strong></td><td><details><summary>3 files</summary><table> <tr> <td><strong>cacher.rs</strong><dd><code>Adjust tests for new response fields</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-8c387b22b2c0a282600ee41ca702b3db60f5a284d507c0feb7418608ff062c6c">+6/-0</a> </td> </tr> <tr> <td><strong>SearchResult.spec.ts</strong><dd><code>Skip flaky resetPlotChart unit test</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-8ff7d14fc3cdac80eeec5550ecd89a77f015b4ebe928da9d4a76a9585d64d788">+1/-1</a> </td> </tr> <tr> <td><strong>useLogs.spec.ts</strong><dd><code>Skip failing query partitions test</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-842307ae8db03aa51891ddde98cfdbbabc712db1e1614c9eee99a59add395f70">+1/-1</a> </td> </tr> </table></details></td></tr><tr><td><strong>Error handling</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>error_utils.rs</strong><dd><code>Map histogram unavailability to 400 with code</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-d4de63bc9275a19c868f6967de0f7835bd2d7bda74ff55ce97b8fb3f9af54da9">+3/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Miscellaneous</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Export histogram SQL module</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-dc0ab53d0ff551cf3cd12027c73d0da5e66680c0bf2d1830778639a33bac3464">+1/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Bug fix</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>SearchResult.vue</strong><dd><code>UI: robust empty-histogram handling and reset watcher</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-5e65533aa4754bb6d0074fda18ffc1ecacded3b9ca6bc28a161c3f6b12f20195">+17/-15</a> </td> </tr> <tr> <td><strong>SearchBar.vue</strong><dd><code>Cancel running query before applying saved view</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-81893de4edf186b0556f610acb280c87baa2ffe67d3b48a60a866094c7635baf">+2/-1</a> </td> </tr> </table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>query.ts</strong><dd><code>Types: histogram interval and eligibility fields</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-12dd5b3bb9741bbeaa348c0efbbd745d0a582899678738159c6927d34a35191d">+2/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>fga.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-e7c99f37f3abbb5e0507d2760bbf67df010d9218704b38dce84c5477acecd22e">+0/-2</a> </td> </tr> <tr> <td><strong>useLogs.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-1040f61354f92711e0e77ed4e9bfa15586d45a94c7e2ab06cc37372068030e7b">+154/-168</a></td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> |
||
|
Loakesh IndiranandBhargav
|
1bd1caa10a |
fix: limit, hist_alignment by factors of 24 hrs, order_by (#7895)
### **User description** Fixes: - `total` count when limit is reached for limit queries in http2 streaming - Search response to return an array of `order_by` ``` eg: [["x_axis", "asc"], ["y_axis", "asc"]] ``` - Check and validate the histogram interval provided either as a query payload or the histogram query, to generate partitions in aligned 24 hr manner to avoid duplicate. ___ ### **PR Type** Enhancement, Bug fix, Tests ___ ### **Description** - Add `order_by_metadata` to search responses - Validate histogram intervals as 24h factors - Fix total count after results truncation - Implement frontend sorting with metadata ___ ### Diagram Walkthrough ```mermaid flowchart LR SQL["SQL generation"] -- "validate hist interval" --> R["Response (order_by metadata)"] R -- "merge response" --> C["Cache module"] R -- "set result" --> H["HTTP handler"] H -- "streaming data" --> S["Streaming exec"] S -- "apply sort metadata" --> U["Frontend UI"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody></tr></tbody></table> </details> ___ --------- Co-authored-by: Bhargav <BJP232004@GMAIL.COM> |
||
 udevandAshish Kolhe
|
1290512fe3 |
fix: SYNC optimize remote wal [phase 2] (#7664) (#7786)
### **User description** Add separate envs for pipeline wal Improve metrics ___ ### **PR Type** Enhancement ___ ### **Description** - Add batch buffering for pipeline WAL writes - Schedule periodic buffer flush in enterprise mode - Introduce env configs and metric labels - Instrument WAL reader with timing logs ___ ### Diagram Walkthrough ```mermaid flowchart LR A["process_node"] B["group by batch_key"] C["BATCH_BUFFERS"] D["get_pipeline_wal_writer"] E["remote WAL"] F["accumulate"] G["pipeline flush task"] A -- "records" --> B B -- "add to buffer" --> C C -- "should_flush? yes" --> D D -- "write_wal" --> E C -- "should_flush? no" --> F G -- "flush_all_buffers" --> C ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody></tr></tbody></table> </details> ___ --------- Co-authored-by: Ashish Kolhe <ashish.j.kolhe@gmail.com> |
||
|
Hengfei Yang
|
b7e1a9d702 | perf: pushdown plan (#7778) | ||
 Huaijin
|
b41bb71779 | feat: upgrade datafusion to v49 (#7649) | ||
 Anand Kumar Dubey
|
ad345086d4 | fix: validate password length when creating & updating users (#6782) | ||
|
Ashish KolheandShrinath Rao
|
91bc704dc0 |
fix: RBAC default role (#7560)
### **PR Type** Bug fix ___ ### **Description** - Include role in user-org tuple creation - Pass `org.role` to `get_user_org_tuple` - Add default role in custom role mapping ___ ### **Changes diagram** ```mermaid flowchart LR PT["process_token"] -- "pass role" --> GUT["get_user_org_tuple"] MG["map_group_to_custom_role"] -- "pass default role" --> GUT ``` ___ ### **Changes walkthrough** 📝 <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Bug fix</strong></td><td><table> <tr> <td> <details> <summary><strong>jwt.rs</strong><dd><code>Add role parameter to tuple function</code> </dd></summary> <hr> src/handler/http/auth/jwt.rs <li>Added role parameter to <code>get_user_org_tuple</code><br> <li> Passed <code>org.role</code> in <code>process_token</code><br> <li> Included default role in custom role mapping </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/7560/files#diff-232cab74f4cbf135af424744272e17732fbda8ac005d33c0acb4597e65479d02">+15/-3</a> </td> </tr> </table></td></tr></tr></tbody></table> ___ > <details> <summary> Need help?</summary><li>Type <code>/help how to ...</code> in the comments thread for any questions about PR-Agent usage.</li><li>Check out the <a href="https://qodo-merge-docs.qodo.ai/usage-guide/">documentation</a> for more information.</li></details> --------- Co-authored-by: Shrinath Rao <shnath@openobserve.ai> |
||
|
Hengfei Yang
|
5b2148b043 | fix: remove sql_mode support for FE (#7516) | ||
 
|
c2c7ea6a95 |
test: match_all queries pytest (#7208)
### **User description** match_all queries pytest ___ ### **PR Type** Tests ___ ### **Description** - Add tests for match_all streaming queries ___ ### **Changes walkthrough** 📝 <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table> <tr> <td> <details> <summary><strong>test_streaming.py</strong><dd><code>Add match_all streaming test cases</code> </dd></summary> <hr> tests/api-testing/tests/test_streaming.py <li>Added "match_all query one" test case<br> <li> Added "match_all query two" test case<br> <li> Added "match_all query three" test case </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/7208/files#diff-f15e9b733130f72112380fda658264545435fbf956b08869b0ad5a9435d222f7">+21/-3</a> </td> </tr> </table></td></tr></tr></tbody></table> ___ > <details> <summary> Need help?</summary><li>Type <code>/help how to ...</code> in the comments thread for any questions about PR-Agent usage.</li><li>Check out the <a href="https://qodo-merge-docs.qodo.ai/usage-guide/">documentation</a> for more information.</li></details> --------- Co-authored-by: Taiming Liu <liutaiming3@gmail.com> Co-authored-by: Shrinath Rao <shnath@openobserve.ai> |
||
|
Loakesh IndiranandHengfei Yang
|
600c0d3b7e |
feat: aggregation cache (#6923)
Cache Intermediate record batches for aggregate sql queries ## TODO - [ ] Add in memory cache file index - [ ] Modify exiting delete stream results api to also delete `aggregations` cache - [ ] use bounded Chanel for `CACHE_QUEUE` and and an env for queue size --------- Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com> |
||
|
Shyam Panjiyar
|
111298017c |
test: pagination pytest (#7126)
### **PR Type** Tests, Enhancement ___ ### **Description** - Comment out SQL mode calls in streaming tests - Disable Chromium browser tests in config - Add WebKit (Safari) project configuration - Adjust viewport settings for Safari tests ___ ### **Changes walkthrough** 📝 <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table> <tr> <td> <details> <summary><strong>streaming.spec.js</strong><dd><code>Disable SQL mode in streaming tests</code> </dd></summary> <hr> tests/ui-testing/playwright-tests/streaming.spec.js <li>Commented out <code>await logsPage.enableSQLMode()</code> calls<br> <li> Removed SQL mode requirement from streaming tests </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/7126/files#diff-7ee1f433f2c0f92476e64e6777e719a3cbfd2c60c7e1b4fb77891445e3cd7b91">+13/-13</a> </td> </tr> </table></td></tr><tr><td><strong>Configuration changes</strong></td><td><table> <tr> <td> <details> <summary><strong>playwright.config.js</strong><dd><code>Add Safari (WebKit) project to Playwright config</code> </dd></summary> <hr> tests/ui-testing/playwright.config.js <li>Commented out Chromium project configuration<br> <li> Added WebKit (Safari) project settings<br> <li> Set viewport for Desktop Safari device </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/7126/files#diff-91cb9615068914b2e4f5d66fcbed50d3e9771ea2d882f8e8048bbe54eab3c414">+13/-6</a> </td> </tr> </table></td></tr></tr></tbody></table> ___ > <details> <summary> Need help?</summary><li>Type <code>/help how to ...</code> in the comments thread for any questions about PR-Agent usage.</li><li>Check out the <a href="https://qodo-merge-docs.qodo.ai/usage-guide/">documentation</a> for more information.</li></details> |
||
|
Shyam Panjiyar
|
138e7e296d |
test: re_match, partition search and values API pytest (#7051)
re_match, partition search and values API pytest |