mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-05 08:09:22 +00:00
e17e8bc5e6a83d6a547db18cfe25bd79ebfebb21
2615
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Subhra264
|
e17e8bc5e6 |
fix: when retrying allow user to choose whether to resume from last byte or start from scratch (#9792)
### **PR Type**
Enhancement
When the job is failed, and the user wants to retry manually, ask the
user whether resume should be from the last successful position or from
start again.
___
### **Description**
- Support resume option for enrichment import
- Parse `resume` query param server-side
- Custom UI for resume or fresh restart
- Update JS service to include `resume` param
- Add `RetryJobDialog.vue` component
___
### Diagram Walkthrough
```mermaid
flowchart LR
UI["UI: RetryJobDialog"] -- "confirm resume choice" --> API["POST /api/{org}/enrichment_tables/{table}/url?append=&resume="]
API -- "parse `resume` param" --> Handler["save_enrichment_table_from_url"]
Handler -- "resume logic" --> DB["Job record (resumed or new)"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>jstransform.ts</strong><dd><code>Add `resume` param in
request service</code>
</dd></summary>
<hr>
web/src/services/jstransform.ts
<ul><li>Add <code>resume</code> parameter with default false<br> <li>
Append <code>resume</code> to POST request URL</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9792/files#diff-301add2528e0f32323ca4c503bd7d460ab15ca6b9a9537192a02c5e52fd82926">+2/-1</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>mod.rs</strong><dd><code>Support resume logic in import
handler</code>
</dd></summary>
<hr>
src/handler/http/request/enrichment_table/mod.rs
<ul><li>Parse <code>resume</code> query parameter<br> <li> Store
<code>existing_job</code> for resume handling<br> <li> Resume failed
ranged jobs preserving progress<br> <li> Skip range support check when
resuming</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9792/files#diff-7b5b203002d1c3e5f671392a2a8ed84f45225896b652c2dac94ac26e72a43124">+79/-42</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>EnrichmentTableList.vue</strong><dd><code>Integrate
retry dialog into table list</code>
</dd></summary>
<hr>
web/src/components/functions/EnrichmentTableList.vue
<ul><li>Import and register <code>RetryJobDialog</code><br> <li> Manage
<code>showRetryDialog</code> and <code>retryJobData</code><br> <li>
Update <code>retryUrlJob</code> to open dialog<br> <li> Adjust
<code>handleRetryConfirm</code> to pass <code>resume</code></ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9792/files#diff-6a29e253f59405d3c0f756dba36afd7483f10ad86927cbbc3348ede6372c3077">+64/-34</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>RetryJobDialog.vue</strong><dd><code>Add RetryJobDialog
component</code>
</dd></summary>
<hr>
web/src/components/functions/RetryJobDialog.vue
<ul><li>New dialog to choose resume or restart<br> <li> Handles focus
trap and ESC key<br> <li> Displays range support warning<br> <li> Emits
<code>confirm</code> with resume choice</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9792/files#diff-7bc926ef894b5aed7010006e729ede28a2f4931809b154dc81880999b3a7e72f">+638/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
  
|
78c96c3721 |
fix: support bulk deletes for various resources (#9153)
### **User description** fixes https://github.com/openobserve/openobserve/issues/9116 This adds support for bulk delete operation from API and UI side for various resources. ___ ### **PR Type** Enhancement ___ ### **Description** - Add bulk delete endpoints for multiple resources - Enforce per-item permission checks before delete - Aggregate successful/failed items in responses - Register bulk-delete routes in HTTP router - Extend UI: add bulk-delete buttons, dialogs, service calls ___ ### Diagram Walkthrough ```mermaid flowchart LR UI["\"UI: Bulk Delete Trigger\""] API["\"HTTP DELETE /{resource}/bulk Handler\""] AUTH["\"Permission Checks per Item\""] DEL["\"Perform Delete Operations Loop\""] RESP["\"Aggregate Response: successes/ failures\""] UI_FE["\"FE: Show Notifications\""] UI -- "calls" --> API API -- "checks" --> AUTH AUTH -- "allowed" --> DEL DEL -- "deletes items" --> RESP RESP -- "returns" --> UI_FE ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>14 files</summary><table> <tr> <td><strong>fga.rs</strong><dd><code>Add bulk delete roles and groups endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-e7c99f37f3abbb5e0507d2760bbf67df010d9218704b38dce84c5477acecd22e">+218/-6</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Implement delete_alert_bulk handler</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-83431801d27626ef27482e67b62661f900043544a45ddf958bd027cf2fa3bd8d">+87/-6</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add delete_function_bulk and error types</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-a5e1b3644cef9feb689dfe66382e44e5434ecadbff3469127346a333d4733102">+97/-3</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add delete_dashboard_bulk endpoint</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-1b3c257e1d174c826718499faa4a1d4e93465dd56cb2b7b273b690ff7e3223fd">+78/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Implement re_patterns delete_bulk logic</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-27d37a8738de89c81b540daa978e684caf2cf28803e1208d90cb4d42528ed28b">+116/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add users bulk delete handler</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-56897e42f9436a640bc70fddc25686485ba8ae1223635b32d252a0333309f748">+80/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add service_accounts delete_bulk API</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-095d70f65b6e69bf7de0180fcbcddcd7c89c4cf00095d7b06f039a07e73916bb">+86/-1</a> </td> </tr> <tr> <td><strong>action.rs</strong><dd><code>Implement delete_action_bulk endpoint</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-616df3014c553f763ea57e925f1bff7bb47f9467c632940d93a45b7b1e4c3a07">+90/-1</a> </td> </tr> <tr> <td><strong>destinations.rs</strong><dd><code>Add delete_destination_bulk for alerts</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-76b00293c621036de31cf499c1c276efb9704c2bf8cf1b728f49cfdd43f9fd1b">+75/-4</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add delete_bulk for cipher_keys</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-043222e433d1e5825a4b7bfc2c7feedba417f2630f0082fa67aff64976736396">+76/-1</a> </td> </tr> <tr> <td><strong>templates.rs</strong><dd><code>Implement delete_template_bulk handler</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-b08e274a64cc935b0cec402a04271a872b1586027a77dc0ccb84c597c3960f61">+73/-4</a> </td> </tr> <tr> <td><strong>pipeline.rs</strong><dd><code>Add delete_pipeline_bulk API handler</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-c69f4f94eee97ce226caddc497b2adf89e26869f2d72b0d0206dc31ab2714daf">+70/-2</a> </td> </tr> <tr> <td><strong>reports.rs</strong><dd><code>Add delete_report_bulk endpoint</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-00435e0d9301342b51cb57ae313f1413e7ed973642e0badf3f3959125d0677d1">+75/-1</a> </td> </tr> <tr> <td><strong>ActionScripts.vue</strong><dd><code>Integrate bulk delete UI in action scripts</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-770f71fcb92a13b3b8b1041567b885fac39903c814825c878595cafa624ebbd8">+284/-5</a> </td> </tr> </table></details></td></tr><tr><td><strong>Configuration changes</strong></td><td><details><summary>3 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Register bulk delete services in router</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+14/-0</a> </td> </tr> <tr> <td><strong>iam.ts</strong><dd><code>Add bulkDeleteGroups and bulkDeleteRoles calls</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-3712c1bb0eecb664b6691d724c3673b8c1befa41efbd8c5de626c5849f8ace98">+10/-0</a> </td> </tr> <tr> <td><strong>dashboards.ts</strong><dd><code>Add bulkDelete service for dashboards</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-913b3f54367036b1b3960aa3eddd7e17b509b1d6011c5ab68ae0e935653179c5">+7/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Error handling</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>auth.rs</strong><dd><code>Allow bulk-delete paths in auth extractor</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-9bc2b1841504a435f66ad094f72ddbe7b7fa75e5e1557b8adbfe36f7f0592e1b">+17/-1</a> </td> </tr> </table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>user.rs</strong><dd><code>Define bulk delete request/response structs</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-e13b3a981fd70182b9738a6b0ecfdee915c5b36757703a87741e1eddf3f6e786">+36/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add PipelineBulkDelete meta types</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-c4bd542e4e07ed7706b0643086f5a7d4edbb3647c7ceedbe4bea516c5274d8bd">+12/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>40 files</summary><table> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-6f9c2e6dbc40f8d40e3d3ecbee23240813a3d4856d49bd5d979f5b98551fdc3d">+12/-0</a> </td> </tr> <tr> <td><strong>service_account.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-878ad520e1c84500cc21ff6fc4088c8da64ff2febd79e60c0d4cc8c695c49798">+11/-0</a> </td> </tr> <tr> <td><strong>action.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-fc87c75a7df9539d7e7ee6184dfab02d4a8932323e81f95d57c05363aa44752d">+15/-0</a> </td> </tr> <tr> <td><strong>reports.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-d2117a3d67d4ec47b0017861845c92143baf917e10e6746df5dd97f925f81eeb">+12/-0</a> </td> </tr> <tr> <td><strong>function.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-27c9f3d93723397a0e234bc45c9f543b43eff5fd518d76e735f78d95a58f3f1c">+12/-0</a> </td> </tr> <tr> <td><strong>requests.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-412f11e06c0e901e2292aab80536cb36edebcc4796dec52923389c3aaa32b25a">+6/-0</a> </td> </tr> <tr> <td><strong>responses.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-224027f90893122eea28e35778467acf1a4b5572ce612fc9bc9ec44ca8ebd9fb">+9/-0</a> </td> </tr> <tr> <td><strong>dashboards.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-250eeebeec9d4ac0ac271466ed83cd33fea3f121080491b340dc95d9af287259">+12/-0</a> </td> </tr> <tr> <td><strong>destinations.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-ecffe114c8200bab742dd2ee4f7f41128aa59438a879634f070edd575a1f917c">+24/-0</a> </td> </tr> <tr> <td><strong>search_stream.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-642de43ff6b19ae41087a9f1efc02d8b0b873809e0c92cbddfac1a30ef3f1184">+1/-0</a> </td> </tr> <tr> <td><strong>utils.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-96bb9f4a8f547c821f26c957e8505d0437095b08d2b36687531f71565d287bd6">+2/-1</a> </td> </tr> <tr> <td><strong>functions.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-ed5611a9aa6b5c033e70d7631ba72fafad3cc842672077a7fcf965d11a4fef23">+22/-31</a> </td> </tr> <tr> <td><strong>AppTable.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-efcf66a9e913b5d3a7c2eea800b7f4e54d9e5268307131482fd40f32e2074d8e">+72/-16</a> </td> </tr> <tr> <td><strong>AlertList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-be3e41b823e163b226ffc270d95cc4981e59c600851e4e0d10ee5e4ee994ac8e">+123/-2</a> </td> </tr> <tr> <td><strong>AlertsDestinationList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-e9d05ba6c071c0e2f5d9ef7acdef757f79910abd6897b29e27b069a10d4ba8de">+142/-5</a> </td> </tr> <tr> <td><strong>PipelinesDestinationList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-088c66f5a34e73a5350a794a772e5d905262277337c937dd36f4295072203ebf">+163/-28</a></td> </tr> <tr> <td><strong>TemplateList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-7df66364d7d69a26cfa25cbc555d0e090c2f73e51c8386d9b0435bae4412564c">+116/-27</a></td> </tr> <tr> <td><strong>EnrichmentTableList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-6a29e253f59405d3c0f756dba36afd7483f10ad86927cbbc3348ede6372c3077">+124/-13</a></td> </tr> <tr> <td><strong>FunctionList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-6ff9af50a1597504b7f0dca91e61cf30c800b4a17b4be2ed2c3a4f7a7b2d7068">+143/-6</a> </td> </tr> <tr> <td><strong>AppGroups.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-402fb2a5338e60fdc3da489f4d50b5cf52f06b4e884a6fe7b82441c9fa5ea22c">+86/-1</a> </td> </tr> <tr> <td><strong>AppRoles.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-191634ef09436199f91f442968110a01f54ae46b5ceee67476f52613ad796db4">+86/-1</a> </td> </tr> <tr> <td><strong>EditRole.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-263b6bc7d29fdf8358c8437cd637f1350aa3733cc2706eae8ff00028686cc0f5">+20/-0</a> </td> </tr> <tr> <td><strong>ServiceAccountsList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-3afe835bcfbec3fda2e9550850c29581092ee0717e7dd8de8888e774a0048d24">+114/-1</a> </td> </tr> <tr> <td><strong>User.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-e973c2a665f9b27b94673ba64cc5b23650300836a727a2e07673c8e26305aee5">+133/-3</a> </td> </tr> <tr> <td><strong>PipelinesList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-85e3ef322f3eea0fc6d74d89431ba338a1153cb82ee06bef0f7c5f439c04267d">+108/-2</a> </td> </tr> <tr> <td><strong>ReportList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-f7d700f7e67fbaba3c58ab5d2c614925db743101828214445ce9dae35dc29067">+157/-5</a> </td> </tr> <tr> <td><strong>CipherKeys.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-950b715cff2c964d8542512692030c370733b57d698abf40e1dcb3cce2cfaf7a">+104/-13</a></td> </tr> <tr> <td><strong>RegexPatternList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-524927acd9f5bb07b71bfac489e747470108574e21a6dde63ea7c1f219cea853">+109/-14</a></td> </tr> <tr> <td><strong>action_scripts.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-6c1a011ac23290134801b22feb2b50c2b19cabf2a009f37a62b4dd9ca6e415e3">+3/-0</a> </td> </tr> <tr> <td><strong>alert_destination.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-377d28822e9ad4e9e80327ecc2d09bf1981c7997c5f1c1de120c6179eef19c6e">+3/-0</a> </td> </tr> <tr> <td><strong>alert_templates.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-fd8c4260abc3df601e4331d36b0c211eca568ee70139ad90c87247bc552fe4e2">+3/-0</a> </td> </tr> <tr> <td><strong>alerts.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-7a62782cceacac92b787e058219b3c53ea7406d4be5780def899943a01f500c1">+7/-0</a> </td> </tr> <tr> <td><strong>cipher_keys.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-0a0ef3a63c29b2a74b7f904c54c3fb32bee421608474aa32334b7c55133ab1ef">+3/-0</a> </td> </tr> <tr> <td><strong>jstransform.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-301add2528e0f32323ca4c503bd7d460ab15ca6b9a9537192a02c5e52fd82926">+3/-0</a> </td> </tr> <tr> <td><strong>pipelines.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-e8626670bb4e46e073f9858ade6b3dc77eaadf39d6721d9c0006eccc1b430612">+8/-0</a> </td> </tr> <tr> <td><strong>regex_pattern.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-2a0b0828141d39ab0385e0941314ead1cdd2ec3ec7b3baf5fd0c42f8f010728a">+3/-0</a> </td> </tr> <tr> <td><strong>reports.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-32d3299b2b21dc57c39d01080baba03765d853bd05c82d6158834b54713ae825">+3/-0</a> </td> </tr> <tr> <td><strong>service_accounts.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-4038191b0dc0cad335f3d8f7f703505df905ad843fe950096aefb8abbd139a1e">+4/-1</a> </td> </tr> <tr> <td><strong>users.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-a334d8d570d9bdcaa3d4c8c2a993eff009a1ec8eb499c8a3d1a292d9224159d0">+3/-0</a> </td> </tr> <tr> <td><strong>Dashboards.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-8da3187db8ce3d795d0f196ede6b2290221ffa3e27b15dcc2b645e44dd196213">+122/-6</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> |
||
 Loakesh Indiran
|
0c7d3f7485 |
refactor: cache version v3 centralize cache file_path computation (#9653)
## **Refactor: Centralize cache file path computation v3**
Moves cache file path and histogram metadata calculation to
`prepare_cache_response()` to ensure consistent values across all code
paths.
### **Key changes:**
- Extract `compute_cache_file_path()` and
`refine_is_descending_for_histogram()` helper functions
- Compute `file_path`, `ts_column`, and `is_descending` once before
calling `check_cache()`
- Remove duplicate logic that was recalculating these values in multiple
locations
- and **fixes a critical bug** where the same query generated different
cache hashes.
### **Why:**
Previously, file path components (histogram interval, ts_column) were
computed at different stages, leading to inconsistent cache lookups.
This refactor ensures cache metadata is computed once with the same
refined values, preventing cache misses and duplicate cache entries.
### **Bug Fixed:**
The main branch had inconsistent SQL hashing that caused cache misses
and duplicate cache entries:
- **First request**: SQL hashed before normalization →
`histogram(_timestamp)` → Hash A
- **Subsequent requests**: SQL hashed after normalization →
`histogram(_timestamp,'3600 second')` → Hash B
- **Result**: Same query generates different hashes, causing cache
misses and duplicate cache files
**Proof from logs:**
```
Main branch (buggy):
Request 1: hash_input = "v2,SELECT histogram(_timestamp) AS..." ← Hash: 680371199371727182
Request 2: hash_input = "v2,SELECT histogram(_timestamp,'3600 second')..." ← Hash: 9214012975698861238
Fix branch (correct):
Request 1: hash_input = "v2,SELECT histogram(_timestamp,'3600 second')..." ← Hash: 9214012975698861238
Request 2: hash_input = "v2,SELECT histogram(_timestamp,'3600 second')..." ← Hash: 9214012975698861238
```
___
### **PR Type**
Enhancement, Bug fix
___
### **Description**
- Pre-parse SQL and normalize histogram
- Compute ts_column, is_descending, file_path early
- Simplify check_cache signature and logic
- Add helper funcs for file_path and descending
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["prepare_cache_response"] --> B["parse SQL & normalize histogram"]
B --> C["compute hash"]
C --> D["refine ts_column & is_descending"]
D --> E["compute_file_path"]
E --> F["check_cache"]
F --> G["MultiCachedQueryResponse"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>cacher.rs</strong><dd><code>Simplify check_cache
signature and logic</code>
</dd></summary>
<hr>
src/service/search/cache/cacher.rs
<ul><li>Updated <code>check_cache</code> signature and parameters<br>
<li> Removed inline SQL parsing and path logic<br> <li> Uses precomputed
<code>sql</code>, <code>result_ts_col</code>,
<code>is_descending</code>, <code>file_path</code><br> <li> Added
helpers <code>compute_cache_file_path</code> and
<br><code>refine_is_descending_for_histogram</code></ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9653/files#diff-8c387b22b2c0a282600ee41ca702b3db60f5a284d507c0feb7418608ff062c6c">+123/-92</a></td>
</tr>
<tr>
<td>
<details>
<summary><strong>mod.rs</strong><dd><code>Precompute SQL metadata & file
path</code>
</dd></summary>
<hr>
src/service/search/cache/mod.rs
<ul><li>Moved SQL parsing and histogram normalization to
<br><code>prepare_cache_response</code><br> <li> Refined
<code>ts_column</code> and <code>is_descending</code> before hash
computation<br> <li> Computed <code>file_path</code> once using new
helper<br> <li> Simplified cache vs non-cache branches</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9653/files#diff-f07bfacc0501b9c234e64b16c1dd8eb0ae8fcbe231f90c81fed72bcc94946f74">+83/-33</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
 Subhra264andHengfei Yang
|
6cc7caf6c3 |
fix: only origin region ingesters should process the enrichment table url job (#9782)
### **User description** Currently, when the enrichment table url job is created, it is only processed by the ingesters of the same origin region which is correct. But, the background job responsible for reprocessing the stale jobs can process enrichment table url from other regions as well which is not ideal. This pr, adds a is_local_region field to the enrichment_table_urls table to indicate whether the db record was created in the same region. The background job in ingesters to process stale jobs, hence, now only process the jobs created in that region. This change applies to only the background jobs that processes stale jobs to ensure enrichment table data is present only in the origin region. The actual job processing (when the user creates the job) happens through router and hence is always processed by the ingesters of origin region. ___ ### **PR Type** Bug fix, Enhancement ___ ### **Description** - Add `is_local_region` flag to URL jobs - Filter stale-job recovery by `is_local_region` - Initialize URL processor on ingester startup - Mark remote jobs with `is_local_region=false` ___ ### Diagram Walkthrough ```mermaid flowchart LR JC["URL Job Created\n(is_local_region=true)"] RC["Remote Job Received\n(is_local_region=false)"] DB["enrichment_table_urls\ndatabase record"] SJ["Stale Job Recovery\n(filters is_local_region=true)"] JC -- "save_url_job" --> DB RC -- "super_cluster_queue.put" --> DB SJ -- "claim_stale_jobs" --> DB ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>6 files</summary><table> <tr> <td><strong>enrichment_table_urls.rs</strong><dd><code>Add and propagate `is_local_region` field</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9782/files#diff-e9ba1ba6e742e8d7652c3f31765de43bee061e7983bca67ffefecf8590210188">+11/-6</a> </td> </tr> <tr> <td><strong>enrichment_table_urls.rs</strong><dd><code>Include `is_local_region` in ORM model</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9782/files#diff-9c3081476927bd337b2cbed8310578869845a6137a2c5305f677affb7604d39f">+1/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Initialize URL processor at startup</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9782/files#diff-f2e4c29b8eef22189ec704a832633c4bd135a24444f054864c98064bd0963681">+7/-0</a> </td> </tr> <tr> <td><strong>enrichment_table.rs</strong><dd><code>Set `is_local_region=true` on new jobs</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9782/files#diff-6ce27ddf8a0d8b4e790bda90d088ff692a5da31a8d330beea8a9dd843294572b">+1/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Re-export `init_url_processor` function</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9782/files#diff-3e2b6d3b8da1f9e73aca279052a97e683184d68256c23215b18ffee885836bbf">+3/-0</a> </td> </tr> <tr> <td><strong>url_processor.rs</strong><dd><code>Add `init_url_processor` for recovery task</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9782/files#diff-d39589e34157fe7930563b9bc5740128e168b8d8ac0a90371a4bdf1e52ce058b">+39/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Configuration changes</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>m20251226_000001_add_enrichment_table_urls_is_local_region.rs</strong><dd><code>Migration: add `is_local_region` column</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9782/files#diff-25f7d4220c44499608e8b139ba530335a1426f81988fae2e48d1b41aaacdfb05">+80/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Register new region-flag migration</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9782/files#diff-7ee42a714d3d56c663da0412d15514613222055aebb5b50dafbc051a656a36f6">+2/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Bug fix</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>pipelines.rs</strong><dd><code>Mark remote jobs `is_local_region=false`</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9782/files#diff-7c8aa678045dd00de89d4c2345cf4a0511ce9bfac5d6f64e7404178c81e541ad">+3/-1</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com> |
||
|
Hengfei Yang
|
511f2389c6 | feat: new db migration (#9774) | ||
|
Hengfei Yang
|
e6e86cade3 |
fix: histogram cache issue (#9776)
### **PR Type**
Bug fix
___
### **Description**
- Correct histogram interval application to end_time only
- Remove interval offset from last_ts computation
- Ensure start_time and end_time remain consistent
___
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Bug
fix</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>cacher.rs</strong><dd><code>Adjust histogram interval
offset logic</code>
</dd></summary>
<hr>
src/service/search/cache/cacher.rs
<ul><li>Removed histogram_interval offset from last_ts<br> <li> Added
histogram_interval addition to end_time<br> <li> Ensured correct
start_time and end_time calculation</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9776/files#diff-8c387b22b2c0a282600ee41ca702b3db60f5a284d507c0feb7418608ff062c6c">+2/-3</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
|
Hengfei Yang
|
27f66010e1 | feat: add configuration for index value length limits (#9753) | ||
|
Hengfei Yang
|
e59474aa1b | perf: reduce nats events (#9712) | ||
 Huaijin
|
6082edf321 | feat: follower querier timeout for UI (#9744) | ||
 Zhao.wei
|
ad215f57c2 | fix: Use from_utf8_lossy to handle potential invalid UTF-8 characters (#9730) | ||
|
Loakesh Indiran
|
781f81d4e4 |
fix: generate empty ttv file when index field not present in parquet (#9707)
# Fix: Create Empty Tantivy Index Files for Parquets Without Indexed Fields Scenarios: ### Context: - **Default FTS fields:** `log`, `message`, `msg`, `content`, `data`, `body`, `json`, `error`, `errors` (always included) - **Default secondary index fields:** `trace_id`, `service_name`, `operation_name` (always included) - **latest_schema:** Stream schema with all fields (including user-defined) - **Parquet data:** Actual data in the specific parquet file --- ### **Scenario 1:** All fields present in data - `latest_schema` has: default fields + user fields ✅ - FTS/Index settings: defaults + user configured fields ✅ - Parquet data has: ALL the configured fields ✅ - **Result:** Create .ttv with full data ✅ --- ### **Scenario 2:** Some configured fields missing from parquet data - `latest_schema` has: `body`, `continent`, `name` ✅ - FTS/Index settings: `body` (FTS), `continent`, `name` (index) ✅ - Parquet data has: `body`, `name` only (missing `continent`) ⚠️ - **Result:** Create .ttv with `body` and `name` data, `continent` as empty placeholder ✅ --- ### **Scenario 3:** All configured fields missing from parquet data (but parquet has other data) - `latest_schema` has: `body`, `continent` ✅ - FTS/Index settings: `body` (FTS), `continent` (index) ✅ - Parquet data has: `other_field_1`, `other_field_2` (10 rows, but no `body` or `continent`) ❌ - **Result:** Create .ttv as empty marker (has documents but no field data) ✅ --- ### **Scenario 4:** Parquet file is empty (0 rows) - `latest_schema` has: `body`, `continent` ✅ - FTS/Index settings: `body` (FTS), `continent` (index) ✅ - Parquet data has: 0 rows (empty file) ❌ - **Result:** Create .ttv as empty marker ✅ --- **Key Insight:** We only create empty .ttv files when the configured fields are valid in the stream schema (2). This ensures the .ttv file acts as a proper marker that "this parquet was checked for these fields, they just happen to be empty in this specific file." ## Problem When a stream has full-text search (FTS) or index fields configured in settings, but incoming data stops containing those fields, the system fails to create `.ttv` (tantivy index) files. This causes significant query performance degradation. ### Impact - Files **with** `.ttv` → fast tantivy index search - Files **without** `.ttv` → **slow DataFusion full table scan** with `LIKE '%pattern%'` ### Root Cause Two issues: 1. Code checked if configured fields exist in **parquet schema** (not stream schema) → skipped indexing if fields missing 2. Used parquet schema instead of stream schema for index generation → missing fields excluded from index **Result:** Parquets without indexed fields have `index_size = 0`, forcing expensive DataFusion scans during queries. ## Solution 1. **Always create `.ttv` files** when FTS/index fields are configured, even if parquet data doesn't contain those fields 2. **Use stream schema** (not parquet schema) for index generation to include all configured fields 3. Create empty index as marker: "this parquet was indexed but has no matching data" ## Test Scenarios ### Scenario 1: All Configured Fields Missing - **Config:** `body` for FTS - **Data:** No `body` field - **Result:** Empty `.ttv` created (0 documents) - **Query:** File skipped, no DataFusion scan ### Scenario 2: Partial Fields Missing (Secondary Index) - **Config:** `continent`, `name`, `flag_url` as secondary index - **Data:** Only `name` and `flag_url` present (no `continent`) - **Result:** `.ttv` created with all 3 fields: - `name`: indexed (has data) - `flag_url`: indexed (has data) - `continent`: empty column (no data) - **Query on `continent`:** Returns 0 matches (field exists but empty) ### **PR Type** Bug fix, Enhancement ___ ### **Description** - Align skip logic to stream‐configured index fields - Add `should_create_empty_index` flag in Tantivy module - Generate empty index file if no matching data fields - Enhance debug logs with empty‐index context ___ ### Diagram Walkthrough ```mermaid flowchart LR A["Stream settings"] --> B{"Configured FTS/index fields?"} B -- "No" --> C["Skip index generation"] B -- "Yes" --> D["merge_files calls generate_tantivy_index"] D --> E{"Schema contains fields?"} E -- "Yes" --> F["Build full index"] E -- "No" --> G["Build empty .ttv index"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table> <tr> <td> <details> <summary><strong>parquet.rs</strong><dd><code>Use configured fields check for indexing</code> </dd></summary> <hr> src/job/files/parquet.rs <ul><li>Removed schema‐field presence check<br> <li> Skip index only if no configured fields<br> <li> Updated debug log with config note</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9707/files#diff-27f710981cfe1b938aeeb53abaed05961596c32e2edf3b8a2a59d3e72b2f542b">+6/-12</a> </td> </tr> <tr> <td> <details> <summary><strong>merge.rs</strong><dd><code>Simplify need_index evaluation</code> </dd></summary> <hr> src/service/compact/merge.rs <ul><li>Replace schema lookup with config‐based flag<br> <li> Skip logs now include config context<br> <li> Retain <code>need_index</code> logic for compactor</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9707/files#diff-0ab016dbd5129c9c285c9c459bf78928147bce97dc92a03aa754da3da71168d7">+6/-10</a> </td> </tr> <tr> <td> <details> <summary><strong>mod.rs</strong><dd><code>Support empty index creation flag</code> </dd></summary> <hr> src/service/tantivy/mod.rs <ul><li>Introduce <code>should_create_empty_index</code> variable<br> <li> Only skip when no configured fields exist<br> <li> Log creation of empty index structure<br> <li> Update row‐count debug logging</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9707/files#diff-68837d2f393b453002ec2d95f6882cfcaf70d9ef6547be2c9a890e8b0fc3c559">+17/-3</a> </td> </tr> </table></td></tr></tr></tbody></table> </details> ___ |
||
 Harsh Mahajan
|
2bb67c3b7e |
fix: new kv table for kv store (#9459)
This PR make necessary changing for using new KV table for storing PKCE states and also removes the existing states from meta table |
||
    
|
488a6915be |
feat: dashboard promql support for other charts (#9696)
### **User description** Design at : dashboards ### **PR Type** Enhancement ___ ### **Description** - Introduce modular PromQL chart conversion system - Add data processor and shared utilities - Implement converters for pie, table, heatmap, etc. - Update UI to configure new PromQL chart types ___ ### Diagram Walkthrough ```mermaid flowchart LR A["PromQL Responses"] -- "processPromQLData" --> B["Processed Data"] B -- "convertPromQLChartData" --> C["Converter Registry"] C -- "select converter" --> D["Chart-specific Converter"] D -- "output options & extras" --> E["ECharts Options"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>20 files</summary><table> <tr> <td><strong>dataProcessor.ts</strong><dd><code>Common data preprocessing for PromQL responses</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-363d102e4c9cc1beceb56816113bae703cde4c104e915770f918b16688c602d1">+248/-0</a> </td> </tr> <tr> <td><strong>convertPromQLChartData.ts</strong><dd><code>Main router for PromQL chart conversion</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-7a96651b77954f37d362daf6441ac4f8caff47f0827941ef4618a25774490642">+225/-0</a> </td> </tr> <tr> <td><strong>convertPromQLTimeSeriesChart.ts</strong><dd><code>Add time-series chart converter class</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-c04ec8aaa18dbda0d66caba0a05d06529872dbf7e0628a4ee6ffca56210b45e4">+254/-0</a> </td> </tr> <tr> <td><strong>convertPromQLPieChart.ts</strong><dd><code>Implement pie and donut chart converter</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-9f433d0f746e3c8f07eb27be45b17dad91518720833c0f58b86c85d491eadfeb">+119/-0</a> </td> </tr> <tr> <td><strong>convertPromQLTableChart.ts</strong><dd><code>Implement table chart converter logic</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-94123beacdf1e10ef812b4a6fda870004aa30f65eeb9e5ef6662f907dc50092d">+192/-0</a> </td> </tr> <tr> <td><strong>convertPromQLGaugeChart.ts</strong><dd><code>Add gauge chart converter support</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-91706cfd8de0e4dd864bd1efe1fbd6dc220690963c278a8c3ac46aec51d974d6">+205/-0</a> </td> </tr> <tr> <td><strong>convertPromQLMetricChart.ts</strong><dd><code>Implement single-value metric converter</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-d99617406e007e2185e0552465f456d5f60646e893fa6c45235cf12d0ca055b6">+97/-0</a> </td> </tr> <tr> <td><strong>convertPromQLHeatmapChart.ts</strong><dd><code>Add heatmap chart converter</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-be005732a54fbe2914c68c052407adde3c35b11a81b914ae602e322b4bd63302">+120/-0</a> </td> </tr> <tr> <td><strong>convertPromQLBarChart.ts</strong><dd><code>Support bar and horizon-bar converters</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-15ade277873baedc98e922520bed7e9d9095ecb3ab24ea69f342cdae6c1e7458">+133/-0</a> </td> </tr> <tr> <td><strong>convertPromQLGeoChart.ts</strong><dd><code>Add geographic map chart converter</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-6eb91a67a23394322bc5577c8750a91c19b3e53f54c088afd2dcb017acccfb03">+147/-0</a> </td> </tr> <tr> <td><strong>convertPromQLSankeyChart.ts</strong><dd><code>Implement Sankey diagram converter</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-4790b82ec6453ffad790f22d8f2c2a9747db1b1cf5b397c7ef4e53991a5301dc">+137/-0</a> </td> </tr> <tr> <td><strong>types.ts</strong><dd><code>Define PromQL types and conversion interfaces</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-395705e34877a642f353dcdd90733ac4a9e1a2188304a79dd2b5bb322fde77a8">+138/-0</a> </td> </tr> <tr> <td><strong>axisBuilder.ts</strong><dd><code>Build axis configurations for ECharts</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-2806b132f6ff85d110f34517e7f9dc86566f784fe85593ca2ce5f51bf2cc68df">+162/-0</a> </td> </tr> <tr> <td><strong>legendBuilder.ts</strong><dd><code>Add legend name templating utility</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-e66fbf5127520890aa3e3e8fa82362dc69aa84acd3ba6b639814d250787eb47f">+69/-0</a> </td> </tr> <tr> <td><strong>convertPromQLData.ts</strong><dd><code>Integrate new modular PromQL converter</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-68d70ba6c622911d48cdb567940dc404a4a6468b86c6d24fb54e5f1d8aa5ca2a">+72/-0</a> </td> </tr> <tr> <td><strong>convertPanelData.ts</strong><dd><code>Route table panel by promql queryType</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-15264039050c04ff085ea42722654dfef0ef537b7bef1e9cea59e1c1bb0ce748">+21/-4</a> </td> </tr> <tr> <td><strong>PromQLChartConfig.vue</strong><dd><code>Add UI config for PromQL chart types</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-305fcbfc89618c91ca0ee6c8c52cfa596169ca2dced6778eea6cd3c89f071573">+571/-0</a> </td> </tr> <tr> <td><strong>ChartSelection.vue</strong><dd><code>Include new chart types in selection</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-384ce178717731e2c85f1312f12a54b42ceabd0b86f7e2befb0e421ef568a6a3">+9/-0</a> </td> </tr> <tr> <td><strong>PanelSchemaRenderer.vue</strong><dd><code>Compute and log tableRendererData for PromQL</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-e5376e7e77bf3aaf6590c2522d8c2dbffcc95a8006ae0d6dc0040a37825367d6">+44/-5</a> </td> </tr> <tr> <td><strong>TableRenderer.vue</strong><dd><code>Add debug logging for table data changes</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-794706ea9da3a70012a0327c0464f9fd16802f78ece26055fe951716c2d73d59">+16/-1</a> </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>6 files</summary><table> <tr> <td><strong>PROMQL_CHART_EXTENSION_DESIGN_SPEC.md</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-c28f23229dbc72f164b9986421cd37311a9df8b2c0fe63ad87e8c518a8b61163">+1905/-0</a></td> </tr> <tr> <td><strong>ConfigPanel.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-c3a6e64dff02b58bb7cdc3f0668990b41ff41544b4c5e991bbcd989ad4bba3f8">+10/-0</a> </td> </tr> <tr> <td><strong>PromQLTableChart.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-9c39051461b6f7deae94e8847f91443c4c68541f52137a423bb54b6ab5cd3317">+263/-0</a> </td> </tr> <tr> <td><strong>en.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-892aa3d75ab6e129ef9116aa21bec1b9a15ba69ce5366d2a19128078c690d824">+16/-0</a> </td> </tr> <tr> <td><strong>convertDataIntoUnitValue.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-14c6fe442ef231566ec411f9e0262cdd24ec912fc67b9154eec743282ab7211a">+9/-0</a> </td> </tr> <tr> <td><strong>index.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-bb77f007e906b4ef04c9a4d73dd94adc0e9fb9910a48b872d4587ab37cf3a51e">+65/-0</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ ___ ### **PR Type** Enhancement, Tests ___ ### **Description** - Add PromQL label & operation schema in backend - Define PromQL operations for query builder - Introduce country code mapping util for map charts - Extend useDashboardPanel with PromQL label fetching - Update TableRenderer and add queryModeller tests ___ <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Configuration changes</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Add PromQL filter & operation schema to dashboard meta</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-cbf3fe971525ba58892400391dfd7dac7e43d6049d6b4f5a9840ad104ebc5e08">+52/-0</a> </td> </tr> <tr> <td><strong>convertMapsData.ts</strong><dd><code>Replace inline country list with shared mapper</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-5f61730637542ed11dfb6ef6e4bc72480289b16636243c98d1b628a97cbdfa55">+1/-227</a> </td> </tr> </table></details></td></tr><tr><td><strong>Enhancement</strong></td><td><details><summary>3 files</summary><table> <tr> <td><strong>index.ts</strong><dd><code>Define full set of PromQL operation definitions</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-962f834d1d04c23122d5226deb40f08aa67f41937d70105f2d644b8c15d3dceb">+832/-0</a> </td> </tr> <tr> <td><strong>countryMappings.ts</strong><dd><code>Add ISO country code to name mapping util</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-0c9975df0349e4b2c425b5568f9559537dc40052805c9b4475b9f73bd8752798">+255/-0</a> </td> </tr> <tr> <td><strong>useDashboardPanel.ts</strong><dd><code>Extend dashboard panel composable for PromQL support</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-fbba1700750053b9eaf38ddd30dab84531bc44131c795ebb713811e4930cd7cb">+144/-29</a></td> </tr> </table></details></td></tr><tr><td><strong>Tests</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>TableRenderer.spec.ts</strong><dd><code>Reformat and extend table renderer unit tests</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-550a0a669838c47473442a7d37407fead8b7cb2a05121880dd2c0522022c8c3f">+127/-91</a></td> </tr> <tr> <td><strong>queryModeller.spec.ts</strong><dd><code>Add comprehensive PromQL queryModeller tests</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-1037f836d59596dcc1c3e65b19b5a740e2dfdba9fa813feb8044bc139b1d30e0">+624/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>63 files</summary><table> <tr> <td><strong>PROMQL_CHART_EXTENSION_DESIGN_SPEC.md</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-c28f23229dbc72f164b9986421cd37311a9df8b2c0fe63ad87e8c518a8b61163">+2126/-0</a></td> </tr> <tr> <td><strong>PanelSchemaRenderer.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-e5376e7e77bf3aaf6590c2522d8c2dbffcc95a8006ae0d6dc0040a37825367d6">+45/-2</a> </td> </tr> <tr> <td><strong>ChartSelection.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-384ce178717731e2c85f1312f12a54b42ceabd0b86f7e2befb0e421ef568a6a3">+9/-0</a> </td> </tr> <tr> <td><strong>ConfigPanel.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-c3a6e64dff02b58bb7cdc3f0668990b41ff41544b4c5e991bbcd989ad4bba3f8">+119/-49</a></td> </tr> <tr> <td><strong>DashboardQueryBuilder.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-a4f9abbeaad1156ab6ba5740d59ccb869d6a9172ca1c64b89cff78f3d7783409">+123/-0</a> </td> </tr> <tr> <td><strong>DashboardQueryEditor.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-68bf1b319223d103f1b99e0b681764e4eb76942399397779c81c2cd18f6485ff">+29/-14</a> </td> </tr> <tr> <td><strong>FieldList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-a071c06e3e26629cb4ae4751450acc39c61fa49b07a1cc4be3711697829668d4">+7/-0</a> </td> </tr> <tr> <td><strong>OverrideConfig.spec.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-e05709bd1fededda3d3ed90a47d9fddb8ae171e87c7f25d07ecca1e6a8ab33ee">+19/-3</a> </td> </tr> <tr> <td><strong>OverrideConfig.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-54a6038f5c684ade27b88e3c6a134874b51501904441fa7849a532d10aa7d533">+45/-4</a> </td> </tr> <tr> <td><strong>PromQLChartConfig.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-305fcbfc89618c91ca0ee6c8c52cfa596169ca2dced6778eea6cd3c89f071573">+879/-0</a> </td> </tr> <tr> <td><strong>QueryTypeSelector.spec.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-3094340ad57af51f1bcbad3b63a8ab860a46905ab79997289b89b601553934fd">+1/-1</a> </td> </tr> <tr> <td><strong>QueryTypeSelector.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-0dfd7075714453289e8f65d7f085ff9314148c02ca64fb591d0b8d8cbe1ef0b3">+63/-18</a> </td> </tr> <tr> <td><strong>PromQLTableChart.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-9c39051461b6f7deae94e8847f91443c4c68541f52137a423bb54b6ab5cd3317">+307/-0</a> </td> </tr> <tr> <td><strong>TableRenderer.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-794706ea9da3a70012a0327c0464f9fd16802f78ece26055fe951716c2d73d59">+24/-7</a> </td> </tr> <tr> <td><strong>LabelFilterEditor.spec.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-a7753ad2d6fe4a1878d454dff2b4cb353afca8eabfdc7e131104fa8d2eaf8ef1">+354/-0</a> </td> </tr> <tr> <td><strong>LabelFilterEditor.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-6f8f873d354390719997b0a63051445658997aafab2d36b268e20a1c07b2f845">+327/-0</a> </td> </tr> <tr> <td><strong>MetricSelector.spec.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-2129c149ce2ebddb6a9a17a8a63104057cd138ad6656ce0ccef7e56d25ba1a83">+488/-0</a> </td> </tr> <tr> <td><strong>MetricSelector.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-8c2adc3e91f44534029c70c61cab50e193882fd1d11bb6c75e3a7f0641101238">+172/-0</a> </td> </tr> <tr> <td><strong>OperationsList.spec.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-21e51d0ccdf0744ac3702582d77f4342fc0af0bd916cd953d53be24b9c4dcd13">+448/-0</a> </td> </tr> <tr> <td><strong>OperationsList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-0aeab8d5b16334d3a11d9071ed3b799e866fa0ba2e9b9318f39c3b0c92b5f7e8">+361/-0</a> </td> </tr> <tr> <td><strong>PromQLBuilderOptions.spec.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-0f591de06f0b72c4035885b0c11234c01f8920ea9e6f98336202f7a51e46db58">+462/-0</a> </td> </tr> <tr> <td><strong>PromQLBuilderOptions.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-0118a096d9bd174a3fbe091f62af6f94d6fc6f51c10ad00a1c8660bd5a14f2e9">+236/-0</a> </td> </tr> <tr> <td><strong>queryModeller.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-677776e7803f48b130926c3929a0e9e764b0a57aa4501d0ff5e25123f7b712e1">+246/-0</a> </td> </tr> <tr> <td><strong>index.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-94d2a6c3a1e12028714c8f9e287f562f92d911d04ea1bf375d8535be5d7ab1e8">+197/-0</a> </td> </tr> <tr> <td><strong>usePanelDataLoader.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-5a42e9493a7bcd722b88e61fd38883c4f61001bc9cafa399f65f8eafda3aace9">+2/-1</a> </td> </tr> <tr> <td><strong>router.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-a1827956e5a817edc7efb5f4516e7cc4d453e15ea9e9d970aafb8360b09fc78e">+13/-0</a> </td> </tr> <tr> <td><strong>de.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-78f24ecb2dbcfc38fdf3ec22b2559a5b40d236ea689d1c55f5d3e87168310264">+29/-2</a> </td> </tr> <tr> <td><strong>en.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-892aa3d75ab6e129ef9116aa21bec1b9a15ba69ce5366d2a19128078c690d824">+29/-2</a> </td> </tr> <tr> <td><strong>es.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-ac2a2167b8b973e3f776f9fe3b43bef4576470d8ad33f3d7cfaff6edff0e5778">+29/-2</a> </td> </tr> <tr> <td><strong>fr.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-db80e762b88ddf183820c050cdc6b215e6b02b365b5e73f5152af46b0d26f1a2">+29/-2</a> </td> </tr> <tr> <td><strong>it.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-2de78074a5f887afddce60d694ead9322e6ab223eb70251c13819f3028bdc70d">+29/-2</a> </td> </tr> <tr> <td><strong>ja.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-092e49c5f9fbc1b539ab9c2866d108d0ce1d314022b4c294eb4611b285e7bbd3">+29/-2</a> </td> </tr> <tr> <td><strong>ko.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-50e27ca1b4825a5026f3a251bfcf79c97cab07cfade2ebd512d332453d707c48">+29/-2</a> </td> </tr> <tr> <td><strong>nl.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-a1706021687ae5d8a1c006c49eab3946174fecd895f9b70a89c2e76d97250906">+29/-2</a> </td> </tr> <tr> <td><strong>pt.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-38e804f64cb5e0e451849fc84e9b46e53c954a4a4263d51c18665ad45f4b4815">+29/-2</a> </td> </tr> <tr> <td><strong>tr.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-6f0f131e9e2591480c4c5d1eaffc72d02dc31cbd8d3f90d8f012f08cabac8818">+29/-2</a> </td> </tr> <tr> <td><strong>zh.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-8818055db58f83230aeea4ebfd4cec31d924e8c47b6f91298e356959ab25339e">+29/-2</a> </td> </tr> <tr> <td><strong>Index.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-a6734928ea4dcb5244e90de5a16fe6f5cafa95ccf58430ac7b548a065daf3d61">+1/-1</a> </td> </tr> <tr> <td><strong>convertDataIntoUnitValue.spec.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-e9d8a984ed78af13e4d4424c605a3a55fe6ad2a3f7e403fad156ce17ffdc3bdd">+0/-1</a> </td> </tr> <tr> <td><strong>convertDataIntoUnitValue.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-14c6fe442ef231566ec411f9e0262cdd24ec912fc67b9154eec743282ab7211a">+11/-2</a> </td> </tr> <tr> <td><strong>convertPanelData.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-15264039050c04ff085ea42722654dfef0ef537b7bef1e9cea59e1c1bb0ce748">+63/-12</a> </td> </tr> <tr> <td><strong>convertPromQLData.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-68d70ba6c622911d48cdb567940dc404a4a6468b86c6d24fb54e5f1d8aa5ca2a">+49/-0</a> </td> </tr> <tr> <td><strong>convertPromQLBarChart.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-15ade277873baedc98e922520bed7e9d9095ecb3ab24ea69f342cdae6c1e7458">+241/-0</a> </td> </tr> <tr> <td><strong>convertPromQLChartData.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-7a96651b77954f37d362daf6441ac4f8caff47f0827941ef4618a25774490642">+247/-0</a> </td> </tr> <tr> <td><strong>convertPromQLGaugeChart.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-91706cfd8de0e4dd864bd1efe1fbd6dc220690963c278a8c3ac46aec51d974d6">+206/-0</a> </td> </tr> <tr> <td><strong>convertPromQLGeoChart.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-6eb91a67a23394322bc5577c8750a91c19b3e53f54c088afd2dcb017acccfb03">+208/-0</a> </td> </tr> <tr> <td><strong>convertPromQLHeatmapChart.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-be005732a54fbe2914c68c052407adde3c35b11a81b914ae602e322b4bd63302">+168/-0</a> </td> </tr> <tr> <td><strong>convertPromQLMapsChart.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-8961325d132fe026290bbb3a1de46f058b68e41b4033b0549cdeac5d030a2094">+171/-0</a> </td> </tr> <tr> <td><strong>convertPromQLMetricChart.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-d99617406e007e2185e0552465f456d5f60646e893fa6c45235cf12d0ca055b6">+98/-0</a> </td> </tr> <tr> <td><strong>convertPromQLPieChart.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-9f433d0f746e3c8f07eb27be45b17dad91518720833c0f58b86c85d491eadfeb">+177/-0</a> </td> </tr> <tr> <td><strong>convertPromQLTableChart.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-94123beacdf1e10ef812b4a6fda870004aa30f65eeb9e5ef6662f907dc50092d">+453/-0</a> </td> </tr> <tr> <td><strong>convertPromQLTimeSeriesChart.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-c04ec8aaa18dbda0d66caba0a05d06529872dbf7e0628a4ee6ffca56210b45e4">+241/-0</a> </td> </tr> <tr> <td><strong>index.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-bb77f007e906b4ef04c9a4d73dd94adc0e9fb9910a48b872d4587ab37cf3a51e">+65/-0</a> </td> </tr> <tr> <td><strong>axisBuilder.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-2806b132f6ff85d110f34517e7f9dc86566f784fe85593ca2ce5f51bf2cc68df">+292/-0</a> </td> </tr> <tr> <td><strong>dataProcessor.spec.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-ae46ef8d3c0c85add59609034907f8788d5a41964f492d1c0638f805566d9822">+632/-0</a> </td> </tr> <tr> <td><strong>dataProcessor.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-363d102e4c9cc1beceb56816113bae703cde4c104e915770f918b16688c602d1">+245/-0</a> </td> </tr> <tr> <td><strong>gridBuilder.spec.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-0c4241a8c8473bb69f9eeeefcfdde6fd71c12556c54b73567407136f22825eb3">+690/-0</a> </td> </tr> <tr> <td><strong>gridBuilder.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-1b8f609ff2cd6a36ba25f8bf58f8977c16480da0f92ac1071a06c4751e6c3a4d">+244/-0</a> </td> </tr> <tr> <td><strong>legendBuilder.spec.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-f3f0d6e2c05b75076a68df34e18bb53b2f73d22762205c8013a4fd101688f838">+257/-0</a> </td> </tr> <tr> <td><strong>legendBuilder.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-e66fbf5127520890aa3e3e8fa82362dc69aa84acd3ba6b639814d250787eb47f">+69/-0</a> </td> </tr> <tr> <td><strong>types.spec.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-5af785b1b9d835b13ec80fff7372a9dedb3f5f7612e86afa926f1d8735cdc7fc">+535/-0</a> </td> </tr> <tr> <td><strong>types.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-395705e34877a642f353dcdd90733ac4a9e1a2188304a79dd2b5bb322fde77a8">+148/-0</a> </td> </tr> <tr> <td><strong>QueryBuilder.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9696/files#diff-4d2c58124ad5c6c376c0e07e5ca8f16c1c0f0f0859baf55c8c73b1350c6e8123">+199/-0</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Kirtan Thakkar <kirtan@kiara.tech> Co-authored-by: omkark06 <omkar@zinclabs.io> Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech> |
||
 
|
384aa5ba62 |
fix: issue 9355 (#9706)
### **User description** This PR fixes the issue in #9355 Alert doesn't get triggered if the field seleceted in a non uds field ___ ### **PR Type** Bug fix, Enhancement ___ ### **Description** - Enforce UDS fields in alerts and pipelines UIs - Filter stream fields by defined_schema_fields - Always include timestamp and _all fields - Add `validate` flag to search API calls - Backend search utils: error on non-UDS fields - New alert UI validation: `validateConditionsAgainstUDS` ___ ### Diagram Walkthrough ```mermaid flowchart LR A["Select stream"] --> B["Fetch schema & settings"] B -- "Apply UDS filter" --> C["Available fields list"] C --> D["Build alert conditions"] D --> E["validateConditionsAgainstUDS"] E -- valid --> F["Submit alert"] E -- invalid --> G["Show error notification"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Bug fix</strong></td><td><details><summary>4 files</summary><table> <tr> <td><strong>utils.rs</strong><dd><code>Return error on non-UDS search fields</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9706/files#diff-96bb9f4a8f547c821f26c957e8505d0437095b08d2b36687531f71565d287bd6">+1/-2</a> </td> </tr> <tr> <td><strong>alertValidation.ts</strong><dd><code>Always pass `validate:true` in alert SQL validation</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9706/files#diff-3164a4a58c1865cfcbbba084ac339664c99564545be4d57504dd406d59d8ad4a">+1/-0</a> </td> </tr> <tr> <td><strong>QueryEditorDialog.vue</strong><dd><code>Include `validate:true` in query trigger search</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9706/files#diff-6c524f609ed374aa29e075e8d046ac06be0edda7bfa1914b0a2bbc24d255e5eb">+1/-0</a> </td> </tr> <tr> <td><strong>Query.vue</strong><dd><code>Deep copy routing data and validate SQL</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9706/files#diff-6247cbd506bcf963372f2e429507fc579af95f003df246889b478da11326a657">+20/-10</a> </td> </tr> </table></details></td></tr><tr><td><strong>Enhancement</strong></td><td><details><summary>5 files</summary><table> <tr> <td><strong>search.ts</strong><dd><code>Add `validate` flag to search URL builder</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9706/files#diff-885ab872dbc6e7b20264df815e3ef4727f0abc86ce90d2fb47e4aff572814c75">+4/-0</a> </td> </tr> <tr> <td><strong>AddAlert.vue</strong><dd><code>Filter and validate conditions against UDS</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9706/files#diff-7b426bb23e9ed0250c3877d6d97b099c42b2d20e22ddf4459915381b2233a0a8">+187/-0</a> </td> </tr> <tr> <td><strong>FilterCondition.vue</strong><dd><code>Add column-exists validation rule per module</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9706/files#diff-53916e2402d0ee094fcd63f0ec0c99ceffde4fa042f9001203d2e2e1b7709e10">+28/-1</a> </td> </tr> <tr> <td><strong>FilterGroup.vue</strong><dd><code>Propagate `module` prop to nested filter groups</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9706/files#diff-8d5a73bea3aa0b5c86d48cda4a61ee3e30f27797ac2a5add1ee8bd66c75bef23">+8/-0</a> </td> </tr> <tr> <td><strong>ScheduledPipeline.vue</strong><dd><code>Adjust submit handler and add UDS filter</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9706/files#diff-63d8f838bb880356e7804b4a57fd9223d4f2c189de7aadc7cf33cba723b29aaf">+4/-4</a> </td> </tr> </table></details></td></tr><tr><td><strong>Configuration changes</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>QueryConfig.vue</strong><dd><code>Tag QueryConfig with `module="alerts"`</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9706/files#diff-7273c8de49b8ebf706d7bee5b275c75818c0493f41c73f00c55efe667cbd00d9">+1/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>Condition.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9706/files#diff-4c3e624233c9e0355c7ded3b26dc74971b867be63e24f92a5e639284f60639dd">+37/-1</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Co-authored-by: loaki07 <rustyloaki@gmail.com> Co-authored-by: Shrinath Rao <shnath@openobserve.ai> |
||
|
|
e40938a08c |
feat: enrichment table file url support (#9697)
### **PR Type**
Enhancement
Design at : enrichment-file-url
___
### **Description**
- Async URL-based enrichment table processing
- Event-driven MPSC, streaming CSV, batching
- Retry with backoff, status updates
- Database and ORM additions
- `enrichment_table_urls` table and migration
- HTTP API endpoints
- POST `/url` import and GET status
- Frontend enhancements
- Tabs, status icons, retry button
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["HTTP POST /{org}/enrichment_tables/{table}/url"] --> B["DB: save_url_job"]
B --> C["trigger_url_job_processing()"]
C --> D["MPSC Channel"]
D --> E["process_url_jobs()"]
E --> F["process_single_url_job()"]
F --> G["validate_url() & fetch_and_process_batches()"]
G --> H["parse_csv_chunk() & save_enrichment_batch()"]
H --> I["save_url_job() update status"]
I --> J["notify_update()"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>14
files</summary><table>
<tr>
<td><strong>url_processor.rs</strong><dd><code>Implement async URL
enrichment table processor</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-d39589e34157fe7930563b9bc5740128e168b8d8ac0a90371a4bdf1e52ce058b">+1351/-0</a></td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Add URL enrichment table HTTP
endpoints</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-7b5b203002d1c3e5f671392a2a8ed84f45225896b652c2dac94ac26e72a43124">+301/-1</a>
</td>
</tr>
<tr>
<td><strong>enrichment_table.rs</strong><dd><code>Add URL job CRUD
persistence functions</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-6ce27ddf8a0d8b4e790bda90d088ff692a5da31a8d330beea8a9dd843294572b">+159/-2</a>
</td>
</tr>
<tr>
<td><strong>enrichment_table_urls.rs</strong><dd><code>SeaORM logic for
URL job records</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-e9ba1ba6e742e8d7652c3f31765de43bee061e7983bca67ffefecf8590210188">+173/-0</a>
</td>
</tr>
<tr>
<td><strong>enrichment_table.rs</strong><dd><code>Define URL job status
and job struct</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-29818b7d921497260251df82a49ae9ab7a93bbaedc0ca03abdf2b8c0726e4085">+90/-0</a>
</td>
</tr>
<tr>
<td><strong>pipelines.rs</strong><dd><code>Handle super-cluster URL job
events</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-7c8aa678045dd00de89d4c2345cf4a0511ce9bfac5d6f64e7404178c81e541ad">+23/-1</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Expose url_processor module in
service</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-3e2b6d3b8da1f9e73aca279052a97e683184d68256c23215b18ffee885836bbf">+6/-0</a>
</td>
</tr>
<tr>
<td><strong>enrichment_table_urls.rs</strong><dd><code>Define SeaORM
entity for URL jobs</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-9c3081476927bd337b2cbed8310578869845a6137a2c5305f677affb7604d39f">+26/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Register URL handlers in
router</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+2/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Expose enrichment_table_urls entity
module</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-c539d7557d1425986d5c09849d8dbca42d1b309c2bc33d49db641d6a783098f6">+1/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Add enrichment_table_urls to table
module</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-af023301203a27a1ef34f698d5a329ad7bfced65ac65e3829b1c0905a0c74fb2">+1/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Include enrichment_table meta
module</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-bab79e8da2d0d9f903cb1a6187d922c6de63d3e4561e8ce633660b664b4e859f">+1/-0</a>
</td>
</tr>
<tr>
<td><strong>jstransform.ts</strong><dd><code>Add client methods for URL
jobs API</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-301add2528e0f32323ca4c503bd7d460ab15ca6b9a9537192a02c5e52fd82926">+18/-0</a>
</td>
</tr>
<tr>
<td><strong>EnrichmentTableList.vue</strong><dd><code>Enhance UI for
URL-based enrichment tables</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-6a29e253f59405d3c0f756dba36afd7483f10ad86927cbbc3348ede6372c3077">+327/-74</a></td>
</tr>
</table></details></td></tr><tr><td><strong>Configuration
changes</strong></td><td><details><summary>3 files</summary><table>
<tr>
<td><strong>m20251221_000001_create_enrichment_table_urls.rs</strong><dd><code>Create
migration for URL jobs table</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-330b4b7ebbda5001f4f28050d50e219b91961d80d8369222d94d058cced57702">+189/-0</a>
</td>
</tr>
<tr>
<td><strong>config.rs</strong><dd><code>Add URL fetch configuration
options</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+24/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Register URL jobs migration in
migrator</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-7ee42a714d3d56c663da0412d15514613222055aebb5b50dafbc051a656a36f6">+2/-0</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>1
files</summary><table>
<tr>
<td><strong>openapi.rs</strong><dd><code>Document URL endpoints in
OpenAPI spec</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-edb8ce232c0cd08366879f642788a75d99f6bab64b4c0e77391281fdb9241e90">+5/-0</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Bug
fix</strong></td><td><details><summary>1 files</summary><table>
<tr>
<td><strong>auth.rs</strong><dd><code>Extend auth for enrichment_tables
URLs</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-9bc2b1841504a435f66ad094f72ddbe7b7fa75e5e1557b8adbfe36f7f0592e1b">+9/-1</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Additional
files</strong></td><td><details><summary>1 files</summary><table>
<tr>
<td><strong>AddEnrichmentTable.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-9aad551dbc051edfdb3994a664506ed6d5e27dcb64f9c2fa917b02e51a35d86e">+159/-42</a></td>
</tr>
</table></details></td></tr></tr></tbody></table>
</details>
___
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
|
||
|
Huaijin
|
6a19b96234 | fix: query metrics from grafana (#9718) | ||
 udev
|
07ad3ef921 |
feat: org level concurrency group (#9702)
### **User description**
- **init org level concurrency group**
- **fix seaorm query**
___
### **PR Type**
Enhancement, Tests
___
### **Description**
- Add `org_id` column and composite index migration
- Implement unified `count_all_levels` query function
- Update service logic to pass `org_id` in concurrency checks
- Add tests covering multi-level concurrency scenarios
___
### Diagram Walkthrough
```mermaid
flowchart LR
Migration["Add org_id column and composite index"]
Entity["Update search_queue entity schema"]
Count["Implement count_all_levels function"]
Service["Update service to pass org_id in checks"]
Migration -- "alters table" --> Entity
Entity -- "enables" --> Count
Count -- "used by" --> Service
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>search_queue.rs</strong><dd><code>Add org_id to
search_queue entity</code>
</dd></summary>
<hr>
src/infra/src/table/entity/search_queue.rs
- Add new `org_id: String` field to `Model`
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9702/files#diff-77ac78cec82fad033d11d57153c0fbad0b6d845ee37981b8f4806f008ef2e0ec">+1/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>search_queue.rs</strong><dd><code>Implement
count_all_levels and org_id support</code>
</dd></summary>
<hr>
src/infra/src/table/search_queue.rs
<ul><li>Extend <code>add</code> to accept and set
<code>org_id</code><br> <li> Introduce <code>count_all_levels</code> for
global/org/user counts<br> <li> Add <code>CountResult</code> and SQL
window query logic<br> <li> Add comprehensive tests for concurrency
levels</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9702/files#diff-fa07a79bb4c866a1f50f71e35c35fca84313cea36ac444b91d8934737c4c4349">+281/-3</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>mod.rs</strong><dd><code>Include org_id in cluster
work_group_need_wait</code>
</dd></summary>
<hr>
src/service/search/cluster/mod.rs
<ul><li>Pass <code>org_id</code> to <code>need_wait</code> calls<br>
<li> Update log messages to show global/org/user status</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9702/files#diff-ec9344fb91e5bf19a8c3e44e5ec0b9b00386840776937b6ac7fd24642aca5b2a">+17/-5</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>work_group.rs</strong><dd><code>Update work_group
concurrency to use org_id</code>
</dd></summary>
<hr>
src/service/search/work_group.rs
<ul><li>Consolidate global/org/user concurrency checks<br> <li> Pass
<code>org_id</code> to <code>need_wait</code>, <code>process</code>, and
<code>done</code><br> <li> Simplify skip logic for background tasks</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9702/files#diff-e8a58c4516b0c137403092a06d9aa1922a435d652860cc977709d9d9c1416cfd">+26/-25</a>
</td>
</tr>
</table></td></tr><tr><td><strong>Configuration
changes</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>m20251219_000001_add_org_id_to_search_queue.rs</strong><dd><code>Create
migration for org_id and index</code>
</dd></summary>
<hr>
src/infra/src/table/migration/m20251219_000001_add_org_id_to_search_queue.rs
<ul><li>Create migration adding <code>org_id</code> column (not null,
default "")<br> <li> Create composite index on (work_group, org_id,
user_id)<br> <li> Drop index and column in <code>down</code></ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9702/files#diff-2f973c0036bc473fe1cd676b5a6d8e90c06150ced4b19699abb1882bb1ae7a88">+91/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>mod.rs</strong><dd><code>Register new org_id
migration</code>
</dd></summary>
<hr>
src/infra/src/table/migration/mod.rs
<ul><li>Register
<code>m20251219_000001_add_org_id_to_search_queue</code> module<br> <li>
Include new migration in <code>MigratorTrait</code> list</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9702/files#diff-7ee42a714d3d56c663da0412d15514613222055aebb5b50dafbc051a656a36f6">+2/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
|
Hengfei Yang
|
1714f6c570 | fix: metrics stream can't be deleted (#9710) | ||
|
Loakesh Indiran
|
adcc0ca2b6 |
fix: feat add validate sql query param for _search and _multi_search (#9652)
### **PR Type**
Enhancement, Tests
___
### **Description**
- Add `validate` query parameter to HTTP search handlers
- Invoke `validate_query_fields` before executing search
- Provide `get_bool_from_request` boolean extractor utility
- Add unit tests for query validation helpers
### Existing behavior:
1. lets assume sql `select continent from oly`
2. for the above sql initial run the stream oly does not have any uds ->
UI calls _search/ _search_multi to validate the query in the alerts and
pipelines it will return records with the col continent
### Problem:
1. when user now ads uds to oly without the continent col
2. the apis return response with just _timestamp col
3. and alerts and pipelines fail silently without the actual field
### Solution:
- add a validate query param to the api, when true we parse the sql get
the cols check if part of uds or not
with this approach we also needs to handle the case for queries like
joins, ctes, subqueries, union, etc (where table 1 can have the field ,
where as table 2 does not have the field part of uds)
So to handle this we honor what datafusion does, I did the testing with
datafusion, results are shared in the pr description
- Error message
```json
{
"code": 20004,
"message": "Search field not found: continent. Field exists but not in User-Defined Schema (UDS). Available UDS fields: [\"body\", \"id\", \"name\", \"total_medals\", \"unique_id\"]",
"error_detail": "",
"trace_id": "019b35dba10073aa8b9dc5064442b417"
}
```
### Testing Results
# DataFusion CLI Test Results - Field Validation Behavior
**Date:** 2025-12-20
**DataFusion CLI Version:** v51.0.0
**Purpose:** Determine field validation behavior to align OpenObserve
UDS validation with DataFusion standards
---
## Test Setup
```sql
CREATE TABLE oly (id VARCHAR, name VARCHAR, continent VARCHAR, gold_medals INT, total_medals INT);
INSERT INTO oly VALUES
('USA', 'EUA', 'AME', 40, 126),
('CHN', 'China', 'ASI', 40, 91),
('JPN', 'Japão', 'ASI', 20, 45),
('AUS', 'Austrália', 'OCE', 18, 53),
('FRA', 'França', 'EUR', 16, 64),
('NED', 'Países Baixos', 'EUR', 15, 34),
('GBR', 'Grã-Bretanha', 'EUR', 14, 65),
('KOR', 'República da Coréia', 'ASI', 13, 32),
('ITA', 'Itália', 'EUR', 12, 40),
('GER', 'Alemanha', 'EUR', 12, 33);
CREATE TABLE test1 AS SELECT * FROM oly;
```
**Result:** ✅ 10 rows inserted, test1 table created successfully
---
## Test Results
### Test 1: Basic SELECT with Existing Field
**Query:**
```sql
SELECT continent FROM oly LIMIT 5;
```
**Result:** ✅ **SUCCESS**
```
+-----------+
| continent |
+-----------+
| AME |
| ASI |
| ASI |
| OCE |
| EUR |
+-----------+
5 row(s) fetched.
```
**Conclusion:** DataFusion allows querying existing schema fields.
---
### Test 2: Basic SELECT with Non-Existent Field
**Query:**
```sql
SELECT non_existent_field FROM oly;
```
**Result:** ❌ **ERROR**
```
Schema error: No field named non_existent_field. Valid fields are oly.id, oly.name, oly.continent, oly.gold_medals, oly.total_medals.
```
**Conclusion:** DataFusion **strictly validates** that fields exist in
schema at query time.
---
### Test 3: JOIN with Existing Fields
**Query:**
```sql
SELECT a.continent AS "oly_continent", b.continent AS "test1_continent"
FROM oly AS a
JOIN test1 AS b ON a.name = b.name
LIMIT 5;
```
**Result:** ✅ **SUCCESS**
```
+---------------+-----------------+
| oly_continent | test1_continent |
+---------------+-----------------+
| AME | AME |
| ASI | ASI |
| ASI | ASI |
| OCE | OCE |
| EUR | EUR |
+---------------+-----------------+
5 row(s) fetched.
```
**Conclusion:** DataFusion allows JOIN queries when all fields exist in
their respective table schemas.
---
### Test 4: JOIN with Non-Existent Field
**Query:**
```sql
SELECT a.non_existent_field, b.continent
FROM oly AS a
JOIN test1 AS b ON a.name = b.name;
```
**Result:** ❌ **ERROR**
```
Schema error: No field named a.non_existent_field. Valid fields are a.id, a.name, a.continent, a.gold_medals, a.total_medals, b.id, b.name, b.continent, b.gold_medals, b.total_medals.
```
**Conclusion:** DataFusion **validates per-table fields** in JOIN
queries. If a field doesn't exist in table 'a', it fails even if table
'b' has valid fields.
---
### Test 5: Subquery with Existing Field
**Query:**
```sql
SELECT a.name
FROM oly AS a
WHERE a.continent IN (
SELECT b.continent FROM test1 AS b
)
LIMIT 5;
```
**Result:** ✅ **SUCCESS**
```
+-----------+
| name |
+-----------+
| EUA |
| China |
| Japão |
| Austrália |
| França |
+-----------+
5 row(s) fetched.
```
**Conclusion:** DataFusion allows subqueries when all fields exist in
their respective schemas.
---
### Test 6: Subquery with Non-Existent Field
**Query:**
```sql
SELECT a.name
FROM oly AS a
WHERE a.non_existent_field IN (
SELECT b.continent FROM test1 AS b
);
```
**Result:** ❌ **ERROR**
```
Schema error: No field named a.non_existent_field. Valid fields are a.id, a.name, a.continent, a.gold_medals, a.total_medals.
```
**Conclusion:** DataFusion **validates fields in subquery WHERE
clauses**. Non-existent fields cause immediate errors.
---
### Test 7: UNION with Existing Fields
**Query:**
```sql
SELECT continent FROM oly
UNION
SELECT continent FROM test1
LIMIT 5;
```
**Result:** ✅ **SUCCESS**
```
+-----------+
| continent |
+-----------+
| OCE |
| EUR |
| AME |
| ASI |
+-----------+
4 row(s) fetched.
```
**Conclusion:** DataFusion allows UNION queries when fields exist in all
involved tables.
---
### Test 8: CTE with Existing Field
**Query:**
```sql
WITH oly_data AS (
SELECT name, continent FROM oly
)
SELECT continent, COUNT(*) as cnt
FROM oly_data
GROUP BY continent;
```
**Result:** ✅ **SUCCESS**
```
+-----------+-----+
| continent | cnt |
+-----------+-----+
| AME | 1 |
| ASI | 3 |
| EUR | 5 |
| OCE | 1 |
+-----------+-----+
4 row(s) fetched.
```
**Conclusion:** DataFusion allows CTEs when all fields exist in the base
table schema.
---
### Test 9: CTE with Non-Existent Field
**Query:**
```sql
WITH oly_data AS (
SELECT name, non_existent_field FROM oly
)
SELECT non_existent_field
FROM oly_data;
```
**Result:** ❌ **ERROR**
```
Schema error: No field named non_existent_field. Valid fields are oly.id, oly.name, oly.continent, oly.gold_medals, oly.total_medals.
```
**Conclusion:** DataFusion **validates fields in CTE definitions**.
Non-existent fields in CTEs cause immediate errors.
---
## Summary of Findings
| Test Case | Query Type | Field Status | DataFusion Behavior |
|-----------|-----------|--------------|---------------------|
| 1 | Basic SELECT | Exists | ✅ SUCCESS |
| 2 | Basic SELECT | Non-existent | ❌ ERROR |
| 3 | JOIN | Both exist | ✅ SUCCESS |
| 4 | JOIN | Non-existent in one table | ❌ ERROR |
| 5 | Subquery | Exists | ✅ SUCCESS |
| 6 | Subquery | Non-existent | ❌ ERROR |
| 7 | UNION | Exists | ✅ SUCCESS |
| 8 | CTE | Exists | ✅ SUCCESS |
| 9 | CTE | Non-existent | ❌ ERROR |
---
## Key Conclusions
1. **Strict Schema Validation:** DataFusion performs **strict schema
validation** at query planning time.
2. **No Permissive Behavior:** DataFusion does **not allow** queries
with non-existent fields, regardless of query complexity.
3. **Per-Table Validation:** In JOIN queries, DataFusion validates
fields **per table** - each table reference is validated against its own
schema.
4. **Early Validation:** All field validation happens during query
planning, **before execution**.
5. **Clear Error Messages:** DataFusion provides helpful error messages
listing valid fields when validation fails.
---
## Recommendation for OpenObserve
**OpenObserve should adopt DataFusion's strict validation approach:**
- ✅ Validate all queried fields exist in stream schema
- ✅ Validate per-table in JOIN queries (each table against its own
schema/UDS)
- ✅ Reject queries with non-existent fields at query planning time
- ✅ Apply same validation to subqueries, CTEs, and UNION queries
- ✅ Provide clear error messages listing available fields
This aligns OpenObserve behavior with industry-standard SQL engines and
prevents confusing runtime errors.
---
**Test completed successfully. All results documented for validation
strategy decision.**
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["Receive search request"] --> B["Extract `validate` flag"]
B -- "true" --> C["validate_query_fields"]
C -- "error" --> D["map_error_to_http_response"]
C -- "ok" --> E["Proceed to search execution"]
B -- "false" --> E
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>mod.rs</strong><dd><code>Add query validation in search
handler</code>
</dd></summary>
<hr>
src/handler/http/request/search/mod.rs
<ul><li>Extract <code>validate_query</code> flag from query
parameters<br> <li> Call <code>utils::validate_query_fields</code> when
flag is true<br> <li> Return HTTP error on validation failure</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9652/files#diff-96630a0dbb6dd7b2b0248fa325c7c7041ce31340c32826675a4a70772e9d29f6">+11/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>multi_streams.rs</strong><dd><code>Add query validation
in multi-stream handler</code>
</dd></summary>
<hr>
src/handler/http/request/search/multi_streams.rs
<ul><li>Extract <code>validate_query</code> flag in multi-stream
search<br> <li> Validate SQL fields with
<code>validate_query_fields</code><br> <li> Short-circuit and map errors
if validation fails</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9652/files#diff-c3233cdf1e68cb2455f91457719b9c2f477e735ee16546fdfe111e76768bd054">+15/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>utils.rs</strong><dd><code>Implement query validation
utilities</code>
</dd></summary>
<hr>
src/handler/http/request/search/utils.rs
<ul><li>Implement <code>get_bool_from_request</code> boolean
extractor<br> <li> Add <code>validate_query_fields</code> to check SQL
columns against schema<br> <li> Provide helper functions: range adjust,
stream resolve, system-field <br>check<br> <li> Include unit tests for
query helpers</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9652/files#diff-96bb9f4a8f547c821f26c957e8505d0437095b08d2b36687531f71565d287bd6">+192/-2</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
|
27704a0d4c |
fix: add alert revamp v2 (#9694)
### **PR Type**
Enhancement, Tests
___
### **Description**
- Introduce `generate_sql` API endpoint for alerts
- Implement request/response models & service linkage
- Register route & update OpenAPI documentation
- Add extensive front-end tests for alert wizard
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["POST /v2/{org_id}/alerts/generate_sql"] -- "validate & check perms" --> B["build_sql service"]
B -- "return generated SQL & metadata" --> C["GenerateSqlResponseBody"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>4
files</summary><table>
<tr>
<td><strong>mod.rs</strong><dd><code>Add `generate_sql` alert API
endpoint</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-83431801d27626ef27482e67b62661f900043544a45ddf958bd027cf2fa3bd8d">+150/-6</a>
</td>
</tr>
<tr>
<td><strong>responses.rs</strong><dd><code>Add `GenerateSqlResponseBody`
and metadata</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-224027f90893122eea28e35778467acf1a4b5572ce612fc9bc9ec44ca8ebd9fb">+25/-0</a>
</td>
</tr>
<tr>
<td><strong>requests.rs</strong><dd><code>Add `GenerateSqlRequestBody`
request model</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-412f11e06c0e901e2292aab80536cb36edebcc4796dec52923389c3aaa32b25a">+16/-1</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Make `build_sql` function
public</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-ba9b65130898444f7815fd6068bdb0dc590612f74cc893f79d91b0afc44cc851">+1/-1</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Configuration
changes</strong></td><td><details><summary>1 files</summary><table>
<tr>
<td><strong>mod.rs</strong><dd><code>Register `generate_sql` service
route</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+1/-0</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>1
files</summary><table>
<tr>
<td><strong>openapi.rs</strong><dd><code>Include `generate_sql` in
OpenAPI spec</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-edb8ce232c0cd08366879f642788a75d99f6bab64b4c0e77391281fdb9241e90">+1/-0</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Tests</strong></td><td><details><summary>3
files</summary><table>
<tr>
<td><strong>AlertSettings.spec.ts</strong><dd><code>Add comprehensive
unit tests for AlertSettings</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-4ca37593c11ef263e0b9a9e1733678e2b81f01c8da5f4a8d729306c176802356">+1363/-0</a></td>
</tr>
<tr>
<td><strong>AddAlert.spec.ts</strong><dd><code>Extend AddAlert tests for
SQL & wizard flows</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-84362398fea8abd9acf30f4f40ba8d4cf27348677454017bd4e6f568d5176ac6">+808/-56</a></td>
</tr>
<tr>
<td><strong>CompareWithPast.spec.ts</strong><dd><code>Add tests for
CompareWithPast component</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-5718f3c114808926e2cced290580cb8ae1610338042eebad001e9a7d5429ee77">+596/-0</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Additional
files</strong></td><td><details><summary>45 files</summary><table>
<tr>
<td><strong>AddAlert.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-7b426bb23e9ed0250c3877d6d97b099c42b2d20e22ddf4459915381b2233a0a8">+1295/-409</a></td>
</tr>
<tr>
<td><strong>AlertList.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-be3e41b823e163b226ffc270d95cc4981e59c600851e4e0d10ee5e4ee994ac8e">+58/-63</a>
</td>
</tr>
<tr>
<td><strong>AlertSummary.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-c22aa1c5934b763f1d4a5a61a7e810b6bcb1d60b76b64c61cb3ce89dc0a04499">+169/-0</a>
</td>
</tr>
<tr>
<td><strong>AlertWizardRightColumn.spec.ts</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-eb14abefd15bc1d3db947347b49899fe264138b7aea63729feae27ad7abed606">+851/-0</a>
</td>
</tr>
<tr>
<td><strong>AlertWizardRightColumn.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-f599a1960d1ac37e352854ad368d23e080b3cec2b217df22f40d0e15a4db6986">+291/-0</a>
</td>
</tr>
<tr>
<td><strong>CustomConfirmDialog.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-0fa5a1da2050cbab522801af8ade04a0e4c27e87933d21f6c32a82603c77a0a3">+179/-0</a>
</td>
</tr>
<tr>
<td><strong>DeduplicationConfig.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-63eb39111409780e4dc064ea80056dfe096038ee76b389cdd8e9d14c0ae29a32">+3/-2</a>
</td>
</tr>
<tr>
<td><strong>FilterGroup.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-8d5a73bea3aa0b5c86d48cda4a61ee3e30f27797ac2a5add1ee8bd66c75bef23">+39/-39</a>
</td>
</tr>
<tr>
<td><strong>HorizontalStepper.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-ccca09440e61bf97621fe06ca606ade4f7fc7c1df8f8a3742cafa4c352f2c23a">+370/-0</a>
</td>
</tr>
<tr>
<td><strong>OrganizationDeduplicationSettings.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-f526d40b50f1e96f025973c6fc251f7a50f698c641c7749b32e6bb90b9ba032f">+19/-21</a>
</td>
</tr>
<tr>
<td><strong>PreviewAlert.spec.ts</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-6d5bfc82662dd55768e258ab92676c2559c054d235ef0e48beb4bd90f31ca60b">+795/-0</a>
</td>
</tr>
<tr>
<td><strong>PreviewAlert.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-760f5f0a261910f17eb167f81802e8a52477b99a83d2d7fc1aeefc7185801795">+41/-16</a>
</td>
</tr>
<tr>
<td><strong>QueryEditorDialog.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-6c524f609ed374aa29e075e8d046ac06be0edda7bfa1914b0a2bbc24d255e5eb">+981/-0</a>
</td>
</tr>
<tr>
<td><strong>RealTimeAlert.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-c3ae6b47832eb33f9ca877cc7165d079de14850e5c5d43a227102ba40326b9d8">+43/-24</a>
</td>
</tr>
<tr>
<td><strong>ScheduledAlert.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-5e143a17ab0fa5d7817f6d9301667fb5597236f27184248c7a71728e85b8d999">+304/-270</a></td>
</tr>
<tr>
<td><strong>Advanced.spec.ts</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-6dacd8b2947f53e7c39a54e990ee26ff6d63478b3a68897ab3afbc937c65c647">+683/-0</a>
</td>
</tr>
<tr>
<td><strong>Advanced.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-eae849f0bf6c8868f5640ce1d74862284d8e8a5330e5046c0c437e6672f4f676">+376/-0</a>
</td>
</tr>
<tr>
<td><strong>AlertSettings.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-1d5bfd6a1ea672884193649b1271d751d6f3b8ec84af759071c9bfd169a8ab28">+1441/-0</a></td>
</tr>
<tr>
<td><strong>AlertSetup.spec.ts</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-ed5a1c54e5fb03ce1d8121223d40606c74419b998b6c071667037347a2debde9">+878/-0</a>
</td>
</tr>
<tr>
<td><strong>AlertSetup.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-4f89438c74f329de730b0e7451d4a8d7df3bab52cdeaded776e8ae18d94294eb">+290/-0</a>
</td>
</tr>
<tr>
<td><strong>CompareWithPast.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-6606848e30caef2cda13d65cb30d6590205556259e7e6daf39cada825502d55b">+416/-0</a>
</td>
</tr>
<tr>
<td><strong>Deduplication.spec.ts</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-d3281cd7632a0fa7ce5c5054e80efafb2aa8d272bb2c586d723436b644565ba4">+591/-0</a>
</td>
</tr>
<tr>
<td><strong>Deduplication.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-f8d256bcec370851c9799c13ffa2b7571c07ca27af1b5a74c335e7084157c9b1">+204/-0</a>
</td>
</tr>
<tr>
<td><strong>QueryConfig.spec.ts</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-5a3ac3e9d93e87bc95c87a20f08c9972158a5882d66ecfb658d38255fe1d8f71">+1000/-0</a></td>
</tr>
<tr>
<td><strong>QueryConfig.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-7273c8de49b8ebf706d7bee5b275c75818c0493f41c73f00c55efe667cbd00d9">+594/-0</a>
</td>
</tr>
<tr>
<td><strong>de.json</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-78f24ecb2dbcfc38fdf3ec22b2559a5b40d236ea689d1c55f5d3e87168310264">+70/-0</a>
</td>
</tr>
<tr>
<td><strong>en.json</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-892aa3d75ab6e129ef9116aa21bec1b9a15ba69ce5366d2a19128078c690d824">+89/-15</a>
</td>
</tr>
<tr>
<td><strong>es.json</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-ac2a2167b8b973e3f776f9fe3b43bef4576470d8ad33f3d7cfaff6edff0e5778">+70/-0</a>
</td>
</tr>
<tr>
<td><strong>fr.json</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-db80e762b88ddf183820c050cdc6b215e6b02b365b5e73f5152af46b0d26f1a2">+70/-0</a>
</td>
</tr>
<tr>
<td><strong>it.json</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-2de78074a5f887afddce60d694ead9322e6ab223eb70251c13819f3028bdc70d">+70/-0</a>
</td>
</tr>
<tr>
<td><strong>ja.json</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-092e49c5f9fbc1b539ab9c2866d108d0ce1d314022b4c294eb4611b285e7bbd3">+70/-0</a>
</td>
</tr>
<tr>
<td><strong>ko.json</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-50e27ca1b4825a5026f3a251bfcf79c97cab07cfade2ebd512d332453d707c48">+70/-0</a>
</td>
</tr>
<tr>
<td><strong>nl.json</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-a1706021687ae5d8a1c006c49eab3946174fecd895f9b70a89c2e76d97250906">+70/-0</a>
</td>
</tr>
<tr>
<td><strong>pt.json</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-38e804f64cb5e0e451849fc84e9b46e53c954a4a4263d51c18665ad45f4b4815">+70/-0</a>
</td>
</tr>
<tr>
<td><strong>tr.json</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-6f0f131e9e2591480c4c5d1eaffc72d02dc31cbd8d3f90d8f012f08cabac8818">+70/-0</a>
</td>
</tr>
<tr>
<td><strong>zh.json</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-8818055db58f83230aeea4ebfd4cec31d924e8c47b6f91298e356959ab25339e">+70/-0</a>
</td>
</tr>
<tr>
<td><strong>alerts.ts</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-7a62782cceacac92b787e058219b3c53ea7406d4be5780def899943a01f500c1">+9/-0</a>
</td>
</tr>
<tr>
<td><strong>app.scss</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-5703d12504b0395cbfbbfad02a592bc391496026e5f03743102ce06c210ea05f">+7/-0</a>
</td>
</tr>
<tr>
<td><strong>alertQueryBuilder.ts</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-5c829ce5fd3ad2f5ef359df5976e6adfc52eaa6b5f249f594c8b8a588acf8b3e">+10/-49</a>
</td>
</tr>
<tr>
<td><strong>alertSummaryGenerator.ts</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-a4464ad7df3ac3afdca4e6fc583122d701bace10e3a2daf083cb6c902565b20f">+393/-0</a>
</td>
</tr>
<tr>
<td><strong>alertValidation.ts</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-3164a4a58c1865cfcbbba084ac339664c99564545be4d57504dd406d59d8ad4a">+8/-8</a>
</td>
</tr>
<tr>
<td><strong>conditionsFormatter.spec.ts</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-12eb070ce171e81117000fefaf2a1037af4c337423344f280c21c77e1836f5e3">+237/-0</a>
</td>
</tr>
<tr>
<td><strong>conditionsFormatter.ts</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-d5df71a7ab2451925cd220828fc3776039e56c206a07ac7b72d4ce5e282c257c">+460/-0</a>
</td>
</tr>
<tr>
<td><strong>focusManager.ts</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-987334b24e56735ac755d8e1add737ffcd35fe11612e31586c21824cbea88c36">+179/-0</a>
</td>
</tr>
<tr>
<td><strong>zincutils.ts</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9694/files#diff-d3d59688cccdcc6c84ebe88b698c6910030abecc3125f4c0ef81b0df023699ba">+11/-0</a>
</td>
</tr>
</table></details></td></tr></tr></tbody></table>
</details>
___
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Ashish Kolhe <ashish.j.kolhe@gmail.com>
Co-authored-by: Uddhav Dave <uddhav@openobserve.ai>
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
|
||
|
ac60e181dd |
feat: dashboard join support for auto sql (#9343)
# Nested Functions and Joins - User Guide ## Overview This guide covers two powerful features that enable sophisticated data analysis in OpenObserve: - **Nested Functions**: Combine multiple functions together to create complex calculations - **Joins**: Merge data from multiple streams (data sources) based on common fields These features work in the **Auto SQL** mode of dashboard panels, allowing you to write advanced queries without deep SQL knowledge. --- ## What's New ### Why These Features? **Before:** - Limited function composition - you could only use basic, single-level functions - No way to combine data from multiple streams in one query - Complex analytics required exporting data and manual processing **Now:** - Compose unlimited nested functions with full DataFusion support - Join multiple streams directly in the dashboard panel editor - Create sophisticated analytics queries with a visual interface ### Key Benefits ✓ Calculate complex metrics (error rates, percentiles, conditional aggregations) ✓ Build advanced dashboards without custom code ✓ Support for all DataFusion aggregate and scalar functions --- ## Nested Functions ### What Are Nested Functions? Nested functions let you combine multiple functions together - using the output of one function as input to another. Think of it like a cooking recipe where each step builds on the previous one. **Simple Example:** - Basic: `AVG(response_time)` - calculates average response time - Nested: `ROUND(AVG(response_time), 2)` - calculates average AND rounds it to 2 decimal places ### Supported Functions OpenObserve supports **all DataFusion functions**, giving you powerful analytics capabilities: **Aggregate Functions** (work across multiple rows): - `count`, `sum`, `avg`, `min`, `max` - basic aggregations - `median`, `stddev`, `variance` - statistical functions - `percentile_cont` - calculate percentiles (e.g., p95, p99) - `string_agg` - combine text values **Mathematical Functions**: - `round`, `ceil`, `floor` - rounding operations - `abs`, `power`, `sqrt` - mathematical operations - `log`, `log10`, `log2` - logarithmic functions - `sin`, `cos`, `tan` - trigonometric functions **String Functions**: - `concat`, `upper`, `lower`, `trim` - `substring`, `length`, `replace` **Conditional Functions**: - `CASE WHEN ... THEN ... ELSE ... END` - conditional logic - `COALESCE` - return first non-null value 📖 **Full Reference**: [DataFusion Aggregate Functions](https://datafusion.apache.org/user-guide/sql/aggregate_functions.html) | [DataFusion Scalar Functions](https://datafusion.apache.org/user-guide/sql/scalar_functions.html) --- ### Using Nested Functions (Step-by-Step) #### Step 1: Create or Edit a Dashboard Panel Open your dashboard and create a new panel or edit an existing one. Make sure you're in **Auto SQL** mode (not Custom SQL). #### Step 2: Select a Function In the Y-Axis or column configuration, click **Select Function** to choose your base function. <img width="795" alt="Nested Function - Adding Layers" src="https://github.com/user-attachments/assets/0b6b134e-59b8-48b1-a4ae-52fb4800d772" /> In this example, we're using the `Concat` function to combine multiple fields (namespace and pod name). #### Step 3: Add Nested Layers Click the function icon to add another function layer. You can nest functions multiple times. <img width="793" alt="Nested Function - Select Base Function" src="https://github.com/user-attachments/assets/4bd9e03e-1104-4e82-b2ec-a34e6bb64870" /> Here we're building a nested function: - Inner layer: `Avg(severity)` - calculates average severity - Outer layer: `Ceil()` - rounds up the average to the nearest integer #### Step 4: Build Complex Expressions For advanced analytics, you can combine nested functions with conditional logic using `CASE WHEN` statements. <img width="1340" alt="Complex Nested Function with CASE WHEN" src="https://github.com/user-attachments/assets/4c3e3623-bfad-44c4-a760-e10cc94d4a95" /> This example shows a sophisticated calculation: ```sql ROUND(100.0 * SUM(CASE WHEN COALESCE(severity, 0) >= 4 THEN 1 ELSE 0 END) / COUNT(_timestamp), 2) ``` **What this does:** 1. `COALESCE(severity, 0)` - replaces null severity with 0 2. `CASE WHEN severity >= 4 THEN 1 ELSE 0` - returns 1 for high severity, 0 otherwise 3. `SUM(...)` - counts high severity events 4. Divide by `COUNT(_timestamp)` - calculates percentage 5. Multiply by `100.0` - converts to percentage 6. `ROUND(..., 2)` - rounds to 2 decimal places **Result**: Percentage of high-severity events, rounded to 2 decimals #### Step 5: View the Generated SQL Switch to the **Raw** tab to see the complete SQL query that was generated from your nested function configuration. <img width="1388" alt="View Generated SQL" src="https://github.com/user-attachments/assets/d7f6111a-26c6-4dd0-9e9f-e26092fe6368" /> You can review the auto-generated SQL in the Query section at the bottom of the panel editor. The chart visualization updates in real-time as you build your nested functions. --- ## Joins ### What Are Joins? Joins allow you to combine data from two or more different streams (data sources) based on common fields between them. **Example:** **employee_data_join** stream: ```json [ {"employee_id": 1, "name": "Alice", "department_id": 1}, {"employee_id": 2, "name": "Bob", "department_id": 2}, {"employee_id": 3, "name": "Charlie", "department_id": null}, {"employee_id": 4, "name": "David", "department_id": 1}, {"employee_id": 5, "name": "Eve", "department_id": 3} ] ``` **dept_data_join** stream: ```json [ {"department_id": 1, "department_name": "HR"}, {"department_id": 2, "department_name": "Engineering"}, {"department_id": 3, "department_name": "Sales"}, {"department_id": 4, "department_name": "Marketing"} ] ``` A join lets you combine these streams to see the department name alongside each employee, matched by `department_id`. ### Types of Joins OpenObserve supports three types of joins: | Join Type | Description | When to Use | |-----------|-------------|-------------| | **INNER JOIN** | Returns only records with matches in both streams | When you need data that exists in both sources | | **LEFT JOIN** | Returns all records from left stream + matching records from right | When you want all data from your primary source, with supplemental info where available | | **RIGHT JOIN** | Returns all records from right stream + matching records from left | When you want all data from your secondary source, with supplemental info where available | > **Note:** UNION and CROSS joins are not currently supported. --- ### Using Joins (Step-by-Step) #### Step 1: Access the Joins Section In your dashboard panel editor, locate the **Joins** section. Click the **+** button to add a join. <img width="1725" alt="Access Joins Interface" src="https://github.com/user-attachments/assets/35db4f28-c526-43a2-bdd1-470e188f53d7" /> In this example: - **Primary stream**: `employee_data_join` (contains employee names) - We'll join it with `dept_data_join` (contains department information) - The interface shows available fields from both streams in the left panel #### Step 2: Configure the Join Click on the **Joins** dropdown and select the stream you want to join. Then configure: 1. **Joining Stream**: Select the stream to join (e.g., `employee_data_join`) 2. **Join Type**: Choose from LEFT, INNER, or RIGHT using the visual toggle 3. **On Stream**: Select the target stream (e.g., `dept_data_join`) <img width="1725" alt="Configure Join Type" src="https://github.com/user-attachments/assets/35db4f28-c526-43a2-bdd1-470e188f53d7" /> The interface displays: **"Performing a Left Join between employee_data_join and dept_data_join"** Notice the visual toggle showing three join types: - **Left** (selected in this example) - **Inner** - **Right** #### Step 3: Define the Joining Clause Set up the join condition by specifying which fields to match between the two streams: **Clause 1:** - **Select Field** (Left): Choose a field from the first stream (e.g., `department_id`) - **Select Operator**: Choose comparison operator (usually `=`) - **Select Field** (Right): Choose the matching field from the second stream (e.g., `department_id`) You can add multiple clauses using the **+** button for complex join conditions (e.g., joining on multiple fields or adding additional filters). #### Step 4: View the Results After configuring the join, the data table will show combined results from both streams. <img width="1724" alt="Join Results" src="https://github.com/user-attachments/assets/94ca9f33-6cfa-4ad1-9b84-bee9db34f912" /> In this example: - **Name** column from `employee_data_join` (Alice, Bob, Charlie, David, Eve) - **Department Name** column from `dept_data_join` (HR, Engineering, HR, Sales) - Data successfully joined on `department_id` #### Step 5: Review the Generated SQL Check the **Query** section at the bottom to see the generated SQL: ```sql SELECT employee_data_join.name as "x_axis_1", stream_0.department_name as "x_axis_2" FROM "employee_data_join" LEFT JOIN "dept_data_join" AS stream_0 ON employee_data_join.department_id = stream_0.department_id ``` The SQL query shows exactly how OpenObserve is combining your data streams. --- --- ## Examples ### Example 1: String Concatenation with Nested Functions **Use Case**: Combine multiple fields into a single display value ```sql Concat(k8s_namespace_name, '-', k8s_pod_name) ``` This joins namespace and pod names with a hyphen separator. ### Example 2: Statistical Analysis with Nested Math Functions **Use Case**: Calculate rounded average of a metric ```sql Ceil(Avg(severity)) ``` This calculates the average severity level and rounds up to the nearest whole number. ### Example 3: Conditional Percentage Calculation **Use Case**: Calculate percentage of high-severity events ```sql ROUND(100.0 * SUM(CASE WHEN COALESCE(severity, 0) >= 4 THEN 1 ELSE 0 END) / COUNT(_timestamp), 2) ``` **Breakdown:** - Handles null values with `COALESCE` - Counts events where severity >= 4 - Calculates percentage of total events - Rounds to 2 decimal places ### Example 4: Employee-Department Join **Use Case**: Show employee names with their department information ```sql SELECT employee_data_join.name as "Name", dept_data_join.department_name as "Department Name" FROM employee_data_join LEFT JOIN dept_data_join ON employee_data_join.department_id = dept_data_join.department_id ``` This displays all employees with their department names, showing null for employees without a department assignment. ### Example 5: Complex Analytics - Joins + Nested Functions **Use Case**: Calculate error rate per service with deployment information ```sql SELECT logs.service_name, deployments.version, COUNT(*) as total_requests, ROUND((COUNT(CASE WHEN logs.status_code >= 500 THEN 1 END) * 100.0 / COUNT(*)), 2) as error_rate_percent FROM service_logs as logs LEFT JOIN deployments ON logs.service_name = deployments.service_name GROUP BY logs.service_name, deployments.version ORDER BY error_rate_percent DESC ``` This combines: - Join between service logs and deployment data - Nested functions for conditional counting and percentage calculation - Grouping and sorting for analysis --- ## Tips and Best Practices ### Nested Functions Best Practices ✅ **Start Simple**: Build your function one layer at a time - Begin with inner function: `AVG(response_time)` - Add outer layers: `ROUND(AVG(response_time), 2)` ✅ **Test Each Layer**: Verify results before adding complexity - Test the inner function first - Add one layer at a time and check output ✅ **Handle Null Values**: Use `COALESCE` to avoid unexpected nulls ```sql COALESCE(AVG(response_time), 0) ``` ✅ **Use Clear Aliases**: Name your calculated fields descriptively ```sql ROUND(AVG(response_time), 2) as avg_response_ms ``` ✅ **Check for Division by Zero**: Protect against zero denominators ```sql CASE WHEN COUNT(*) > 0 THEN (SUM(errors) * 100.0 / COUNT(*)) ELSE 0 END as error_rate ``` ### Joins Best Practices ✅ **Choose the Right Join Type**: - **INNER JOIN**: Both streams must have matching data - **LEFT JOIN**: Keep all records from primary stream - **RIGHT JOIN**: Keep all records from secondary stream ✅ **Verify Field Compatibility**: - Match data types (don't join string to number) - Ensure field names are correct - Check that values actually match between streams ✅ **Consider Performance**: - Join on indexed fields when possible - Test with limited data first --- ## FAQ ### Q: Can I nest more than 2 functions? **A**: Yes! You can nest as many functions as needed. Example: ```sql ROUND(CEIL(AVG(COALESCE(response_time, 0))), 2) ``` ### Q: What's the difference between LEFT and RIGHT joins? **A**: - **LEFT JOIN**: Keeps all records from the first (left) stream - **RIGHT JOIN**: Keeps all records from the second (right) stream - Choose based on which dataset you want to preserve completely ### Q: Can I join more than 2 streams? **A**: Yes! Click the **+** button in the Joins section to add additional joins. You can chain multiple joins together. ### Q: Do nested functions impact performance? **A**: Complex nested functions may take longer to compute, especially on large datasets. Test with small time ranges first and optimize your expressions. ### Q: Can I use Custom SQL instead of Auto SQL? **A**: Yes! If you prefer writing raw SQL, switch to **Custom SQL** mode. However, Auto SQL provides a visual interface that's easier for building queries. --- ## Additional Resources 📚 **DataFusion Documentation**: - [Aggregate Functions](https://datafusion.apache.org/user-guide/sql/aggregate_functions.html) - COUNT, SUM, AVG, etc. - [Scalar Functions](https://datafusion.apache.org/user-guide/sql/scalar_functions.html) - ROUND, CONCAT, CASE, etc. 🔗 **OpenObserve Resources**: - [GitHub Repository](https://github.com/openobserve/openobserve) - [PR #9343 - Feature Implementation](https://github.com/openobserve/openobserve/pull/9343) - [OpenObserve Documentation](https://openobserve.ai/docs/) 💬 **Get Help**: - [GitHub Issues](https://github.com/openobserve/openobserve/issues) - Report bugs or request features - [Community Discussions](https://github.com/openobserve/openobserve/discussions) - Ask questions and share ideas --- Design at: https://github.com/openobserve/designs/tree/main/joins --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Shrinath Rao <shnath@openobserve.ai> |
||
|
Loakesh Indiran
|
cdc49125a3 |
fix: clear_cache file path (#9674)
|
||
 Sai NikhilandByteBaker
|
e1e91ad54f |
fix: incident analysis changes (#9673)
### **PR Type**
Enhancement
___
### **Description**
- Backend: refactor agent_chat_stream for pooling & immediate status
- Frontend: rotating “Observing…” UX & stream event handling
- Incident drawer → full-screen dialog with two-column tabs layout
- Minor UI tweaks: dialogs maximized & list wrapper update
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["agent_chat_stream endpoint"] -->|"get_agent_client()" B["Global pooled client"]
B -->|"yield processing status" C["text/event-stream start"]
C -->|"client.query_stream()" D["External RCA Agent Service"]
D -->|"stream bytes" E["Forward chunks to client"]
D -->|"on error" F["Yield error event"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>chat.rs</strong><dd><code>Refactor agent chat streaming
& pooling</code>
</dd></summary>
<hr>
src/handler/http/request/agent/chat.rs
<ul><li>Re-export enterprise AgentChat types under
cfg(feature="enterprise")<br> <li> Provide stub types &
init_agent_client for OSS builds<br> <li> Use global OnceCell client for
connection pooling<br> <li> Rewrite streaming: send immediate
"processing" status and handle <br>errors</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9673/files#diff-a0e13f17674a3657690f9e5829da2f1c4a6a957fe8a95f9c44b854643b5fe9b7">+68/-52</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>main.rs</strong><dd><code>Extend AI init to Agent
client</code>
</dd></summary>
<hr>
src/main.rs
<ul><li>Extend AI init to include agent client startup<br> <li> Update
OpenAPI init log messages</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9673/files#diff-42cb6807ad74b3e201c5a7ca98b911c5fa08380e942be6e4ac5807f8377f87fc">+3/-3</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>SREChat.vue</strong><dd><code>Enhance chat loading UX &
stream handler</code>
</dd></summary>
<hr>
web/src/components/SREChat.vue
<ul><li>Replace static loading text with rotating “Observing…”
messages<br> <li> Add start/stop rotation & updateAssistantMessage
helpers<br> <li> Support new event types: status, tool_call,
tool_response, error<br> <li> Improve spacing, formatting and error
handling in stream parser</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9673/files#diff-a24e3f6392181aa741979fe1c0cbc6f0f02337de20e6ad2fd6668dd7abe9bada">+131/-62</a></td>
</tr>
<tr>
<td>
<details>
<summary><strong>IncidentDetailDrawer.vue</strong><dd><code>Refactor
incident drawer to dialog UI</code>
</dd></summary>
<hr>
web/src/components/alerts/IncidentDetailDrawer.vue
<ul><li>Switch from QDrawer to full-screen QDialog wrapper<br> <li>
Implement two-column layout with tabs for analysis & triggers<br> <li>
Add tile components for status, severity, alerts, timeline<br> <li>
Themed styling, responsive scrollbars and updated actions</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9673/files#diff-6973d58a5f49658b07d3533d4db5d32cc6430433b51732481e7497243dadb0b0">+531/-179</a></td>
</tr>
<tr>
<td>
<details>
<summary><strong>IncidentList.vue</strong><dd><code>Wrap incident detail
in QDialog</code>
</dd></summary>
<hr>
web/src/components/alerts/IncidentList.vue
<ul><li>Wrap IncidentDetailDrawer in QDialog instead of drawer<br> <li>
Update v-model and close event handling</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9673/files#diff-953ad015996758a8c86e489499cd3b0a614501b99b0c9c5c242a299e34c24ff2">+11/-4</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>TracesAnalysisDashboard.vue</strong><dd><code>Maximize
traces dashboard dialog</code>
</dd></summary>
<hr>
web/src/plugins/traces/metrics/TracesAnalysisDashboard.vue
- Change dialog maximized prop from false to true
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9673/files#diff-d27dd619532ec278c376c09a05d5106fdb38863c3aa1c56e109fc2145a8a5a3a">+1/-1</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
---------
Co-authored-by: ByteBaker <42913098+ByteBaker@users.noreply.github.com>
|
||
|
udev
|
61dbc6ff7e |
fix: pipeline/alert history and trigger schema merge (#9665)
### **User description**
We need to ensure that whenever a trigger stream is being created (be it
meta or non-meta org) we have the complete schema of the stream. This
will prevent history queries from failing with `field not present` error
We need to make pipeline history query against own orgs trigger stream
we need to create filter with the alert ID for single alert
___
### **PR Type**
Bug fix, Enhancement
___
### **Description**
- fix alert history filtering to use key suffix
- fix pipeline history query to own org triggers
- enhance triggers schema initialization logic
___
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Bug
fix</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>history.rs</strong><dd><code>Use key suffix filter for
alert history</code>
</dd></summary>
<hr>
src/handler/http/request/alerts/history.rs
<ul><li>updated alert_id filter to use suffix match<br> <li> removed
LIKE prefix filter escaping logic</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9665/files#diff-1c7483ca7a66b00f13cefdecadd02afbdf4baf93df5629adb8abe5942cfe5357">+3/-4</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>history.rs</strong><dd><code>Query pipeline history
from own org</code>
</dd></summary>
<hr>
src/handler/http/request/pipelines/history.rs
<ul><li>replaced META_ORG_ID with dynamic org_id<br> <li> applied own
org context for count and data queries</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9665/files#diff-aa1f4b2b5539a10dbabffb51724c916aac3d6b60e25ec150b54a32a06a6fc5d4">+2/-3</a>
</td>
</tr>
</table></td></tr><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>triggers_schema.rs</strong><dd><code>Enhance triggers
schema initialization logic</code>
</dd></summary>
<hr>
src/service/self_reporting/triggers_schema.rs
<ul><li>removed META_ORG_ID conditional exclusion<br> <li> eliminated
pre-existence schema check<br> <li> added expected_schema equality check
before merge</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9665/files#diff-135de2980c6c20b6e5c8294e6bfd2c1c3dedb5995a6671a74726579e77ac983c">+16/-21</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
|
06df3fb784 |
feat: Issue 9646 promql streaming (#9648)
### **User description**
Design at: dashboard
___
### **PR Type**
Enhancement
___
### **Description**
- Enable HTTP/2 streaming for PromQL queries
- Extend streaming search util with PromQL support
- Add PromQL streaming endpoints in service layer
- Define TypeScript interfaces for PromQL streaming
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["Dashboard Loader"] -- "calls stream" --> B["fetchQueryDataWithHttpStream"]
B -- "chunk data" --> C["handlePromQLResponse"]
B -- "error" --> D["handlePromQLError"]
B -- "complete" --> E["handlePromQLComplete"]
C --> F["state.data update"]
D --> G["state.errorDetail update"]
E --> H["final loading false"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>usePanelDataLoader.ts</strong><dd><code>Implement
PromQL streaming in data loader</code>
</dd></summary>
<hr>
web/src/composables/dashboard/usePanelDataLoader.ts
<ul><li>Remove non-streaming PromQL calls<br> <li> Initialize state for
streaming queries<br> <li> Process each query via
fetchQueryDataWithHttpStream<br> <li> Handle streaming chunks, errors,
completion</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9648/files#diff-5a42e9493a7bcd722b88e61fd38883c4f61001bc9cafa399f65f8eafda3aace9">+193/-73</a></td>
</tr>
<tr>
<td>
<details>
<summary><strong>useStreamingSearch.ts</strong><dd><code>Add PromQL
streaming to HTTP streaming hook</code>
</dd></summary>
<hr>
web/src/composables/useStreamingSearch.ts
<ul><li>Extend StreamResponseType with <code>promql_response</code><br>
<li> Support <code>promql</code> type in fetch/initiate functions<br>
<li> Build GET URL for PromQL streaming endpoint<br> <li> Map
<code>promql_response</code> via convertToPromQLResponse</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9648/files#diff-8fce3e25a161cd458e734bd9b1f98aadefe3482da7b6529a5b1f4fa20ec115e2">+51/-16</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>streaming_search.ts</strong><dd><code>Add PromQL
streaming endpoints to service</code>
</dd></summary>
<hr>
web/src/services/streaming_search.ts
<ul><li>Add <code>promqlQueryRangeStreamUrl</code> endpoint builder<br>
<li> Add <code>promqlQueryRange</code> streaming request method<br> <li>
Include query, time window and metadata params</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9648/files#diff-6a8e19a226bbf9e7e04903d5818c8fb5c9005a48988419f94d3e63a61ade7c57">+78/-0</a>
</td>
</tr>
</table></td></tr><tr><td><strong>Documentation</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>query.ts</strong><dd><code>Define PromQL streaming
TypeScript interfaces</code>
</dd></summary>
<hr>
web/src/ts/interfaces/query.ts
<ul><li>Introduce <code>PromQLStreamingPayload</code> and
<code>PromQLQueryPayload</code><br> <li> Add
<code>PromQLStreamingResponse</code> interface</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9648/files#diff-12dd5b3bb9741bbeaa348c0efbbd745d0a582899678738159c6927d34a35191d">+21/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
---------
Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech>
Co-authored-by: Huaijin <haohuaijin@gmail.com>
Co-authored-by: Abhay Padamani <141122205+ktx-abhay@users.noreply.github.com>
|
||
 Ashish Kolhe
|
bd647bd341 |
fix: organizations ep permisions issue (#9667) (#9669)
fixes: https://github.com/openobserve/openobserve/issues/9670 ### **PR Type** Bug fix ___ ### **Description** - Prioritize `META_ORG_ID` user in token validator - Prioritize `META_ORG_ID` user in credentials validators - Fallback to first available user when no META membership ___ ### Diagram Walkthrough ```mermaid flowchart LR A["Incoming request"] --> B["get_all_users()"] B --> C{"META_ORG_ID user exists?"} C -- "yes" --> D["Select META org user"] C -- "no" --> E["Select first user"] D --> F["Continue validation"] E --> F ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Bug fix</strong></td><td><table> <tr> <td> <details> <summary><strong>token.rs</strong><dd><code>Prioritize META org user in token validation</code> </dd></summary> <hr> src/handler/http/auth/token.rs <ul><li>Added logic to prioritize <code>META_ORG_ID</code> user<br> <li> Fallback to first user if META org missing<br> <li> Inserted explanatory comment for permission rationale</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9669/files#diff-3339b4a964c121766ef87fcec0070f27a2b7085cf81281b53154e53d33cb28cf">+6/-1</a> </td> </tr> <tr> <td> <details> <summary><strong>validator.rs</strong><dd><code>Prioritize META org user in credentials validators</code> </dd></summary> <hr> src/handler/http/auth/validator.rs <ul><li>Applied META org prioritization logic<br> <li> Replaced <code>all_users.first()</code> with META find or fallback<br> <li> Updated both <code>validate_credentials</code> and <code>validate_credentials_ext</code></ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9669/files#diff-f4d4b51574b378a8c372345ea48875834a4eadacfac33fa4d0ceca1e0ef96b35">+14/-2</a> </td> </tr> </table></td></tr></tr></tbody></table> </details> ___ |
||
|
sktrpathi
|
a9b5f8174f |
fix: eliminate HashMap conversion in service streams for memory reduction (#9662)
### **User description** ## Summary Replace HashMap conversion with Arrow-native processing to eliminate 350 MB memory waste per 100K-row file during service discovery, resolving OOM issues during high-volume ingestion. ## Problem Service correlation was causing OOM due to HashMap conversion overhead: - Converting Arrow RecordBatch → Vec<HashMap> wasted 350 MB per 100K-row file - System memory reached 90% threshold with just 2 concurrent files - OOM kills during peak ingestion periods ## Solution Process Arrow RecordBatch directly without HashMap conversion: - Send RecordBatch through channel instead of Vec<HashMap> - Call `process_arrow_batch()` instead of `process_records()` - Remove `record_batch_to_hashmap()` helper (no longer needed) - Remove `record_batch_size` config (no longer needed) ## Changes - **parquet.rs**: Updated ingestion path to use Arrow-native processing - **merge.rs**: Updated compaction path to use Arrow-native processing - Channel protocol changed from `Vec<HashMap>` to `RecordBatch` - Consumer calls `process_arrow_batch()` on enterprise processor ## Impact | Metric | Before | After | Improvement | |--------|--------|-------|-------------| | Memory per 100K-row file | 565 MB | 80 MB | **86% reduction** | | Service discovery time | 2.5s | 1.5s | **1.7× faster** | | Pattern learning time | 250ms | 2ms | **119× faster** | | Concurrent file capacity | 2 files (OOM) | 10+ files | **5× increase** | | System memory threshold | 90% | 60% | **Stable** | ## Architecture All core Arrow-native logic is implemented in o2-enterprise: - `process_arrow_batch()` - Direct Arrow processing - `observe_arrow_batch()` - Vectorized pattern learning - Multi-type support: String, Int64, Boolean, Float64 - Zero-copy operations wherever possible This PR contains only the OSS integration layer that calls these enterprise methods. ## Testing - ✅ Builds successfully (both OSS and enterprise) - ✅ All existing tests pass - ✅ Feature-gated with `#[cfg(feature = "enterprise")]` - ✅ Zero impact on non-enterprise builds ## Environment Variables - ❌ Removed: `O2_SERVICE_STREAMS_RECORD_BATCH_SIZE` (no longer needed) - ✅ Updated: `O2_SERVICE_STREAMS_CHANNEL_CAPACITY` help text (now describes Arrow batches) - ✅ All other variables unchanged ## Deployment Impact - **Backward compatible** - existing configs continue to work - **No action required** - system automatically uses optimized code path - **Can safely remove** `O2_SERVICE_STREAMS_RECORD_BATCH_SIZE` if explicitly set ## Related - Depends on: o2-enterprise#XXXX (Arrow-native implementation) - Resolves: OOM issues during high-volume ingestion - Implements: PRD_ARROW_NATIVE_SERVICE_DISCOVERY_V2.md ## Checklist - [x] Code compiles without errors - [x] No warnings introduced - [x] Enterprise feature properly gated - [x] Backward compatible - [x] Memory reduction verified - [x] Performance improvement verified --- **Review focus:** - Integration points with enterprise code - Channel protocol changes - Feature flag usage - Memory/performance impact ___ ### **PR Type** Enhancement, Bug fix ___ ### **Description** - Send Arrow RecordBatch via channel directly - Producer: drop HashMap conversion overhead - Consumer: call process_arrow_batch on batches - Remove record_batch_size config and conversion helper |
||
|
udev
|
d8aaefc9c6 |
fix: trigger stream schema (#9654)
ingest function does not consider fields with null value as part of
schema. for example:
```
{
"log": "request id : camelcase",
"test": null
}
```
the inferred schema will only contain `log` . The first push to triggers
stream does not have all the fields populated, fields like `query_took`
and `delay_in_secs` are set to `None` , which are serialized to json
with `null` values.
The fn `ensure_triggers_stream_initialized` updates the schema in the
meta table prior to pushing the actual event such that even if the
actual event does not have all fields or has fields with null values
triggers stream will still have the complete schema.
The problem:
When reporting to own org, we do not run
`ensure_triggers_stream_initialized` for the reporting org. Hence the
inferred schema is incomplete and alert/pipeline history calls fail with
`field does not exist` error.
### **PR Type**
Bug fix
___
### **Description**
- Ensure triggers stream schema before data ingestion
- Add warning log if schema initialization fails
- Refine schema existence check by field count
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["ingest_buffered_data"] --> B["ensure_triggers_stream_initialized"]
B -- "schema ok or warn" --> C["ingest_reporting_data"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Bug
fix</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>queues.rs</strong><dd><code>Add schema initialization
before ingestion</code>
</dd></summary>
<hr>
src/service/self_reporting/queues.rs
<ul><li>Insert schema initialization call before ingestion<br> <li>
Handle errors with warning logs<br> <li> Maintain existing ingestion
logic on failure</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9654/files#diff-ed15a56d4d3becc2dfb3d05af2dac7f92962ec3091199728fa953e6aed7ed767">+8/-0</a>
</td>
</tr>
</table></td></tr><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>triggers_schema.rs</strong><dd><code>Enhance schema
existence check</code>
</dd></summary>
<hr>
src/service/self_reporting/triggers_schema.rs
<ul><li>Check existing schema fields before skipping creation<br> <li>
Count fields to verify non-empty schema<br> <li> Improve logs with field
count and creation info</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9654/files#diff-135de2980c6c20b6e5c8294e6bfd2c1c3dedb5995a6671a74726579e77ac983c">+12/-6</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
|
Yashodhan JoshiandYashodhan Joshi
|
14a41bf63c |
fix: handling of cache cleaning for user delete (#9656)
### **User description** This fixes the issue of how we handle the in-mem cache cleanup for user deletion, specifically in case where use only has a single org attributed to them. This also adds few more error logs so if something goes wrong, we know that it went wrong instead of silently ignoring that. Cherry pick of https://github.com/openobserve/openobserve/pull/9655 for main branch ___ ### **PR Type** Bug fix, Enhancement ___ ### **Description** - Propagate errors and log failures for NATS notifications - Fix cache watch removal logic for multi-org users - Add error handling in HTTP user-remove handler ___ <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Error handling</strong></td><td><table> <tr> <td> <details> <summary><strong>org_users.rs</strong><dd><code>Add error logging and fix cache watch removal</code> </dd></summary> <hr> src/service/db/org_users.rs <ul><li>Replace ignored NATS delete calls with <code>inspect_err</code> logging<br> <li> Propagate errors from <code>delete_from_db_coordinator</code><br> <li> Revise <code>watch</code> to remove multi-org user records from cache</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9656/files#diff-3cbb516ea021b4c03cf4b5f9de2ebcd4a8afee0722a4bcc5c39cb78f485900b6">+24/-14</a> </td> </tr> <tr> <td> <details> <summary><strong>user.rs</strong><dd><code>Log and propagate user delete notifications</code> </dd></summary> <hr> src/service/db/user.rs <ul><li>Turn ignored delete notification into logged error via <code>inspect_err</code><br> <li> Propagate errors from <code>delete_from_db_coordinator</code><br> <li> Maintain super-cluster delete under enterprise flag</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9656/files#diff-6ee6b0fc9ed03224d4ddf0b7b631c6f81ff7d1d579bdb98d2c5d4453852eb5d6">+3/-1</a> </td> </tr> <tr> <td> <details> <summary><strong>users.rs</strong><dd><code>Enhance HTTP handler error handling for removals</code> </dd></summary> <hr> src/service/users.rs <ul><li>Replace <code>match</code> with <code>if let Err</code> for invite deletion logging<br> <li> Add error-check and early return for <code>db::user::delete</code><br> <li> Handle <code>org_users::remove</code> errors with logging and HTTP error response</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9656/files#diff-98da2588a28d62bb2e301d989836125f89f89b76272cd7f75e08ec4deeea5f38">+49/-33</a> </td> </tr> </table></td></tr></tr></tbody></table> </details> ___ --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> |
||
|
Yashodhan Joshi
|
330a179d1c | refactor: file list dump and stream stats (#9236) | ||
|
Harsh MahajanandHengfei Yang
|
a0c9d319fb |
fix: adding description as variable to alerts (#9505)
resolves https://github.com/openobserve/openobserve/issues/9460 adding description as variable to alerts https://github.com/openobserve/openobserve/issues/9460#issuecomment-3616618836 Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com> |
||
|
udev
|
19446a3d32 |
fix: empty user email on grpc metrics ingest (#9608)
check auth can return req without populating metadata, this pr fixes the case and adds an `IngestUser` type to mark all system initiated ingestion calls and user initiated ingestion calls |
||
|
Loakesh Indiran
|
8277305f16 | fix: auto uds to also include fts and indes fields from stream settings (#9512) | ||
|
Hengfei YangandLoakesh Indiran
|
c473752df0 |
fix: cache delta calculate (#9623)
Co-authored-by: Loakesh Indiran <66478092+Loaki07@users.noreply.github.com> |
||
|
Huaijin
|
91ae9301e0 | refactor: enhance ingester search handle with memtable id (#9625) | ||
|
Huaijin
|
6246b7bb27 | feat: add peak memory usage to usage report (#9609) | ||
|
sktrpathiandsai nikhil kethe
|
e52d000217 |
feat: add incident correlation system (#9507)
## Summary Introduces comprehensive incident management system to reduce alert fatigue through intelligent correlation and AI-powered root cause analysis. **Design at:** https://github.com/openobserve/designs/tree/main/alert-correlation --- ## Incident Correlation **3-Tier Correlation Priority:** 1. **Trace-based**: Groups alerts from same distributed trace (cross-service) 2. **Service Discovery**: Uses semantic service matching with normalization 3. **Dimension-based**: Blake3 hash from stable alert dimensions (fallback) **Features:** - Automatic alert grouping into incidents - Real-time incident status tracking (open/acknowledged/resolved/suppressed) - Topology enrichment via Service Graph integration - Cross-environment correlation prevention via dimension matching ## Incident Analysis (RCA) **Automated Analysis:** - Background job queries open incidents and triggers AI agent - Periodic analysis via configurable interval - Results displayed in incident detail view with markdown support **On-Demand Analysis:** - Manual trigger via API: `POST /api/v2/{org}/alerts/incidents/{id}/rca` - Real-time results with loading indicators - One-click "Analyze" button in UI ## SRE Chat Assistant **Conversational AI for Alerts:** - Generic `/api/{org}/agent/chat` endpoint with history support - Context-aware assistance for alerts and incidents - Sparkle icons on alert/incident rows for quick access - Persistent chat history via IndexedDB - Includes RCA analysis in conversation context --- ## Configuration ```bash # Incident Management O2_INCIDENTS_AUTO_RESOLVE_AFTER_MINUTES=-1 # -1=disabled (default), >0=auto-resolve # RCA Agent Integration O2_INCIDENTS_RCA_ENABLED=true O2_INCIDENTS_RCA_AGENT_URL=http://localhost:8000 O2_INCIDENTS_RCA_INTERVAL_SECS=600 ``` ## API Endpoints - `GET /api/v2/{org}/alerts/incidents` - List incidents with pagination - `GET /api/v2/{org}/alerts/incidents/{id}` - Get incident details - `PATCH /api/v2/{org}/alerts/incidents/{id}` - Update incident status - `GET /api/v2/{org}/alerts/incidents/stats` - Get incident statistics - `POST /api/v2/{org}/alerts/incidents/{id}/rca` - Trigger RCA analysis - `POST /api/{org}/agent/chat` - Generic agent chat endpoint - `POST /api/{org}/agent/chat/stream` - Streaming agent chat ## Database - **New tables**: `alert_incidents`, `alert_incident_alerts` - **Migration**: `m20251204_000001_create_alert_incidents_table` - **ORM**: SeaORM entities with full CRUD support ## UI Components - **IncidentList.vue**: Status filtering, pagination, statistics dashboard - **IncidentDetailDrawer.vue**: Full incident view with triggers/alerts/analysis - **SREChat.vue**: AI assistant panel with markdown rendering - Incidents tab in AlertList with unified view - AI sparkle icons for contextual chat access --- ## Enterprise Feature This is an **enterprise-only** feature with OSS stubs for compilation compatibility. - Non-enterprise builds: Handlers return `403 Forbidden` - Enterprise builds: Full functionality enabled ## Test Plan - [x] Build passes without `--features enterprise` - [x] Build passes with `--features enterprise` - [x] Create alerts that fire and verify incidents are created - [x] Verify incident list/detail APIs work correctly - [x] Verify status updates (open → acknowledged → resolved) - [x] Verify correlation logic (trace-based, service discovery, dimension-based) - [x] Verify RCA analysis triggers and displays correctly - [x] Verify SRE chat integration with alert/incident context --------- Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> |
||
|
Hengfei Yang
|
91cab4395c | fix: allow merge files for the job which have no file (#9604) | ||
|
Hengfei Yang
|
24b89ebf50 | fix: enhance stream stats handling and reset functionality (#9595) | ||
|
Loakesh Indiran
|
afc0d18d9b |
fix: return 401 Unauthorized instead of 403 Forbidden for authentication failures (#9564)
## Problem APIs were returning **403 Forbidden** when tokens were invalid or expired, which is incorrect. According to HTTP specifications: - **401 Unauthorized**: Authentication failure (invalid/expired token, user not found) - **403 Forbidden**: Authorization failure (authenticated but lacks permissions) ## Changes - [x] Previously, short URL paths would redirect to `/web/` on auth failure. Now they consistently return 401 like all other endpoints. ### token.rs - Changed `ErrorForbidden` to `ErrorUnauthorized` when token validation fails (`is_valid == false`) - Changed `ErrorForbidden` to `ErrorUnauthorized` when user does not exist in the system - Changed `ErrorForbidden` to `ErrorUnauthorized` for non-enterprise builds (unsupported auth method) - Kept `ErrorForbidden` only for `check_permissions()` failures (true authorization issues) |
||
|
Yashodhan JoshiandYashodhan Joshi
|
4780a9edc6 |
fix: rbac for native user for license call (#9529)
Fixes an issue where a native user in non-default account would get 401 for license api call, whereas we want all users to have get access to license call. --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> |
||
|
Huaijin
|
fb4ea9ab4f | feat: approx peak memory usage for datafusion (#9562) | ||
|
Yashodhan JoshiandYashodhan Joshi
|
6493f2b003 |
fix: cloud org related fixes (#9416)
Fixes for various p2 bugs and UI issues, some other functionality changes as well. NOTE: because of some CLA issues, I have squashed all the original commits into a single one. There are no code changes compared to before. Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> |
||
|
Subhra264
|
0e772b1ac3 |
fix: scheduled pipelines should use the alert timeout (#9358)
Currently, scheduled pipeline uses the report timeout when run by the alert manager which is not intended. Report timeout is generally bigger as it is a time-taking job. The Scheduled pipeline should use the alert timeout instead, as it is similar to alert, only ingestion request is extra. |
||
|
Ashish Kolheandgithub-actions[bot]
|
34939d0769 |
fix: fqn co relation user settings (#9536)
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
|
Huaijin
|
7484b46089 | perf: gc stringview before send recordbatch to leader (#9525) | ||
|
Huaijin
|
fe0bc6c297 |
fix: temp disable datafusion topk optimizer (#9519)
releated to https://github.com/apache/datafusion/issues/19219 |
||
|
79bbe72ee3 |
feat: co-relation phase 2 (#9513)
Design at : streams-corelation --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> |
||
|
Huaijin
|
3f4e98fd05 | feat: remove filter for metrics (#9508) | ||
|
Hengfei Yang
|
2124d122a6 | perf: improve realtime trigger (#9492) |