991 Commits

Include renames

Author	SHA1	Message	Date
Hengfei Yang	11d855a149	fix: migration of router (#10008)	2026-01-13 19:52:20 +08:00
Hengfei Yang	89e067778b	refactor: simplify header extraction in search handlers (#10006)	2026-01-13 17:24:34 +08:00
Hengfei Yang	33eecaefe3	refactor: migrate actix-web to axum (#9912)	2026-01-13 16:44:53 +08:00
Subhra264	b25b7eb772	fix: dashboard should not be created using PUT dashboard update api (#9999) ... ### **PR Type** Bug fix, Enhancement ___ ### **Description** - Add `folder` query param for all dashboard endpoints - Return 404, 403, and 409 HTTP errors - Enforce dashboard-folder ownership in service ___ ### Diagram Walkthrough ```mermaid flowchart LR Client["Client request with folder param"] -- "calls" --> Handler["HTTP handler"] Handler -- "validates folder & params" --> Service["update_dashboard"] Service -- "checks dashboard in folder" --> DB["dashboards table"] DB -- "not found" --> Service Service -- "returns DashboardNotFound" --> Handler Handler -- "sends 404/403/409 response" --> Client ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table> <tr> <td> <details> <summary><strong>mod.rs</strong><dd><code>Add folder param and error responses</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> src/handler/http/request/dashboards/mod.rs <ul><li>Added <code>folder</code> Option<String> query parameter to endpoints<br> <li> Introduced 404, 403, and 409 response statuses<br> <li> Updated API docs with RBAC requirement details</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9999/files#diff-1b3c257e1d174c826718499faa4a1d4e93465dd56cb2b7b273b690ff7e3223fd">+13/-0</a>&nbsp; &nbsp; </td> </tr> </table></td></tr><tr><td><strong>Bug fix</strong></td><td><table> <tr> <td> <details> <summary><strong>mod.rs</strong><dd><code>Enforce folder existence in update</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> src/service/dashboards/mod.rs <ul><li>Check dashboard exists within specified folder<br> <li> Return <code>DashboardNotFound</code> if none found<br> <li> Added explanatory comments for foreign key logic</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9999/files#diff-6ae9e22ced40c50675dde5348b8d696f6559655bbbded915d7e6e88bf64b71a6">+9/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> </table></td></tr></tr></tbody></table> </details> ___	2026-01-13 13:40:53 +05:30
Subhra264	36860c7a59	fix: allow replacing the failed enrichment table url (#9958) ... ### **User description** If there are multiple successful urls added for an enrichment table, but the last append was failed, user now can replace the failed url with a new one to rectify if any url issue is there. ___ ### **PR Type** Bug fix, Enhancement ___ ### **Description** - Add `replace_failed` flag to enrichment table API - Backend: allow replacing only failed job URL - UI: new Replace Failed URL update mode - Adjust client and handler logic for replace_failed ___ ### Diagram Walkthrough ```mermaid flowchart LR UI["Vue UI"] -->|select replace_failed| Client["jstransform Service"] Client -->|POST replace_failed| API["Request Handler"] API -->|replace_failed| ProcessFailed["Replace Failed Job"] API -->|append & has_failed| RejectAppend["Reject Append Request"] API -->|retry| RetryAll["Retry All Jobs"] API -->|else| ReplaceAll["Replace All Jobs"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table> <tr> <td> <details> <summary><strong>jstransform.ts</strong><dd><code>Add replace_failed to API client method</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> web/src/services/jstransform.ts <ul><li>Added <code>replace_failed</code> parameter to method signature<br> <li> Included <code>replace_failed</code> in POST request body</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9958/files#diff-301add2528e0f32323ca4c503bd7d460ab15ca6b9a9537192a02c5e52fd82926">+2/-1</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td> <details> <summary><strong>mod.rs</strong><dd><code>Support replace_failed logic in handler</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> src/handler/http/request/enrichment_table/mod.rs <ul><li>Added <code>replace_failed</code> field with serde default<br> <li> Extracted <code>replace_failed</code> from request body<br> <li> Updated append validation to allow replace_failed<br> <li> Added branch for replacing only failed job URL</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9958/files#diff-7b5b203002d1c3e5f671392a2a8ed84f45225896b652c2dac94ac26e72a43124">+49/-4</a>&nbsp; &nbsp; </td> </tr> <tr> <td> <details> <summary><strong>AddEnrichmentTable.vue</strong><dd><code>UI for Replace Failed URL mode</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> web/src/components/functions/AddEnrichmentTable.vue <ul><li>Added replace_failed updateMode option and styling<br> <li> Conditional URL input for replace_failed mode<br> <li> Computed <code>hasFailedJob</code> and <code>updateModeOptions</code><br> <li> Forwarded <code>replaceFailedFlag</code> in service call</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9958/files#diff-9aad551dbc051edfdb3994a664506ed6d5e27dcb64f9c2fa917b02e51a35d86e">+42/-10</a>&nbsp; </td> </tr> </table></td></tr></tr></tbody></table> </details> ___	2026-01-12 13:18:16 +05:30
Huaijin	789ff8915d	fix: streaming partition generate partition start equal end (#9951)	2026-01-09 20:22:29 +08:00
Subhra264	02f93a729f	fix: rbac for backfill jobs (#9940)	2026-01-09 19:22:12 +08:00
3 people	d81752d48f	feat: support multiple jobs for enrichment table urls (#9895) ... Design at: pipelines/enrichment-file-url --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>	2026-01-09 16:34:28 +05:30
Ashish KolheandClaude Opus 4.5	ce09cd0a4d	feat: add js for claim parser (#9873) ... ### **User description** Design at : security/sso-claim-parser/configuration.md ___ ### **PR Type** Enhancement, Tests ___ ### **Description** - Introduce rquickjs-based JS runtime for functions - Add compile_js_function and apply_js_fn utilities - Extend pipeline to support JS via CompiledFunctionRuntime - Update web UI: select JS/VRL transforms, show JS errors ___ ### Diagram Walkthrough ```mermaid flowchart LR A["FunctionNode"] -->|JS| B["compile_js_function"] A -->|VRL| C["compile_vrl_function"] B --> D["apply_js_fn"] C --> E["apply_vrl_fn"] D --> F["send to children"] E --> F ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>13 files</summary><table> <tr> <td><strong>function.ts</strong><dd><code>Add optional `trans_type` to TestFunctionPayload</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-c6766f6c671adce05f9eef5d4a57a32371fa7da35628ef5c142fa503d191cb16">+16/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>js.rs</strong><dd><code>Introduce JS runtime, compile and apply functions</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-88e490bed5fa6c9c8e0aa2738b95c5284bb97899298f54b9cd587713e3ef0109">+756/-0</a>&nbsp; </td> </tr> <tr> <td><strong>batch_execution.rs</strong><dd><code>Extend pipeline execution to handle JS functions</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-3f6824a6eddaf1603e5cb0f1a4311ac6a45e1ffb967266ccc1f0bba944625e69">+465/-96</a></td> </tr> <tr> <td><strong>functions.rs</strong><dd><code>Validate and test-run JS transforms, restrict to _meta</code>&nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-ed5611a9aa6b5c033e70d7631ba72fafad3cc842672077a7fcf965d11a4fef23">+319/-35</a></td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Forbid JavaScript functions in pipelines globally</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-c186c796a7bdfd9d1301b1429df11456430f1d0d4758783810b9e2e44384e3b2">+39/-1</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>jwt.rs</strong><dd><code>Add JS claim parser execution function</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-232cab74f4cbf135af424744272e17732fbda8ac005d33c0acb4597e65479d02">+25/-2</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Wrap JS compile/apply in ingestion API</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-7c7b5bca85f0ced652965cae66540c96f56dfcca4f4c16b77bf7b45aac203d1b">+24/-1</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Include `trans_type` in test_function API payload</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-a5e1b3644cef9feb689dfe66382e44e5434ecadbff3469127346a333d4733102">+8/-3</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>AddFunction.vue</strong><dd><code>UI: add JS/VRL transform type selector</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-ebfd6c538f3cd3ecb9ab17550fd76e2e89e7a1abccc83a7505077e4af3eee110">+42/-25</a>&nbsp; </td> </tr> <tr> <td><strong>FunctionsToolbar.vue</strong><dd><code>Toolbar: transform type radio buttons</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-8283d2ff9c4a66b4b59eac306672aca2d54655a462e2ddb88e14e039d79dcdaf">+51/-3</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>DomainManagement.vue</strong><dd><code>Track claim-parser function changes for save button</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-5e47cd01fcb63f7ef27dc64d2ecf66c39e17097e29944951c3ae1bfe715bb97c">+17/-9</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>TestFunction.vue</strong><dd><code>Support JS error messaging in test UI</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-b9af9a2d1bf04d23422c26a13e6a763c6953c8713357a2f64ed8073ce6c926a3">+15/-5</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>PipelineEditor.vue</strong><dd><code>Filter out JS functions in pipeline editor UI</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-68c21f647336067eb79737ce9aca2d49632f2fa1c0878bcfdae55607e3473a15">+12/-4</a>&nbsp; &nbsp; </td> </tr> </table></details></td></tr><tr><td><strong>Configuration changes</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>function.rs</strong><dd><code>Extend `Transform` model for JS support</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-27c9f3d93723397a0e234bc45c9f543b43eff5fd518d76e735f78d95a58f3f1c">+16/-6</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Register new `js` utils module</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-2aa59ce4464e7dbabdb9367fc8bb8c1b489c914b8690577d4916870463ab12db">+1/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> </table></details></td></tr><tr><td><strong>Bug fix</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>errors.rs</strong><dd><code>Rename `MemoryTableOverflowError` display message</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-bf45932466265b732f13d0cd9b9cffbaefbe37c5861ace9315449eeca9250713">+1/-1</a>&nbsp; &nbsp; &nbsp; </td> </tr> </table></details></td></tr><tr><td><strong>Tests</strong></td><td><details><summary>3 files</summary><table> <tr> <td><strong>result-array-functions.spec.js</strong><dd><code>Add E2E tests for `#ResultArray#` functions in UI</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-e9911625ad2bab76f2b760f0ad254791331993a6a874b96f6a11fc6d88038b10">+470/-0</a>&nbsp; </td> </tr> <tr> <td><strong>cleanup.spec.js</strong><dd><code>Clean up JS function test data in UI tests</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-c214275a512fa285a7e4f2ab61dfcb42d71a0436b0e4241f94d7b45693587983">+31/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>page-manager.js</strong><dd><code>Register `FunctionsPage` in `PageManager`</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-565eab8b3c767ae50c9d05e591621d14f66b81af84d6b5ea6efe136c1c3933df">+6/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> </table></details></td></tr><tr><td><strong>Dependencies</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>Cargo.toml</strong><dd><code>Add `rquickjs` dependency for JS runtime</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542">+2/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>18 files</summary><table> <tr> <td><strong>test_functions_meta_org_restriction.py</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-99ef1207d6ae6d7eeaed37d70c8cf0c4770ba78f6f51fbdc067d35492d2fe9ea">+401/-0</a>&nbsp; </td> </tr> <tr> <td><strong>test_js_enrichment.py</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-c5547d46bec60b888d947d1e9b7c5445aee72cf8e837182f7138e0fc903eedbb">+356/-0</a>&nbsp; </td> </tr> <tr> <td><strong>test_js_result_array.py</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-9ed1363a1f58c94e018bb2b0aaa9d08fa71f89e7e55a6ff6be4a82bbeed8985c">+390/-0</a>&nbsp; </td> </tr> <tr> <td><strong>functionsPage.js</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-7bab9cc2eb8f36d3ee5d0aab453c353ec6d6dfecbc401cf401322ff57faa8440">+326/-0</a>&nbsp; </td> </tr> <tr> <td><strong>javascript-functions.spec.js</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-bdae6529346a7222bdec19f75f490cb9d0f3578a260ba835225efcd27db3f64e">+314/-0</a>&nbsp; </td> </tr> <tr> <td><strong>meta-org-js-restriction.spec.js</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-16cc2373f6f1955f8344a59d5f2edcd6ce121c4fdee1d432b1f0ec8e27f25881">+483/-0</a>&nbsp; </td> </tr> <tr> <td><strong>FunctionList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-6ff9af50a1597504b7f0dca91e61cf30c800b4a17b4be2ed2c3a4f7a7b2d7068">+1/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>de.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-78f24ecb2dbcfc38fdf3ec22b2559a5b40d236ea689d1c55f5d3e87168310264">+30/-18</a>&nbsp; </td> </tr> <tr> <td><strong>en.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-892aa3d75ab6e129ef9116aa21bec1b9a15ba69ce5366d2a19128078c690d824">+15/-7</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>es.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-ac2a2167b8b973e3f776f9fe3b43bef4576470d8ad33f3d7cfaff6edff0e5778">+30/-18</a>&nbsp; </td> </tr> <tr> <td><strong>fr.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-db80e762b88ddf183820c050cdc6b215e6b02b365b5e73f5152af46b0d26f1a2">+30/-18</a>&nbsp; </td> </tr> <tr> <td><strong>it.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-2de78074a5f887afddce60d694ead9322e6ab223eb70251c13819f3028bdc70d">+30/-18</a>&nbsp; </td> </tr> <tr> <td><strong>ja.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-092e49c5f9fbc1b539ab9c2866d108d0ce1d314022b4c294eb4611b285e7bbd3">+30/-18</a>&nbsp; </td> </tr> <tr> <td><strong>ko.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-50e27ca1b4825a5026f3a251bfcf79c97cab07cfade2ebd512d332453d707c48">+30/-18</a>&nbsp; </td> </tr> <tr> <td><strong>nl.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-a1706021687ae5d8a1c006c49eab3946174fecd895f9b70a89c2e76d97250906">+30/-18</a>&nbsp; </td> </tr> <tr> <td><strong>pt.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-38e804f64cb5e0e451849fc84e9b46e53c954a4a4263d51c18665ad45f4b4815">+30/-18</a>&nbsp; </td> </tr> <tr> <td><strong>tr.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-6f0f131e9e2591480c4c5d1eaffc72d02dc31cbd8d3f90d8f012f08cabac8818">+30/-18</a>&nbsp; </td> </tr> <tr> <td><strong>zh.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9873/files#diff-8818055db58f83230aeea4ebfd4cec31d924e8c47b6f91298e356959ab25339e">+254/-242</a></td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>	2026-01-08 22:47:35 +05:30
Ashish Kolhe	0f4dcbe971	fix: 9882 add option to limit max series return for promql (#9896)	2026-01-08 23:42:48 +08:00
Yashodhan JoshiandYashodhan Joshi	68759b4354	fix: sync distinct stream retention to original stream (#9871) ... ### **User description** This adds functionality to always sync original stream retention to the distinct stream created for that stream. This also includes a once-off migration to sync the retention for existing streams ___ ### **PR Type** Enhancement ___ ### **Description** - Bump DB schema version to 24 - Add migration syncing existing distinct stream retention - Sync retention on settings update for distinct streams - Refactor distinct stream naming logic ___ ### Diagram Walkthrough ```mermaid flowchart LR A["Original Stream Settings"] --> B["Generate Distinct Stream Name"] C["Migration: Sync Retention"] --> B D["Settings Update Handler"] --> B ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Configuration changes</strong></td><td><table> <tr> <td> <details> <summary><strong>config.rs</strong><dd><code>Bump database schema version</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> src/config/src/config.rs - bumped `DB_SCHEMA_VERSION` from 23 to 24 </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9871/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+1/-1</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td> <details> <summary><strong>mod.rs</strong><dd><code>Register new migration</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> src/infra/src/table/migration/mod.rs - register new migration in migrator list </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9871/files#diff-7ee42a714d3d56c663da0412d15514613222055aebb5b50dafbc051a656a36f6">+2/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> </table></td></tr><tr><td><strong>Enhancement</strong></td><td><table> <tr> <td> <details> <summary><strong>m20260107_000001_sync_distinct_stream_retention.rs</strong><dd><code>Add migration for distinct stream retention</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> src/infra/src/table/migration/m20260107_000001_sync_distinct_stream_retention.rs <ul><li>add new migration file for retention sync<br> <li> iterate all streams and update distinct retention<br> <li> support enterprise super-cluster propagation</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9871/files#diff-e077e2b4452a91f484015bd939eae29bec994be01a67eb4e71e5f111c761c755">+115/-0</a>&nbsp; </td> </tr> <tr> <td> <details> <summary><strong>distinct_values.rs</strong><dd><code>Refactor naming and retention logic</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> src/service/metadata/distinct_values.rs <ul><li>extract <code>get_distinct_stream_name</code> function<br> <li> use helper instead of inline formatting<br> <li> update distinct stream retention during ingest</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9871/files#diff-3fb2320197c6e8e900658f5a24158e922610eadcbf081ab0df230f770ef27e07">+35/-6</a>&nbsp; &nbsp; </td> </tr> <tr> <td> <details> <summary><strong>stream.rs</strong><dd><code>Sync retention on settings save</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> src/service/stream.rs <ul><li>on <code>save_stream_settings</code>, sync distinct stream retention<br> <li> add <code>get_stream_retention</code> helper function<br> <li> skip metadata streams when syncing</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9871/files#diff-41452a7fb29b4634437174fd6aecf6a9050dbfdef41b9077a21d59af3d79e9ed">+68/-0</a>&nbsp; &nbsp; </td> </tr> </table></td></tr></tr></tbody></table> </details> ___ --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>	2026-01-07 18:32:30 +05:30
Ashish Kolheandgithub-actions[bot]	0284e5ab5a	feat: tenant management using service accounts (#9838) ... ### **User description** Design at : service_account_enhancements/FEATURE_TENANT_MANAGEMENT_SERVICE_ACCOUNTS ___ ### **PR Type** Enhancement, Tests ___ ### **Description** - Add session expiry cache with cleanup - Implement `assume_service_account` endpoint - Enforce `allow_static_token` policy for service accounts - Automate service account creation and tenant admin role ___ ### Diagram Walkthrough ```mermaid flowchart LR A["Create Organization"] -- "adds service account" --> B["DB org_users (allow_static_token=false)"] B -- "assign tenant admin role" --> C["OpenFGA"] A -- "respond with ServiceAccountTokenInfo" --> D["Client"] E["AssumeServiceAccount API"] -- "create session" --> F["DB sessions with expiry"] F -- "cache token & expiry" --> G["USER_SESSIONS & USER_SESSIONS_EXPIRY"] H["AuthExtractor"] -- "validate session token + expiry" --> I["Request Auth"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Configuration changes</strong></td><td><details><summary>5 files</summary><table> <tr> <td><strong>config.rs</strong><dd><code>Add session expiry cache mapping</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-cda0d9179ed982fa5f0989c9b26f21cfa49dd1de28195a98135650ffcc7c602a">+1/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>config.rs</strong><dd><code>Bump DB schema version to 23</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+1/-1</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Import new session meta module</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-bab79e8da2d0d9f903cb1a6187d922c6de63d3e4561e8ce633660b664b4e859f">+1/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>m20251230_000001_add_allow_static_token_to_org_users.rs</strong><dd><code>Migration: add allow_static_token to org_users</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-2c713c6a1665f7cc5b19eda98ed6663e4908201052e31bd3c6643aa48c03130b">+55/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Register new migration for allow_static_token</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-7ee42a714d3d56c663da0412d15514613222055aebb5b50dafbc051a656a36f6">+2/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> </table></details></td></tr><tr><td><strong>Enhancement</strong></td><td><details><summary>20 files</summary><table> <tr> <td><strong>organization.rs</strong><dd><code>Add service_account fields and response types</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-4f089715a7ac4f78b45039d773f2fccbb4fbc6a409e0ab9ceee1fcf7911bc44b">+24/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>user.rs</strong><dd><code>Include `org_name` in `UserOrg` defaults</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-e13b3a981fd70182b9738a6b0ecfdee915c5b36757703a87741e1eddf3f6e786">+3/-1</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>auth.rs</strong><dd><code>Enhance session auth extraction and expiry logic</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-9bc2b1841504a435f66ad094f72ddbe7b7fa75e5e1557b8adbfe36f7f0592e1b">+101/-8</a>&nbsp; </td> </tr> <tr> <td><strong>session.rs</strong><dd><code>Define assume_service_account request/response schemas</code>&nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-cc73e168e23e2cd5d98749492c12493cff611bc9cecfa3d45e8c6a0e9a3ba135">+50/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>user.rs</strong><dd><code>Add `org_name` field to `UserOrg` meta</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-fc77f1b0e696889335f118cd5c1d69b7e3b385f48dd8e94c02a69912242e689b">+2/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>jwt.rs</strong><dd><code>Populate `org_name` in JWT auth mapping</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-232cab74f4cbf135af424744272e17732fbda8ac005d33c0acb4597e65479d02">+8/-1</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>validator.rs</strong><dd><code>Enforce service account static token policy</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-f4d4b51574b378a8c372345ea48875834a4eadacfac33fa4d0ceca1e0ef96b35">+58/-11</a>&nbsp; </td> </tr> <tr> <td><strong>assume_service_account.rs</strong><dd><code>Implement `assume_service_account` HTTP endpoint</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-3cdbc33e83fda4c90aae59b6213861646ce7031b092e1f052c4c3c8e94ac468b">+208/-0</a>&nbsp; </td> </tr> <tr> <td><strong>org.rs</strong><dd><code>Return service_account info on org creation</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-b47c0c1a8a78524196176430268eecdbfaf6f46d6b95873d3cbe70cfd67182b0">+12/-5</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Register assume_service_account route</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+1/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>org_users.rs</strong><dd><code>Add `allow_static_token` to OrgUsers entity</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-aec40965f56c43f49b6c162e7f7cfd549af23e0dcd9980f1e863ddb30eba357b">+1/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>org_users.rs</strong><dd><code>Support `allow_static_token` in ORM models</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-8e62ecba19308fdaaecfdb3d1f23021c926e244a510d9452dfbd71ac8f35ee9f">+33/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>sessions.rs</strong><dd><code>Add session expiry persistence and cleanup</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-945b60afd1f074c5f3bf875317896181829b19153938184bad9098c75ab44b48">+11/-12</a>&nbsp; </td> </tr> <tr> <td><strong>org_users.rs</strong><dd><code>Add `add_with_flags` for org_users</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-3cbb516ea021b4c03cf4b5f9de2ebcd4a8afee0722a4bcc5c39cb78f485900b6">+73/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>organization.rs</strong><dd><code>Default `service_account` in org DB functions</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-ce65cea926a09721a8fea14d07fe188bc2b887218e95d74a8bffefb54001da50">+3/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>session.rs</strong><dd><code>Enhance session caching with expiry checks</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-64311ff8ef53118e7611a2a2a28c5380a0e8b8213139b2c2108c728b1d460a2b">+152/-17</a></td> </tr> <tr> <td><strong>user.rs</strong><dd><code>Map `org_name` in DB user mapping</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-6ee6b0fc9ed03224d4ddf0b7b631c6f81ff7d1d579bdb98d2c5d4453852eb5d6">+4/-2</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>organization.rs</strong><dd><code>Add service account creation & tenant role logic</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-6b3c3c2b027c1158c0d41302cc69dfcd20ab68a6a4fa365cda403f838940bbb4">+270/-7</a>&nbsp; </td> </tr> <tr> <td><strong>session.rs</strong><dd><code>Use `set_with_expiry` for session creation</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-2d2f0c7a3be0a3d7d953c01974be5e093c8d42953952e435bc7b3a0faed6bf38">+3/-1</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>users.rs</strong><dd><code>Add `create_service_account_if_not_exists` helper</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-98da2588a28d62bb2e301d989836125f89f89b76272cd7f75e08ec4deeea5f38">+56/-7</a>&nbsp; &nbsp; </td> </tr> </table></details></td></tr><tr><td><strong>Tests</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Include `allow_static_token` in test fixtures</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-2ba3bceb0d80c8008da06e45374c1f744441375e2c28b55e6a55a1be3337e79b">+1/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> </table></details></td></tr><tr><td><strong>Formatting</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Export assume_service_account module</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-69dda3b2107f353d44361f3f47fad6f504470d26f888dec797664fb7de35ab08">+1/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> </table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Mask static tokens in API docs and logic</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-095d70f65b6e69bf7de0180fcbcddcd7c89c4cf00095d7b06f039a07e73916bb">+13/-3</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>openapi.rs</strong><dd><code>Document assume_service_account in OpenAPI</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9838/files#diff-edb8ce232c0cd08366879f642788a75d99f6bab64b4c0e77391281fdb9241e90">+3/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>	2026-01-06 13:38:50 +05:30
Hengfei Yang	c96605a093	fix: metrics streaming partition (#9839)	2026-01-06 01:07:10 +08:00
ByteBaker	20438076dd	refactor: deduplicate ingestion handler code ... Extract process time tracking and ingestion check logic into helper functions: - `get_process_time()` - consolidates slow log threshold check - `insert_process_time_header()` - adds o2_process_time header - Replace `match` with `if let Err` for cleaner error handling - Remove unused `_in_req` parameters from handler functions Reduces code duplication across `bulk`, `multi`, `json`, `hec`, `otlp_logs_write` handlers.	2026-01-05 14:05:53 +05:30
Hengfei Yang	ec021ac29e	perf: optimize MCP tools - reduce size from 643KB to 123KB (#9825)	2026-01-03 14:47:15 +08:00
Sai Nikhilandgithub-actions[bot]	8beb75de63	fix: share url disabled when web url is not configured (#9826) ... Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>	2026-01-03 01:11:47 +05:30
ByteBakerandsai nikhil kethe	09ef3a8f88	feat: incident detail drawer enhancements - telemetry tabs and alert graph (#9719) ... ### **User description** ## Summary Enhances the incident detail drawer with two new features to improve incident investigation: 1. **Telemetry Correlation Tabs** - Logs, Metrics, and Traces tabs showing related telemetry 2. **Alert Graph Visualization** - Interactive graph of services and dependencies Design at : https://github.com/openobserve/designs/tree/main/alert-correlation --- ## Feature 1: Telemetry Correlation Tabs Adds three new tabs (Logs, Metrics, Traces) that display correlated telemetry streams based on the incident's `stable_dimensions`. **Frontend Changes:** - New service method `getCorrelatedStreams()` in `web/src/services/incidents.ts` - Integrates `TelemetryCorrelationDashboard` component in embedded mode - Lazy loading: data fetched only on first tab click - Data caching: subsequent tab switches reuse loaded data - Refresh button for manual data reload - Auto time range from `first_alert_at` to `last_alert_at` - Empty state with manual search fallback - Memory cleanup on drawer close **No backend changes** - uses existing `POST /api/{org}/service_streams/_correlate` endpoint --- ## Feature 2: Alert Graph Visualization Adds "Alert Graph" tab showing interactive visualization of all services involved in an incident. **Backend Changes:** - New endpoint: `GET /api/v2/{org}/alerts/incidents/{id}/service_graph` - Returns graph nodes (services with alert counts) and edges (dependencies) - Builds from existing `topology_context` field - Gated by `O2_INCIDENTS_ALERT_GRAPH_ENABLED` env variable (default: false) - Enterprise-only with 403 handling for non-enterprise builds **Frontend Changes:** - New `IncidentServiceGraph.vue` component using ECharts - Force-directed and circular layouts - Visual encoding: node size = alert count, color = severity/root cause - Purple border for primary incident service - Stats banner and hover tooltips - Empty state when topology unavailable **Configuration:** ```bash export O2_INCIDENTS_ALERT_GRAPH_ENABLED=true # Required to enable ``` ___ ### **PR Type** Enhancement ___ ### **Description** - Add `IncidentServiceGraph` Rust models and API - Implement `get_service_graph` service logic - Extend `incidentsService` with correlation methods - Enhance `IncidentDetailDrawer` with telemetry and graph tabs --------- Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>	2026-01-02 22:02:45 +05:30
Subhra264	509bf7b6fb	feat: pipeline backfill (#9462) ... Design at : pipeline-gap-filler # feat: Add Pipeline Backfill Jobs Feature This PR introduces a complete pipeline backfill system that allows users to reprocess historical data through existing pipelines. ## Key Features ### Backend - **Dual-table architecture**: Separate `backfill_jobs` table for static config and `scheduled_jobs` for scheduling state - **RESTful API endpoints** under `/api/{org_id}/pipelines/{pipeline_id}/backfill`: - Create, list, pause, resume, update, and delete backfill jobs - **Robust error handling**: Retry mechanism matching derived stream pattern (immediate retries → scheduled retries, never delete jobs) - **Progress tracking**: Chunk-based processing with real-time position updates - **Optional deletion phase**: Delete existing data before backfilling with separate status tracking - **Super cluster support**: Proper trigger synchronization across regions ### Frontend - **Backfill Jobs List Page**: Table view with status/pipeline filters, action buttons (pause/resume/edit/delete) - **Create Dialog**: Time range picker (absolute dates only), configurable chunk periods and delays - **Edit Dialog**: Update paused/completed jobs with reset to fresh state - **Job Details Dialog**: Real-time progress, timeline visualization, deletion status tracking - **Dark mode support**: Full theme compatibility across all components - **DateTime integration**: Reuses existing DateTime component with microsecond precision ## Technical Highlights - Cross-database support (PostgreSQL, MySQL, SQLite) - Frontend-only filtering for performance - Status mapping (waiting/pending → "running" display) - Timestamp formatting: `YYYY-MM-DDTHH:mm:ssZ` - Default chunk period matches pipeline schedule frequency - Navigation integration from PipelinesList component ## Files Changed ### Backend - **Migration**: `src/infra/src/table/migration/m20250109_000001_create_backfill_jobs.rs` - **Table CRUD**: `src/infra/src/table/backfill_jobs.rs` - **Service logic**: `src/service/alerts/backfill.rs` - **Scheduler handlers**: `src/service/alerts/scheduler/handlers.rs` - **API routes**: `src/handler/http/request/pipelines/backfill.rs` - **Router integration**: `src/handler/http/router/mod.rs`, `src/handler/http/router/openapi.rs` - **Super cluster sync**: `src/super_cluster_queue/scheduler.rs` ### Frontend - **Components**: - `web/src/components/pipelines/BackfillJobsList.vue` - `web/src/components/pipelines/CreateBackfillJobDialog.vue` - `web/src/components/pipelines/EditBackfillJobDialog.vue` - `web/src/components/pipelines/BackfillJobDetails.vue` - **API service**: `web/src/services/backfill.ts` - **DateTime enhancement**: `web/src/components/DateTime.vue` ## Testing - Tested across PostgreSQL, MySQL, and SQLite databases - Verified dark mode compatibility - Tested pause/resume/edit/delete operations - Validated progress tracking and completion status - Confirmed super cluster synchronization	2026-01-01 23:05:23 +05:30
Subhra264	e17e8bc5e6	fix: when retrying allow user to choose whether to resume from last byte or start from scratch (#9792) ... ### **PR Type** Enhancement When the job is failed, and the user wants to retry manually, ask the user whether resume should be from the last successful position or from start again. ___ ### **Description** - Support resume option for enrichment import - Parse `resume` query param server-side - Custom UI for resume or fresh restart - Update JS service to include `resume` param - Add `RetryJobDialog.vue` component ___ ### Diagram Walkthrough ```mermaid flowchart LR UI["UI: RetryJobDialog"] -- "confirm resume choice" --> API["POST /api/{org}/enrichment_tables/{table}/url?append=&resume="] API -- "parse `resume` param" --> Handler["save_enrichment_table_from_url"] Handler -- "resume logic" --> DB["Job record (resumed or new)"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table> <tr> <td> <details> <summary><strong>jstransform.ts</strong><dd><code>Add `resume` param in request service</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> web/src/services/jstransform.ts <ul><li>Add <code>resume</code> parameter with default false<br> <li> Append <code>resume</code> to POST request URL</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9792/files#diff-301add2528e0f32323ca4c503bd7d460ab15ca6b9a9537192a02c5e52fd82926">+2/-1</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td> <details> <summary><strong>mod.rs</strong><dd><code>Support resume logic in import handler</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> src/handler/http/request/enrichment_table/mod.rs <ul><li>Parse <code>resume</code> query parameter<br> <li> Store <code>existing_job</code> for resume handling<br> <li> Resume failed ranged jobs preserving progress<br> <li> Skip range support check when resuming</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9792/files#diff-7b5b203002d1c3e5f671392a2a8ed84f45225896b652c2dac94ac26e72a43124">+79/-42</a>&nbsp; </td> </tr> <tr> <td> <details> <summary><strong>EnrichmentTableList.vue</strong><dd><code>Integrate retry dialog into table list</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> web/src/components/functions/EnrichmentTableList.vue <ul><li>Import and register <code>RetryJobDialog</code><br> <li> Manage <code>showRetryDialog</code> and <code>retryJobData</code><br> <li> Update <code>retryUrlJob</code> to open dialog<br> <li> Adjust <code>handleRetryConfirm</code> to pass <code>resume</code></ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9792/files#diff-6a29e253f59405d3c0f756dba36afd7483f10ad86927cbbc3348ede6372c3077">+64/-34</a>&nbsp; </td> </tr> <tr> <td> <details> <summary><strong>RetryJobDialog.vue</strong><dd><code>Add RetryJobDialog component</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> web/src/components/functions/RetryJobDialog.vue <ul><li>New dialog to choose resume or restart<br> <li> Handles focus trap and ESC key<br> <li> Displays range support warning<br> <li> Emits <code>confirm</code> with resume choice</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9792/files#diff-7bc926ef894b5aed7010006e729ede28a2f4931809b154dc81880999b3a7e72f">+638/-0</a>&nbsp; </td> </tr> </table></td></tr></tr></tbody></table> </details> ___	2025-12-29 17:57:15 +05:30
3 people	78c96c3721	fix: support bulk deletes for various resources (#9153) ... ### **User description** fixes https://github.com/openobserve/openobserve/issues/9116 This adds support for bulk delete operation from API and UI side for various resources. ___ ### **PR Type** Enhancement ___ ### **Description** - Add bulk delete endpoints for multiple resources - Enforce per-item permission checks before delete - Aggregate successful/failed items in responses - Register bulk-delete routes in HTTP router - Extend UI: add bulk-delete buttons, dialogs, service calls ___ ### Diagram Walkthrough ```mermaid flowchart LR UI["\"UI: Bulk Delete Trigger\""] API["\"HTTP DELETE /{resource}/bulk Handler\""] AUTH["\"Permission Checks per Item\""] DEL["\"Perform Delete Operations Loop\""] RESP["\"Aggregate Response: successes/ failures\""] UI_FE["\"FE: Show Notifications\""] UI -- "calls" --> API API -- "checks" --> AUTH AUTH -- "allowed" --> DEL DEL -- "deletes items" --> RESP RESP -- "returns" --> UI_FE ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>14 files</summary><table> <tr> <td><strong>fga.rs</strong><dd><code>Add bulk delete roles and groups endpoints</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-e7c99f37f3abbb5e0507d2760bbf67df010d9218704b38dce84c5477acecd22e">+218/-6</a>&nbsp; </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Implement delete_alert_bulk handler</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-83431801d27626ef27482e67b62661f900043544a45ddf958bd027cf2fa3bd8d">+87/-6</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add delete_function_bulk and error types</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-a5e1b3644cef9feb689dfe66382e44e5434ecadbff3469127346a333d4733102">+97/-3</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add delete_dashboard_bulk endpoint</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-1b3c257e1d174c826718499faa4a1d4e93465dd56cb2b7b273b690ff7e3223fd">+78/-1</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Implement re_patterns delete_bulk logic</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-27d37a8738de89c81b540daa978e684caf2cf28803e1208d90cb4d42528ed28b">+116/-0</a>&nbsp; </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add users bulk delete handler</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-56897e42f9436a640bc70fddc25686485ba8ae1223635b32d252a0333309f748">+80/-1</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add service_accounts delete_bulk API</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-095d70f65b6e69bf7de0180fcbcddcd7c89c4cf00095d7b06f039a07e73916bb">+86/-1</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>action.rs</strong><dd><code>Implement delete_action_bulk endpoint</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-616df3014c553f763ea57e925f1bff7bb47f9467c632940d93a45b7b1e4c3a07">+90/-1</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>destinations.rs</strong><dd><code>Add delete_destination_bulk for alerts</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-76b00293c621036de31cf499c1c276efb9704c2bf8cf1b728f49cfdd43f9fd1b">+75/-4</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add delete_bulk for cipher_keys</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-043222e433d1e5825a4b7bfc2c7feedba417f2630f0082fa67aff64976736396">+76/-1</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>templates.rs</strong><dd><code>Implement delete_template_bulk handler</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-b08e274a64cc935b0cec402a04271a872b1586027a77dc0ccb84c597c3960f61">+73/-4</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>pipeline.rs</strong><dd><code>Add delete_pipeline_bulk API handler</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-c69f4f94eee97ce226caddc497b2adf89e26869f2d72b0d0206dc31ab2714daf">+70/-2</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>reports.rs</strong><dd><code>Add delete_report_bulk endpoint</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-00435e0d9301342b51cb57ae313f1413e7ed973642e0badf3f3959125d0677d1">+75/-1</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>ActionScripts.vue</strong><dd><code>Integrate bulk delete UI in action scripts</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-770f71fcb92a13b3b8b1041567b885fac39903c814825c878595cafa624ebbd8">+284/-5</a>&nbsp; </td> </tr> </table></details></td></tr><tr><td><strong>Configuration changes</strong></td><td><details><summary>3 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Register bulk delete services in router</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+14/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>iam.ts</strong><dd><code>Add bulkDeleteGroups and bulkDeleteRoles calls</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-3712c1bb0eecb664b6691d724c3673b8c1befa41efbd8c5de626c5849f8ace98">+10/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>dashboards.ts</strong><dd><code>Add bulkDelete service for dashboards</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-913b3f54367036b1b3960aa3eddd7e17b509b1d6011c5ab68ae0e935653179c5">+7/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> </table></details></td></tr><tr><td><strong>Error handling</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>auth.rs</strong><dd><code>Allow bulk-delete paths in auth extractor</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-9bc2b1841504a435f66ad094f72ddbe7b7fa75e5e1557b8adbfe36f7f0592e1b">+17/-1</a>&nbsp; &nbsp; </td> </tr> </table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>user.rs</strong><dd><code>Define bulk delete request/response structs</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-e13b3a981fd70182b9738a6b0ecfdee915c5b36757703a87741e1eddf3f6e786">+36/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add PipelineBulkDelete meta types</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-c4bd542e4e07ed7706b0643086f5a7d4edbb3647c7ceedbe4bea516c5274d8bd">+12/-0</a>&nbsp; &nbsp; </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>40 files</summary><table> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-6f9c2e6dbc40f8d40e3d3ecbee23240813a3d4856d49bd5d979f5b98551fdc3d">+12/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>service_account.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-878ad520e1c84500cc21ff6fc4088c8da64ff2febd79e60c0d4cc8c695c49798">+11/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>action.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-fc87c75a7df9539d7e7ee6184dfab02d4a8932323e81f95d57c05363aa44752d">+15/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>reports.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-d2117a3d67d4ec47b0017861845c92143baf917e10e6746df5dd97f925f81eeb">+12/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>function.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-27c9f3d93723397a0e234bc45c9f543b43eff5fd518d76e735f78d95a58f3f1c">+12/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>requests.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-412f11e06c0e901e2292aab80536cb36edebcc4796dec52923389c3aaa32b25a">+6/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>responses.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-224027f90893122eea28e35778467acf1a4b5572ce612fc9bc9ec44ca8ebd9fb">+9/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>dashboards.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-250eeebeec9d4ac0ac271466ed83cd33fea3f121080491b340dc95d9af287259">+12/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>destinations.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-ecffe114c8200bab742dd2ee4f7f41128aa59438a879634f070edd575a1f917c">+24/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>search_stream.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-642de43ff6b19ae41087a9f1efc02d8b0b873809e0c92cbddfac1a30ef3f1184">+1/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>utils.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-96bb9f4a8f547c821f26c957e8505d0437095b08d2b36687531f71565d287bd6">+2/-1</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>functions.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-ed5611a9aa6b5c033e70d7631ba72fafad3cc842672077a7fcf965d11a4fef23">+22/-31</a>&nbsp; </td> </tr> <tr> <td><strong>AppTable.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-efcf66a9e913b5d3a7c2eea800b7f4e54d9e5268307131482fd40f32e2074d8e">+72/-16</a>&nbsp; </td> </tr> <tr> <td><strong>AlertList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-be3e41b823e163b226ffc270d95cc4981e59c600851e4e0d10ee5e4ee994ac8e">+123/-2</a>&nbsp; </td> </tr> <tr> <td><strong>AlertsDestinationList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-e9d05ba6c071c0e2f5d9ef7acdef757f79910abd6897b29e27b069a10d4ba8de">+142/-5</a>&nbsp; </td> </tr> <tr> <td><strong>PipelinesDestinationList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-088c66f5a34e73a5350a794a772e5d905262277337c937dd36f4295072203ebf">+163/-28</a></td> </tr> <tr> <td><strong>TemplateList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-7df66364d7d69a26cfa25cbc555d0e090c2f73e51c8386d9b0435bae4412564c">+116/-27</a></td> </tr> <tr> <td><strong>EnrichmentTableList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-6a29e253f59405d3c0f756dba36afd7483f10ad86927cbbc3348ede6372c3077">+124/-13</a></td> </tr> <tr> <td><strong>FunctionList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-6ff9af50a1597504b7f0dca91e61cf30c800b4a17b4be2ed2c3a4f7a7b2d7068">+143/-6</a>&nbsp; </td> </tr> <tr> <td><strong>AppGroups.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-402fb2a5338e60fdc3da489f4d50b5cf52f06b4e884a6fe7b82441c9fa5ea22c">+86/-1</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>AppRoles.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-191634ef09436199f91f442968110a01f54ae46b5ceee67476f52613ad796db4">+86/-1</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>EditRole.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-263b6bc7d29fdf8358c8437cd637f1350aa3733cc2706eae8ff00028686cc0f5">+20/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>ServiceAccountsList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-3afe835bcfbec3fda2e9550850c29581092ee0717e7dd8de8888e774a0048d24">+114/-1</a>&nbsp; </td> </tr> <tr> <td><strong>User.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-e973c2a665f9b27b94673ba64cc5b23650300836a727a2e07673c8e26305aee5">+133/-3</a>&nbsp; </td> </tr> <tr> <td><strong>PipelinesList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-85e3ef322f3eea0fc6d74d89431ba338a1153cb82ee06bef0f7c5f439c04267d">+108/-2</a>&nbsp; </td> </tr> <tr> <td><strong>ReportList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-f7d700f7e67fbaba3c58ab5d2c614925db743101828214445ce9dae35dc29067">+157/-5</a>&nbsp; </td> </tr> <tr> <td><strong>CipherKeys.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-950b715cff2c964d8542512692030c370733b57d698abf40e1dcb3cce2cfaf7a">+104/-13</a></td> </tr> <tr> <td><strong>RegexPatternList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-524927acd9f5bb07b71bfac489e747470108574e21a6dde63ea7c1f219cea853">+109/-14</a></td> </tr> <tr> <td><strong>action_scripts.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-6c1a011ac23290134801b22feb2b50c2b19cabf2a009f37a62b4dd9ca6e415e3">+3/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>alert_destination.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-377d28822e9ad4e9e80327ecc2d09bf1981c7997c5f1c1de120c6179eef19c6e">+3/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>alert_templates.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-fd8c4260abc3df601e4331d36b0c211eca568ee70139ad90c87247bc552fe4e2">+3/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>alerts.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-7a62782cceacac92b787e058219b3c53ea7406d4be5780def899943a01f500c1">+7/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>cipher_keys.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-0a0ef3a63c29b2a74b7f904c54c3fb32bee421608474aa32334b7c55133ab1ef">+3/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>jstransform.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-301add2528e0f32323ca4c503bd7d460ab15ca6b9a9537192a02c5e52fd82926">+3/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>pipelines.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-e8626670bb4e46e073f9858ade6b3dc77eaadf39d6721d9c0006eccc1b430612">+8/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>regex_pattern.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-2a0b0828141d39ab0385e0941314ead1cdd2ec3ec7b3baf5fd0c42f8f010728a">+3/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>reports.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-32d3299b2b21dc57c39d01080baba03765d853bd05c82d6158834b54713ae825">+3/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>service_accounts.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-4038191b0dc0cad335f3d8f7f703505df905ad843fe950096aefb8abbd139a1e">+4/-1</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>users.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-a334d8d570d9bdcaa3d4c8c2a993eff009a1ec8eb499c8a3d1a292d9224159d0">+3/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>Dashboards.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-8da3187db8ce3d795d0f196ede6b2290221ffa3e27b15dcc2b645e44dd196213">+122/-6</a>&nbsp; </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>	2025-12-29 16:37:29 +05:30
Hengfei Yang	e59474aa1b	perf: reduce nats events (#9712)	2025-12-25 19:10:31 +08:00
Huaijin	6082edf321	feat: follower querier timeout for UI (#9744)	2025-12-25 15:54:24 +08:00
Harsh Mahajan	2bb67c3b7e	fix: new kv table for kv store (#9459) ... This PR make necessary changing for using new KV table for storing PKCE states and also removes the existing states from meta table	2025-12-24 11:54:22 +05:30
3 people	384aa5ba62	fix: issue 9355 (#9706) ... ### **User description** This PR fixes the issue in #9355 Alert doesn't get triggered if the field seleceted in a non uds field ___ ### **PR Type** Bug fix, Enhancement ___ ### **Description** - Enforce UDS fields in alerts and pipelines UIs - Filter stream fields by defined_schema_fields - Always include timestamp and _all fields - Add `validate` flag to search API calls - Backend search utils: error on non-UDS fields - New alert UI validation: `validateConditionsAgainstUDS` ___ ### Diagram Walkthrough ```mermaid flowchart LR A["Select stream"] --> B["Fetch schema & settings"] B -- "Apply UDS filter" --> C["Available fields list"] C --> D["Build alert conditions"] D --> E["validateConditionsAgainstUDS"] E -- valid --> F["Submit alert"] E -- invalid --> G["Show error notification"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Bug fix</strong></td><td><details><summary>4 files</summary><table> <tr> <td><strong>utils.rs</strong><dd><code>Return error on non-UDS search fields</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9706/files#diff-96bb9f4a8f547c821f26c957e8505d0437095b08d2b36687531f71565d287bd6">+1/-2</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>alertValidation.ts</strong><dd><code>Always pass `validate:true` in alert SQL validation</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9706/files#diff-3164a4a58c1865cfcbbba084ac339664c99564545be4d57504dd406d59d8ad4a">+1/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>QueryEditorDialog.vue</strong><dd><code>Include `validate:true` in query trigger search</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9706/files#diff-6c524f609ed374aa29e075e8d046ac06be0edda7bfa1914b0a2bbc24d255e5eb">+1/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>Query.vue</strong><dd><code>Deep copy routing data and validate SQL</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9706/files#diff-6247cbd506bcf963372f2e429507fc579af95f003df246889b478da11326a657">+20/-10</a>&nbsp; </td> </tr> </table></details></td></tr><tr><td><strong>Enhancement</strong></td><td><details><summary>5 files</summary><table> <tr> <td><strong>search.ts</strong><dd><code>Add `validate` flag to search URL builder</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9706/files#diff-885ab872dbc6e7b20264df815e3ef4727f0abc86ce90d2fb47e4aff572814c75">+4/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>AddAlert.vue</strong><dd><code>Filter and validate conditions against UDS</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9706/files#diff-7b426bb23e9ed0250c3877d6d97b099c42b2d20e22ddf4459915381b2233a0a8">+187/-0</a>&nbsp; </td> </tr> <tr> <td><strong>FilterCondition.vue</strong><dd><code>Add column-exists validation rule per module</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9706/files#diff-53916e2402d0ee094fcd63f0ec0c99ceffde4fa042f9001203d2e2e1b7709e10">+28/-1</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>FilterGroup.vue</strong><dd><code>Propagate `module` prop to nested filter groups</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9706/files#diff-8d5a73bea3aa0b5c86d48cda4a61ee3e30f27797ac2a5add1ee8bd66c75bef23">+8/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>ScheduledPipeline.vue</strong><dd><code>Adjust submit handler and add UDS filter</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9706/files#diff-63d8f838bb880356e7804b4a57fd9223d4f2c189de7aadc7cf33cba723b29aaf">+4/-4</a>&nbsp; &nbsp; &nbsp; </td> </tr> </table></details></td></tr><tr><td><strong>Configuration changes</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>QueryConfig.vue</strong><dd><code>Tag QueryConfig with `module="alerts"`</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9706/files#diff-7273c8de49b8ebf706d7bee5b275c75818c0493f41c73f00c55efe667cbd00d9">+1/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>Condition.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9706/files#diff-4c3e624233c9e0355c7ded3b26dc74971b867be63e24f92a5e639284f60639dd">+37/-1</a>&nbsp; &nbsp; </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Co-authored-by: loaki07 <rustyloaki@gmail.com> Co-authored-by: Shrinath Rao <shnath@openobserve.ai>	2025-12-23 23:47:36 +05:30
3 people	e40938a08c	feat: enrichment table file url support (#9697) ... ### **PR Type** Enhancement Design at : enrichment-file-url ___ ### **Description** - Async URL-based enrichment table processing - Event-driven MPSC, streaming CSV, batching - Retry with backoff, status updates - Database and ORM additions - `enrichment_table_urls` table and migration - HTTP API endpoints - POST `/url` import and GET status - Frontend enhancements - Tabs, status icons, retry button ___ ### Diagram Walkthrough ```mermaid flowchart LR A["HTTP POST /{org}/enrichment_tables/{table}/url"] --> B["DB: save_url_job"] B --> C["trigger_url_job_processing()"] C --> D["MPSC Channel"] D --> E["process_url_jobs()"] E --> F["process_single_url_job()"] F --> G["validate_url() & fetch_and_process_batches()"] G --> H["parse_csv_chunk() & save_enrichment_batch()"] H --> I["save_url_job() update status"] I --> J["notify_update()"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>14 files</summary><table> <tr> <td><strong>url_processor.rs</strong><dd><code>Implement async URL enrichment table processor</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9697/files#diff-d39589e34157fe7930563b9bc5740128e168b8d8ac0a90371a4bdf1e52ce058b">+1351/-0</a></td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add URL enrichment table HTTP endpoints</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9697/files#diff-7b5b203002d1c3e5f671392a2a8ed84f45225896b652c2dac94ac26e72a43124">+301/-1</a>&nbsp; </td> </tr> <tr> <td><strong>enrichment_table.rs</strong><dd><code>Add URL job CRUD persistence functions</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9697/files#diff-6ce27ddf8a0d8b4e790bda90d088ff692a5da31a8d330beea8a9dd843294572b">+159/-2</a>&nbsp; </td> </tr> <tr> <td><strong>enrichment_table_urls.rs</strong><dd><code>SeaORM logic for URL job records</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9697/files#diff-e9ba1ba6e742e8d7652c3f31765de43bee061e7983bca67ffefecf8590210188">+173/-0</a>&nbsp; </td> </tr> <tr> <td><strong>enrichment_table.rs</strong><dd><code>Define URL job status and job struct</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9697/files#diff-29818b7d921497260251df82a49ae9ab7a93bbaedc0ca03abdf2b8c0726e4085">+90/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>pipelines.rs</strong><dd><code>Handle super-cluster URL job events</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9697/files#diff-7c8aa678045dd00de89d4c2345cf4a0511ce9bfac5d6f64e7404178c81e541ad">+23/-1</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Expose url_processor module in service</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9697/files#diff-3e2b6d3b8da1f9e73aca279052a97e683184d68256c23215b18ffee885836bbf">+6/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>enrichment_table_urls.rs</strong><dd><code>Define SeaORM entity for URL jobs</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9697/files#diff-9c3081476927bd337b2cbed8310578869845a6137a2c5305f677affb7604d39f">+26/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Register URL handlers in router</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9697/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+2/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Expose enrichment_table_urls entity module</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9697/files#diff-c539d7557d1425986d5c09849d8dbca42d1b309c2bc33d49db641d6a783098f6">+1/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add enrichment_table_urls to table module</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9697/files#diff-af023301203a27a1ef34f698d5a329ad7bfced65ac65e3829b1c0905a0c74fb2">+1/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Include enrichment_table meta module</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9697/files#diff-bab79e8da2d0d9f903cb1a6187d922c6de63d3e4561e8ce633660b664b4e859f">+1/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>jstransform.ts</strong><dd><code>Add client methods for URL jobs API</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9697/files#diff-301add2528e0f32323ca4c503bd7d460ab15ca6b9a9537192a02c5e52fd82926">+18/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>EnrichmentTableList.vue</strong><dd><code>Enhance UI for URL-based enrichment tables</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9697/files#diff-6a29e253f59405d3c0f756dba36afd7483f10ad86927cbbc3348ede6372c3077">+327/-74</a></td> </tr> </table></details></td></tr><tr><td><strong>Configuration changes</strong></td><td><details><summary>3 files</summary><table> <tr> <td><strong>m20251221_000001_create_enrichment_table_urls.rs</strong><dd><code>Create migration for URL jobs table</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9697/files#diff-330b4b7ebbda5001f4f28050d50e219b91961d80d8369222d94d058cced57702">+189/-0</a>&nbsp; </td> </tr> <tr> <td><strong>config.rs</strong><dd><code>Add URL fetch configuration options</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9697/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+24/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Register URL jobs migration in migrator</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9697/files#diff-7ee42a714d3d56c663da0412d15514613222055aebb5b50dafbc051a656a36f6">+2/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> </table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>openapi.rs</strong><dd><code>Document URL endpoints in OpenAPI spec</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9697/files#diff-edb8ce232c0cd08366879f642788a75d99f6bab64b4c0e77391281fdb9241e90">+5/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> </table></details></td></tr><tr><td><strong>Bug fix</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>auth.rs</strong><dd><code>Extend auth for enrichment_tables URLs</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9697/files#diff-9bc2b1841504a435f66ad094f72ddbe7b7fa75e5e1557b8adbfe36f7f0592e1b">+9/-1</a>&nbsp; &nbsp; &nbsp; </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>AddEnrichmentTable.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9697/files#diff-9aad551dbc051edfdb3994a664506ed6d5e27dcb64f9c2fa917b02e51a35d86e">+159/-42</a></td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Shrinath Rao <shnath@openobserve.ai>	2025-12-23 21:55:20 +05:30
Hengfei Yang	1714f6c570	fix: metrics stream can't be deleted (#9710)	2025-12-23 14:49:04 +08:00
Loakesh Indiran	adcc0ca2b6	fix: feat add validate sql query param for _search and _multi_search (#9652) ... ### **PR Type** Enhancement, Tests ___ ### **Description** - Add `validate` query parameter to HTTP search handlers - Invoke `validate_query_fields` before executing search - Provide `get_bool_from_request` boolean extractor utility - Add unit tests for query validation helpers ### Existing behavior: 1. lets assume sql `select continent from oly` 2. for the above sql initial run the stream oly does not have any uds -> UI calls _search/ _search_multi to validate the query in the alerts and pipelines it will return records with the col continent ### Problem: 1. when user now ads uds to oly without the continent col 2. the apis return response with just _timestamp col 3. and alerts and pipelines fail silently without the actual field ### Solution: - add a validate query param to the api, when true we parse the sql get the cols check if part of uds or not with this approach we also needs to handle the case for queries like joins, ctes, subqueries, union, etc (where table 1 can have the field , where as table 2 does not have the field part of uds) So to handle this we honor what datafusion does, I did the testing with datafusion, results are shared in the pr description - Error message ```json { "code": 20004, "message": "Search field not found: continent. Field exists but not in User-Defined Schema (UDS). Available UDS fields: [\"body\", \"id\", \"name\", \"total_medals\", \"unique_id\"]", "error_detail": "", "trace_id": "019b35dba10073aa8b9dc5064442b417" } ``` ### Testing Results # DataFusion CLI Test Results - Field Validation Behavior **Date:** 2025-12-20 **DataFusion CLI Version:** v51.0.0 **Purpose:** Determine field validation behavior to align OpenObserve UDS validation with DataFusion standards --- ## Test Setup ```sql CREATE TABLE oly (id VARCHAR, name VARCHAR, continent VARCHAR, gold_medals INT, total_medals INT); INSERT INTO oly VALUES ('USA', 'EUA', 'AME', 40, 126), ('CHN', 'China', 'ASI', 40, 91), ('JPN', 'Japão', 'ASI', 20, 45), ('AUS', 'Austrália', 'OCE', 18, 53), ('FRA', 'França', 'EUR', 16, 64), ('NED', 'Países Baixos', 'EUR', 15, 34), ('GBR', 'Grã-Bretanha', 'EUR', 14, 65), ('KOR', 'República da Coréia', 'ASI', 13, 32), ('ITA', 'Itália', 'EUR', 12, 40), ('GER', 'Alemanha', 'EUR', 12, 33); CREATE TABLE test1 AS SELECT * FROM oly; ``` **Result:** ✅ 10 rows inserted, test1 table created successfully --- ## Test Results ### Test 1: Basic SELECT with Existing Field **Query:** ```sql SELECT continent FROM oly LIMIT 5; ``` **Result:** ✅ **SUCCESS** ``` +-----------+ | continent | +-----------+ | AME | | ASI | | ASI | | OCE | | EUR | +-----------+ 5 row(s) fetched. ``` **Conclusion:** DataFusion allows querying existing schema fields. --- ### Test 2: Basic SELECT with Non-Existent Field **Query:** ```sql SELECT non_existent_field FROM oly; ``` **Result:** ❌ **ERROR** ``` Schema error: No field named non_existent_field. Valid fields are oly.id, oly.name, oly.continent, oly.gold_medals, oly.total_medals. ``` **Conclusion:** DataFusion **strictly validates** that fields exist in schema at query time. --- ### Test 3: JOIN with Existing Fields **Query:** ```sql SELECT a.continent AS "oly_continent", b.continent AS "test1_continent" FROM oly AS a JOIN test1 AS b ON a.name = b.name LIMIT 5; ``` **Result:** ✅ **SUCCESS** ``` +---------------+-----------------+ | oly_continent | test1_continent | +---------------+-----------------+ | AME | AME | | ASI | ASI | | ASI | ASI | | OCE | OCE | | EUR | EUR | +---------------+-----------------+ 5 row(s) fetched. ``` **Conclusion:** DataFusion allows JOIN queries when all fields exist in their respective table schemas. --- ### Test 4: JOIN with Non-Existent Field **Query:** ```sql SELECT a.non_existent_field, b.continent FROM oly AS a JOIN test1 AS b ON a.name = b.name; ``` **Result:** ❌ **ERROR** ``` Schema error: No field named a.non_existent_field. Valid fields are a.id, a.name, a.continent, a.gold_medals, a.total_medals, b.id, b.name, b.continent, b.gold_medals, b.total_medals. ``` **Conclusion:** DataFusion **validates per-table fields** in JOIN queries. If a field doesn't exist in table 'a', it fails even if table 'b' has valid fields. --- ### Test 5: Subquery with Existing Field **Query:** ```sql SELECT a.name FROM oly AS a WHERE a.continent IN ( SELECT b.continent FROM test1 AS b ) LIMIT 5; ``` **Result:** ✅ **SUCCESS** ``` +-----------+ | name | +-----------+ | EUA | | China | | Japão | | Austrália | | França | +-----------+ 5 row(s) fetched. ``` **Conclusion:** DataFusion allows subqueries when all fields exist in their respective schemas. --- ### Test 6: Subquery with Non-Existent Field **Query:** ```sql SELECT a.name FROM oly AS a WHERE a.non_existent_field IN ( SELECT b.continent FROM test1 AS b ); ``` **Result:** ❌ **ERROR** ``` Schema error: No field named a.non_existent_field. Valid fields are a.id, a.name, a.continent, a.gold_medals, a.total_medals. ``` **Conclusion:** DataFusion **validates fields in subquery WHERE clauses**. Non-existent fields cause immediate errors. --- ### Test 7: UNION with Existing Fields **Query:** ```sql SELECT continent FROM oly UNION SELECT continent FROM test1 LIMIT 5; ``` **Result:** ✅ **SUCCESS** ``` +-----------+ | continent | +-----------+ | OCE | | EUR | | AME | | ASI | +-----------+ 4 row(s) fetched. ``` **Conclusion:** DataFusion allows UNION queries when fields exist in all involved tables. --- ### Test 8: CTE with Existing Field **Query:** ```sql WITH oly_data AS ( SELECT name, continent FROM oly ) SELECT continent, COUNT(*) as cnt FROM oly_data GROUP BY continent; ``` **Result:** ✅ **SUCCESS** ``` +-----------+-----+ | continent | cnt | +-----------+-----+ | AME | 1 | | ASI | 3 | | EUR | 5 | | OCE | 1 | +-----------+-----+ 4 row(s) fetched. ``` **Conclusion:** DataFusion allows CTEs when all fields exist in the base table schema. --- ### Test 9: CTE with Non-Existent Field **Query:** ```sql WITH oly_data AS ( SELECT name, non_existent_field FROM oly ) SELECT non_existent_field FROM oly_data; ``` **Result:** ❌ **ERROR** ``` Schema error: No field named non_existent_field. Valid fields are oly.id, oly.name, oly.continent, oly.gold_medals, oly.total_medals. ``` **Conclusion:** DataFusion **validates fields in CTE definitions**. Non-existent fields in CTEs cause immediate errors. --- ## Summary of Findings | Test Case | Query Type | Field Status | DataFusion Behavior | |-----------|-----------|--------------|---------------------| | 1 | Basic SELECT | Exists | ✅ SUCCESS | | 2 | Basic SELECT | Non-existent | ❌ ERROR | | 3 | JOIN | Both exist | ✅ SUCCESS | | 4 | JOIN | Non-existent in one table | ❌ ERROR | | 5 | Subquery | Exists | ✅ SUCCESS | | 6 | Subquery | Non-existent | ❌ ERROR | | 7 | UNION | Exists | ✅ SUCCESS | | 8 | CTE | Exists | ✅ SUCCESS | | 9 | CTE | Non-existent | ❌ ERROR | --- ## Key Conclusions 1. **Strict Schema Validation:** DataFusion performs **strict schema validation** at query planning time. 2. **No Permissive Behavior:** DataFusion does **not allow** queries with non-existent fields, regardless of query complexity. 3. **Per-Table Validation:** In JOIN queries, DataFusion validates fields **per table** - each table reference is validated against its own schema. 4. **Early Validation:** All field validation happens during query planning, **before execution**. 5. **Clear Error Messages:** DataFusion provides helpful error messages listing valid fields when validation fails. --- ## Recommendation for OpenObserve **OpenObserve should adopt DataFusion's strict validation approach:** - ✅ Validate all queried fields exist in stream schema - ✅ Validate per-table in JOIN queries (each table against its own schema/UDS) - ✅ Reject queries with non-existent fields at query planning time - ✅ Apply same validation to subqueries, CTEs, and UNION queries - ✅ Provide clear error messages listing available fields This aligns OpenObserve behavior with industry-standard SQL engines and prevents confusing runtime errors. --- **Test completed successfully. All results documented for validation strategy decision.** ___ ### Diagram Walkthrough ```mermaid flowchart LR A["Receive search request"] --> B["Extract `validate` flag"] B -- "true" --> C["validate_query_fields"] C -- "error" --> D["map_error_to_http_response"] C -- "ok" --> E["Proceed to search execution"] B -- "false" --> E ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table> <tr> <td> <details> <summary><strong>mod.rs</strong><dd><code>Add query validation in search handler</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> src/handler/http/request/search/mod.rs <ul><li>Extract <code>validate_query</code> flag from query parameters<br> <li> Call <code>utils::validate_query_fields</code> when flag is true<br> <li> Return HTTP error on validation failure</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9652/files#diff-96630a0dbb6dd7b2b0248fa325c7c7041ce31340c32826675a4a70772e9d29f6">+11/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td> <details> <summary><strong>multi_streams.rs</strong><dd><code>Add query validation in multi-stream handler</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> src/handler/http/request/search/multi_streams.rs <ul><li>Extract <code>validate_query</code> flag in multi-stream search<br> <li> Validate SQL fields with <code>validate_query_fields</code><br> <li> Short-circuit and map errors if validation fails</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9652/files#diff-c3233cdf1e68cb2455f91457719b9c2f477e735ee16546fdfe111e76768bd054">+15/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td> <details> <summary><strong>utils.rs</strong><dd><code>Implement query validation utilities</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> src/handler/http/request/search/utils.rs <ul><li>Implement <code>get_bool_from_request</code> boolean extractor<br> <li> Add <code>validate_query_fields</code> to check SQL columns against schema<br> <li> Provide helper functions: range adjust, stream resolve, system-field <br>check<br> <li> Include unit tests for query helpers</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9652/files#diff-96bb9f4a8f547c821f26c957e8505d0437095b08d2b36687531f71565d287bd6">+192/-2</a>&nbsp; </td> </tr> </table></td></tr></tr></tbody></table> </details> ___	2025-12-22 16:32:31 +05:30
5 people	27704a0d4c	fix: add alert revamp v2 (#9694) ... ### **PR Type** Enhancement, Tests ___ ### **Description** - Introduce `generate_sql` API endpoint for alerts - Implement request/response models & service linkage - Register route & update OpenAPI documentation - Add extensive front-end tests for alert wizard ___ ### Diagram Walkthrough ```mermaid flowchart LR A["POST /v2/{org_id}/alerts/generate_sql"] -- "validate & check perms" --> B["build_sql service"] B -- "return generated SQL & metadata" --> C["GenerateSqlResponseBody"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>4 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Add `generate_sql` alert API endpoint</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-83431801d27626ef27482e67b62661f900043544a45ddf958bd027cf2fa3bd8d">+150/-6</a>&nbsp; </td> </tr> <tr> <td><strong>responses.rs</strong><dd><code>Add `GenerateSqlResponseBody` and metadata</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-224027f90893122eea28e35778467acf1a4b5572ce612fc9bc9ec44ca8ebd9fb">+25/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>requests.rs</strong><dd><code>Add `GenerateSqlRequestBody` request model</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-412f11e06c0e901e2292aab80536cb36edebcc4796dec52923389c3aaa32b25a">+16/-1</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Make `build_sql` function public</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-ba9b65130898444f7815fd6068bdb0dc590612f74cc893f79d91b0afc44cc851">+1/-1</a>&nbsp; &nbsp; &nbsp; </td> </tr> </table></details></td></tr><tr><td><strong>Configuration changes</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Register `generate_sql` service route</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+1/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> </table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>openapi.rs</strong><dd><code>Include `generate_sql` in OpenAPI spec</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-edb8ce232c0cd08366879f642788a75d99f6bab64b4c0e77391281fdb9241e90">+1/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> </table></details></td></tr><tr><td><strong>Tests</strong></td><td><details><summary>3 files</summary><table> <tr> <td><strong>AlertSettings.spec.ts</strong><dd><code>Add comprehensive unit tests for AlertSettings</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-4ca37593c11ef263e0b9a9e1733678e2b81f01c8da5f4a8d729306c176802356">+1363/-0</a></td> </tr> <tr> <td><strong>AddAlert.spec.ts</strong><dd><code>Extend AddAlert tests for SQL & wizard flows</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-84362398fea8abd9acf30f4f40ba8d4cf27348677454017bd4e6f568d5176ac6">+808/-56</a></td> </tr> <tr> <td><strong>CompareWithPast.spec.ts</strong><dd><code>Add tests for CompareWithPast component</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-5718f3c114808926e2cced290580cb8ae1610338042eebad001e9a7d5429ee77">+596/-0</a>&nbsp; </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>45 files</summary><table> <tr> <td><strong>AddAlert.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-7b426bb23e9ed0250c3877d6d97b099c42b2d20e22ddf4459915381b2233a0a8">+1295/-409</a></td> </tr> <tr> <td><strong>AlertList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-be3e41b823e163b226ffc270d95cc4981e59c600851e4e0d10ee5e4ee994ac8e">+58/-63</a>&nbsp; </td> </tr> <tr> <td><strong>AlertSummary.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-c22aa1c5934b763f1d4a5a61a7e810b6bcb1d60b76b64c61cb3ce89dc0a04499">+169/-0</a>&nbsp; </td> </tr> <tr> <td><strong>AlertWizardRightColumn.spec.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-eb14abefd15bc1d3db947347b49899fe264138b7aea63729feae27ad7abed606">+851/-0</a>&nbsp; </td> </tr> <tr> <td><strong>AlertWizardRightColumn.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-f599a1960d1ac37e352854ad368d23e080b3cec2b217df22f40d0e15a4db6986">+291/-0</a>&nbsp; </td> </tr> <tr> <td><strong>CustomConfirmDialog.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-0fa5a1da2050cbab522801af8ade04a0e4c27e87933d21f6c32a82603c77a0a3">+179/-0</a>&nbsp; </td> </tr> <tr> <td><strong>DeduplicationConfig.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-63eb39111409780e4dc064ea80056dfe096038ee76b389cdd8e9d14c0ae29a32">+3/-2</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>FilterGroup.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-8d5a73bea3aa0b5c86d48cda4a61ee3e30f27797ac2a5add1ee8bd66c75bef23">+39/-39</a>&nbsp; </td> </tr> <tr> <td><strong>HorizontalStepper.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-ccca09440e61bf97621fe06ca606ade4f7fc7c1df8f8a3742cafa4c352f2c23a">+370/-0</a>&nbsp; </td> </tr> <tr> <td><strong>OrganizationDeduplicationSettings.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-f526d40b50f1e96f025973c6fc251f7a50f698c641c7749b32e6bb90b9ba032f">+19/-21</a>&nbsp; </td> </tr> <tr> <td><strong>PreviewAlert.spec.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-6d5bfc82662dd55768e258ab92676c2559c054d235ef0e48beb4bd90f31ca60b">+795/-0</a>&nbsp; </td> </tr> <tr> <td><strong>PreviewAlert.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-760f5f0a261910f17eb167f81802e8a52477b99a83d2d7fc1aeefc7185801795">+41/-16</a>&nbsp; </td> </tr> <tr> <td><strong>QueryEditorDialog.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-6c524f609ed374aa29e075e8d046ac06be0edda7bfa1914b0a2bbc24d255e5eb">+981/-0</a>&nbsp; </td> </tr> <tr> <td><strong>RealTimeAlert.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-c3ae6b47832eb33f9ca877cc7165d079de14850e5c5d43a227102ba40326b9d8">+43/-24</a>&nbsp; </td> </tr> <tr> <td><strong>ScheduledAlert.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-5e143a17ab0fa5d7817f6d9301667fb5597236f27184248c7a71728e85b8d999">+304/-270</a></td> </tr> <tr> <td><strong>Advanced.spec.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-6dacd8b2947f53e7c39a54e990ee26ff6d63478b3a68897ab3afbc937c65c647">+683/-0</a>&nbsp; </td> </tr> <tr> <td><strong>Advanced.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-eae849f0bf6c8868f5640ce1d74862284d8e8a5330e5046c0c437e6672f4f676">+376/-0</a>&nbsp; </td> </tr> <tr> <td><strong>AlertSettings.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-1d5bfd6a1ea672884193649b1271d751d6f3b8ec84af759071c9bfd169a8ab28">+1441/-0</a></td> </tr> <tr> <td><strong>AlertSetup.spec.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-ed5a1c54e5fb03ce1d8121223d40606c74419b998b6c071667037347a2debde9">+878/-0</a>&nbsp; </td> </tr> <tr> <td><strong>AlertSetup.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-4f89438c74f329de730b0e7451d4a8d7df3bab52cdeaded776e8ae18d94294eb">+290/-0</a>&nbsp; </td> </tr> <tr> <td><strong>CompareWithPast.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-6606848e30caef2cda13d65cb30d6590205556259e7e6daf39cada825502d55b">+416/-0</a>&nbsp; </td> </tr> <tr> <td><strong>Deduplication.spec.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-d3281cd7632a0fa7ce5c5054e80efafb2aa8d272bb2c586d723436b644565ba4">+591/-0</a>&nbsp; </td> </tr> <tr> <td><strong>Deduplication.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-f8d256bcec370851c9799c13ffa2b7571c07ca27af1b5a74c335e7084157c9b1">+204/-0</a>&nbsp; </td> </tr> <tr> <td><strong>QueryConfig.spec.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-5a3ac3e9d93e87bc95c87a20f08c9972158a5882d66ecfb658d38255fe1d8f71">+1000/-0</a></td> </tr> <tr> <td><strong>QueryConfig.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-7273c8de49b8ebf706d7bee5b275c75818c0493f41c73f00c55efe667cbd00d9">+594/-0</a>&nbsp; </td> </tr> <tr> <td><strong>de.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-78f24ecb2dbcfc38fdf3ec22b2559a5b40d236ea689d1c55f5d3e87168310264">+70/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>en.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-892aa3d75ab6e129ef9116aa21bec1b9a15ba69ce5366d2a19128078c690d824">+89/-15</a>&nbsp; </td> </tr> <tr> <td><strong>es.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-ac2a2167b8b973e3f776f9fe3b43bef4576470d8ad33f3d7cfaff6edff0e5778">+70/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>fr.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-db80e762b88ddf183820c050cdc6b215e6b02b365b5e73f5152af46b0d26f1a2">+70/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>it.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-2de78074a5f887afddce60d694ead9322e6ab223eb70251c13819f3028bdc70d">+70/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>ja.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-092e49c5f9fbc1b539ab9c2866d108d0ce1d314022b4c294eb4611b285e7bbd3">+70/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>ko.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-50e27ca1b4825a5026f3a251bfcf79c97cab07cfade2ebd512d332453d707c48">+70/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>nl.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-a1706021687ae5d8a1c006c49eab3946174fecd895f9b70a89c2e76d97250906">+70/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>pt.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-38e804f64cb5e0e451849fc84e9b46e53c954a4a4263d51c18665ad45f4b4815">+70/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>tr.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-6f0f131e9e2591480c4c5d1eaffc72d02dc31cbd8d3f90d8f012f08cabac8818">+70/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>zh.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-8818055db58f83230aeea4ebfd4cec31d924e8c47b6f91298e356959ab25339e">+70/-0</a>&nbsp; &nbsp; </td> </tr> <tr> <td><strong>alerts.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-7a62782cceacac92b787e058219b3c53ea7406d4be5780def899943a01f500c1">+9/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>app.scss</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-5703d12504b0395cbfbbfad02a592bc391496026e5f03743102ce06c210ea05f">+7/-0</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>alertQueryBuilder.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-5c829ce5fd3ad2f5ef359df5976e6adfc52eaa6b5f249f594c8b8a588acf8b3e">+10/-49</a>&nbsp; </td> </tr> <tr> <td><strong>alertSummaryGenerator.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-a4464ad7df3ac3afdca4e6fc583122d701bace10e3a2daf083cb6c902565b20f">+393/-0</a>&nbsp; </td> </tr> <tr> <td><strong>alertValidation.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-3164a4a58c1865cfcbbba084ac339664c99564545be4d57504dd406d59d8ad4a">+8/-8</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td><strong>conditionsFormatter.spec.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-12eb070ce171e81117000fefaf2a1037af4c337423344f280c21c77e1836f5e3">+237/-0</a>&nbsp; </td> </tr> <tr> <td><strong>conditionsFormatter.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-d5df71a7ab2451925cd220828fc3776039e56c206a07ac7b72d4ce5e282c257c">+460/-0</a>&nbsp; </td> </tr> <tr> <td><strong>focusManager.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-987334b24e56735ac755d8e1add737ffcd35fe11612e31586c21824cbea88c36">+179/-0</a>&nbsp; </td> </tr> <tr> <td><strong>zincutils.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9694/files#diff-d3d59688cccdcc6c84ebe88b698c6910030abecc3125f4c0ef81b0df023699ba">+11/-0</a>&nbsp; &nbsp; </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Ashish Kolhe <ashish.j.kolhe@gmail.com> Co-authored-by: Uddhav Dave <uddhav@openobserve.ai> Co-authored-by: Shrinath Rao <shnath@openobserve.ai>	2025-12-22 15:44:00 +05:30
3 people	ac60e181dd	feat: dashboard join support for auto sql (#9343) ... # Nested Functions and Joins - User Guide ## Overview This guide covers two powerful features that enable sophisticated data analysis in OpenObserve: - **Nested Functions**: Combine multiple functions together to create complex calculations - **Joins**: Merge data from multiple streams (data sources) based on common fields These features work in the **Auto SQL** mode of dashboard panels, allowing you to write advanced queries without deep SQL knowledge. --- ## What's New ### Why These Features? **Before:** - Limited function composition - you could only use basic, single-level functions - No way to combine data from multiple streams in one query - Complex analytics required exporting data and manual processing **Now:** - Compose unlimited nested functions with full DataFusion support - Join multiple streams directly in the dashboard panel editor - Create sophisticated analytics queries with a visual interface ### Key Benefits ✓ Calculate complex metrics (error rates, percentiles, conditional aggregations) ✓ Build advanced dashboards without custom code ✓ Support for all DataFusion aggregate and scalar functions --- ## Nested Functions ### What Are Nested Functions? Nested functions let you combine multiple functions together - using the output of one function as input to another. Think of it like a cooking recipe where each step builds on the previous one. **Simple Example:** - Basic: `AVG(response_time)` - calculates average response time - Nested: `ROUND(AVG(response_time), 2)` - calculates average AND rounds it to 2 decimal places ### Supported Functions OpenObserve supports **all DataFusion functions**, giving you powerful analytics capabilities: **Aggregate Functions** (work across multiple rows): - `count`, `sum`, `avg`, `min`, `max` - basic aggregations - `median`, `stddev`, `variance` - statistical functions - `percentile_cont` - calculate percentiles (e.g., p95, p99) - `string_agg` - combine text values **Mathematical Functions**: - `round`, `ceil`, `floor` - rounding operations - `abs`, `power`, `sqrt` - mathematical operations - `log`, `log10`, `log2` - logarithmic functions - `sin`, `cos`, `tan` - trigonometric functions **String Functions**: - `concat`, `upper`, `lower`, `trim` - `substring`, `length`, `replace` **Conditional Functions**: - `CASE WHEN ... THEN ... ELSE ... END` - conditional logic - `COALESCE` - return first non-null value 📖 **Full Reference**: [DataFusion Aggregate Functions](https://datafusion.apache.org/user-guide/sql/aggregate_functions.html) | [DataFusion Scalar Functions](https://datafusion.apache.org/user-guide/sql/scalar_functions.html) --- ### Using Nested Functions (Step-by-Step) #### Step 1: Create or Edit a Dashboard Panel Open your dashboard and create a new panel or edit an existing one. Make sure you're in **Auto SQL** mode (not Custom SQL). #### Step 2: Select a Function In the Y-Axis or column configuration, click **Select Function** to choose your base function. <img width="795" alt="Nested Function - Adding Layers" src="https://github.com/user-attachments/assets/0b6b134e-59b8-48b1-a4ae-52fb4800d772" /> In this example, we're using the `Concat` function to combine multiple fields (namespace and pod name). #### Step 3: Add Nested Layers Click the function icon to add another function layer. You can nest functions multiple times. <img width="793" alt="Nested Function - Select Base Function" src="https://github.com/user-attachments/assets/4bd9e03e-1104-4e82-b2ec-a34e6bb64870" /> Here we're building a nested function: - Inner layer: `Avg(severity)` - calculates average severity - Outer layer: `Ceil()` - rounds up the average to the nearest integer #### Step 4: Build Complex Expressions For advanced analytics, you can combine nested functions with conditional logic using `CASE WHEN` statements. <img width="1340" alt="Complex Nested Function with CASE WHEN" src="https://github.com/user-attachments/assets/4c3e3623-bfad-44c4-a760-e10cc94d4a95" /> This example shows a sophisticated calculation: ```sql ROUND(100.0 * SUM(CASE WHEN COALESCE(severity, 0) >= 4 THEN 1 ELSE 0 END) / COUNT(_timestamp), 2) ``` **What this does:** 1. `COALESCE(severity, 0)` - replaces null severity with 0 2. `CASE WHEN severity >= 4 THEN 1 ELSE 0` - returns 1 for high severity, 0 otherwise 3. `SUM(...)` - counts high severity events 4. Divide by `COUNT(_timestamp)` - calculates percentage 5. Multiply by `100.0` - converts to percentage 6. `ROUND(..., 2)` - rounds to 2 decimal places **Result**: Percentage of high-severity events, rounded to 2 decimals #### Step 5: View the Generated SQL Switch to the **Raw** tab to see the complete SQL query that was generated from your nested function configuration. <img width="1388" alt="View Generated SQL" src="https://github.com/user-attachments/assets/d7f6111a-26c6-4dd0-9e9f-e26092fe6368" /> You can review the auto-generated SQL in the Query section at the bottom of the panel editor. The chart visualization updates in real-time as you build your nested functions. --- ## Joins ### What Are Joins? Joins allow you to combine data from two or more different streams (data sources) based on common fields between them. **Example:** **employee_data_join** stream: ```json [ {"employee_id": 1, "name": "Alice", "department_id": 1}, {"employee_id": 2, "name": "Bob", "department_id": 2}, {"employee_id": 3, "name": "Charlie", "department_id": null}, {"employee_id": 4, "name": "David", "department_id": 1}, {"employee_id": 5, "name": "Eve", "department_id": 3} ] ``` **dept_data_join** stream: ```json [ {"department_id": 1, "department_name": "HR"}, {"department_id": 2, "department_name": "Engineering"}, {"department_id": 3, "department_name": "Sales"}, {"department_id": 4, "department_name": "Marketing"} ] ``` A join lets you combine these streams to see the department name alongside each employee, matched by `department_id`. ### Types of Joins OpenObserve supports three types of joins: | Join Type | Description | When to Use | |-----------|-------------|-------------| | **INNER JOIN** | Returns only records with matches in both streams | When you need data that exists in both sources | | **LEFT JOIN** | Returns all records from left stream + matching records from right | When you want all data from your primary source, with supplemental info where available | | **RIGHT JOIN** | Returns all records from right stream + matching records from left | When you want all data from your secondary source, with supplemental info where available | > **Note:** UNION and CROSS joins are not currently supported. --- ### Using Joins (Step-by-Step) #### Step 1: Access the Joins Section In your dashboard panel editor, locate the **Joins** section. Click the **+** button to add a join. <img width="1725" alt="Access Joins Interface" src="https://github.com/user-attachments/assets/35db4f28-c526-43a2-bdd1-470e188f53d7" /> In this example: - **Primary stream**: `employee_data_join` (contains employee names) - We'll join it with `dept_data_join` (contains department information) - The interface shows available fields from both streams in the left panel #### Step 2: Configure the Join Click on the **Joins** dropdown and select the stream you want to join. Then configure: 1. **Joining Stream**: Select the stream to join (e.g., `employee_data_join`) 2. **Join Type**: Choose from LEFT, INNER, or RIGHT using the visual toggle 3. **On Stream**: Select the target stream (e.g., `dept_data_join`) <img width="1725" alt="Configure Join Type" src="https://github.com/user-attachments/assets/35db4f28-c526-43a2-bdd1-470e188f53d7" /> The interface displays: **"Performing a Left Join between employee_data_join and dept_data_join"** Notice the visual toggle showing three join types: - **Left** (selected in this example) - **Inner** - **Right** #### Step 3: Define the Joining Clause Set up the join condition by specifying which fields to match between the two streams: **Clause 1:** - **Select Field** (Left): Choose a field from the first stream (e.g., `department_id`) - **Select Operator**: Choose comparison operator (usually `=`) - **Select Field** (Right): Choose the matching field from the second stream (e.g., `department_id`) You can add multiple clauses using the **+** button for complex join conditions (e.g., joining on multiple fields or adding additional filters). #### Step 4: View the Results After configuring the join, the data table will show combined results from both streams. <img width="1724" alt="Join Results" src="https://github.com/user-attachments/assets/94ca9f33-6cfa-4ad1-9b84-bee9db34f912" /> In this example: - **Name** column from `employee_data_join` (Alice, Bob, Charlie, David, Eve) - **Department Name** column from `dept_data_join` (HR, Engineering, HR, Sales) - Data successfully joined on `department_id` #### Step 5: Review the Generated SQL Check the **Query** section at the bottom to see the generated SQL: ```sql SELECT employee_data_join.name as "x_axis_1", stream_0.department_name as "x_axis_2" FROM "employee_data_join" LEFT JOIN "dept_data_join" AS stream_0 ON employee_data_join.department_id = stream_0.department_id ``` The SQL query shows exactly how OpenObserve is combining your data streams. --- --- ## Examples ### Example 1: String Concatenation with Nested Functions **Use Case**: Combine multiple fields into a single display value ```sql Concat(k8s_namespace_name, '-', k8s_pod_name) ``` This joins namespace and pod names with a hyphen separator. ### Example 2: Statistical Analysis with Nested Math Functions **Use Case**: Calculate rounded average of a metric ```sql Ceil(Avg(severity)) ``` This calculates the average severity level and rounds up to the nearest whole number. ### Example 3: Conditional Percentage Calculation **Use Case**: Calculate percentage of high-severity events ```sql ROUND(100.0 * SUM(CASE WHEN COALESCE(severity, 0) >= 4 THEN 1 ELSE 0 END) / COUNT(_timestamp), 2) ``` **Breakdown:** - Handles null values with `COALESCE` - Counts events where severity >= 4 - Calculates percentage of total events - Rounds to 2 decimal places ### Example 4: Employee-Department Join **Use Case**: Show employee names with their department information ```sql SELECT employee_data_join.name as "Name", dept_data_join.department_name as "Department Name" FROM employee_data_join LEFT JOIN dept_data_join ON employee_data_join.department_id = dept_data_join.department_id ``` This displays all employees with their department names, showing null for employees without a department assignment. ### Example 5: Complex Analytics - Joins + Nested Functions **Use Case**: Calculate error rate per service with deployment information ```sql SELECT logs.service_name, deployments.version, COUNT(*) as total_requests, ROUND((COUNT(CASE WHEN logs.status_code >= 500 THEN 1 END) * 100.0 / COUNT(*)), 2) as error_rate_percent FROM service_logs as logs LEFT JOIN deployments ON logs.service_name = deployments.service_name GROUP BY logs.service_name, deployments.version ORDER BY error_rate_percent DESC ``` This combines: - Join between service logs and deployment data - Nested functions for conditional counting and percentage calculation - Grouping and sorting for analysis --- ## Tips and Best Practices ### Nested Functions Best Practices ✅ **Start Simple**: Build your function one layer at a time - Begin with inner function: `AVG(response_time)` - Add outer layers: `ROUND(AVG(response_time), 2)` ✅ **Test Each Layer**: Verify results before adding complexity - Test the inner function first - Add one layer at a time and check output ✅ **Handle Null Values**: Use `COALESCE` to avoid unexpected nulls ```sql COALESCE(AVG(response_time), 0) ``` ✅ **Use Clear Aliases**: Name your calculated fields descriptively ```sql ROUND(AVG(response_time), 2) as avg_response_ms ``` ✅ **Check for Division by Zero**: Protect against zero denominators ```sql CASE WHEN COUNT(*) > 0 THEN (SUM(errors) * 100.0 / COUNT(*)) ELSE 0 END as error_rate ``` ### Joins Best Practices ✅ **Choose the Right Join Type**: - **INNER JOIN**: Both streams must have matching data - **LEFT JOIN**: Keep all records from primary stream - **RIGHT JOIN**: Keep all records from secondary stream ✅ **Verify Field Compatibility**: - Match data types (don't join string to number) - Ensure field names are correct - Check that values actually match between streams ✅ **Consider Performance**: - Join on indexed fields when possible - Test with limited data first --- ## FAQ ### Q: Can I nest more than 2 functions? **A**: Yes! You can nest as many functions as needed. Example: ```sql ROUND(CEIL(AVG(COALESCE(response_time, 0))), 2) ``` ### Q: What's the difference between LEFT and RIGHT joins? **A**: - **LEFT JOIN**: Keeps all records from the first (left) stream - **RIGHT JOIN**: Keeps all records from the second (right) stream - Choose based on which dataset you want to preserve completely ### Q: Can I join more than 2 streams? **A**: Yes! Click the **+** button in the Joins section to add additional joins. You can chain multiple joins together. ### Q: Do nested functions impact performance? **A**: Complex nested functions may take longer to compute, especially on large datasets. Test with small time ranges first and optimize your expressions. ### Q: Can I use Custom SQL instead of Auto SQL? **A**: Yes! If you prefer writing raw SQL, switch to **Custom SQL** mode. However, Auto SQL provides a visual interface that's easier for building queries. --- ## Additional Resources 📚 **DataFusion Documentation**: - [Aggregate Functions](https://datafusion.apache.org/user-guide/sql/aggregate_functions.html) - COUNT, SUM, AVG, etc. - [Scalar Functions](https://datafusion.apache.org/user-guide/sql/scalar_functions.html) - ROUND, CONCAT, CASE, etc. 🔗 **OpenObserve Resources**: - [GitHub Repository](https://github.com/openobserve/openobserve) - [PR #9343 - Feature Implementation](https://github.com/openobserve/openobserve/pull/9343) - [OpenObserve Documentation](https://openobserve.ai/docs/) 💬 **Get Help**: - [GitHub Issues](https://github.com/openobserve/openobserve/issues) - Report bugs or request features - [Community Discussions](https://github.com/openobserve/openobserve/discussions) - Ask questions and share ideas --- Design at: https://github.com/openobserve/designs/tree/main/joins --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Shrinath Rao <shnath@openobserve.ai>	2025-12-18 14:17:23 +05:30
Sai NikhilandByteBaker	e1e91ad54f	fix: incident analysis changes (#9673) ... ### **PR Type** Enhancement ___ ### **Description** - Backend: refactor agent_chat_stream for pooling & immediate status - Frontend: rotating “Observing…” UX & stream event handling - Incident drawer → full-screen dialog with two-column tabs layout - Minor UI tweaks: dialogs maximized & list wrapper update ___ ### Diagram Walkthrough ```mermaid flowchart LR A["agent_chat_stream endpoint"] -->|"get_agent_client()" B["Global pooled client"] B -->|"yield processing status" C["text/event-stream start"] C -->|"client.query_stream()" D["External RCA Agent Service"] D -->|"stream bytes" E["Forward chunks to client"] D -->|"on error" F["Yield error event"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table> <tr> <td> <details> <summary><strong>chat.rs</strong><dd><code>Refactor agent chat streaming & pooling</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> src/handler/http/request/agent/chat.rs <ul><li>Re-export enterprise AgentChat types under cfg(feature="enterprise")<br> <li> Provide stub types & init_agent_client for OSS builds<br> <li> Use global OnceCell client for connection pooling<br> <li> Rewrite streaming: send immediate "processing" status and handle <br>errors</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9673/files#diff-a0e13f17674a3657690f9e5829da2f1c4a6a957fe8a95f9c44b854643b5fe9b7">+68/-52</a>&nbsp; </td> </tr> <tr> <td> <details> <summary><strong>main.rs</strong><dd><code>Extend AI init to Agent client</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> src/main.rs <ul><li>Extend AI init to include agent client startup<br> <li> Update OpenAPI init log messages</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9673/files#diff-42cb6807ad74b3e201c5a7ca98b911c5fa08380e942be6e4ac5807f8377f87fc">+3/-3</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td> <details> <summary><strong>SREChat.vue</strong><dd><code>Enhance chat loading UX & stream handler</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> web/src/components/SREChat.vue <ul><li>Replace static loading text with rotating “Observing…” messages<br> <li> Add start/stop rotation & updateAssistantMessage helpers<br> <li> Support new event types: status, tool_call, tool_response, error<br> <li> Improve spacing, formatting and error handling in stream parser</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9673/files#diff-a24e3f6392181aa741979fe1c0cbc6f0f02337de20e6ad2fd6668dd7abe9bada">+131/-62</a></td> </tr> <tr> <td> <details> <summary><strong>IncidentDetailDrawer.vue</strong><dd><code>Refactor incident drawer to dialog UI</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> web/src/components/alerts/IncidentDetailDrawer.vue <ul><li>Switch from QDrawer to full-screen QDialog wrapper<br> <li> Implement two-column layout with tabs for analysis & triggers<br> <li> Add tile components for status, severity, alerts, timeline<br> <li> Themed styling, responsive scrollbars and updated actions</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9673/files#diff-6973d58a5f49658b07d3533d4db5d32cc6430433b51732481e7497243dadb0b0">+531/-179</a></td> </tr> <tr> <td> <details> <summary><strong>IncidentList.vue</strong><dd><code>Wrap incident detail in QDialog</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> web/src/components/alerts/IncidentList.vue <ul><li>Wrap IncidentDetailDrawer in QDialog instead of drawer<br> <li> Update v-model and close event handling</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9673/files#diff-953ad015996758a8c86e489499cd3b0a614501b99b0c9c5c242a299e34c24ff2">+11/-4</a>&nbsp; &nbsp; </td> </tr> <tr> <td> <details> <summary><strong>TracesAnalysisDashboard.vue</strong><dd><code>Maximize traces dashboard dialog</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> web/src/plugins/traces/metrics/TracesAnalysisDashboard.vue - Change dialog maximized prop from false to true </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9673/files#diff-d27dd619532ec278c376c09a05d5106fdb38863c3aa1c56e109fc2145a8a5a3a">+1/-1</a>&nbsp; &nbsp; &nbsp; </td> </tr> </table></td></tr></tr></tbody></table> </details> ___ --------- Co-authored-by: ByteBaker <42913098+ByteBaker@users.noreply.github.com>	2025-12-17 22:07:40 +05:30
udev	61dbc6ff7e	fix: pipeline/alert history and trigger schema merge (#9665) ... ### **User description** We need to ensure that whenever a trigger stream is being created (be it meta or non-meta org) we have the complete schema of the stream. This will prevent history queries from failing with `field not present` error We need to make pipeline history query against own orgs trigger stream we need to create filter with the alert ID for single alert ___ ### **PR Type** Bug fix, Enhancement ___ ### **Description** - fix alert history filtering to use key suffix - fix pipeline history query to own org triggers - enhance triggers schema initialization logic ___ <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Bug fix</strong></td><td><table> <tr> <td> <details> <summary><strong>history.rs</strong><dd><code>Use key suffix filter for alert history</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> src/handler/http/request/alerts/history.rs <ul><li>updated alert_id filter to use suffix match<br> <li> removed LIKE prefix filter escaping logic</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9665/files#diff-1c7483ca7a66b00f13cefdecadd02afbdf4baf93df5629adb8abe5942cfe5357">+3/-4</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td> <details> <summary><strong>history.rs</strong><dd><code>Query pipeline history from own org</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> src/handler/http/request/pipelines/history.rs <ul><li>replaced META_ORG_ID with dynamic org_id<br> <li> applied own org context for count and data queries</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9665/files#diff-aa1f4b2b5539a10dbabffb51724c916aac3d6b60e25ec150b54a32a06a6fc5d4">+2/-3</a>&nbsp; &nbsp; &nbsp; </td> </tr> </table></td></tr><tr><td><strong>Enhancement</strong></td><td><table> <tr> <td> <details> <summary><strong>triggers_schema.rs</strong><dd><code>Enhance triggers schema initialization logic</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> src/service/self_reporting/triggers_schema.rs <ul><li>removed META_ORG_ID conditional exclusion<br> <li> eliminated pre-existence schema check<br> <li> added expected_schema equality check before merge</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9665/files#diff-135de2980c6c20b6e5c8294e6bfd2c1c3dedb5995a6671a74726579e77ac983c">+16/-21</a>&nbsp; </td> </tr> </table></td></tr></tr></tbody></table> </details> ___	2025-12-17 21:13:38 +05:30
4 people	06df3fb784	feat: Issue 9646 promql streaming (#9648) ... ### **User description** Design at: dashboard ___ ### **PR Type** Enhancement ___ ### **Description** - Enable HTTP/2 streaming for PromQL queries - Extend streaming search util with PromQL support - Add PromQL streaming endpoints in service layer - Define TypeScript interfaces for PromQL streaming ___ ### Diagram Walkthrough ```mermaid flowchart LR A["Dashboard Loader"] -- "calls stream" --> B["fetchQueryDataWithHttpStream"] B -- "chunk data" --> C["handlePromQLResponse"] B -- "error" --> D["handlePromQLError"] B -- "complete" --> E["handlePromQLComplete"] C --> F["state.data update"] D --> G["state.errorDetail update"] E --> H["final loading false"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table> <tr> <td> <details> <summary><strong>usePanelDataLoader.ts</strong><dd><code>Implement PromQL streaming in data loader</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> web/src/composables/dashboard/usePanelDataLoader.ts <ul><li>Remove non-streaming PromQL calls<br> <li> Initialize state for streaming queries<br> <li> Process each query via fetchQueryDataWithHttpStream<br> <li> Handle streaming chunks, errors, completion</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9648/files#diff-5a42e9493a7bcd722b88e61fd38883c4f61001bc9cafa399f65f8eafda3aace9">+193/-73</a></td> </tr> <tr> <td> <details> <summary><strong>useStreamingSearch.ts</strong><dd><code>Add PromQL streaming to HTTP streaming hook</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> web/src/composables/useStreamingSearch.ts <ul><li>Extend StreamResponseType with <code>promql_response</code><br> <li> Support <code>promql</code> type in fetch/initiate functions<br> <li> Build GET URL for PromQL streaming endpoint<br> <li> Map <code>promql_response</code> via convertToPromQLResponse</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9648/files#diff-8fce3e25a161cd458e734bd9b1f98aadefe3482da7b6529a5b1f4fa20ec115e2">+51/-16</a>&nbsp; </td> </tr> <tr> <td> <details> <summary><strong>streaming_search.ts</strong><dd><code>Add PromQL streaming endpoints to service</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> web/src/services/streaming_search.ts <ul><li>Add <code>promqlQueryRangeStreamUrl</code> endpoint builder<br> <li> Add <code>promqlQueryRange</code> streaming request method<br> <li> Include query, time window and metadata params</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9648/files#diff-6a8e19a226bbf9e7e04903d5818c8fb5c9005a48988419f94d3e63a61ade7c57">+78/-0</a>&nbsp; &nbsp; </td> </tr> </table></td></tr><tr><td><strong>Documentation</strong></td><td><table> <tr> <td> <details> <summary><strong>query.ts</strong><dd><code>Define PromQL streaming TypeScript interfaces</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> web/src/ts/interfaces/query.ts <ul><li>Introduce <code>PromQLStreamingPayload</code> and <code>PromQLQueryPayload</code><br> <li> Add <code>PromQLStreamingResponse</code> interface</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9648/files#diff-12dd5b3bb9741bbeaa348c0efbbd745d0a582899678738159c6927d34a35191d">+21/-0</a>&nbsp; &nbsp; </td> </tr> </table></td></tr></tr></tbody></table> </details> ___ --------- Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech> Co-authored-by: Huaijin <haohuaijin@gmail.com> Co-authored-by: Abhay Padamani <141122205+ktx-abhay@users.noreply.github.com>	2025-12-17 19:17:18 +05:30
Ashish Kolhe	bd647bd341	fix: organizations ep permisions issue (#9667) (#9669) ... fixes: https://github.com/openobserve/openobserve/issues/9670 ### **PR Type** Bug fix ___ ### **Description** - Prioritize `META_ORG_ID` user in token validator - Prioritize `META_ORG_ID` user in credentials validators - Fallback to first available user when no META membership ___ ### Diagram Walkthrough ```mermaid flowchart LR A["Incoming request"] --> B["get_all_users()"] B --> C{"META_ORG_ID user exists?"} C -- "yes" --> D["Select META org user"] C -- "no" --> E["Select first user"] D --> F["Continue validation"] E --> F ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Bug fix</strong></td><td><table> <tr> <td> <details> <summary><strong>token.rs</strong><dd><code>Prioritize META org user in token validation</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> src/handler/http/auth/token.rs <ul><li>Added logic to prioritize <code>META_ORG_ID</code> user<br> <li> Fallback to first user if META org missing<br> <li> Inserted explanatory comment for permission rationale</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9669/files#diff-3339b4a964c121766ef87fcec0070f27a2b7085cf81281b53154e53d33cb28cf">+6/-1</a>&nbsp; &nbsp; &nbsp; </td> </tr> <tr> <td> <details> <summary><strong>validator.rs</strong><dd><code>Prioritize META org user in credentials validators</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary> <hr> src/handler/http/auth/validator.rs <ul><li>Applied META org prioritization logic<br> <li> Replaced <code>all_users.first()</code> with META find or fallback<br> <li> Updated both <code>validate_credentials</code> and <code>validate_credentials_ext</code></ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/9669/files#diff-f4d4b51574b378a8c372345ea48875834a4eadacfac33fa4d0ceca1e0ef96b35">+14/-2</a>&nbsp; &nbsp; </td> </tr> </table></td></tr></tr></tbody></table> </details> ___	2025-12-17 18:09:47 +05:30
Yashodhan Joshi	330a179d1c	refactor: file list dump and stream stats (#9236)	2025-12-17 09:44:42 +08:00
udev	19446a3d32	fix: empty user email on grpc metrics ingest (#9608) ... check auth can return req without populating metadata, this pr fixes the case and adds an `IngestUser` type to mark all system initiated ingestion calls and user initiated ingestion calls	2025-12-16 10:16:37 +05:30
Huaijin	6246b7bb27	feat: add peak memory usage to usage report (#9609)	2025-12-13 17:30:51 +08:00
sktrpathiandsai nikhil kethe	e52d000217	feat: add incident correlation system (#9507) ... ## Summary Introduces comprehensive incident management system to reduce alert fatigue through intelligent correlation and AI-powered root cause analysis. **Design at:** https://github.com/openobserve/designs/tree/main/alert-correlation --- ## Incident Correlation **3-Tier Correlation Priority:** 1. **Trace-based**: Groups alerts from same distributed trace (cross-service) 2. **Service Discovery**: Uses semantic service matching with normalization 3. **Dimension-based**: Blake3 hash from stable alert dimensions (fallback) **Features:** - Automatic alert grouping into incidents - Real-time incident status tracking (open/acknowledged/resolved/suppressed) - Topology enrichment via Service Graph integration - Cross-environment correlation prevention via dimension matching ## Incident Analysis (RCA) **Automated Analysis:** - Background job queries open incidents and triggers AI agent - Periodic analysis via configurable interval - Results displayed in incident detail view with markdown support **On-Demand Analysis:** - Manual trigger via API: `POST /api/v2/{org}/alerts/incidents/{id}/rca` - Real-time results with loading indicators - One-click "Analyze" button in UI ## SRE Chat Assistant **Conversational AI for Alerts:** - Generic `/api/{org}/agent/chat` endpoint with history support - Context-aware assistance for alerts and incidents - Sparkle icons on alert/incident rows for quick access - Persistent chat history via IndexedDB - Includes RCA analysis in conversation context --- ## Configuration ```bash # Incident Management O2_INCIDENTS_AUTO_RESOLVE_AFTER_MINUTES=-1 # -1=disabled (default), >0=auto-resolve # RCA Agent Integration O2_INCIDENTS_RCA_ENABLED=true O2_INCIDENTS_RCA_AGENT_URL=http://localhost:8000 O2_INCIDENTS_RCA_INTERVAL_SECS=600 ``` ## API Endpoints - `GET /api/v2/{org}/alerts/incidents` - List incidents with pagination - `GET /api/v2/{org}/alerts/incidents/{id}` - Get incident details - `PATCH /api/v2/{org}/alerts/incidents/{id}` - Update incident status - `GET /api/v2/{org}/alerts/incidents/stats` - Get incident statistics - `POST /api/v2/{org}/alerts/incidents/{id}/rca` - Trigger RCA analysis - `POST /api/{org}/agent/chat` - Generic agent chat endpoint - `POST /api/{org}/agent/chat/stream` - Streaming agent chat ## Database - **New tables**: `alert_incidents`, `alert_incident_alerts` - **Migration**: `m20251204_000001_create_alert_incidents_table` - **ORM**: SeaORM entities with full CRUD support ## UI Components - **IncidentList.vue**: Status filtering, pagination, statistics dashboard - **IncidentDetailDrawer.vue**: Full incident view with triggers/alerts/analysis - **SREChat.vue**: AI assistant panel with markdown rendering - Incidents tab in AlertList with unified view - AI sparkle icons for contextual chat access --- ## Enterprise Feature This is an **enterprise-only** feature with OSS stubs for compilation compatibility. - Non-enterprise builds: Handlers return `403 Forbidden` - Enterprise builds: Full functionality enabled ## Test Plan - [x] Build passes without `--features enterprise` - [x] Build passes with `--features enterprise` - [x] Create alerts that fire and verify incidents are created - [x] Verify incident list/detail APIs work correctly - [x] Verify status updates (open → acknowledged → resolved) - [x] Verify correlation logic (trace-based, service discovery, dimension-based) - [x] Verify RCA analysis triggers and displays correctly - [x] Verify SRE chat integration with alert/incident context --------- Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>	2025-12-12 22:14:18 +05:30
Loakesh Indiran	afc0d18d9b	fix: return 401 Unauthorized instead of 403 Forbidden for authentication failures (#9564) ... ## Problem APIs were returning **403 Forbidden** when tokens were invalid or expired, which is incorrect. According to HTTP specifications: - **401 Unauthorized**: Authentication failure (invalid/expired token, user not found) - **403 Forbidden**: Authorization failure (authenticated but lacks permissions) ## Changes - [x] Previously, short URL paths would redirect to `/web/` on auth failure. Now they consistently return 401 like all other endpoints. ### token.rs - Changed `ErrorForbidden` to `ErrorUnauthorized` when token validation fails (`is_valid == false`) - Changed `ErrorForbidden` to `ErrorUnauthorized` when user does not exist in the system - Changed `ErrorForbidden` to `ErrorUnauthorized` for non-enterprise builds (unsupported auth method) - Kept `ErrorForbidden` only for `check_permissions()` failures (true authorization issues)	2025-12-10 18:28:42 +05:30
Yashodhan JoshiandYashodhan Joshi	4780a9edc6	fix: rbac for native user for license call (#9529) ... Fixes an issue where a native user in non-default account would get 401 for license api call, whereas we want all users to have get access to license call. --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>	2025-12-10 17:07:31 +05:30
Yashodhan JoshiandYashodhan Joshi	6493f2b003	fix: cloud org related fixes (#9416) ... Fixes for various p2 bugs and UI issues, some other functionality changes as well. NOTE: because of some CLA issues, I have squashed all the original commits into a single one. There are no code changes compared to before. Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>	2025-12-10 15:15:29 +05:30
Ashish Kolheandgithub-actions[bot]	34939d0769	fix: fqn co relation user settings (#9536) ... Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>	2025-12-10 06:38:21 +05:30
3 people	79bbe72ee3	feat: co-relation phase 2 (#9513) ... Design at : streams-corelation --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>	2025-12-08 19:05:33 +05:30
4 people	9056a2c92d	feat: revamp Conditions in alerts and pipelines (#9267) ... Co-authored-by: Uddhav Dave <uddhav@MacBook-Pro.local> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> Co-authored-by: Shrinath Rao <shnath@openobserve.ai>	2025-12-08 11:36:56 +05:30
Hengfei Yang	3435b80d6d	feat: Add cache reload API and CLI command (#9497)	2025-12-05 16:01:01 +08:00
udev	0829ee89fe	feat: add metrics workgroup (#9400)	2025-12-04 21:43:31 +05:30
Hengfei Yang	04b15af48f	refactor: unify bulk API to use the same ingestion function (#9467)	2025-12-04 18:21:58 +08:00
Ashish Kolheandgithub-actions[bot]	1353ea653b	feat: service discovery (#9469) ... Design at: streams-corelation/SERVICE_CORRELATION_ARCHITECTURE.md --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>	2025-12-04 13:58:09 +05:30
Hengfei Yang	9c249f2c3c	fix: rollback scc (#9442)	2025-12-03 11:08:23 +08:00
ByteBaker	ada80858e8	feat: migrate service graph to SQL-based daemon architecture ... Replace in-memory EdgeStore with daemon-based architecture that periodically queries trace streams using SQL aggregation in DataFusion. Architecture: - Daemon runs in compactor job every 1 hour (default, configurable) - Queries last 60 minutes of trace data (configurable window) - Uses DataFusion SQL with CTE for efficient aggregation - Calculates p50, p95, p99 percentiles using approx functions - Writes aggregated edges to internal `_o2_service_graph` stream - Zero impact on trace ingestion performance Key Changes: - Add `processor.rs` with SQL-based daemon that queries trace streams - Add `aggregator.rs` to write SQL results to `_o2_service_graph` stream (uses json! macro) - Refactor `api.rs` to query pre-aggregated data from stream - Move business logic to enterprise repo (`build_topology()`, `span_to_graph_span()`) - Remove ALL inline processing during trace ingestion - Delete in-memory EdgeStore, worker threads, and buffering code - Feature-gated for enterprise builds with non-enterprise stubs SQL Query: - CTE extracts client/server from span_kind ('2'=SERVER, '3'=CLIENT) - Aggregates by (client, server) with COUNT, percentiles, error rate - Filters: span_kind IN ('2','3') AND peer_service IS NOT NULL Configuration: - O2_SERVICE_GRAPH_ENABLED (default: false) - O2_SERVICE_GRAPH_PROCESSING_INTERVAL_SECS (default: 3600) - Daemon run frequency - O2_SERVICE_GRAPH_QUERY_TIME_RANGE_MINUTES (default: 60) - Query window size Technical Details: - Stream name: `_o2_service_graph` (internal stream) - Edge timestamp: Uses query end time for API compatibility - Span kinds: '2' (SERVER), '3' (CLIENT) as VARCHAR in parquet - Percentiles: approx_median(), approx_percentile_cont(0.95), approx_percentile_cont(0.99)	2025-12-01 17:06:53 +05:30
Loakesh Indiran	baa7520d90	fix: overwrite aggs cache instead of delete and rewrite cache (#9390) ... How I run my query 1. nov 26 10:30 - 11:30 - interval 30 mins 2. nov 26 10:30 - 16:30 - interval 60 mins 3. nov 26 10:30 - 16:30 - interval 60 mins -- refresh cache Cache dir 1. on first run - has 2 caches 30 mins 2. on 2nd run - will havve mixed 30mins cache and 60 mins cache TODO: - [x] Problem 1: on step 3 - the interval for that duration is 60 mins - but the discovery of cache will also include the 30 mins cache [@mod.rs](file:///Users/lin/o2/openobserve/src/service/search/mod.rs) `discover_cache_for_query` - the overwrite will over write the 60 mins cache only. How to address this problem? - [x] Problem 2: - fix: add rbac control only user with permussion to edit stream can `refresh cache` Proposed Solution - On `refresh_cache` run streaming aggs without using cache - at `streaming/execution.rs` , at the end of steaming aggs delete all the cache files for the given duration when there are miss matched cache interval files - then remove the `streaming_id` from the `GLOBAL_CACHE` - eg: nov 26 10:30 - 16:30 - interval 60 mins -- refresh cache - Cache consists of 30mins or 4hr interval files all of them will be deleted - 60 mins cache interval files will be overwritten or created a new. - This way on the next run there will be no query results mismatch when the query runs again with the newly updated cache, especially delayed data for the same duration. - add rbac control to only allow users with edit permissions to stream to use this feature	2025-11-29 12:49:39 +05:30