mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-05 16:10:43 +00:00
e17e8bc5e6a83d6a547db18cfe25bd79ebfebb21
727
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  
|
78c96c3721 |
fix: support bulk deletes for various resources (#9153)
### **User description** fixes https://github.com/openobserve/openobserve/issues/9116 This adds support for bulk delete operation from API and UI side for various resources. ___ ### **PR Type** Enhancement ___ ### **Description** - Add bulk delete endpoints for multiple resources - Enforce per-item permission checks before delete - Aggregate successful/failed items in responses - Register bulk-delete routes in HTTP router - Extend UI: add bulk-delete buttons, dialogs, service calls ___ ### Diagram Walkthrough ```mermaid flowchart LR UI["\"UI: Bulk Delete Trigger\""] API["\"HTTP DELETE /{resource}/bulk Handler\""] AUTH["\"Permission Checks per Item\""] DEL["\"Perform Delete Operations Loop\""] RESP["\"Aggregate Response: successes/ failures\""] UI_FE["\"FE: Show Notifications\""] UI -- "calls" --> API API -- "checks" --> AUTH AUTH -- "allowed" --> DEL DEL -- "deletes items" --> RESP RESP -- "returns" --> UI_FE ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>14 files</summary><table> <tr> <td><strong>fga.rs</strong><dd><code>Add bulk delete roles and groups endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-e7c99f37f3abbb5e0507d2760bbf67df010d9218704b38dce84c5477acecd22e">+218/-6</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Implement delete_alert_bulk handler</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-83431801d27626ef27482e67b62661f900043544a45ddf958bd027cf2fa3bd8d">+87/-6</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add delete_function_bulk and error types</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-a5e1b3644cef9feb689dfe66382e44e5434ecadbff3469127346a333d4733102">+97/-3</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add delete_dashboard_bulk endpoint</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-1b3c257e1d174c826718499faa4a1d4e93465dd56cb2b7b273b690ff7e3223fd">+78/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Implement re_patterns delete_bulk logic</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-27d37a8738de89c81b540daa978e684caf2cf28803e1208d90cb4d42528ed28b">+116/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add users bulk delete handler</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-56897e42f9436a640bc70fddc25686485ba8ae1223635b32d252a0333309f748">+80/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add service_accounts delete_bulk API</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-095d70f65b6e69bf7de0180fcbcddcd7c89c4cf00095d7b06f039a07e73916bb">+86/-1</a> </td> </tr> <tr> <td><strong>action.rs</strong><dd><code>Implement delete_action_bulk endpoint</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-616df3014c553f763ea57e925f1bff7bb47f9467c632940d93a45b7b1e4c3a07">+90/-1</a> </td> </tr> <tr> <td><strong>destinations.rs</strong><dd><code>Add delete_destination_bulk for alerts</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-76b00293c621036de31cf499c1c276efb9704c2bf8cf1b728f49cfdd43f9fd1b">+75/-4</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add delete_bulk for cipher_keys</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-043222e433d1e5825a4b7bfc2c7feedba417f2630f0082fa67aff64976736396">+76/-1</a> </td> </tr> <tr> <td><strong>templates.rs</strong><dd><code>Implement delete_template_bulk handler</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-b08e274a64cc935b0cec402a04271a872b1586027a77dc0ccb84c597c3960f61">+73/-4</a> </td> </tr> <tr> <td><strong>pipeline.rs</strong><dd><code>Add delete_pipeline_bulk API handler</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-c69f4f94eee97ce226caddc497b2adf89e26869f2d72b0d0206dc31ab2714daf">+70/-2</a> </td> </tr> <tr> <td><strong>reports.rs</strong><dd><code>Add delete_report_bulk endpoint</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-00435e0d9301342b51cb57ae313f1413e7ed973642e0badf3f3959125d0677d1">+75/-1</a> </td> </tr> <tr> <td><strong>ActionScripts.vue</strong><dd><code>Integrate bulk delete UI in action scripts</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-770f71fcb92a13b3b8b1041567b885fac39903c814825c878595cafa624ebbd8">+284/-5</a> </td> </tr> </table></details></td></tr><tr><td><strong>Configuration changes</strong></td><td><details><summary>3 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Register bulk delete services in router</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+14/-0</a> </td> </tr> <tr> <td><strong>iam.ts</strong><dd><code>Add bulkDeleteGroups and bulkDeleteRoles calls</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-3712c1bb0eecb664b6691d724c3673b8c1befa41efbd8c5de626c5849f8ace98">+10/-0</a> </td> </tr> <tr> <td><strong>dashboards.ts</strong><dd><code>Add bulkDelete service for dashboards</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-913b3f54367036b1b3960aa3eddd7e17b509b1d6011c5ab68ae0e935653179c5">+7/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Error handling</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>auth.rs</strong><dd><code>Allow bulk-delete paths in auth extractor</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-9bc2b1841504a435f66ad094f72ddbe7b7fa75e5e1557b8adbfe36f7f0592e1b">+17/-1</a> </td> </tr> </table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>user.rs</strong><dd><code>Define bulk delete request/response structs</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-e13b3a981fd70182b9738a6b0ecfdee915c5b36757703a87741e1eddf3f6e786">+36/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add PipelineBulkDelete meta types</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-c4bd542e4e07ed7706b0643086f5a7d4edbb3647c7ceedbe4bea516c5274d8bd">+12/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>40 files</summary><table> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-6f9c2e6dbc40f8d40e3d3ecbee23240813a3d4856d49bd5d979f5b98551fdc3d">+12/-0</a> </td> </tr> <tr> <td><strong>service_account.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-878ad520e1c84500cc21ff6fc4088c8da64ff2febd79e60c0d4cc8c695c49798">+11/-0</a> </td> </tr> <tr> <td><strong>action.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-fc87c75a7df9539d7e7ee6184dfab02d4a8932323e81f95d57c05363aa44752d">+15/-0</a> </td> </tr> <tr> <td><strong>reports.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-d2117a3d67d4ec47b0017861845c92143baf917e10e6746df5dd97f925f81eeb">+12/-0</a> </td> </tr> <tr> <td><strong>function.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-27c9f3d93723397a0e234bc45c9f543b43eff5fd518d76e735f78d95a58f3f1c">+12/-0</a> </td> </tr> <tr> <td><strong>requests.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-412f11e06c0e901e2292aab80536cb36edebcc4796dec52923389c3aaa32b25a">+6/-0</a> </td> </tr> <tr> <td><strong>responses.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-224027f90893122eea28e35778467acf1a4b5572ce612fc9bc9ec44ca8ebd9fb">+9/-0</a> </td> </tr> <tr> <td><strong>dashboards.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-250eeebeec9d4ac0ac271466ed83cd33fea3f121080491b340dc95d9af287259">+12/-0</a> </td> </tr> <tr> <td><strong>destinations.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-ecffe114c8200bab742dd2ee4f7f41128aa59438a879634f070edd575a1f917c">+24/-0</a> </td> </tr> <tr> <td><strong>search_stream.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-642de43ff6b19ae41087a9f1efc02d8b0b873809e0c92cbddfac1a30ef3f1184">+1/-0</a> </td> </tr> <tr> <td><strong>utils.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-96bb9f4a8f547c821f26c957e8505d0437095b08d2b36687531f71565d287bd6">+2/-1</a> </td> </tr> <tr> <td><strong>functions.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-ed5611a9aa6b5c033e70d7631ba72fafad3cc842672077a7fcf965d11a4fef23">+22/-31</a> </td> </tr> <tr> <td><strong>AppTable.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-efcf66a9e913b5d3a7c2eea800b7f4e54d9e5268307131482fd40f32e2074d8e">+72/-16</a> </td> </tr> <tr> <td><strong>AlertList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-be3e41b823e163b226ffc270d95cc4981e59c600851e4e0d10ee5e4ee994ac8e">+123/-2</a> </td> </tr> <tr> <td><strong>AlertsDestinationList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-e9d05ba6c071c0e2f5d9ef7acdef757f79910abd6897b29e27b069a10d4ba8de">+142/-5</a> </td> </tr> <tr> <td><strong>PipelinesDestinationList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-088c66f5a34e73a5350a794a772e5d905262277337c937dd36f4295072203ebf">+163/-28</a></td> </tr> <tr> <td><strong>TemplateList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-7df66364d7d69a26cfa25cbc555d0e090c2f73e51c8386d9b0435bae4412564c">+116/-27</a></td> </tr> <tr> <td><strong>EnrichmentTableList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-6a29e253f59405d3c0f756dba36afd7483f10ad86927cbbc3348ede6372c3077">+124/-13</a></td> </tr> <tr> <td><strong>FunctionList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-6ff9af50a1597504b7f0dca91e61cf30c800b4a17b4be2ed2c3a4f7a7b2d7068">+143/-6</a> </td> </tr> <tr> <td><strong>AppGroups.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-402fb2a5338e60fdc3da489f4d50b5cf52f06b4e884a6fe7b82441c9fa5ea22c">+86/-1</a> </td> </tr> <tr> <td><strong>AppRoles.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-191634ef09436199f91f442968110a01f54ae46b5ceee67476f52613ad796db4">+86/-1</a> </td> </tr> <tr> <td><strong>EditRole.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-263b6bc7d29fdf8358c8437cd637f1350aa3733cc2706eae8ff00028686cc0f5">+20/-0</a> </td> </tr> <tr> <td><strong>ServiceAccountsList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-3afe835bcfbec3fda2e9550850c29581092ee0717e7dd8de8888e774a0048d24">+114/-1</a> </td> </tr> <tr> <td><strong>User.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-e973c2a665f9b27b94673ba64cc5b23650300836a727a2e07673c8e26305aee5">+133/-3</a> </td> </tr> <tr> <td><strong>PipelinesList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-85e3ef322f3eea0fc6d74d89431ba338a1153cb82ee06bef0f7c5f439c04267d">+108/-2</a> </td> </tr> <tr> <td><strong>ReportList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-f7d700f7e67fbaba3c58ab5d2c614925db743101828214445ce9dae35dc29067">+157/-5</a> </td> </tr> <tr> <td><strong>CipherKeys.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-950b715cff2c964d8542512692030c370733b57d698abf40e1dcb3cce2cfaf7a">+104/-13</a></td> </tr> <tr> <td><strong>RegexPatternList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-524927acd9f5bb07b71bfac489e747470108574e21a6dde63ea7c1f219cea853">+109/-14</a></td> </tr> <tr> <td><strong>action_scripts.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-6c1a011ac23290134801b22feb2b50c2b19cabf2a009f37a62b4dd9ca6e415e3">+3/-0</a> </td> </tr> <tr> <td><strong>alert_destination.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-377d28822e9ad4e9e80327ecc2d09bf1981c7997c5f1c1de120c6179eef19c6e">+3/-0</a> </td> </tr> <tr> <td><strong>alert_templates.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-fd8c4260abc3df601e4331d36b0c211eca568ee70139ad90c87247bc552fe4e2">+3/-0</a> </td> </tr> <tr> <td><strong>alerts.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-7a62782cceacac92b787e058219b3c53ea7406d4be5780def899943a01f500c1">+7/-0</a> </td> </tr> <tr> <td><strong>cipher_keys.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-0a0ef3a63c29b2a74b7f904c54c3fb32bee421608474aa32334b7c55133ab1ef">+3/-0</a> </td> </tr> <tr> <td><strong>jstransform.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-301add2528e0f32323ca4c503bd7d460ab15ca6b9a9537192a02c5e52fd82926">+3/-0</a> </td> </tr> <tr> <td><strong>pipelines.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-e8626670bb4e46e073f9858ade6b3dc77eaadf39d6721d9c0006eccc1b430612">+8/-0</a> </td> </tr> <tr> <td><strong>regex_pattern.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-2a0b0828141d39ab0385e0941314ead1cdd2ec3ec7b3baf5fd0c42f8f010728a">+3/-0</a> </td> </tr> <tr> <td><strong>reports.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-32d3299b2b21dc57c39d01080baba03765d853bd05c82d6158834b54713ae825">+3/-0</a> </td> </tr> <tr> <td><strong>service_accounts.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-4038191b0dc0cad335f3d8f7f703505df905ad843fe950096aefb8abbd139a1e">+4/-1</a> </td> </tr> <tr> <td><strong>users.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-a334d8d570d9bdcaa3d4c8c2a993eff009a1ec8eb499c8a3d1a292d9224159d0">+3/-0</a> </td> </tr> <tr> <td><strong>Dashboards.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-8da3187db8ce3d795d0f196ede6b2290221ffa3e27b15dcc2b645e44dd196213">+122/-6</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> |
||
 Hengfei Yang
|
511f2389c6 | feat: new db migration (#9774) | ||
|
Hengfei Yang
|
e59474aa1b | perf: reduce nats events (#9712) | ||
  
|
e40938a08c |
feat: enrichment table file url support (#9697)
### **PR Type**
Enhancement
Design at : enrichment-file-url
___
### **Description**
- Async URL-based enrichment table processing
- Event-driven MPSC, streaming CSV, batching
- Retry with backoff, status updates
- Database and ORM additions
- `enrichment_table_urls` table and migration
- HTTP API endpoints
- POST `/url` import and GET status
- Frontend enhancements
- Tabs, status icons, retry button
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["HTTP POST /{org}/enrichment_tables/{table}/url"] --> B["DB: save_url_job"]
B --> C["trigger_url_job_processing()"]
C --> D["MPSC Channel"]
D --> E["process_url_jobs()"]
E --> F["process_single_url_job()"]
F --> G["validate_url() & fetch_and_process_batches()"]
G --> H["parse_csv_chunk() & save_enrichment_batch()"]
H --> I["save_url_job() update status"]
I --> J["notify_update()"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>14
files</summary><table>
<tr>
<td><strong>url_processor.rs</strong><dd><code>Implement async URL
enrichment table processor</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-d39589e34157fe7930563b9bc5740128e168b8d8ac0a90371a4bdf1e52ce058b">+1351/-0</a></td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Add URL enrichment table HTTP
endpoints</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-7b5b203002d1c3e5f671392a2a8ed84f45225896b652c2dac94ac26e72a43124">+301/-1</a>
</td>
</tr>
<tr>
<td><strong>enrichment_table.rs</strong><dd><code>Add URL job CRUD
persistence functions</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-6ce27ddf8a0d8b4e790bda90d088ff692a5da31a8d330beea8a9dd843294572b">+159/-2</a>
</td>
</tr>
<tr>
<td><strong>enrichment_table_urls.rs</strong><dd><code>SeaORM logic for
URL job records</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-e9ba1ba6e742e8d7652c3f31765de43bee061e7983bca67ffefecf8590210188">+173/-0</a>
</td>
</tr>
<tr>
<td><strong>enrichment_table.rs</strong><dd><code>Define URL job status
and job struct</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-29818b7d921497260251df82a49ae9ab7a93bbaedc0ca03abdf2b8c0726e4085">+90/-0</a>
</td>
</tr>
<tr>
<td><strong>pipelines.rs</strong><dd><code>Handle super-cluster URL job
events</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-7c8aa678045dd00de89d4c2345cf4a0511ce9bfac5d6f64e7404178c81e541ad">+23/-1</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Expose url_processor module in
service</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-3e2b6d3b8da1f9e73aca279052a97e683184d68256c23215b18ffee885836bbf">+6/-0</a>
</td>
</tr>
<tr>
<td><strong>enrichment_table_urls.rs</strong><dd><code>Define SeaORM
entity for URL jobs</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-9c3081476927bd337b2cbed8310578869845a6137a2c5305f677affb7604d39f">+26/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Register URL handlers in
router</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+2/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Expose enrichment_table_urls entity
module</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-c539d7557d1425986d5c09849d8dbca42d1b309c2bc33d49db641d6a783098f6">+1/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Add enrichment_table_urls to table
module</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-af023301203a27a1ef34f698d5a329ad7bfced65ac65e3829b1c0905a0c74fb2">+1/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Include enrichment_table meta
module</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-bab79e8da2d0d9f903cb1a6187d922c6de63d3e4561e8ce633660b664b4e859f">+1/-0</a>
</td>
</tr>
<tr>
<td><strong>jstransform.ts</strong><dd><code>Add client methods for URL
jobs API</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-301add2528e0f32323ca4c503bd7d460ab15ca6b9a9537192a02c5e52fd82926">+18/-0</a>
</td>
</tr>
<tr>
<td><strong>EnrichmentTableList.vue</strong><dd><code>Enhance UI for
URL-based enrichment tables</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-6a29e253f59405d3c0f756dba36afd7483f10ad86927cbbc3348ede6372c3077">+327/-74</a></td>
</tr>
</table></details></td></tr><tr><td><strong>Configuration
changes</strong></td><td><details><summary>3 files</summary><table>
<tr>
<td><strong>m20251221_000001_create_enrichment_table_urls.rs</strong><dd><code>Create
migration for URL jobs table</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-330b4b7ebbda5001f4f28050d50e219b91961d80d8369222d94d058cced57702">+189/-0</a>
</td>
</tr>
<tr>
<td><strong>config.rs</strong><dd><code>Add URL fetch configuration
options</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+24/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Register URL jobs migration in
migrator</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-7ee42a714d3d56c663da0412d15514613222055aebb5b50dafbc051a656a36f6">+2/-0</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>1
files</summary><table>
<tr>
<td><strong>openapi.rs</strong><dd><code>Document URL endpoints in
OpenAPI spec</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-edb8ce232c0cd08366879f642788a75d99f6bab64b4c0e77391281fdb9241e90">+5/-0</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Bug
fix</strong></td><td><details><summary>1 files</summary><table>
<tr>
<td><strong>auth.rs</strong><dd><code>Extend auth for enrichment_tables
URLs</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-9bc2b1841504a435f66ad094f72ddbe7b7fa75e5e1557b8adbfe36f7f0592e1b">+9/-1</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Additional
files</strong></td><td><details><summary>1 files</summary><table>
<tr>
<td><strong>AddEnrichmentTable.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-9aad551dbc051edfdb3994a664506ed6d5e27dcb64f9c2fa917b02e51a35d86e">+159/-42</a></td>
</tr>
</table></details></td></tr></tr></tbody></table>
</details>
___
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>
|
||
|
Yashodhan Joshi
|
330a179d1c | refactor: file list dump and stream stats (#9236) | ||
 udev
|
19446a3d32 |
fix: empty user email on grpc metrics ingest (#9608)
check auth can return req without populating metadata, this pr fixes the case and adds an `IngestUser` type to mark all system initiated ingestion calls and user initiated ingestion calls |
||
|
Yashodhan JoshiandYashodhan Joshi
|
6493f2b003 |
fix: cloud org related fixes (#9416)
Fixes for various p2 bugs and UI issues, some other functionality changes as well. NOTE: because of some CLA issues, I have squashed all the original commits into a single one. There are no code changes compared to before. Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> |
||
 
|
79bbe72ee3 |
feat: co-relation phase 2 (#9513)
Design at : streams-corelation --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> |
||
|
Hengfei Yang
|
04b15af48f | refactor: unify bulk API to use the same ingestion function (#9467) | ||
|
Ashish Kolheandgithub-actions[bot]
|
1353ea653b |
feat: service discovery (#9469)
Design at: streams-corelation/SERVICE_CORRELATION_ARCHITECTURE.md --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
|
udev
|
99c3391575 |
fix: audit (#9417)
This brings changes to the MMDB parser logic. Need to test via VRL functions. |
||
|
Hengfei Yang
|
9c249f2c3c | fix: rollback scc (#9442) | ||
|
Hengfei Yang
|
fa0672a663 | feat: add super cluster support for metrics (#9225) | ||
 Huaijin
|
9190a9d650 | feat: upgrade datafusion to 51 (#9202) | ||
 Harsh Mahajan
|
1f06d4a46a | feat:adding configuration ENV for colour theme (#9165) | ||
|
Ashish Kolhe
|
9059afdb2b | feat: sso claim parser (#9156) | ||
 Simha G
|
6ca371ec99 | fix: Concurrency control for file locking has multiple anti-concurrent design patterns in the way (#8523) | ||
|
Hengfei Yang
|
b950b1cec1 |
fix: debug metrics stream keep update (#9141)
I saw a lot of logs like this:
```
DEBUG openobserve::service::db::schema: [Schema:watch] Received event: Put(EventData { key: "/schema/default/metrics/http_server_duration", value: Some(b""), start_dt: None
```
This PR optimized the metadata check logic also try to debug this case.
|
||
|
Sai Nikhil
|
803434bd30 | fix: theme customization (#9077) | ||
|
|
85a8fc1648 |
feat: license check (#8522)
### **User description** This adds license check job ___ ### **PR Type** Enhancement, Bug fix ___ ### **Description** - Add enterprise license HTTP APIs and DB watch - Enforce quota checks on search and ingestion - Always enable usage reporting; simplify modes - Expose license expiry in config and UI ___ ### Diagram Walkthrough ```mermaid flowchart LR Backend["Backend: License APIs + quota checks"] DB["DB: License watch + dist lock"] Config["Config: usage/reporting & expiry"] UI["UI: Settings License page + banner"] Search["Search/PromQL: quota enforcement"] Backend -- "watch updates" --> DB Backend -- "expose expiry" --> Config UI -- "GET/POST /api/license" --> Backend Search -- "check_search_allowed" --> Backend ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>21 files</summary><table> <tr> <td><strong>config.rs</strong><dd><code>Validate usage mode and publish interval bounds</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+16/-4</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add license GET/POST endpoints with validation</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-d569000673e415b222a20a6f4f50a93c278c189a7d6e43231d5ab8807e93021f">+75/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Register license routes for enterprise builds</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-a1cfd40da1ab6cd4db0181fd8664965034298047b3b7c3320c5385897a7c4aed">+2/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Enforce search quota in PromQL endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-0ad5729a1fbce00ff53076728c51822400c13af9918e1d8554f41be56d060fa0">+24/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add enterprise search quota checks in endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-96630a0dbb6dd7b2b0248fa325c7c7041ce31340c32826675a4a70772e9d29f6">+51/-1</a> </td> </tr> <tr> <td><strong>search_stream.rs</strong><dd><code>Enforce quota in streaming search endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-642de43ff6b19ae41087a9f1efc02d8b0b873809e0c92cbddfac1a30ef3f1184">+21/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Expose license expiry; always enable usage flag</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-ca09c9c9b020049bdbc3382c466bfcca80daac92e98760d9a9a76172abf34049">+8/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Wire license endpoints into router</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+3/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add license check locking and startup watchers</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-f2e4c29b8eef22189ec704a832633c4bd135a24444f054864c98064bd0963681">+57/-0</a> </td> </tr> <tr> <td><strong>license.rs</strong><dd><code>Implement license update broadcast and watch</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-43bb2d7d70d03657337dfabb078079de71fae8a7f58962a10ac68c03594e0f81">+70/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Export license DB module for enterprise</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-b7e3bf7f0ac23dd084c0c817a310a4d5e7a64ff532746d182986ef940fa03fd0">+2/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Enforce ingestion quota for enterprise</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-7c7b5bca85f0ced652965cae66540c96f56dfcca4f4c16b77bf7b45aac203d1b">+10/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Provide search quota check helper</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-c3b85ea662a885fbd39797968928b291cb59a58f53a03f7b7dfd7ba8b6a7985a">+18/-0</a> </td> </tr> <tr> <td><strong>ingestion.rs</strong><dd><code>Simplify reporting mode; always requeue on error</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-3edd5f4ed1eed0f765b7b450ef0fbf587632e6b22302006a5e397049e297057f">+19/-50</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Always enable usage queue and publishing</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-c6255010f7a87d6acdb3ad04954fc9f2c27350ae319c53a1857b120aa06408a3">+2/-25</a> </td> </tr> <tr> <td><strong>useManagementRoutes.ts</strong><dd><code>Add license management route</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-b71a1d39b1fab0749f0668d5f16382536fbda6e2f0c92c56a972fb5d8b5835f5">+8/-0</a> </td> </tr> <tr> <td><strong>license_server.ts</strong><dd><code>Add client for license APIs</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-3ac3fe8faca5f5a6aebd9d5e8885837f1511834eb7c78d0830af8f00c70523cb">+16/-0</a> </td> </tr> <tr> <td><strong>License.vue</strong><dd><code>New license management UI with usage</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-614c7b48f6f2ab1e8041d20c7583b6864285eacf9db032c27508b76f6de8467f">+469/-0</a> </td> </tr> <tr> <td><strong>index.vue</strong><dd><code>Add License tab and routing guards</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-699c63979f851c48ad440a078e93282fd797a18fc065e5331b2e41122ddb8b69">+27/-0</a> </td> </tr> <tr> <td><strong>LicensePeriod.vue</strong><dd><code>License expiry banner component</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-14f02914aaa78ef3e82717173fa95455b7f1092faff2d0dbd51e243663850ad4">+111/-0</a> </td> </tr> <tr> <td><strong>HomeView.vue</strong><dd><code>Show license expiry banner and navigate</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-9af441af191d3254ef5f9db603dca75486de7638cda41a3c050e7b3c33d68280">+9/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Bug fix</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>schema.rs</strong><dd><code>Remove usage_enabled dependency in org creation</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-4696a44abde2df4ac48fed716f930bfdcc01759bdfcfd8d802821c6dd857bff9">+1/-3</a> </td> </tr> </table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>en.json</strong><dd><code>Add i18n key for License tab label</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-892aa3d75ab6e129ef9116aa21bec1b9a15ba69ce5366d2a19128078c690d824">+2/-1</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> |
||
 Loakesh Indiran
|
6b76fbf785 | feat: clear cache main (#8919) | ||
 ByteBaker
|
1593db0e22 |
feat: skip aud validation for MCP-invoked requests
- Add `x-o2-mcp` header support for MCP request identification in token validator - Wrap OpenAPI schema types with `inline()` for better documentation generation - Update AI module imports to use new agent namespace - fix compilation for `cloud` feature, also return early if user isn't in trial - Enable service graph by default |
||
 Ashish Kolheandomkark06
|
196512057c |
feat: log patterns (#8927)
PR introduces below features: - [x] Reporting triggers to source org - [x] Alerts Insights for detailed visibility in alerts Enterprise features: - [x] Patterns extraction during query - UI and API - [x] Patterns extraction as part of ingestion process (No Sampling today) - [x] Settings to enable pattern extraction --------- Co-authored-by: omkark06 <omkar@zinclabs.io> |
||
|
Hengfei Yang
|
f56b2b8f17 | fix: metrics can't search parquet in wal (#8896) | ||
|
Loakesh Indiran
|
875d1f3005 |
feat: cache hist query with non ts col order by (#8837)
### **User description** ref: #8731 ___ ### **PR Type** Enhancement, Bug fix ___ ### **Description** - Cache support for histogram with non-TS ORDER BY - Scan all hits to compute time range - Preserve ORDER BY metadata in cache writes - Refactor utils for ORDER BY timestamp checks ___ ### Diagram Walkthrough ```mermaid flowchart LR SQL["SQL parse (histogram, ORDER BY)"] Check["check_cache: detect non-TS histogram order"] Req["CacheQueryRequest + is_histogram_non_ts_order"] Utils["result_utils: is_timestamp_field, has_non_timestamp_ordering, extract_timestamp_range"] Write["write_results: scan hits for min/max when needed"] WS["streaming cache: propagate flag"] Mod["search mod: cache write integration"] SQL -- "order_by, ts_column" --> Check Check -- "set flag" --> Req Check -- "use timestamp detection" --> Utils Req -- "persist flag" --> Write Mod -- "compute is_histogram_non_ts_order" --> Write WS -- "compute and pass flag" --> Write Write -- "min/max via Utils" --> Utils ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table> <tr> <td> <details> <summary><strong>search.rs</strong><dd><code>Add non-TS histogram order flag to request</code> </dd></summary> <hr> src/common/meta/search.rs <ul><li>Add <code>is_histogram_non_ts_order</code> to <code>CacheQueryRequest</code><br> <li> Document behavior requiring full-hit scan</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/8837/files#diff-a36a6683e327641569d8098ee0d4f13e894e1b9d51230dc3569abe42e87ba310">+5/-0</a> </td> </tr> <tr> <td> <details> <summary><strong>cacher.rs</strong><dd><code>Detect and flag non-TS ordered histograms</code> </dd></summary> <hr> src/service/search/cache/cacher.rs <ul><li>Import new utils for ORDER BY checks<br> <li> Allow histogram caching without TS ORDER BY<br> <li> Determine <code>is_histogram_non_ts_order</code><br> <li> Propagate flag in cache write paths</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/8837/files#diff-8c387b22b2c0a282600ee41ca702b3db60f5a284d507c0feb7418608ff062c6c">+33/-3</a> </td> </tr> <tr> <td> <details> <summary><strong>mod.rs</strong><dd><code>Compute and pass histogram order flag to writer</code> </dd></summary> <hr> src/service/search/cache/mod.rs <ul><li>Import <code>extract_timestamp_range</code><br> <li> Compute <code>is_histogram_non_ts_order</code> on write<br> <li> Pass flag to <code>write_results</code></ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/8837/files#diff-f07bfacc0501b9c234e64b16c1dd8eb0ae8fcbe231f90c81fed72bcc94946f74">+18/-5</a> </td> </tr> <tr> <td> <details> <summary><strong>result_utils.rs</strong><dd><code>Utilities for timestamp/order checks and range extraction</code></dd></summary> <hr> src/service/search/cache/result_utils.rs <ul><li>Add <code>is_timestamp_field</code> helper<br> <li> Add <code>has_non_timestamp_ordering</code><br> <li> Add <code>extract_timestamp_range</code> with scan fallback</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/8837/files#diff-fb601e2f37657029b6a4dcb82d16aeff5916e13660d284e7fb00db6002dc0f73">+59/-0</a> </td> </tr> <tr> <td> <details> <summary><strong>cache.rs</strong><dd><code>Streaming cache supports non-TS histogram ordering</code> </dd></summary> <hr> src/service/search/streaming/cache.rs <ul><li>Detect non-TS histogram order in streaming path<br> <li> Pass flag to cache <code>write_results</code></ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/8837/files#diff-55769b7cb67b550daf45363c6d7d569b39d4a6235f6a68c091beb40f16b6511e">+10/-0</a> </td> </tr> </table></td></tr><tr><td><strong>Miscellaneous</strong></td><td><table> <tr> <td> <details> <summary><strong>mod.rs</strong><dd><code>Hardcode partition timing and count</code> </dd></summary> <hr> src/service/search/mod.rs <ul><li>Force <code>total_secs = 4</code> (debug change)<br> <li> Force <code>part_num = 10</code> (debug change)</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/8837/files#diff-c3b85ea662a885fbd39797968928b291cb59a58f53a03f7b7dfd7ba8b6a7985a">+2/-0</a> </td> </tr> </table></td></tr></tr></tbody></table> </details> ___ --------- Signed-off-by: loaki07 <rustyloaki@gmail.com> |
||
|
Hengfei Yang
|
e850ea2f29 | fix: read user session from db if the cache is missing (#8846) | ||
|
ByteBaker
|
bb725e94ee |
feat: add MCP OAuth 2.0 support and auth improvements
- Implement OAuth 2.0 Authorization Server Metadata endpoint (RFC 8414) - Add JWT token validation for MCP endpoints with dynamic clients - Add `WWW-Authenticate` header support for unauthorized responses - Add `AuthTokensExt::has_expired()` method for token validation - Refactor base64 decode to accept generic byte inputs - Update authentication flow to allow MCP users without DB records - Clippy delinting |
||
|
Hengfei Yang
|
5dee87a8d5 | feat: add update fields api (#8826) | ||
 Loakesh Indiranandktx-abhay
|
826cbbd276 |
feat: partial err warning if results exceeds query limit only for (#8802)
### **User description** - show warning in dashboards if the ZO_QUERY_DEFAULT_LIMIT is reached only dashboards - truncate the results exactly to the query limit on error - remove query result error if search_type is ui ___ ### **PR Type** Enhancement, Bug fix ___ ### **Description** - Add capped results warning on default limit - Hide cachable function errors for UI - Truncate results when default limit exceeded - Fix partition function naming in cluster paths ___ ### Diagram Walkthrough ```mermaid flowchart LR DF["DataFusion execution"] -- "row count" --> CL["check_query_default_limit_exceeded"] CL -- "set partial_err with capped msg" --> PR["Partial response propagation"] CL -- "truncate rows" --> HC["HTTP cluster response"] UI["UI search requests"] -- "filter function_error" --> HF["HTTP search handlers"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>9 files</summary><table> <tr> <td><strong>search.rs</strong><dd><code>Add capped results warning constant</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8802/files#diff-a36a6683e327641569d8098ee0d4f13e894e1b9d51230dc3569abe42e87ba310">+2/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Hide cachable function errors for UI</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8802/files#diff-96630a0dbb6dd7b2b0248fa325c7c7041ce31340c32826675a4a70772e9d29f6">+9/-1</a> </td> </tr> <tr> <td><strong>search_stream.rs</strong><dd><code>Filter cachable errors in streaming UI responses</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8802/files#diff-642de43ff6b19ae41087a9f1efc02d8b0b873809e0c92cbddfac1a30ef3f1184">+15/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Standardize partial error message usage and mark partial on fn errors</code></dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8802/files#diff-f07bfacc0501b9c234e64b16c1dd8eb0ae8fcbe231f90c81fed72bcc94946f74">+3/-2</a> </td> </tr> <tr> <td><strong>flight.rs</strong><dd><code>Add default-limit check and rename partition function</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8802/files#diff-dbffb1a03b65f6a9163b18c5bb1241ba2966735657be39bbe9f4d0e3260d8c1d">+12/-5</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Over-fetch default limit by 5 to detect cap</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8802/files#diff-3b1c9f0a84041235ca3765361e8eadfd7e9309a52319da882dee0540c826eb3d">+2/-1</a> </td> </tr> <tr> <td><strong>execution.rs</strong><dd><code>Simplify partial error handling logic</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8802/files#diff-75d432351865224f0b00d7cc819554c2ad5fa1f777e9c645d91a3fd5c1113bc0">+0/-2</a> </td> </tr> <tr> <td><strong>leader.rs</strong><dd><code>Inject default-limit partial warning via DF path</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8802/files#diff-21a32af318adcd10e62fb0d45944993f623fb3d2d6df4bb70d882dfc8f677fdc">+10/-3</a> </td> </tr> <tr> <td><strong>utils.rs</strong><dd><code>Add limit-exceeded checks and cachable-error helper</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8802/files#diff-bfdb4ff6a6e6bebf219b885f74dcbd3058ada7d2a7aa4c306b2e604aa1471474">+80/-1</a> </td> </tr> </table></details></td></tr><tr><td><strong>Bug fix</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>http.rs</strong><dd><code>Truncate results when default limit exceeded</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8802/files#diff-fe0a1bbec7ef6418120544b589a531e5144b28cd2650f99135ed8a70ac17ac03">+12/-2</a> </td> </tr> <tr> <td><strong>follower.rs</strong><dd><code>Use corrected partition function name</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8802/files#diff-59e0cc386582cf8baf201977475fd9a7e479fd8ed000bbaa6746a51ecb28632f">+2/-2</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Signed-off-by: loaki07 <rustyloaki@gmail.com> Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech> |
||
 Simha GandOpenObserve
|
78efd236ed |
feat: Introduce query optimization recommendation service to background jobs (#8791)
### **User description**
# PR Type
Enhancement
# Description
This PR introduces a configurable query recommendation service / back
ground job that will take org wise usage data from metadata and generate
recommendations for the same and reingests this data into _meta org's
query_recommendation stream.
___
### **PR Type**
Enhancement
___
### **Description**
- Add enterprise query recommendation background job
- Expose config for schedule and limits
- Map meta stream to enterprise type
- Wire job init under enterprise feature
___
### Diagram Walkthrough
```mermaid
flowchart LR
CFG["Config limits: recommendation params"]
JOB["Background job: QueryRecommendationService"]
CTX["OptimizerContext (engine impl)"]
ORGS["List org IDs"]
STREAMS["Get stream settings"]
SEARCH["Run usage search"]
INGEST["Ingest recommendations to _meta/query_recommendations"]
CFG -- interval/duration/top_k --> JOB
JOB -- uses --> CTX
CTX -- fetch --> ORGS
CTX -- fetch --> STREAMS
CTX -- analyze --> SEARCH
CTX -- publish --> INGEST
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>stream.rs</strong><dd><code>Add enterprise Stream
conversion for recommendations</code>
</dd></summary>
<hr>
src/common/meta/stream.rs
<ul><li>Gate enterprise import behind feature flag<br> <li> Implement
From<Stream> to enterprise meta Stream</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8791/files#diff-6f5ee19bbcc9cc7fb699fc1c0f43fc96d42698ae1168a2827b5c65f7948d5dbc">+14/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>mod.rs</strong><dd><code>Wire query recommendation job
into init</code>
</dd></summary>
<hr>
src/job/mod.rs
<ul><li>Add enterprise-only module include<br> <li> Spawn recommendation
job at init</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8791/files#diff-f2e4c29b8eef22189ec704a832633c4bd135a24444f054864c98064bd0963681">+5/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>query_optimization_recommendation.rs</strong><dd><code>Implement
enterprise query recommendation service job</code>
</dd></summary>
<hr>
src/job/query_optimization_recommendation.rs
<ul><li>Add enterprise-only background job implementation<br> <li>
Implement engine trait with org/streams/search<br> <li> Ingest
recommendations into meta stream<br> <li> Run service using configurable
schedule</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8791/files#diff-85c42193130f71a11dc1d54d1a80a5888f1311857d5899b48f8c92973c21ae21">+124/-0</a>
</td>
</tr>
</table></td></tr><tr><td><strong>Configuration
changes</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>config.rs</strong><dd><code>Configure query
recommendation scheduling and limits</code>
</dd></summary>
<hr>
src/config/src/config.rs
<ul><li>Add recommendation duration config<br> <li> Add analysis
interval config<br> <li> Add top-k limit config<br> <li> Set sensible
defaults via env vars</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8791/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+7/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
Co-authored-by: OpenObserve <o2@OpenObserves-MacBook-Pro.local>
|
||
|
ByteBaker
|
c868605736 | chore: add tests, fmt, clippy | ||
|
ByteBaker
|
da96d301ab |
refactor: extract reusable Headers extractor for actix-web handlers
This commit: - Creates a new extractors.rs module with a generic Headers<T> extractor - Implements FromRequest trait for type-safe header deserialization - Centralizes header parsing logic to reduce code duplication across HTTP handlers |
||
|
udev
|
c0e8efe084 |
fix: node health api (#8744)
### **PR Type**
Bug fix
___
### **Description**
- Initialize health map on node add
- Remove from health map on node delete
- Fail when online node missing health entry
- Fix heartbeat comment typo
___
### Diagram Walkthrough
```mermaid
flowchart LR
add["Node added (watch_node_list)"] -- "insert" --> healthMap["NODES_HEALTH_CHECK[uuid]=0"]
del["Node deleted (watch_node_list)"] -- "remove" --> healthMap
check["check_nodes_status"] -- "missing entry" --> err["Return error + warn log"]
check -- "present entry" --> incr["Increment failure counter"]
note["Comment fix"] -- "typo" --> hb["heartbeat"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Bug
fix</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>mod.rs</strong><dd><code>Health check map lifecycle and
validation fixes</code>
</dd></summary>
<hr>
src/common/infra/cluster/mod.rs
<ul><li>Initialize <code>NODES_HEALTH_CHECK</code> on node add.<br> <li>
Remove <code>NODES_HEALTH_CHECK</code> entry on node delete.<br> <li>
Validate presence in health map; error if missing.<br> <li> Fix comment
typo: heartbeat.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8744/files#diff-d15434a27e2b8406811f5a707b605203b603dad4a262c0642326be860bd2b568">+16/-4</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
|
Subhra264
|
41dc18866a |
feat: use cache for scheduled pipeline (#8707)
### **User description**
Currently we call the db when we process a scheduled pipeline and hence
increases db calls from the alert manager. This pr maintains a cache for
scheduled pipeline, so that alert manager does not make unnecessary db
calls.
___
### **PR Type**
Enhancement, Tests
___
### **Description**
- Add cache for scheduled pipelines
- Use cache in scheduler handler
- Extend pipeline cache/watch logic
- Add tests for cache behavior
___
### Diagram Walkthrough
```mermaid
flowchart LR
Scheduler["Scheduler handler"] -- "get/cache pipeline" --> PipelineDB["DB pipeline module"]
PipelineDB -- "reads/writes" --> ScheduledCache["Scheduled pipelines cache"]
RealtimeCache["Realtime exec cache"] -- "unchanged usage" --> Scheduler
Watcher["Pipelines watcher"] -- "update caches" --> ScheduledCache
Watcher -- "update caches" --> RealtimeCache
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>config.rs</strong><dd><code>Introduce global scheduled
pipelines cache</code>
</dd></summary>
<hr>
src/common/infra/config.rs
- Import `Pipeline` type.
- Add `SCHEDULED_PIPELINES` global cache.
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8707/files#diff-cda0d9179ed982fa5f0989c9b26f21cfa49dd1de28195a98135650ffcc7c602a">+3/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>handlers.rs</strong><dd><code>Scheduler uses cache for
pipeline lookups</code>
</dd></summary>
<hr>
src/service/alerts/scheduler/handlers.rs
<ul><li>Use scheduled pipeline cache before DB.<br> <li> Cache
DB-fetched pipelines on miss.<br> <li> Preserve existing error handling
path.<br> <li> Add debug logs for cache hits/misses.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8707/files#diff-5eca8995a88fcecb6d6b8efd3c7c4496f4a96a4b060fb4ed85593574f7cf49eb">+64/-40</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>pipeline.rs</strong><dd><code>Implement scheduled
pipelines cache and watcher integration</code></dd></summary>
<hr>
src/service/db/pipeline.rs
<ul><li>Add cache APIs: get/cache/remove/size/clear/stats.<br> <li>
Populate scheduled cache during full cache load.<br> <li> Extend watcher
to maintain scheduled cache.<br> <li> Improve logging and delete
handling for both caches.<br> <li> Add unit tests for scheduled cache
behavior.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8707/files#diff-0856951eadc646b4d4d3f0417317150cdbdc37b46c2f2a25a3c2bae15601dc50">+229/-57</a></td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
|
ByteBaker
|
33237447c4 |
chore: bump crate versions
- `async-walkdir`: v1.0.0 -> v2.1.0 - `maxminddb`: v0.25 -> v0.26 - `pprof`: v0.14 -> v0.15.0 - `prometheus`: v0.13 -> v0.14 - `promql-parser`: v0.4 -> v0.6 - `rcgen`: v0.13 -> v0.14.4 - `snafu`: v0.7.5 -> v0.8.9 - `tantivy`: v0.24 -> v0.25 - `vector-enrichment`: 063cabb -> 2003d61 - `vrl`: v0.22 -> v0.27.0 Removed `tokio-tungstenite` from dependencies. Also updated `vrl` to use git source from main branch as `vector-enrichment` needed syncing Changes to enrichment tables: - return early for empty conditions, avoiding computation - avoid vector collection before iterating again - added support for `FromDate` & `ToDate` condition types - simplified data processing using iterator chains - update `vector-enrichment` trait impl (add `wildcard` param) |
||
|
Huaijin
|
ade98b64d1 |
perf: improve load speed for enrichment table in EnrichmentExec (#8679)
|
||
|
Hengfei Yang
|
fafd4444ef | fix: search partition alignment (#8581) | ||
|
51d9782e5d |
feat: add separate telemetry url (#8564)
### **PR Type**
Enhancement
___
### **Description**
- Add keyevent telemetry support and API
- Introduce `ZO_KEYEVENT_TELEMETRY_URL` config
- Send both standard and keyevent telemetry
- Update handlers to use new API
___
### Diagram Walkthrough
```mermaid
flowchart LR
Cfg["Config: add ZO_KEYEVENT_TELEMETRY_URL"]
Tel["Telemetry: send_keyevent_track_event()"]
JWT["JWT auth: new user registered"]
Bill["Billing: new subscription started"]
Mkt["Marketing: new user attribution"]
Job["Job: no ingestion after creation"]
Cfg -- "provides URL" --> Tel
JWT -- "send standard + keyevent" --> Tel
Bill -- "send standard + keyevent" --> Tel
Mkt -- "send standard + keyevent" --> Tel
Job -- "send standard + keyevent" --> Tel
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>telemetry.rs</strong><dd><code>Add keyevent telemetry
send method</code>
</dd></summary>
<hr>
src/common/meta/telemetry.rs
<ul><li>Add <code>send_keyevent_track_event</code> method.<br> <li> Use
separate keyevent telemetry URL.<br> <li> Respect telemetry enabled flag
(non-cloud).<br> <li> Robust error/status logging for keyevent
sends.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8564/files#diff-3ff026b46d9431e0657828a3bde4ab118fec006b353c2abc80be4b74633c98b4">+60/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>jwt.rs</strong><dd><code>Emit keyevent telemetry on
user registration</code>
</dd></summary>
<hr>
src/handler/http/auth/jwt.rs
<ul><li>Instantiate telemetry once per flow.<br> <li> Send both standard
and keyevent events for registration.<br> <li> Duplicate payload reuse
for both sends.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8564/files#diff-232cab74f4cbf135af424744272e17732fbda8ac005d33c0acb4597e65479d02">+11/-1</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>billings.rs</strong><dd><code>Add keyevent telemetry
for subscription start</code>
</dd></summary>
<hr>
src/handler/http/request/cloud/billings.rs
<ul><li>Send standard and keyevent for new subscription.<br> <li> Reuse
event data across sends.<br> <li> Single telemetry instance per
request.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8564/files#diff-725936e3d6e955175e6100e8ca6f4437f334a2b554d6f11fa349ee9cbac70425">+11/-1</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>marketing.rs</strong><dd><code>Add keyevent telemetry
for attribution events</code>
</dd></summary>
<hr>
src/handler/http/request/cloud/marketing.rs
<ul><li>Send standard and keyevent for attribution.<br> <li> Share
segment event data between sends.<br> <li> Use single telemetry
instance.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8564/files#diff-424975d70871fd725536b17e4aa2131af8fd29e3c9d30ded551526e8ae71cfcc">+11/-1</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>cloud.rs</strong><dd><code>Add keyevent telemetry to
no-ingestion job</code>
</dd></summary>
<hr>
src/job/cloud.rs
<ul><li>Send standard and keyevent for no-ingestion report.<br> <li>
Reuse event data; single telemetry instance.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8564/files#diff-7fc16d008216c106d63f78f75c867fbefe7102433fcaa8893c314f0b7ea8d0e6">+11/-1</a>
</td>
</tr>
</table></td></tr><tr><td><strong>Configuration
changes</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>config.rs</strong><dd><code>Introduce keyevent
telemetry configuration</code>
</dd></summary>
<hr>
src/config/src/config.rs
<ul><li>Add <code>keyevent_telemetry_url</code> to
<code>Common</code>.<br> <li> Map to
<code>ZO_KEYEVENT_TELEMETRY_URL</code> env var.<br> <li> Default empty
string disables keyevent sending.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8564/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+2/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Prabhat Sharma <hi.prabhat@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
|
||
|
Hengfei Yang
|
24bc37a80e | feat: remove grpc call for get result cache also improve RW logic (#8246) | ||
|
|
6e75dc0e62 |
feat: add bulk enable/disable for alert and pipelines (#8438)
### **User description**
This adds two apis, one for bulk enable/disable of pipeline and one for
bulk enable/disable of alerts.
___
### **PR Type**
Enhancement
___
### **Description**
- Add bulk enable/disable for alerts
- Add bulk enable/disable for pipelines
- Introduce enterprise permission checks helper
- Update auth bypass for bulk endpoints
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["New models: Bulk request/response (alerts, pipelines)"]
B["Alerts: POST /v2/{org_id}/alerts/bulk/enable"]
C["Pipelines: POST /{org_id}/pipelines/bulk/enable"]
D["Enterprise permission check: check_resource_permissions"]
E["Router: register bulk endpoints"]
F["Auth: bypass for /bulk/enable paths"]
A -- "used by" --> B
A -- "used by" --> C
D -- "enforced in" --> B
D -- "enforced in" --> C
E -- "exposes" --> B
E -- "exposes" --> C
F -- "allows unauth extraction for" --> B
F -- "allows unauth extraction for" --> C
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>6
files</summary><table>
<tr>
<td><strong>auth.rs</strong><dd><code>Allow auth extractor bypass for
bulk enable endpoints</code> </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8438/files#diff-9bc2b1841504a435f66ad094f72ddbe7b7fa75e5e1557b8adbfe36f7f0592e1b">+1/-0</a>
</td>
</tr>
<tr>
<td><strong>responses.rs</strong><dd><code>Add alerts bulk enable
request/response models</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8438/files#diff-224027f90893122eea28e35778467acf1a4b5572ce612fc9bc9ec44ca8ebd9fb">+12/-0</a>
</td>
</tr>
<tr>
<td><strong>pipelines.rs</strong><dd><code>Add pipelines bulk enable
request/response models</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8438/files#diff-61bb28d7091130297c917a0530d96202fcb4b073b514410531f9fe71acfc814f">+12/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Implement alerts bulk enable
endpoint with permissions</code> </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8438/files#diff-83431801d27626ef27482e67b62661f900043544a45ddf958bd027cf2fa3bd8d">+91/-2</a>
</td>
</tr>
<tr>
<td><strong>pipeline.rs</strong><dd><code>Implement pipelines bulk
enable endpoint with permissions</code></dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8438/files#diff-c69f4f94eee97ce226caddc497b2adf89e26869f2d72b0d0206dc31ab2714daf">+89/-1</a>
</td>
</tr>
<tr>
<td><strong>utils.rs</strong><dd><code>Add generic enterprise resource
permission checker</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8438/files#diff-96bb9f4a8f547c821f26c957e8505d0437095b08d2b36687531f71565d287bd6">+38/-0</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Configuration
changes</strong></td><td><details><summary>1 files</summary><table>
<tr>
<td><strong>mod.rs</strong><dd><code>Register bulk enable endpoints in
service routes</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8438/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+2/-0</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>1
files</summary><table>
<tr>
<td><strong>openapi.rs</strong><dd><code>Expose bulk enable endpoints in
OpenAPI</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8438/files#diff-edb8ce232c0cd08366879f642788a75d99f6bab64b4c0e77391281fdb9241e90">+2/-0</a>
</td>
</tr>
</table></details></td></tr></tr></tbody></table>
</details>
___
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
|
||
|
Hengfei Yang
|
215cf796cd | feat: remove etcd (#8466) | ||
|
Hengfei Yang
|
97d09eff1e | fix: support prefix star match_all (#8451) | ||
|
Hengfei Yang
|
237c757c7c | fix: improve nats queue with limit policy (#8436) | ||
|
Ashish Kolhe
|
34a7341e87 |
chore: node list & user session refresh apis (#8198) (#8200)
### **PR Type** Enhancement ___ ### **Description** - Add CLI commands to refresh caches - Expose HTTP endpoints for refresh actions - Cache node list building in cluster module - Replace retention flag with mode enum ___ ### Diagram Walkthrough ```mermaid flowchart LR CLI["CLI subcommands"] -- triggers --> HTTP["HTTP endpoints"] HTTP -- calls --> Cluster["cluster::cache_node_list"] HTTP -- calls --> Sessions["db::session::cache()"] Config["Config: file_list_deleted_mode"] -- controls --> Retention["Compactor retention logic"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>8 files</summary><table> <tr> <td><strong>cli.rs</strong><dd><code>New CLI subcommands for cache refresh</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8200/files#diff-cb1eb3e74723d891353ccf1e2e5e0d246d2e78a5e69dbe4c73006ea8412d65e0">+6/-0</a> </td> </tr> <tr> <td><strong>http.rs</strong><dd><code>HTTP client helpers for refresh endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8200/files#diff-790b7780a51e8eecd4754b623af8d15ad87baaaf29455ba646e2e410f83f5498">+22/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Cache node list and update consistent hash</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8200/files#diff-d15434a27e2b8406811f5a707b605203b603dad4a262c0642326be860bd2b568">+33/-0</a> </td> </tr> <tr> <td><strong>config.rs</strong><dd><code>Replace retention flag with mode string config</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8200/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+3/-2</a> </td> </tr> <tr> <td><strong>stream.rs</strong><dd><code>Add FileListBookKeepMode enum and conversions</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8200/files#diff-2438d04c3fe97bc5452e5b74a303163b8723d93455cda32c95f0525435ad46bd">+29/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add refresh nodes and user sessions endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8200/files#diff-ca09c9c9b020049bdbc3382c466bfcca80daac92e98760d9a9a76172abf34049">+20/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Wire refresh endpoints into basic routes</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8200/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+3/-1</a> </td> </tr> <tr> <td><strong>retention.rs</strong><dd><code>Use retention mode for history/deleted behavior</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8200/files#diff-6abd048f5ef4e7163cffbb51cb4bee10c07fb88b871469d5b3724f7cd04743dc">+16/-6</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ |
||
|
Subhra264
|
6f40d2ffc0 | feat: use nats queue for cluster coordination (#8413) | ||
|
Hengfei Yang
|
b72e85e0fe | feat: support nats ttl marker (#8377) | ||
|
0730d5f7a4 |
fix: landing page / Home page revamp (#6729)
### **User description**
1. This PR implements the latest UI of landing page / Home page with 2
charts for showing alerts and pipelines
2.
___
### **PR Type**
Enhancement
___
### **Description**
- Redesigned Home page layout and tiles
- Added alerts and pipelines charts
- Implemented theme-aware icons and styles
- Added loading state for summary fetch
___
### Diagram Walkthrough
```mermaid
flowchart LR
HomeView["HomeView.vue"]
SummaryAPI["Summary fetch & parsing"]
Charts["CustomChartRenderer charts"]
Tiles["Responsive metric tiles"]
Theming["Dark/Light computed icons & styles"]
Loading["Loading spinner state"]
HomeView -- "fetches org summary" --> SummaryAPI
SummaryAPI -- "sets metrics & statuses" --> Tiles
SummaryAPI -- "feeds alert/pipeline data" --> Charts
HomeView -- "applies theme classes" --> Theming
HomeView -- "shows while fetching" --> Loading
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>HomeView.vue</strong><dd><code>Home page redesign with
charts and theme-aware UI</code>
</dd></summary>
<hr>
web/src/views/HomeView.vue
<ul><li>Rebuilt template with responsive sections and tiles.<br> <li>
Added alerts pie and pipelines bar charts via CustomChartRenderer.<br>
<li> Introduced loading state, improved summary parsing, and theme-aware
<br>icons.<br> <li> Added extensive SCSS for dark/light theming and
layout.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/6729/files#diff-9af441af191d3254ef5f9db603dca75486de7638cda41a3c050e7b3c33d68280">+568/-229</a></td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: ByteBaker <42913098+ByteBaker@users.noreply.github.com>
|
||
|
Hengfei Yang
|
55a0c62d3f | fix: root user password reset (#8348) | ||
|
sktrpathi
|
f8cc585408 |
feat: store ratelimit info in vendor extension (#8259)
### **PR Type** Enhancement, Documentation ___ ### **Description** - Replace ratelimit comments with OpenAPI extensions - Add `x-o2-ratelimit` metadata to endpoints - Import `serde_json::json` where needed - Clean up obsolete inline ratelimit markers |
||
|
8af3495611 |
fix: cloud sync (#8222)
### **User description**
This is for syncing the changes done in cloud to main branch. No changes
for non-cloud code.
___
### **PR Type**
Bug fix, Enhancement
___
### **Description**
- Add invite listing and decline APIs
- Bypass auth for user-scoped invite routes
- Improve trial-expiry handling and logging
- Add org cache and invalidation hooks
___
### Diagram Walkthrough
```mermaid
flowchart LR
FE["Login & Invitations UI"] -- "list/accept/decline" --> API["/api/invites*, /{org}/member_subscription"]
API -- "auth bypass for user-scoped invites" --> Auth["AuthExtractor/TokenValidator"]
Auth -- "pending invites flag" --> LoginRedir["/config/redirect builds URL params"]
OrgSvc["Organization Service"] -- "generate/list/decline/delete" --> InvitesDB["Org Invites"]
OrgSvc -- "invalidate cache" --> OrgCache["Org table in-memory cache"]
Ingest["Ingestion services"] -- "trial expired -> 429" --> Clients["Ingestion Clients"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>25
files</summary><table>
<tr>
<td><strong>user.rs</strong><dd><code>Expand invite payload with org and
inviter</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-e13b3a981fd70182b9738a6b0ecfdee915c5b36757703a87741e1eddf3f6e786">+2/-0</a>
</td>
</tr>
<tr>
<td><strong>auth.rs</strong><dd><code>Auth bypass and key handling for
invite routes</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-9bc2b1841504a435f66ad094f72ddbe7b7fa75e5e1557b8adbfe36f7f0592e1b">+36/-0</a>
</td>
</tr>
<tr>
<td><strong>jwt.rs</strong><dd><code>Return new user and pending invite
flags; defer org creation</code></dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-232cab74f4cbf135af424744272e17732fbda8ac005d33c0acb4597e65479d02">+22/-4</a>
</td>
</tr>
<tr>
<td><strong>token.rs</strong><dd><code>Allow invite/member routes
without org user; header injection</code></dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-3339b4a964c121766ef87fcec0070f27a2b7085cf81281b53154e53d33cb28cf">+34/-0</a>
</td>
</tr>
<tr>
<td><strong>ingest.rs</strong><dd><code>Suppress logging for trial
expiry errors</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-24f250254ecfdbe004af9dc0ea23f8a2c0105ae0898690ed3e565089e5d8eef9">+30/-9</a>
</td>
</tr>
<tr>
<td><strong>org.rs</strong><dd><code>Pending-only invites; delete invite
API; trial extend cache poke</code></dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-b47c0c1a8a78524196176430268eecdbfaf6f46d6b95873d3cbe70cfd67182b0">+40/-4</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Include pending_invites flag in
login redirect</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-ca09c9c9b020049bdbc3382c466bfcca80daac92e98760d9a9a76172abf34049">+4/-4</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>List pending invites and decline
invite endpoint</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-56897e42f9436a640bc70fddc25686485ba8ae1223635b32d252a0333309f748">+36/-1</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Wire new invite endpoints into
router</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+2/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Add TrialPeriodExpired error
variant</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-90c725d0e77a6e7d4bcc130cf5783848a5cf0efdb304b351cd63cacf8775fc36">+2/-0</a>
</td>
</tr>
<tr>
<td><strong>organizations.rs</strong><dd><code>Add in-memory org cache
with invalidation</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-4d87f9fe379a5d161a3e7c3c77f7f320be738e6a70c785aa5ce93cb0830c0147">+77/-7</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Start cloud billing watcher on
cloud builds</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-f2e4c29b8eef22189ec704a832633c4bd135a24444f054864c98064bd0963681">+3/-0</a>
</td>
</tr>
<tr>
<td><strong>organization.rs</strong><dd><code>Invalidate org cache on
watch and mutations</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-ce65cea926a09721a8fea14d07fe188bc2b887218e95d74a8bffefb54001da50">+4/-0</a>
</td>
</tr>
<tr>
<td><strong>user.rs</strong><dd><code>Delete invites for user
helper</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-6ee6b0fc9ed03224d4ddf0b7b631c6f81ff7d1d579bdb98d2c5d4453852eb5d6">+4/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Return trial expiry as dedicated
error</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-7c7b5bca85f0ced652965cae66540c96f56dfcca4f4c16b77bf7b45aac203d1b">+1/-3</a>
</td>
</tr>
<tr>
<td><strong>loki.rs</strong><dd><code>Silence trial expiry logs for Loki
ingestion</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-a970c8c4785d4e4d878218615b17eec0cf4b4976121766b002c2b2f76371bfe6">+4/-1</a>
</td>
</tr>
<tr>
<td><strong>json.rs</strong><dd><code>Map trial expiry to HTTP 429 in
JSON ingest</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-506987e61095a54e47f7f208dad89f9f1bce595fa6bf1ba8a7d746f2b2ff6c4a">+15/-6</a>
</td>
</tr>
<tr>
<td><strong>otlp.rs</strong><dd><code>Return 429 for trial expiry;
reduce noise</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-a24391bef75a4678c50724e8b5f305ec0ee8c2c74c68be9d677765de68b5d8dd">+17/-7</a>
</td>
</tr>
<tr>
<td><strong>organization.rs</strong><dd><code>Invite generation
validation; delete/decline invite flows</code></dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-6b3c3c2b027c1158c0d41302cc69dfcd20ab68a6a4fa365cda403f838940bbb4">+48/-1</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Handle trial expiry with 429;
logging tweaks</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-66f3b4f5e7927092fd6e57738687a1091b6f316638f7c58f8d7be9addb72bc30">+34/-14</a>
</td>
</tr>
<tr>
<td><strong>users.rs</strong><dd><code>Enrich invite list; optional
pending filter; cleanup on removal</code></dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-98da2588a28d62bb2e301d989836125f89f89b76272cd7f75e08ec4deeea5f38">+50/-6</a>
</td>
</tr>
<tr>
<td><strong>organizations.ts</strong><dd><code>Add decline subscription
API client</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-2c3bad4c7a9fcedd7a653599381149b82f9c350b177de9d72431013807d4c36f">+10/-7</a>
</td>
</tr>
<tr>
<td><strong>users.ts</strong><dd><code>Add pending/accept/reject invite
APIs</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-a334d8d570d9bdcaa3d4c8c2a993eff009a1ec8eb499c8a3d1a292d9224159d0">+12/-4</a>
</td>
</tr>
<tr>
<td><strong>InvitationList.vue</strong><dd><code>New UI to accept or
reject invites</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-ee97546ff1aba36cc0b24cb20782502c7c88279850cc3d540ea5685209f10702">+409/-0</a>
</td>
</tr>
<tr>
<td><strong>Login.vue</strong><dd><code>First-login invite handling and
logo tweaks</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-a135c379ef6ac8f92199ea5f7ff19a227971a9b0a87ccccf87c0268740522c2c">+178/-58</a></td>
</tr>
</table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>1
files</summary><table>
<tr>
<td><strong>en.json</strong><dd><code>Add invitation-related
translations</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-892aa3d75ab6e129ef9116aa21bec1b9a15ba69ce5366d2a19128078c690d824">+18/-0</a>
</td>
</tr>
</table></details></td></tr></tr></tbody></table>
</details>
___
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Subhra264 <chakrabortysubhradeep556@gmail.com>
Co-authored-by: udev <uddhav@openobserve.ai>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: udev <uddhavdave1@gmail.com>
|