mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-06 08:09:24 +00:00
fc7fdcdcec2ce76f3711080b32ca65336b71dd71
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Abhay PadamaniandClaude Opus 4.8
|
5ec9b89ff8 |
build: FE code format check for improved maintainability (#13351)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
  ByteBakerandHengfei Yang
|
b948361769 |
chore: update copyright year to 2026 (#11166)
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com> |
||
  
|
78c96c3721 |
fix: support bulk deletes for various resources (#9153)
### **User description** fixes https://github.com/openobserve/openobserve/issues/9116 This adds support for bulk delete operation from API and UI side for various resources. ___ ### **PR Type** Enhancement ___ ### **Description** - Add bulk delete endpoints for multiple resources - Enforce per-item permission checks before delete - Aggregate successful/failed items in responses - Register bulk-delete routes in HTTP router - Extend UI: add bulk-delete buttons, dialogs, service calls ___ ### Diagram Walkthrough ```mermaid flowchart LR UI["\"UI: Bulk Delete Trigger\""] API["\"HTTP DELETE /{resource}/bulk Handler\""] AUTH["\"Permission Checks per Item\""] DEL["\"Perform Delete Operations Loop\""] RESP["\"Aggregate Response: successes/ failures\""] UI_FE["\"FE: Show Notifications\""] UI -- "calls" --> API API -- "checks" --> AUTH AUTH -- "allowed" --> DEL DEL -- "deletes items" --> RESP RESP -- "returns" --> UI_FE ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>14 files</summary><table> <tr> <td><strong>fga.rs</strong><dd><code>Add bulk delete roles and groups endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-e7c99f37f3abbb5e0507d2760bbf67df010d9218704b38dce84c5477acecd22e">+218/-6</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Implement delete_alert_bulk handler</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-83431801d27626ef27482e67b62661f900043544a45ddf958bd027cf2fa3bd8d">+87/-6</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add delete_function_bulk and error types</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-a5e1b3644cef9feb689dfe66382e44e5434ecadbff3469127346a333d4733102">+97/-3</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add delete_dashboard_bulk endpoint</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-1b3c257e1d174c826718499faa4a1d4e93465dd56cb2b7b273b690ff7e3223fd">+78/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Implement re_patterns delete_bulk logic</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-27d37a8738de89c81b540daa978e684caf2cf28803e1208d90cb4d42528ed28b">+116/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add users bulk delete handler</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-56897e42f9436a640bc70fddc25686485ba8ae1223635b32d252a0333309f748">+80/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add service_accounts delete_bulk API</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-095d70f65b6e69bf7de0180fcbcddcd7c89c4cf00095d7b06f039a07e73916bb">+86/-1</a> </td> </tr> <tr> <td><strong>action.rs</strong><dd><code>Implement delete_action_bulk endpoint</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-616df3014c553f763ea57e925f1bff7bb47f9467c632940d93a45b7b1e4c3a07">+90/-1</a> </td> </tr> <tr> <td><strong>destinations.rs</strong><dd><code>Add delete_destination_bulk for alerts</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-76b00293c621036de31cf499c1c276efb9704c2bf8cf1b728f49cfdd43f9fd1b">+75/-4</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add delete_bulk for cipher_keys</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-043222e433d1e5825a4b7bfc2c7feedba417f2630f0082fa67aff64976736396">+76/-1</a> </td> </tr> <tr> <td><strong>templates.rs</strong><dd><code>Implement delete_template_bulk handler</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-b08e274a64cc935b0cec402a04271a872b1586027a77dc0ccb84c597c3960f61">+73/-4</a> </td> </tr> <tr> <td><strong>pipeline.rs</strong><dd><code>Add delete_pipeline_bulk API handler</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-c69f4f94eee97ce226caddc497b2adf89e26869f2d72b0d0206dc31ab2714daf">+70/-2</a> </td> </tr> <tr> <td><strong>reports.rs</strong><dd><code>Add delete_report_bulk endpoint</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-00435e0d9301342b51cb57ae313f1413e7ed973642e0badf3f3959125d0677d1">+75/-1</a> </td> </tr> <tr> <td><strong>ActionScripts.vue</strong><dd><code>Integrate bulk delete UI in action scripts</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-770f71fcb92a13b3b8b1041567b885fac39903c814825c878595cafa624ebbd8">+284/-5</a> </td> </tr> </table></details></td></tr><tr><td><strong>Configuration changes</strong></td><td><details><summary>3 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Register bulk delete services in router</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+14/-0</a> </td> </tr> <tr> <td><strong>iam.ts</strong><dd><code>Add bulkDeleteGroups and bulkDeleteRoles calls</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-3712c1bb0eecb664b6691d724c3673b8c1befa41efbd8c5de626c5849f8ace98">+10/-0</a> </td> </tr> <tr> <td><strong>dashboards.ts</strong><dd><code>Add bulkDelete service for dashboards</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-913b3f54367036b1b3960aa3eddd7e17b509b1d6011c5ab68ae0e935653179c5">+7/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Error handling</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>auth.rs</strong><dd><code>Allow bulk-delete paths in auth extractor</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-9bc2b1841504a435f66ad094f72ddbe7b7fa75e5e1557b8adbfe36f7f0592e1b">+17/-1</a> </td> </tr> </table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>user.rs</strong><dd><code>Define bulk delete request/response structs</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-e13b3a981fd70182b9738a6b0ecfdee915c5b36757703a87741e1eddf3f6e786">+36/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add PipelineBulkDelete meta types</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-c4bd542e4e07ed7706b0643086f5a7d4edbb3647c7ceedbe4bea516c5274d8bd">+12/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>40 files</summary><table> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-6f9c2e6dbc40f8d40e3d3ecbee23240813a3d4856d49bd5d979f5b98551fdc3d">+12/-0</a> </td> </tr> <tr> <td><strong>service_account.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-878ad520e1c84500cc21ff6fc4088c8da64ff2febd79e60c0d4cc8c695c49798">+11/-0</a> </td> </tr> <tr> <td><strong>action.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-fc87c75a7df9539d7e7ee6184dfab02d4a8932323e81f95d57c05363aa44752d">+15/-0</a> </td> </tr> <tr> <td><strong>reports.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-d2117a3d67d4ec47b0017861845c92143baf917e10e6746df5dd97f925f81eeb">+12/-0</a> </td> </tr> <tr> <td><strong>function.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-27c9f3d93723397a0e234bc45c9f543b43eff5fd518d76e735f78d95a58f3f1c">+12/-0</a> </td> </tr> <tr> <td><strong>requests.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-412f11e06c0e901e2292aab80536cb36edebcc4796dec52923389c3aaa32b25a">+6/-0</a> </td> </tr> <tr> <td><strong>responses.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-224027f90893122eea28e35778467acf1a4b5572ce612fc9bc9ec44ca8ebd9fb">+9/-0</a> </td> </tr> <tr> <td><strong>dashboards.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-250eeebeec9d4ac0ac271466ed83cd33fea3f121080491b340dc95d9af287259">+12/-0</a> </td> </tr> <tr> <td><strong>destinations.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-ecffe114c8200bab742dd2ee4f7f41128aa59438a879634f070edd575a1f917c">+24/-0</a> </td> </tr> <tr> <td><strong>search_stream.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-642de43ff6b19ae41087a9f1efc02d8b0b873809e0c92cbddfac1a30ef3f1184">+1/-0</a> </td> </tr> <tr> <td><strong>utils.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-96bb9f4a8f547c821f26c957e8505d0437095b08d2b36687531f71565d287bd6">+2/-1</a> </td> </tr> <tr> <td><strong>functions.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-ed5611a9aa6b5c033e70d7631ba72fafad3cc842672077a7fcf965d11a4fef23">+22/-31</a> </td> </tr> <tr> <td><strong>AppTable.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-efcf66a9e913b5d3a7c2eea800b7f4e54d9e5268307131482fd40f32e2074d8e">+72/-16</a> </td> </tr> <tr> <td><strong>AlertList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-be3e41b823e163b226ffc270d95cc4981e59c600851e4e0d10ee5e4ee994ac8e">+123/-2</a> </td> </tr> <tr> <td><strong>AlertsDestinationList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-e9d05ba6c071c0e2f5d9ef7acdef757f79910abd6897b29e27b069a10d4ba8de">+142/-5</a> </td> </tr> <tr> <td><strong>PipelinesDestinationList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-088c66f5a34e73a5350a794a772e5d905262277337c937dd36f4295072203ebf">+163/-28</a></td> </tr> <tr> <td><strong>TemplateList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-7df66364d7d69a26cfa25cbc555d0e090c2f73e51c8386d9b0435bae4412564c">+116/-27</a></td> </tr> <tr> <td><strong>EnrichmentTableList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-6a29e253f59405d3c0f756dba36afd7483f10ad86927cbbc3348ede6372c3077">+124/-13</a></td> </tr> <tr> <td><strong>FunctionList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-6ff9af50a1597504b7f0dca91e61cf30c800b4a17b4be2ed2c3a4f7a7b2d7068">+143/-6</a> </td> </tr> <tr> <td><strong>AppGroups.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-402fb2a5338e60fdc3da489f4d50b5cf52f06b4e884a6fe7b82441c9fa5ea22c">+86/-1</a> </td> </tr> <tr> <td><strong>AppRoles.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-191634ef09436199f91f442968110a01f54ae46b5ceee67476f52613ad796db4">+86/-1</a> </td> </tr> <tr> <td><strong>EditRole.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-263b6bc7d29fdf8358c8437cd637f1350aa3733cc2706eae8ff00028686cc0f5">+20/-0</a> </td> </tr> <tr> <td><strong>ServiceAccountsList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-3afe835bcfbec3fda2e9550850c29581092ee0717e7dd8de8888e774a0048d24">+114/-1</a> </td> </tr> <tr> <td><strong>User.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-e973c2a665f9b27b94673ba64cc5b23650300836a727a2e07673c8e26305aee5">+133/-3</a> </td> </tr> <tr> <td><strong>PipelinesList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-85e3ef322f3eea0fc6d74d89431ba338a1153cb82ee06bef0f7c5f439c04267d">+108/-2</a> </td> </tr> <tr> <td><strong>ReportList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-f7d700f7e67fbaba3c58ab5d2c614925db743101828214445ce9dae35dc29067">+157/-5</a> </td> </tr> <tr> <td><strong>CipherKeys.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-950b715cff2c964d8542512692030c370733b57d698abf40e1dcb3cce2cfaf7a">+104/-13</a></td> </tr> <tr> <td><strong>RegexPatternList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-524927acd9f5bb07b71bfac489e747470108574e21a6dde63ea7c1f219cea853">+109/-14</a></td> </tr> <tr> <td><strong>action_scripts.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-6c1a011ac23290134801b22feb2b50c2b19cabf2a009f37a62b4dd9ca6e415e3">+3/-0</a> </td> </tr> <tr> <td><strong>alert_destination.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-377d28822e9ad4e9e80327ecc2d09bf1981c7997c5f1c1de120c6179eef19c6e">+3/-0</a> </td> </tr> <tr> <td><strong>alert_templates.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-fd8c4260abc3df601e4331d36b0c211eca568ee70139ad90c87247bc552fe4e2">+3/-0</a> </td> </tr> <tr> <td><strong>alerts.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-7a62782cceacac92b787e058219b3c53ea7406d4be5780def899943a01f500c1">+7/-0</a> </td> </tr> <tr> <td><strong>cipher_keys.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-0a0ef3a63c29b2a74b7f904c54c3fb32bee421608474aa32334b7c55133ab1ef">+3/-0</a> </td> </tr> <tr> <td><strong>jstransform.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-301add2528e0f32323ca4c503bd7d460ab15ca6b9a9537192a02c5e52fd82926">+3/-0</a> </td> </tr> <tr> <td><strong>pipelines.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-e8626670bb4e46e073f9858ade6b3dc77eaadf39d6721d9c0006eccc1b430612">+8/-0</a> </td> </tr> <tr> <td><strong>regex_pattern.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-2a0b0828141d39ab0385e0941314ead1cdd2ec3ec7b3baf5fd0c42f8f010728a">+3/-0</a> </td> </tr> <tr> <td><strong>reports.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-32d3299b2b21dc57c39d01080baba03765d853bd05c82d6158834b54713ae825">+3/-0</a> </td> </tr> <tr> <td><strong>service_accounts.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-4038191b0dc0cad335f3d8f7f703505df905ad843fe950096aefb8abbd139a1e">+4/-1</a> </td> </tr> <tr> <td><strong>users.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-a334d8d570d9bdcaa3d4c8c2a993eff009a1ec8eb499c8a3d1a292d9224159d0">+3/-0</a> </td> </tr> <tr> <td><strong>Dashboards.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/9153/files#diff-8da3187db8ce3d795d0f196ede6b2290221ffa3e27b15dcc2b645e44dd196213">+122/-6</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> |
||
 Harsh Mahajanandsai nikhil kethe
|
b267159579 |
feat: adding built-in regex patterns (#8827)
### **PR Type**
Enhancement, Documentation
___
### **Description**
- Add built-in regex patterns fetch API
- Introduce GitHub data service with caching
- Handle duplicate pattern names gracefully
- UI to browse and import built-ins
___
### Diagram Walkthrough
```mermaid
flowchart LR
cfg["Config: regex_patterns_source_url"] -- used by --> ghsvc["GitHubDataService"]
ghsvc -- fetch_json/cache --> adapter["PyWhatAdapter.fetch_built_in_patterns"]
adapter -- returns --> api["GET /api/{org}/re_patterns/built-in"]
api -- consumed by --> websvc["web regexPatterns.getBuiltInPatterns()"]
websvc -- powers --> ui["BuiltInPatternsTab.vue"]
ui -- import selected --> importFlow["ImportRegexPattern.vue"]
importFlow -- create --> dbsvc["db::re_pattern::add (duplicate-safe)"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Configuration
changes</strong></td><td><details><summary>3 files</summary><table>
<tr>
<td><strong>config.rs</strong><dd><code>Add source URL for built-in
regex patterns</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8827/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+6/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Register built-in patterns and
reorder routes</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8827/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+2/-2</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Export github service
module</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8827/files#diff-1b15316505c31acccb26ba66d920fdb0894567d08d2bf1e3457716d128a67b83">+1/-0</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Enhancement</strong></td><td><details><summary>10
files</summary><table>
<tr>
<td><strong>mod.rs</strong><dd><code>New endpoint to fetch built-in
patterns and improved save response</code></dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8827/files#diff-27d37a8738de89c81b540daa978e684caf2cf28803e1208d90cb4d42528ed28b">+103/-4</a>
</td>
</tr>
<tr>
<td><strong>re_pattern.rs</strong><dd><code>Add duplicate-safe creation
with auto-suffix names</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8827/files#diff-0cb2bf6d550d07b66db7e325c6d44c6a598f2ac8e315bf9e168f4a1d9d6e745d">+47/-17</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Add adapters module export for
pattern formats</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8827/files#diff-3a24aa67600735383b8b94a435974dc9b2677b6bb3f5fb502fb7d617af8bdec0">+20/-0</a>
</td>
</tr>
<tr>
<td><strong>pywhat.rs</strong><dd><code>Implement generic pattern
adapter and filters</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8827/files#diff-9ecd12df46c16fca3984bc46122a26568db208839f904e0b216185df43e13e08">+264/-0</a>
</td>
</tr>
<tr>
<td><strong>cache.rs</strong><dd><code>Introduce in-memory cache with
eviction and stats</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8827/files#diff-09bcde8f6abf0cbf88628e1e6e7290e9fde7655c7903569902bf28742e958326">+130/-0</a>
</td>
</tr>
<tr>
<td><strong>client.rs</strong><dd><code>Add GitHub data client with
retry and caching</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8827/files#diff-20a335d645493f38003faf7303124a6e85c3c2dfd765e148811aebee1ebfd890">+246/-0</a>
</td>
</tr>
<tr>
<td><strong>types.rs</strong><dd><code>Define service config, errors,
and cache types</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8827/files#diff-35becfa9ecfcd7711485f645f09a3fe2429231d0c351f0374f20f121225fbaae">+119/-0</a>
</td>
</tr>
<tr>
<td><strong>regex_pattern.ts</strong><dd><code>Add client method to
fetch built-in patterns</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8827/files#diff-2a0b0828141d39ab0385e0941314ead1cdd2ec3ec7b3baf5fd0c42f8f010728a">+18/-0</a>
</td>
</tr>
<tr>
<td><strong>BuiltInPatternsTab.vue</strong><dd><code>New UI to search,
filter, preview, import patterns</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8827/files#diff-0747e124429b322a69f6ae0117ec635044059bd48fb03e90b2328aa1f79b8b28">+451/-0</a>
</td>
</tr>
<tr>
<td><strong>ImportRegexPattern.vue</strong><dd><code>Integrate built-in
tab and import flow updates</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8827/files#diff-e8e1bcb5a02311504d7d571090eab27a270aaa71cae185ad2703023e2e70a31e">+94/-36</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>3
files</summary><table>
<tr>
<td><strong>mod.rs</strong><dd><code>Expose GitHub service modules and
types</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8827/files#diff-90a3518cd7c8b2a8b2a2093535b64a3dc4e6ffead41e5dca7901521e0c0ba910">+39/-0</a>
</td>
</tr>
<tr>
<td><strong>.env.example</strong><dd><code>Document regex patterns
source URL env var</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8827/files#diff-a3046da0d15a27e89f2afe639b25748a7ad4d9290af3e7b1b6c1a5533c8f0a8c">+6/-1</a>
</td>
</tr>
<tr>
<td><strong>en.json</strong><dd><code>Add i18n strings for built-in
patterns UI</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8827/files#diff-892aa3d75ab6e129ef9116aa21bec1b9a15ba69ce5366d2a19128078c690d824">+17/-1</a>
</td>
</tr>
</table></details></td></tr></tr></tbody></table>
</details>
___
---------
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
|
||
  udevandShrinath Rao
|
0c1268a230 |
feat: add hash replace feature in SDR (#8803)
Adds a config variable to control the length of hash UI changes to accomodate Hash pattern policy <img width="720" height="410" alt="image" src="https://github.com/user-attachments/assets/b99e28ec-2b7d-431e-9606-7b855df62f8d" /> <img width="2037" height="1112" alt="image (3)" src="https://github.com/user-attachments/assets/67432a01-c598-4b0a-830b-137a4cd04bf9" /> --------- Co-authored-by: Shrinath Rao <shnath@openobserve.ai> |
||
|
41f330af67 |
feat: re_patterns at ingestion (#7323)
### **User description** This PR adds support for adding and applying regexp patterns at ingestion time to redact or drop the values from fields if matched. This is an ent-only feature, and this PR contains the required table migrations, along with nats code for super cluster. This also has some additions for musl build for ent repo. There are basic crud APIs for the pattersn as well as an test api which can be used to test a specific pattern on given input. ___ ### **PR Type** Enhancement ___ ### **Description** - Add HTTP CRUD and test endpoints for regex patterns - Implement DB models, migrations for patterns associations - Integrate pattern manager into ingestion and streams - Spawn watchers and super-cluster queue for patterns ___ ### **Changes diagram** ```mermaid flowchart LR HTTP["CRUD & test HTTP handlers"] DBService["Service DB layer"] DB["re_patterns & associations tables"] Cluster["Cluster Coordinator"] PatternMgr["Pattern Manager"] Ingest["Ingestion pipeline"] Stream["Stream settings update"] HTTP -- "invoke add/update/delete/test" --> DBService DBService -- "persist & notify" --> DB DBService -- "trigger" --> Cluster Cluster -- "watch events" --> PatternMgr Ingest -- "redact at ingestion" --> PatternMgr Stream -- "update associations" --> PatternMgr ``` ___ ### **Changes walkthrough** 📝 <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>14 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Add HTTP handlers for re_patterns endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-27d37a8738de89c81b540daa978e684caf2cf28803e1208d90cb4d42528ed28b">+459/-0</a> </td> </tr> <tr> <td><strong>re_pattern.rs</strong><dd><code>Implement DB service for re_patterns</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-0cb2bf6d550d07b66db7e325c6d44c6a598f2ac8e315bf9e168f4a1d9d6e745d">+384/-0</a> </td> </tr> <tr> <td><strong>re_pattern_stream_map.rs</strong><dd><code>Add pattern association table ORM</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-13798240832b9110426267b3e0ebb862bd6c2ac03a3462721f5e85b5bfa88f51">+257/-0</a> </td> </tr> <tr> <td><strong>re_pattern.rs</strong><dd><code>Add re_patterns table ORM definitions</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-f113ed92667f5e352ac7a0c1c8b5e48ea858992b92be5817c9b38f65843ef4a2">+185/-0</a> </td> </tr> <tr> <td><strong>stream.rs</strong><dd><code>Integrate pattern associations in streams</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-41452a7fb29b4634437174fd6aecf6a9050dbfdef41b9077a21d59af3d79e9ed">+62/-2</a> </td> </tr> <tr> <td><strong>re_pattern.rs</strong><dd><code>Handle super-cluster queue for patterns</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-9e64341dc2d7d50cc5acffd0340b983f37b46a9aa34c86fb5a25978f9eb761dc">+166/-0</a> </td> </tr> <tr> <td><strong>ingest.rs</strong><dd><code>Redact data via pattern manager at ingestion</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-e62118e45be183266d39ffa08491f255017954235ff0f3d6937683e314da406f">+23/-0</a> </td> </tr> <tr> <td><strong>stream.rs</strong><dd><code>Add pattern associations to Stream metadata</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-6f5ee19bbcc9cc7fb699fc1c0f43fc96d42698ae1168a2827b5c65f7948d5dbc">+4/-1</a> </td> </tr> <tr> <td><strong>stream.rs</strong><dd><code>Define PatternAssociation and settings wrapper</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-2438d04c3fe97bc5452e5b74a303163b8723d93455cda32c95f0525435ad46bd">+14/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Spawn pattern manager watchers on init</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-f2e4c29b8eef22189ec704a832633c4bd135a24444f054864c98064bd0963681">+13/-3</a> </td> </tr> <tr> <td><strong>re_pattern_stream_map.rs</strong><dd><code>Define ORM entity for stream pattern map</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-3c32b9e32bea40b63ed43542aa136cc9000132c4049ed7b6310baabadc55cd79">+38/-0</a> </td> </tr> <tr> <td><strong>re_patterns.rs</strong><dd><code>Define ORM entity for re_patterns table</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-9c4ac843741290c1b999f3101fae6f46d71d1893d7740de50c71e2c29fe1636c">+33/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Register re_pattern processor in super queue</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-6ed0f1dd36b7216fffde573232a49869f247330ecb03b50f94b2ddd723742e46">+2/-0</a> </td> </tr> <tr> <td><strong>prelude.rs</strong><dd><code>Expose re_patterns entities in prelude</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-a818656b9fb01945155f28450270dd2ae20b67c8248c851ffa0f25e859d263ed">+2/-1</a> </td> </tr> </table></details></td></tr><tr><td><strong>Configuration changes</strong></td><td><details><summary>5 files</summary><table> <tr> <td><strong>m20250612_000001_create_re_pattern_table.rs</strong><dd><code>Create migration for re_patterns table</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-db969ee5510136d5b3c712cdea999c151ab04efe55f1b090670c1eed2b36bcfc">+173/-0</a> </td> </tr> <tr> <td><strong>m20250612_000002_create_re_pattern_stream_map_table.rs</strong><dd><code>Create migration for association stream map table</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-f1ec25aa2150a4fc857fdb17e45b8e43e7c31e884df77e8c91fd07636a6931ca">+168/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Register new pattern migrations</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-7ee42a714d3d56c663da0412d15514613222055aebb5b50dafbc051a656a36f6">+5/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add re_patterns services to HTTP router</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+7/-1</a> </td> </tr> <tr> <td><strong>config.rs</strong><dd><code>Bump DB schema version to support patterns</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+1/-1</a> </td> </tr> </table></details></td></tr><tr><td><strong>Error handling</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>auth.rs</strong><dd><code>Add auth rules for re_patterns test endpoint</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-9bc2b1841504a435f66ad094f72ddbe7b7fa75e5e1557b8adbfe36f7f0592e1b">+11/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>23 files</summary><table> <tr> <td><strong>Cross.toml</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-3abca7e5bab5b140e44643dade038033b8a6f6e50771f76c7d48e7bd0b338815">+5/-0</a> </td> </tr> <tr> <td><strong>Dockerfile.musl</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-f47eb18f838cbf757cd4e34ed5357c767b969d4551dcea6a4ab3b64d384a5614">+16/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-a1cfd40da1ab6cd4db0181fd8664965034298047b3b7c3320c5385897a7c4aed">+2/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-c539d7557d1425986d5c09849d8dbca42d1b309c2bc33d49db641d6a783098f6">+2/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-af023301203a27a1ef34f698d5a329ad7bfced65ac65e3829b1c0905a0c74fb2">+2/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-b7e3bf7f0ac23dd084c0c817a310a4d5e7a64ff532746d182986ef940fa03fd0">+2/-0</a> </td> </tr> <tr> <td><strong>O2AIChat.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-a6a125719a7b32b31b53fcfe05c12081d82c139a401699affdda0db8e73887e3">+2/-0</a> </td> </tr> <tr> <td><strong>TemplateList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-7df66364d7d69a26cfa25cbc555d0e090c2f73e51c8386d9b0435bae4412564c">+1/-1</a> </td> </tr> <tr> <td><strong>FullViewContainer.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-c0722446274c88c250042e38b051201b81450cb5da1ee827cd5a4a927424a357">+1/-6</a> </td> </tr> <tr> <td><strong>AssociatedRegexPatterns.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-c6fce5e1c83798991b6ff19effb23c67af59f38fac0b9e56dcab7a09d139f550">+940/-0</a> </td> </tr> <tr> <td><strong>schema.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-9196be2ccae4054a00b0b1aa9d1e87b366c10920d6eaf9ef8c6e6293d50bcf46">+134/-4</a> </td> </tr> <tr> <td><strong>AddRegexPattern.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-a3783cfbdd32609bfff45ecfa23542854cad7ae60c3d29fedb832312a4de03dc">+689/-0</a> </td> </tr> <tr> <td><strong>ImportRegexPattern.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-e8e1bcb5a02311504d7d571090eab27a270aaa71cae185ad2703023e2e70a31e">+771/-0</a> </td> </tr> <tr> <td><strong>NoRegexPatterns.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-279b923f20042948c87e907d2453e9d5a4c8c73672b23df951d7d7d823ff3a88">+103/-0</a> </td> </tr> <tr> <td><strong>RegexPatternList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-524927acd9f5bb07b71bfac489e747470108574e21a6dde63ea7c1f219cea853">+506/-0</a> </td> </tr> <tr> <td><strong>index.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-699c63979f851c48ad440a078e93282fd797a18fc065e5331b2e41122ddb8b69">+27/-2</a> </td> </tr> <tr> <td><strong>useManagementRoutes.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-b71a1d39b1fab0749f0668d5f16382536fbda6e2f0c92c56a972fb5d8b5835f5">+12/-0</a> </td> </tr> <tr> <td><strong>useStreams.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-a083369344f022cdf8d4f536719179b1d920369e33e2e26234e70dd36d151a7c">+42/-1</a> </td> </tr> <tr> <td><strong>MainLayout.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-91e4134141264c4624d4eb4cfa987b33ed31eb6f551e089c0b26f2610dd54362">+3/-1</a> </td> </tr> <tr> <td><strong>en.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-892aa3d75ab6e129ef9116aa21bec1b9a15ba69ce5366d2a19128078c690d824">+24/-1</a> </td> </tr> <tr> <td><strong>JsonPreview.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-3985073134d35ff074742944b114d0905524038b42e6d2ac887fef2c8436fb16">+272/-3</a> </td> </tr> <tr> <td><strong>regex_pattern.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-2a0b0828141d39ab0385e0941314ead1cdd2ec3ec7b3baf5fd0c42f8f010728a">+63/-0</a> </td> </tr> <tr> <td><strong>index.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-955a15c7ee060f1545e82be22911dd0c14307dee5a70071a394e9fd583bd2c55">+9/-0</a> </td> </tr> </table></details></td></tr></tr></tbody></table> ___ > <details> <summary> Need help?</summary><li>Type <code>/help how to ...</code> in the comments thread for any questions about PR-Agent usage.</li><li>Check out the <a href="https://qodo-merge-docs.qodo.ai/usage-guide/">documentation</a> for more information.</li></details> --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> Co-authored-by: Shrinath Rao <shnath@openobserve.ai> Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com> |