mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-04 16:09:05 +00:00
41f330af67
### **User description** This PR adds support for adding and applying regexp patterns at ingestion time to redact or drop the values from fields if matched. This is an ent-only feature, and this PR contains the required table migrations, along with nats code for super cluster. This also has some additions for musl build for ent repo. There are basic crud APIs for the pattersn as well as an test api which can be used to test a specific pattern on given input. ___ ### **PR Type** Enhancement ___ ### **Description** - Add HTTP CRUD and test endpoints for regex patterns - Implement DB models, migrations for patterns associations - Integrate pattern manager into ingestion and streams - Spawn watchers and super-cluster queue for patterns ___ ### **Changes diagram** ```mermaid flowchart LR HTTP["CRUD & test HTTP handlers"] DBService["Service DB layer"] DB["re_patterns & associations tables"] Cluster["Cluster Coordinator"] PatternMgr["Pattern Manager"] Ingest["Ingestion pipeline"] Stream["Stream settings update"] HTTP -- "invoke add/update/delete/test" --> DBService DBService -- "persist & notify" --> DB DBService -- "trigger" --> Cluster Cluster -- "watch events" --> PatternMgr Ingest -- "redact at ingestion" --> PatternMgr Stream -- "update associations" --> PatternMgr ``` ___ ### **Changes walkthrough** 📝 <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>14 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Add HTTP handlers for re_patterns endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-27d37a8738de89c81b540daa978e684caf2cf28803e1208d90cb4d42528ed28b">+459/-0</a> </td> </tr> <tr> <td><strong>re_pattern.rs</strong><dd><code>Implement DB service for re_patterns</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-0cb2bf6d550d07b66db7e325c6d44c6a598f2ac8e315bf9e168f4a1d9d6e745d">+384/-0</a> </td> </tr> <tr> <td><strong>re_pattern_stream_map.rs</strong><dd><code>Add pattern association table ORM</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-13798240832b9110426267b3e0ebb862bd6c2ac03a3462721f5e85b5bfa88f51">+257/-0</a> </td> </tr> <tr> <td><strong>re_pattern.rs</strong><dd><code>Add re_patterns table ORM definitions</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-f113ed92667f5e352ac7a0c1c8b5e48ea858992b92be5817c9b38f65843ef4a2">+185/-0</a> </td> </tr> <tr> <td><strong>stream.rs</strong><dd><code>Integrate pattern associations in streams</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-41452a7fb29b4634437174fd6aecf6a9050dbfdef41b9077a21d59af3d79e9ed">+62/-2</a> </td> </tr> <tr> <td><strong>re_pattern.rs</strong><dd><code>Handle super-cluster queue for patterns</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-9e64341dc2d7d50cc5acffd0340b983f37b46a9aa34c86fb5a25978f9eb761dc">+166/-0</a> </td> </tr> <tr> <td><strong>ingest.rs</strong><dd><code>Redact data via pattern manager at ingestion</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-e62118e45be183266d39ffa08491f255017954235ff0f3d6937683e314da406f">+23/-0</a> </td> </tr> <tr> <td><strong>stream.rs</strong><dd><code>Add pattern associations to Stream metadata</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-6f5ee19bbcc9cc7fb699fc1c0f43fc96d42698ae1168a2827b5c65f7948d5dbc">+4/-1</a> </td> </tr> <tr> <td><strong>stream.rs</strong><dd><code>Define PatternAssociation and settings wrapper</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-2438d04c3fe97bc5452e5b74a303163b8723d93455cda32c95f0525435ad46bd">+14/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Spawn pattern manager watchers on init</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-f2e4c29b8eef22189ec704a832633c4bd135a24444f054864c98064bd0963681">+13/-3</a> </td> </tr> <tr> <td><strong>re_pattern_stream_map.rs</strong><dd><code>Define ORM entity for stream pattern map</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-3c32b9e32bea40b63ed43542aa136cc9000132c4049ed7b6310baabadc55cd79">+38/-0</a> </td> </tr> <tr> <td><strong>re_patterns.rs</strong><dd><code>Define ORM entity for re_patterns table</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-9c4ac843741290c1b999f3101fae6f46d71d1893d7740de50c71e2c29fe1636c">+33/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Register re_pattern processor in super queue</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-6ed0f1dd36b7216fffde573232a49869f247330ecb03b50f94b2ddd723742e46">+2/-0</a> </td> </tr> <tr> <td><strong>prelude.rs</strong><dd><code>Expose re_patterns entities in prelude</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-a818656b9fb01945155f28450270dd2ae20b67c8248c851ffa0f25e859d263ed">+2/-1</a> </td> </tr> </table></details></td></tr><tr><td><strong>Configuration changes</strong></td><td><details><summary>5 files</summary><table> <tr> <td><strong>m20250612_000001_create_re_pattern_table.rs</strong><dd><code>Create migration for re_patterns table</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-db969ee5510136d5b3c712cdea999c151ab04efe55f1b090670c1eed2b36bcfc">+173/-0</a> </td> </tr> <tr> <td><strong>m20250612_000002_create_re_pattern_stream_map_table.rs</strong><dd><code>Create migration for association stream map table</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-f1ec25aa2150a4fc857fdb17e45b8e43e7c31e884df77e8c91fd07636a6931ca">+168/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Register new pattern migrations</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-7ee42a714d3d56c663da0412d15514613222055aebb5b50dafbc051a656a36f6">+5/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add re_patterns services to HTTP router</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+7/-1</a> </td> </tr> <tr> <td><strong>config.rs</strong><dd><code>Bump DB schema version to support patterns</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+1/-1</a> </td> </tr> </table></details></td></tr><tr><td><strong>Error handling</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>auth.rs</strong><dd><code>Add auth rules for re_patterns test endpoint</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-9bc2b1841504a435f66ad094f72ddbe7b7fa75e5e1557b8adbfe36f7f0592e1b">+11/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>23 files</summary><table> <tr> <td><strong>Cross.toml</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-3abca7e5bab5b140e44643dade038033b8a6f6e50771f76c7d48e7bd0b338815">+5/-0</a> </td> </tr> <tr> <td><strong>Dockerfile.musl</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-f47eb18f838cbf757cd4e34ed5357c767b969d4551dcea6a4ab3b64d384a5614">+16/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-a1cfd40da1ab6cd4db0181fd8664965034298047b3b7c3320c5385897a7c4aed">+2/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-c539d7557d1425986d5c09849d8dbca42d1b309c2bc33d49db641d6a783098f6">+2/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-af023301203a27a1ef34f698d5a329ad7bfced65ac65e3829b1c0905a0c74fb2">+2/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-b7e3bf7f0ac23dd084c0c817a310a4d5e7a64ff532746d182986ef940fa03fd0">+2/-0</a> </td> </tr> <tr> <td><strong>O2AIChat.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-a6a125719a7b32b31b53fcfe05c12081d82c139a401699affdda0db8e73887e3">+2/-0</a> </td> </tr> <tr> <td><strong>TemplateList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-7df66364d7d69a26cfa25cbc555d0e090c2f73e51c8386d9b0435bae4412564c">+1/-1</a> </td> </tr> <tr> <td><strong>FullViewContainer.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-c0722446274c88c250042e38b051201b81450cb5da1ee827cd5a4a927424a357">+1/-6</a> </td> </tr> <tr> <td><strong>AssociatedRegexPatterns.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-c6fce5e1c83798991b6ff19effb23c67af59f38fac0b9e56dcab7a09d139f550">+940/-0</a> </td> </tr> <tr> <td><strong>schema.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-9196be2ccae4054a00b0b1aa9d1e87b366c10920d6eaf9ef8c6e6293d50bcf46">+134/-4</a> </td> </tr> <tr> <td><strong>AddRegexPattern.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-a3783cfbdd32609bfff45ecfa23542854cad7ae60c3d29fedb832312a4de03dc">+689/-0</a> </td> </tr> <tr> <td><strong>ImportRegexPattern.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-e8e1bcb5a02311504d7d571090eab27a270aaa71cae185ad2703023e2e70a31e">+771/-0</a> </td> </tr> <tr> <td><strong>NoRegexPatterns.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-279b923f20042948c87e907d2453e9d5a4c8c73672b23df951d7d7d823ff3a88">+103/-0</a> </td> </tr> <tr> <td><strong>RegexPatternList.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-524927acd9f5bb07b71bfac489e747470108574e21a6dde63ea7c1f219cea853">+506/-0</a> </td> </tr> <tr> <td><strong>index.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-699c63979f851c48ad440a078e93282fd797a18fc065e5331b2e41122ddb8b69">+27/-2</a> </td> </tr> <tr> <td><strong>useManagementRoutes.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-b71a1d39b1fab0749f0668d5f16382536fbda6e2f0c92c56a972fb5d8b5835f5">+12/-0</a> </td> </tr> <tr> <td><strong>useStreams.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-a083369344f022cdf8d4f536719179b1d920369e33e2e26234e70dd36d151a7c">+42/-1</a> </td> </tr> <tr> <td><strong>MainLayout.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-91e4134141264c4624d4eb4cfa987b33ed31eb6f551e089c0b26f2610dd54362">+3/-1</a> </td> </tr> <tr> <td><strong>en.json</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-892aa3d75ab6e129ef9116aa21bec1b9a15ba69ce5366d2a19128078c690d824">+24/-1</a> </td> </tr> <tr> <td><strong>JsonPreview.vue</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-3985073134d35ff074742944b114d0905524038b42e6d2ac887fef2c8436fb16">+272/-3</a> </td> </tr> <tr> <td><strong>regex_pattern.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-2a0b0828141d39ab0385e0941314ead1cdd2ec3ec7b3baf5fd0c42f8f010728a">+63/-0</a> </td> </tr> <tr> <td><strong>index.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7323/files#diff-955a15c7ee060f1545e82be22911dd0c14307dee5a70071a394e9fd583bd2c55">+9/-0</a> </td> </tr> </table></details></td></tr></tr></tbody></table> ___ > <details> <summary> Need help?</summary><li>Type <code>/help how to ...</code> in the comments thread for any questions about PR-Agent usage.</li><li>Check out the <a href="https://qodo-merge-docs.qodo.ai/usage-guide/">documentation</a> for more information.</li></details> --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> Co-authored-by: Shrinath Rao <shnath@openobserve.ai> Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
web
This template should help get you started developing with Vue 3 in Vite.
Recommended IDE Setup
VSCode + Volar (and disable Vetur) + TypeScript Vue Plugin (Volar).
Type Support for .vue Imports in TS
TypeScript cannot handle type information for .vue imports by default, so we replace the tsc CLI with vue-tsc for type checking. In editors, we need TypeScript Vue Plugin (Volar) to make the TypeScript language service aware of .vue types.
If the standalone TypeScript plugin doesn't feel fast enough to you, Volar has also implemented a Take Over Mode that is more performant. You can enable it by the following steps:
- Disable the built-in TypeScript Extension
- Run
Extensions: Show Built-in Extensionsfrom VSCode's command palette - Find
TypeScript and JavaScript Language Features, right click and selectDisable (Workspace)
- Run
- Reload the VSCode window by running
Developer: Reload Windowfrom the command palette.
Customize configuration
See Vite Configuration Reference.
Project Setup
npm install
Compile and Hot-Reload for Development
For stand-alone UI development environment, create .env file and which contains below environment variable
VITE_OPENOBSERVE_ENDPOINT=http://localhost:5080/
npm run dev
Type-Check, Compile and Minify for Production
npm run build
Run Unit Tests with Vitest
npm run test:unit
Run End-to-End Tests with Cypress
npm run test:e2e:dev
This runs the end-to-end tests against the Vite development server. It is much faster than the production build.
But it's still recommended to test the production build with test:e2e before deploying (e.g. in CI environments):
npm run build
npm run test:e2e
Lint with ESLint
npm run lint