mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-06 16:09:41 +00:00
e40938a08c
### **PR Type**
Enhancement
Design at : enrichment-file-url
___
### **Description**
- Async URL-based enrichment table processing
- Event-driven MPSC, streaming CSV, batching
- Retry with backoff, status updates
- Database and ORM additions
- `enrichment_table_urls` table and migration
- HTTP API endpoints
- POST `/url` import and GET status
- Frontend enhancements
- Tabs, status icons, retry button
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["HTTP POST /{org}/enrichment_tables/{table}/url"] --> B["DB: save_url_job"]
B --> C["trigger_url_job_processing()"]
C --> D["MPSC Channel"]
D --> E["process_url_jobs()"]
E --> F["process_single_url_job()"]
F --> G["validate_url() & fetch_and_process_batches()"]
G --> H["parse_csv_chunk() & save_enrichment_batch()"]
H --> I["save_url_job() update status"]
I --> J["notify_update()"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>14
files</summary><table>
<tr>
<td><strong>url_processor.rs</strong><dd><code>Implement async URL
enrichment table processor</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-d39589e34157fe7930563b9bc5740128e168b8d8ac0a90371a4bdf1e52ce058b">+1351/-0</a></td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Add URL enrichment table HTTP
endpoints</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-7b5b203002d1c3e5f671392a2a8ed84f45225896b652c2dac94ac26e72a43124">+301/-1</a>
</td>
</tr>
<tr>
<td><strong>enrichment_table.rs</strong><dd><code>Add URL job CRUD
persistence functions</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-6ce27ddf8a0d8b4e790bda90d088ff692a5da31a8d330beea8a9dd843294572b">+159/-2</a>
</td>
</tr>
<tr>
<td><strong>enrichment_table_urls.rs</strong><dd><code>SeaORM logic for
URL job records</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-e9ba1ba6e742e8d7652c3f31765de43bee061e7983bca67ffefecf8590210188">+173/-0</a>
</td>
</tr>
<tr>
<td><strong>enrichment_table.rs</strong><dd><code>Define URL job status
and job struct</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-29818b7d921497260251df82a49ae9ab7a93bbaedc0ca03abdf2b8c0726e4085">+90/-0</a>
</td>
</tr>
<tr>
<td><strong>pipelines.rs</strong><dd><code>Handle super-cluster URL job
events</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-7c8aa678045dd00de89d4c2345cf4a0511ce9bfac5d6f64e7404178c81e541ad">+23/-1</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Expose url_processor module in
service</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-3e2b6d3b8da1f9e73aca279052a97e683184d68256c23215b18ffee885836bbf">+6/-0</a>
</td>
</tr>
<tr>
<td><strong>enrichment_table_urls.rs</strong><dd><code>Define SeaORM
entity for URL jobs</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-9c3081476927bd337b2cbed8310578869845a6137a2c5305f677affb7604d39f">+26/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Register URL handlers in
router</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+2/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Expose enrichment_table_urls entity
module</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-c539d7557d1425986d5c09849d8dbca42d1b309c2bc33d49db641d6a783098f6">+1/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Add enrichment_table_urls to table
module</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-af023301203a27a1ef34f698d5a329ad7bfced65ac65e3829b1c0905a0c74fb2">+1/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Include enrichment_table meta
module</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-bab79e8da2d0d9f903cb1a6187d922c6de63d3e4561e8ce633660b664b4e859f">+1/-0</a>
</td>
</tr>
<tr>
<td><strong>jstransform.ts</strong><dd><code>Add client methods for URL
jobs API</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-301add2528e0f32323ca4c503bd7d460ab15ca6b9a9537192a02c5e52fd82926">+18/-0</a>
</td>
</tr>
<tr>
<td><strong>EnrichmentTableList.vue</strong><dd><code>Enhance UI for
URL-based enrichment tables</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-6a29e253f59405d3c0f756dba36afd7483f10ad86927cbbc3348ede6372c3077">+327/-74</a></td>
</tr>
</table></details></td></tr><tr><td><strong>Configuration
changes</strong></td><td><details><summary>3 files</summary><table>
<tr>
<td><strong>m20251221_000001_create_enrichment_table_urls.rs</strong><dd><code>Create
migration for URL jobs table</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-330b4b7ebbda5001f4f28050d50e219b91961d80d8369222d94d058cced57702">+189/-0</a>
</td>
</tr>
<tr>
<td><strong>config.rs</strong><dd><code>Add URL fetch configuration
options</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+24/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Register URL jobs migration in
migrator</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-7ee42a714d3d56c663da0412d15514613222055aebb5b50dafbc051a656a36f6">+2/-0</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>1
files</summary><table>
<tr>
<td><strong>openapi.rs</strong><dd><code>Document URL endpoints in
OpenAPI spec</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-edb8ce232c0cd08366879f642788a75d99f6bab64b4c0e77391281fdb9241e90">+5/-0</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Bug
fix</strong></td><td><details><summary>1 files</summary><table>
<tr>
<td><strong>auth.rs</strong><dd><code>Extend auth for enrichment_tables
URLs</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-9bc2b1841504a435f66ad094f72ddbe7b7fa75e5e1557b8adbfe36f7f0592e1b">+9/-1</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Additional
files</strong></td><td><details><summary>1 files</summary><table>
<tr>
<td><strong>AddEnrichmentTable.vue</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9697/files#diff-9aad551dbc051edfdb3994a664506ed6d5e27dcb64f9c2fa917b02e51a35d86e">+159/-42</a></td>
</tr>
</table></details></td></tr></tr></tbody></table>
</details>
___
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>