mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-04 16:09:05 +00:00
Ashish Kolhe
d6339499f8
### **PR Type** Enhancement, Bug fix ___ ### **Description** - Surface last pipeline error in API - Persist pipeline errors in new table - Batch upsert pipeline errors from reporting - Safer error messages in VRL failures ___ ### Diagram Walkthrough ```mermaid flowchart LR Reporting["Self-reporting errors"] -- "batch_upsert" --> DB["Table: pipeline_last_errors"] API["GET /pipelines"] -- "join with errors" --> UIData["PipelineList.last_error"] UIData -- "render" --> ListUI["PipelinesList error icon"] UIData -- "per-node errors" --> NodeUI["CustomNode error badge"] Ingestion["VRL apply_vrl_fn"] -- "clean error message" --> Reporting ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Configuration changes</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>config.rs</strong><dd><code>Bump DB schema version to 10</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8695/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+1/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Wire migration for last errors table</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8695/files#diff-7ee42a714d3d56c663da0412d15514613222055aebb5b50dafbc051a656a36f6">+2/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Enhancement</strong></td><td><details><summary>11 files</summary><table> <tr> <td><strong>pipelines.rs</strong><dd><code>Add last_error to pipeline model</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8695/files#diff-61bb28d7091130297c917a0530d96202fcb4b073b514410531f9fe71acfc814f">+22/-3</a> </td> </tr> <tr> <td><strong>pipeline.rs</strong><dd><code>Join pipelines with last error data</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8695/files#diff-c69f4f94eee97ce226caddc497b2adf89e26869f2d72b0d0206dc31ab2714daf">+26/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Register pipeline_last_errors entity</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8695/files#diff-c539d7557d1425986d5c09849d8dbca42d1b309c2bc33d49db641d6a783098f6">+1/-0</a> </td> </tr> <tr> <td><strong>pipeline_last_errors.rs</strong><dd><code>Define ORM entity for last errors</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8695/files#diff-e4ff5af0cb9c6e6343ecaad06fbb03515ace8868be6ee6d192ff0f01ee4d3e17">+22/-0</a> </td> </tr> <tr> <td><strong>prelude.rs</strong><dd><code>Export PipelineLastErrors entity</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8695/files#diff-a818656b9fb01945155f28450270dd2ae20b67c8248c851ffa0f25e859d263ed">+1/-0</a> </td> </tr> <tr> <td><strong>m20250930_000001_create_pipeline_last_errors_table.rs</strong><dd><code>Create pipeline_last_errors table and index</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8695/files#diff-173fe20fccfdcfa68563a13f9d69c2e6cb0901ac257d43386962bbac90f296de">+169/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Expose pipeline_errors DB module</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8695/files#diff-b7e3bf7f0ac23dd084c0c817a310a4d5e7a64ff532746d182986ef940fa03fd0">+1/-0</a> </td> </tr> <tr> <td><strong>pipeline_errors.rs</strong><dd><code>CRUD and batch upsert for pipeline errors</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8695/files#diff-627b6bb9ca334984b8f08f1405a67e5110c238788f165e30abda39b03f7ab68f">+331/-0</a> </td> </tr> <tr> <td><strong>queues.rs</strong><dd><code>Batch upsert pipeline errors during ingest</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8695/files#diff-ed15a56d4d3becc2dfb3d05af2dac7f92962ec3091199728fa953e6aed7ed767">+45/-5</a> </td> </tr> <tr> <td><strong>PipelinesList.vue</strong><dd><code>Show pipeline-level last error indicator</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8695/files#diff-85e3ef322f3eea0fc6d74d89431ba338a1153cb82ee06bef0f7c5f439c04267d">+27/-0</a> </td> </tr> <tr> <td><strong>CustomNode.vue</strong><dd><code>Show node-level error badge and navigation</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8695/files#diff-b9d83cd503b6d3470b238accaaea0b4decf567add8ad4727c63c9e583725f408">+98/-2</a> </td> </tr> </table></details></td></tr><tr><td><strong>Bug fix</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Return sanitized VRL error messages</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8695/files#diff-7c7b5bca85f0ced652965cae66540c96f56dfcca4f4c16b77bf7b45aac203d1b">+12/-8</a> </td> </tr> </table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Clarify queuing of error data</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8695/files#diff-c6255010f7a87d6acdb3ad04954fc9f2c27350ae319c53a1857b120aa06408a3">+2/-0</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___