openobserve

starred/openobserve

Fork 0

mirror of https://github.com/openobserve/openobserve.git synced 2026-11-04 08:09:27 +00:00

Files

T

History

c77a559183 feat: add pipeline usage (#8295 )

### **PR Type**
Enhancement, Bug fix


___

### **Description**
- Add pipeline and remote usage tracking

- Report usage for remote destination WAL writes

- Switch pipeline channels to structured items

- Fix size accounting across pipeline branches


___

### Diagram Walkthrough


```mermaid
flowchart LR
  A["Ingestion (Logs/Metrics)"] -- "pre-pipeline size" --> B["Usage: Pipeline"]
  B -- "execute pipeline" --> C["Pipeline Nodes (Condition/Function/Destination)"]
  C -- "local dest" --> D["Size by stream (no double count)"]
  C -- "remote dest WAL write" --> E["Usage: RemotePipeline"]
```



<details> <summary><h3> File Walkthrough</h3></summary>

<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>3
files</summary><table>
<tr>
<td><strong>usage.rs</strong><dd><code>Add Pipeline and RemotePipeline
usage types and mapping</code>&nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8295/files#diff-b15a6adcb163dc27993c83528105a6726c44c86d8a84c5cf5d1d382c6ec8cc29">+22/-0</a>&nbsp;
&nbsp; </td>

</tr>

<tr>
<td><strong>ingest.rs</strong><dd><code>Report size before pipeline;
avoid double counting</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8295/files#diff-e62118e45be183266d39ffa08491f255017954235ff0f3d6937683e314da406f">+13/-7</a>&nbsp;
&nbsp; </td>

</tr>

<tr>
<td><strong>batch_execution.rs</strong><dd><code>Add usage reporting;
introduce PipelineItem; report remote WAL usage</code></dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8295/files#diff-3f6824a6eddaf1603e5cb0f1a4311ac6a45e1ffb967266ccc1f0bba944625e69">+158/-29</a></td>

</tr>
</table></details></td></tr><tr><td><strong>Bug
fix</strong></td><td><details><summary>1 files</summary><table>
<tr>
<td><strong>handlers.rs</strong><dd><code>Avoid ownership by cloning
stream params key</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8295/files#diff-5eca8995a88fcecb6d6b8efd3c7c4496f4a96a4b060fb4ed85593574f7cf49eb">+1/-1</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>

</table></details></td></tr><tr><td><strong>Formatting</strong></td><td><details><summary>3
files</summary><table>
<tr>
<td><strong>otlp.rs</strong><dd><code>Minor formatting and flow tidy in
loop</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8295/files#diff-183498d98b17ddf344cbab4da6b2d9fe89ac81112db51655c142a89023b46ef3">+1/-0</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>

<tr>
<td><strong>json.rs</strong><dd><code>Minor formatting; prep for
pipeline handling</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8295/files#diff-506987e61095a54e47f7f208dad89f9f1bce595fa6bf1ba8a7d746f2b2ff6c4a">+1/-0</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>

<tr>
<td><strong>prom.rs</strong><dd><code>Minor formatting; prep for
pipeline handling</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8295/files#diff-c4e9e1d49a2bd0fefe67684b3768c4792be63cebcf727d71a48141f5b5957b8c">+1/-0</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>
</table></details></td></tr></tr></tbody></table>

</details>

___

---------

Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
Co-authored-by: Shrinath Rao <shnath@openobserve.ai>

2025-09-17 19:02:41 +05:30

api-testing

test: str_match_coalesce_fix (#8403 )

2025-09-12 18:40:35 +05:30

test-data

test: e2e-cte-queries (#8070 )

2025-08-27 18:24:39 +05:30

ui-testing

feat: add pipeline usage (#8295 )

2025-09-17 19:02:41 +05:30

input.json

docs: Fix typos (#4492 )