openobserve

starred/openobserve

Fork 0

mirror of https://github.com/openobserve/openobserve.git synced 2026-11-05 08:09:22 +00:00

Files

T

History

Yashodhan JoshiandYashodhan Joshi 68759b4354 fix: sync distinct stream retention to original stream (#9871 )

### **User description**
This adds functionality to always sync original stream retention to the
distinct stream created for that stream.
This also includes a once-off migration to sync the retention for
existing streams


___

### **PR Type**
Enhancement


___

### **Description**
- Bump DB schema version to 24

- Add migration syncing existing distinct stream retention

- Sync retention on settings update for distinct streams

- Refactor distinct stream naming logic


___

### Diagram Walkthrough


```mermaid
flowchart LR
  A["Original Stream Settings"] --> B["Generate Distinct Stream Name"]
  C["Migration: Sync Retention"] --> B
  D["Settings Update Handler"] --> B
```



<details> <summary><h3> File Walkthrough</h3></summary>

<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Configuration
changes</strong></td><td><table>
<tr>
  <td>
    <details>
<summary><strong>config.rs</strong><dd><code>Bump database schema
version</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; </dd></summary>
<hr>

src/config/src/config.rs

- bumped `DB_SCHEMA_VERSION` from 23 to 24


</details>


  </td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9871/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+1/-1</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>

<tr>
  <td>
    <details>
<summary><strong>mod.rs</strong><dd><code>Register new
migration</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

src/infra/src/table/migration/mod.rs

- register new migration in migrator list


</details>


  </td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9871/files#diff-7ee42a714d3d56c663da0412d15514613222055aebb5b50dafbc051a656a36f6">+2/-0</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>
</table></td></tr><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
  <td>
    <details>

<summary><strong>m20260107_000001_sync_distinct_stream_retention.rs</strong><dd><code>Add
migration for distinct stream retention</code>&nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; </dd></summary>
<hr>


src/infra/src/table/migration/m20260107_000001_sync_distinct_stream_retention.rs

<ul><li>add new migration file for retention sync<br> <li> iterate all
streams and update distinct retention<br> <li> support enterprise
super-cluster propagation</ul>


</details>


  </td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9871/files#diff-e077e2b4452a91f484015bd939eae29bec994be01a67eb4e71e5f111c761c755">+115/-0</a>&nbsp;
</td>

</tr>

<tr>
  <td>
    <details>
<summary><strong>distinct_values.rs</strong><dd><code>Refactor naming
and retention logic</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

src/service/metadata/distinct_values.rs

<ul><li>extract <code>get_distinct_stream_name</code> function<br> <li>
use helper instead of inline formatting<br> <li> update distinct stream
retention during ingest</ul>


</details>


  </td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9871/files#diff-3fb2320197c6e8e900658f5a24158e922610eadcbf081ab0df230f770ef27e07">+35/-6</a>&nbsp;
&nbsp; </td>

</tr>

<tr>
  <td>
    <details>
<summary><strong>stream.rs</strong><dd><code>Sync retention on settings
save</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></summary>
<hr>

src/service/stream.rs

<ul><li>on <code>save_stream_settings</code>, sync distinct stream
retention<br> <li> add <code>get_stream_retention</code> helper
function<br> <li> skip metadata streams when syncing</ul>


</details>


  </td>
<td><a
href="https://github.com/openobserve/openobserve/pull/9871/files#diff-41452a7fb29b4634437174fd6aecf6a9050dbfdef41b9077a21d59af3d79e9ed">+68/-0</a>&nbsp;
&nbsp; </td>

</tr>
</table></td></tr></tr></tbody></table>

</details>

___

---------

Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>

2026-01-07 18:32:30 +05:30

cipher

fix: support bulk deletes for various resources (#9153 )

2025-12-29 16:37:29 +05:30

cli

feat: new db migration (#9774 )

2025-12-27 21:05:07 +08:00

common

feat: tenant management using service accounts (#9838 )

2026-01-06 13:38:50 +05:30

config

fix: sync distinct stream retention to original stream (#9871 )