openobserve

starred/openobserve
Fork 0
mirror of https://github.com/openobserve/openobserve.git synced 2026-11-06 08:09:24 +00:00
Files
T
History
75f2235d04 feat: create stream with field and data type (#7851 )
### **User description**
closes #6660


___

### **PR Type**
Enhancement, Bug fix


___

### **Description**
- Add field data types and UDS support

- Validate field names with strict regex

- Store defined schema as objects, not strings

- Backward-compatible parsing and UI updates


___

### Diagram Walkthrough


```mermaid
flowchart LR
  API["API: Stream settings/schema"] --> Parse["Parse defined_schema_fields (strings|objects)"]
  Parse -- "normalize to DataField{name,type}" --> Settings["StreamSettings.defined_schema_fields: DataField[]"]
  Settings -- "new stream" --> SchemaAdd["Add fields to Arrow schema with types"]
  Settings -- "existing + UDS" --> UDSUpdate["Add new fields to UDS + persist"]
  Types["DataType enum (Utf8, Int64, Uint64, Float64, Boolean)"] --> Arrow["Arrow DataType mapping"]
  Validate["Regex field name validation"] --> Save["save/update settings endpoints"]
  UI["UI: Add/Edit stream fields"] --> ObjectFmt["Send/handle {name,type} objects"]
  ObjectFmt -- "compat" --> Compat["Store original + derive string list for display"]
```



<details> <summary><h3> File Walkthrough</h3></summary>

<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>10
files</summary><table>
<tr>
<td><strong>stream.rs</strong><dd><code>Rename stream property field to
raw type key</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7851/files#diff-6f5ee19bbcc9cc7fb699fc1c0f43fc96d42698ae1168a2827b5c65f7948d5dbc">+5/-6</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>

<tr>
<td><strong>stream.rs</strong><dd><code>Introduce DataType and
DataField; settings parsing</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7851/files#diff-2438d04c3fe97bc5452e5b74a303163b8723d93455cda32c95f0525435ad46bd">+99/-13</a>&nbsp;
</td>

</tr>

<tr>
<td><strong>mod.rs</strong><dd><code>Optimize settings fetch; adapt
defined fields helpers</code>&nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7851/files#diff-386fd4fad1e48ab0a90516cab1947b1888b270b2fe912f30d2255cc38e6c519b">+13/-6</a>&nbsp;
&nbsp; </td>

</tr>

<tr>
<td><strong>mod.rs</strong><dd><code>Use DataField names when building
UDS sets</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7851/files#diff-7c7b5bca85f0ced652965cae66540c96f56dfcca4f4c16b77bf7b45aac203d1b">+6/-1</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>

<tr>
<td><strong>schema.rs</strong><dd><code>Add DataField flows; new/UDS
logic and conversions</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7851/files#diff-d9596182a0c7a46a6a7ff5cee822554cdae05ceec312f86c93b538db9f73bbfd">+270/-18</a></td>

</tr>

<tr>
<td><strong>stream.rs</strong><dd><code>Validate field names; resolve
defined field types</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7851/files#diff-41452a7fb29b4634437174fd6aecf6a9050dbfdef41b9077a21d59af3d79e9ed">+126/-7</a>&nbsp;
</td>

</tr>

<tr>
<td><strong>AddStream.vue</strong><dd><code>Send defined fields as
objects with type</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7851/files#diff-78b8c84a558e60d942bacffee365633d074042e464be465eb6984b0ee5df82b9">+6/-2</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>

<tr>
<td><strong>StreamFieldInputs.vue</strong><dd><code>UI for field data
type and UX tweaks</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7851/files#diff-c8173413939f320f197b0f8790e619da18666f79009345f72b3325e89bbb7566">+90/-27</a>&nbsp;
</td>

</tr>

<tr>
<td><strong>schema.vue</strong><dd><code>Support object-based defined
fields; sync logic</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7851/files#diff-9196be2ccae4054a00b0b1aa9d1e87b366c10920d6eaf9ef8c6e6293d50bcf46">+119/-25</a></td>

</tr>

<tr>
<td><strong>useStreams.ts</strong><dd><code>Compare/sync object-based
defined fields</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7851/files#diff-a083369344f022cdf8d4f536719179b1d920369e33e2e26234e70dd36d151a7c">+50/-2</a>&nbsp;
&nbsp; </td>

</tr>
</table></details></td></tr><tr><td><strong>Bug
fix</strong></td><td><details><summary>1 files</summary><table>
<tr>
<td><strong>mod.rs</strong><dd><code>Map UDS fields by object name not
string</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7851/files#diff-619eafba0f0509d905b021f4a16659010aca23c62cffd9cd3e6c2bbe204ce200">+1/-1</a>&nbsp;
&nbsp; &nbsp; </td>

</tr>
</table></details></td></tr></tr></tbody></table>

</details>

___

---------

Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
2025-08-20 20:57:04 +08:00
infra
fix: new sso admin/viewer/editor should be able to enjoy given permissions (#8002 )
2025-08-20 13:25:36 +05:30