Hengfei Yang
e59474aa1b
perf: reduce nats events ( #9712 )
2025-12-25 19:10:31 +08:00
Simha G
6ca371ec99
fix: Concurrency control for file locking has multiple anti-concurrent design patterns in the way ( #8523 )
2025-11-17 20:35:43 +08:00
ByteBaker
a288a0604a
chore: update default dashboard version from 1 to 5
...
In `CreateDashboard` API, the default dashboard version when
`version` key was absent in the payload used to be 1, which
was wrong. It is now 5 (for v5).
Going forward, this value should default to the latest version
supported by OpenObserve.
This is tracked by `const LATEST_DASHBOARD_VERSION`, which is
automatically populated at compile time by counting the number
of variants of `enum DashboardRequestBody`, requiring no human
intervention beyond adding the new variant itself to the enum.
2025-10-03 20:09:49 +05:30
Hengfei Yang
41d0d4acb6
fix: add more logs for flower ( #7944 )
...
### **User description**
- [x] Add more logs for flower
- [x] Optimize parquet footer cache calculate
___
### **PR Type**
Enhancement, Bug fix
___
### **Description**
- Switch query partition strategy to enum
- Improve file stats cache size accuracy
- Add follower leader work-group and partition logs
- Minor consistent-hash refactor for clarity
___
### Diagram Walkthrough
```mermaid
flowchart LR
CFG["Config Common.feature_query_partition_strategy (enum)"]
STREAM["meta/stream QueryPartitionStrategy FromStr"]
FLIGHT["search/cluster/flight uses enum directly"]
FOLL["super_cluster/follower adds detailed logs"]
HASH["cluster get_node_from_consistent_hash minor reorder"]
CACHE["file_statistics_cache counts min/max/sum sizes"]
STREAM -- "provides parser" --> CFG
CFG -- "typed value" --> FLIGHT
FLIGHT -- "strategy choice" --> FOLL
CACHE -- "more accurate memory estimate" --> FLIGHT
HASH -- "clarify flow" --> FLIGHT
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>6
files</summary><table>
<tr>
<td><strong>mod.rs</strong><dd><code>Precompute hash before node-set
access</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7944/files#diff-d15434a27e2b8406811f5a707b605203b603dad4a262c0642326be860bd2b568 ">+1/-1</a>
</td>
</tr>
<tr>
<td><strong>config.rs</strong><dd><code>Make query partition strategy a
parsed enum</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7944/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62 ">+7/-3</a>
</td>
</tr>
<tr>
<td><strong>stream.rs</strong><dd><code>Add FromStr and default for
strategy enum</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7944/files#diff-2438d04c3fe97bc5452e5b74a303163b8723d93455cda32c95f0525435ad46bd ">+8/-5</a>
</td>
</tr>
<tr>
<td><strong>flight.rs</strong><dd><code>Use enum-config strategy
directly</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7944/files#diff-dbffb1a03b65f6a9163b18c5bb1241ba2966735657be39bbe9f4d0e3260d8c1d ">+1/-2</a>
</td>
</tr>
<tr>
<td><strong>file_statistics_cache.rs</strong><dd><code>Include
min/max/sum sizes in cache estimate</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7944/files#diff-5b3f948f2ad04a6babe72c15365b95a284d0533720419fa5dc793aa379783a7d ">+5/-1</a>
</td>
</tr>
<tr>
<td><strong>follower.rs</strong><dd><code>Add work-group wait and
partition logs</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7944/files#diff-59e0cc386582cf8baf201977475fd9a7e479fd8ed000bbaa6746a51ecb28632f ">+8/-1</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Miscellaneous</strong></td><td><details><summary>1
files</summary><table>
<tr>
<td><strong>lib.rs</strong><dd><code>Remove unused async_drop feature
gate</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7944/files#diff-b1a35a68f14e696205874893c07fd24fdb88882b47c23cc0e0c80a30c7d53759 ">+0/-1</a>
</td>
</tr>
</table></details></td></tr></tr></tbody></table>
</details>
___
2025-08-11 15:12:22 +08:00
Huaijin
b141504cf8
chore: upgrade rust to 2025-05-20 ( #7316 )
...
### **PR Type**
Enhancement
___
### **Description**
- Use `if let ... && let ...` patterns
- Inline named args in `format!` calls
- Simplify visitor `visit` calls with `_ =`
- Replace string interpolation in logging macros
___
### **Changes walkthrough** 📝
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>5
files</summary><table>
<tr>
<td><strong>sql.rs</strong><dd><code>Refactor SQL visitor and limit
logic</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-d9ccf393c53794b0ef2816eef12742c26e23d2788cde27780af7570662aea446 ">+249/-255</a></td>
</tr>
<tr>
<td><strong>mysql.rs</strong><dd><code>Refactor time_range early-return
patterns</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-06d02c738c6a4dc22ff022c16befef8d46aece6cfaa88bb4a5d3d5f92c2e50a1 ">+51/-51</a>
</td>
</tr>
<tr>
<td><strong>sql.rs</strong><dd><code>Expand UDF list and simplify
visitors</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-a0a09c66a20d878ec7683ae222c919fbe092b8001871ade7f455b89573ce0911 ">+43/-39</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Refactor error mapping with
`std::io::Error::other`</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-8ce79cea5617c3a661e5b64058ad3846abf8652bb44fbb88dfb9117ffcc577fd ">+20/-36</a>
</td>
</tr>
<tr>
<td><strong>engine.rs</strong><dd><code>Simplify unary ops and format
messages</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-a98ac5b55a4c9440c09c4e2dfa3b6f7f24f9e58fcb6bf8b10577a0b4f123ed41 ">+49/-63</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Formatting</strong></td><td><details><summary>4
files</summary><table>
<tr>
<td><strong>mysql.rs</strong><dd><code>Inline format! and logging
braces</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-412dc407af348f4b54b7d98de6a928876b0333ea501792498a0568850f83d7b7 ">+59/-76</a>
</td>
</tr>
<tr>
<td><strong>nats.rs</strong><dd><code>Simplify error mapping and
formatting</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-2d6809ea34ea04477afb9a6f8287c93cef488b45ef8f0ebeec2241571b93994c ">+49/-54</a>
</td>
</tr>
<tr>
<td><strong>sqlite.rs</strong><dd><code>Use brace shorthand in SQL
strings</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-85f17b22460902614e5c28df8966ea3e7073325bd8dd43fa396ed9268b6b4b06 ">+45/-56</a>
</td>
</tr>
<tr>
<td><strong>action.rs</strong><dd><code>Use brace shorthand in Display
impls</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-fc87c75a7df9539d7e7ee6184dfab02d4a8932323e81f95d57c05363aa44752d ">+4/-4</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Additional
files</strong></td><td><details><summary>101 files</summary><table>
<tr>
<td><strong>Cargo.toml</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542 ">+1/-1</a>
</td>
</tr>
<tr>
<td><strong>clippy.toml</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-f3dbfb2563c3490394f44c26b51837018e79a79e75fd31e89b19e943a38317ea ">+1/-0</a>
</td>
</tr>
<tr>
<td><strong>rust-toolchain.toml</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-2b1bde2cf3a858b7bf7424cb8bcbf01f35b94dc80b925d9432cbab3319ca9b4e ">+1/-1</a>
</td>
</tr>
<tr>
<td><strong>cli.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-cb1eb3e74723d891353ccf1e2e5e0d246d2e78a5e69dbe4c73006ea8412d65e0 ">+8/-8</a>
</td>
</tr>
<tr>
<td><strong>http.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-790b7780a51e8eecd4754b623af8d15ad87baaaf29455ba646e2e410f83f5498 ">+1/-1</a>
</td>
</tr>
<tr>
<td><strong>load.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-bfc7e9b23a8fa100bcad8490d697a3090f139440fcd2ce02fb539c1791100d50 ">+4/-4</a>
</td>
</tr>
<tr>
<td><strong>export.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-b9ca11eac019064e747b0c75157c280c8e8fa6d75edb83ca9ae46777e9668cc0 ">+2/-5</a>
</td>
</tr>
<tr>
<td><strong>import.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-e4415a147f0fe6e557c9fee2a5cad9058a46c30cfb79fe131d13802c0c8bb804 ">+2/-2</a>
</td>
</tr>
<tr>
<td><strong>etcd.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-37c2c2803d1c12ebb4ef9299bc79c9e6740fc5f19f7f4d84e7cfbb610b9561a9 ">+4/-4</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-d15434a27e2b8406811f5a707b605203b603dad4a262c0642326be860bd2b568 ">+4/-4</a>
</td>
</tr>
<tr>
<td><strong>nats.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-901773980aa255942e7865b977fe63811f31511ac40bc512d6bbb14dbb54bece ">+4/-4</a>
</td>
</tr>
<tr>
<td><strong>alert_folders.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-6ed0a38075313ced0916343ee6e6b86ab639f8ed59e4968df395aa30d72bd300 ">+3/-3</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-1b8694fad31c8249afeb419d92a9facbd170289db2c89100552883df876bb275 ">+4/-4</a>
</td>
</tr>
<tr>
<td><strong>wal.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-4928ae81597b28facdc3644f786ac14c60b0a406e1c7b8eac7232fbafbcbb6e3 ">+5/-5</a>
</td>
</tr>
<tr>
<td><strong>loki.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-121c166af5eab8b9cb83716afa7b65fa7856c006c809b3d910ced184c52ec587 ">+8/-8</a>
</td>
</tr>
<tr>
<td><strong>telemetry.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-3ff026b46d9431e0657828a3bde4ab118fec006b353c2abc80be4b74633c98b4 ">+2/-2</a>
</td>
</tr>
<tr>
<td><strong>auth.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-9bc2b1841504a435f66ad094f72ddbe7b7fa75e5e1557b8adbfe36f7f0592e1b ">+39/-43</a>
</td>
</tr>
<tr>
<td><strong>auth_tests.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-2688cbdc0b009b3f0dd829c69ab86c6a7d0778ab3e65e916a410a76aa6ae6480 ">+1/-0</a>
</td>
</tr>
<tr>
<td><strong>functions.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-f4df68dbcf485d6b98c0e673c12a19be2edab416e5fc22c7d08396d375cd9126 ">+1/-1</a>
</td>
</tr>
<tr>
<td><strong>http.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-3b50a69c1ec2f63f8eebb5f9880e17826463a586768aed3e3e15242833609d6b ">+2/-3</a>
</td>
</tr>
<tr>
<td><strong>jwt.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-ad8a7b809caff90b4c49885a9b5d49ae03ca3fe00bef37a72d61b08b59df449c ">+5/-5</a>
</td>
</tr>
<tr>
<td><strong>redirect_response.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-f0415572330f30fceb16f230517e07f2c186e834cb8ca634795e5f92c1c57717 ">+3/-3</a>
</td>
</tr>
<tr>
<td><strong>stream.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-2a60b2ae71e0a5c9dcceb588f262738f99df65fc2cb5b7002f2c37f87639e3b6 ">+2/-3</a>
</td>
</tr>
<tr>
<td><strong>websocket.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-6ff41099eded8eca4a0e579142d7af4429efb83350917780e0d3ae7dac13847a ">+16/-17</a>
</td>
</tr>
<tr>
<td><strong>ider.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-6ea42b8eab8ba0708f93f9a8b44056d9a9e6ac42517ffba12f81420d6b0bd2b0 ">+1/-4</a>
</td>
</tr>
<tr>
<td><strong>alert.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-45c7eaf1a764ab52f3e014fe0555c03dc59d77c5ce974684cdee4ed90463a234 ">+2/-5</a>
</td>
</tr>
<tr>
<td><strong>destinations.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-e0ba33d95ef7f0eac13bdac7e750462907a0308f01c027ea206d058e282c977b ">+1/-1</a>
</td>
</tr>
<tr>
<td><strong>inverted_index.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-8bf9d9403f05f53f0c6fd5e6a6dbf0204ea80465b233567552cb7395d08d1c3c ">+1/-1</a>
</td>
</tr>
<tr>
<td><strong>components.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-e2d647c422ae05ac91abc683c7b11a3fb7da61ee30968ca6e84cdba40221bbb9 ">+2/-0</a>
</td>
</tr>
<tr>
<td><strong>promql.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-0893731b3f0345378627894e3ea4a9f8e8735b38622b2b96ab55e81760d188db ">+1/-1</a>
</td>
</tr>
<tr>
<td><strong>ratelimit.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-fa7dc03fb526c09d5eb58021872d821fcd72eeb4d6a670224f5286012485bc3d ">+1/-4</a>
</td>
</tr>
<tr>
<td><strong>search.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-373e83a31e5bbc82fba05d7ae274b0dbd624ed748272232cf7eb8b8c3b2f3afc ">+22/-22</a>
</td>
</tr>
<tr>
<td><strong>sql.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-845dcf22be9d370a278c081df83a72bb8dc4b729f95c201839787088c82b57e7 ">+39/-39</a>
</td>
</tr>
<tr>
<td><strong>stream.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-2438d04c3fe97bc5452e5b74a303163b8723d93455cda32c95f0525435ad46bd ">+1/-1</a>
</td>
</tr>
<tr>
<td><strong>timed_annotations.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-f94b4d6d12d68e7054ad33601e666b08e4594586013dc34384fa21b1df5fe6e8 ">+4/-4</a>
</td>
</tr>
<tr>
<td><strong>async_file.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-8275f369cab85b5e9a97ed11d236de3364850b6989698936f19e70a0144e6a0d ">+19/-23</a>
</td>
</tr>
<tr>
<td><strong>download_utils.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-990aff76afde0d775c850302bf6399998aee73da05c76e719b5fbad6e7ed4b27 ">+3/-3</a>
</td>
</tr>
<tr>
<td><strong>file.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-405c4fb06d186c2423f78d42f3d195b12961877b091feff032f7df500450a72a ">+6/-9</a>
</td>
</tr>
<tr>
<td><strong>flatten.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-d13aed536f8b484d0223d7158e45e3a51ea181846965f73be42dedae975afcc7 ">+1/-1</a>
</td>
</tr>
<tr>
<td><strong>md5.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-f4d317c045b3df075e418dd6a7cd51f275befd1f2d15622afcb41c20024a8ee5 ">+1/-1</a>
</td>
</tr>
<tr>
<td><strong>prom_json_encoder.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-5c6b4c5bb9f3859855269fe2b06a05167469ebc2ac04b84fb45d8d92aba6b825 ">+1/-1</a>
</td>
</tr>
<tr>
<td><strong>query_select_utils.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-2396baca05d001b681c6cc302c2acc901d4dc1e7acf32dd1be6fa9a759578a46 ">+30/-30</a>
</td>
</tr>
<tr>
<td><strong>record_batch_ext.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-f84aa2842a164d6e85aa05a718159e87a325cac139c59e561f6f59313c6d7c5d ">+1/-2</a>
</td>
</tr>
<tr>
<td><strong>schema.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-2431bc157b9e51be2f16836a995b5ed52f41dded921a404bf2eebd506b49fdfb ">+1/-2</a>
</td>
</tr>
<tr>
<td><strong>cgroup.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-167dbe688715e2a299b86aa662fd5170b68e54ec0f6a04e0ebfa0492b3ec6b00 ">+11/-9</a>
</td>
</tr>
<tr>
<td><strong>net.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-f95051aabe608a9348aecbcc7f6866576f683f0e72cb4c84f7ed3e06c7e78b67 ">+7/-8</a>
</td>
</tr>
<tr>
<td><strong>time.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-7b7e5972c321c99c5147af56a71afc1136a7c75844b8442e207de60bc86b7ddf ">+4/-4</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-9a4dddd253d61f35b8d61d372477e24a4523f419aad2a63b226b19a584b8624f ">+2/-5</a>
</td>
</tr>
<tr>
<td><strong>event.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-689769712cbd6bd02beba078c1e1dfc188e2be39ab19a0e232b8bd12bc7f1e78 ">+11/-11</a>
</td>
</tr>
<tr>
<td><strong>ingest.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-27ffe683f384ff836c6aac746240d755bdf17f3c6e0a65d8ec1a1ff82aaf7d5c ">+7/-12</a>
</td>
</tr>
<tr>
<td><strong>jwt.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-232cab74f4cbf135af424744272e17732fbda8ac005d33c0acb4597e65479d02 ">+61/-62</a>
</td>
</tr>
<tr>
<td><strong>validator.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-f4d4b51574b378a8c372345ea48875834a4eadacfac33fa4d0ceca1e0ef96b35 ">+17/-18</a>
</td>
</tr>
<tr>
<td><strong>dashboards.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-250eeebeec9d4ac0ac271466ed83cd33fea3f121080491b340dc95d9af287259 ">+4/-4</a>
</td>
</tr>
<tr>
<td><strong>action.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-616df3014c553f763ea57e925f1bff7bb47f9467c632940d93a45b7b1e4c3a07 ">+5/-6</a>
</td>
</tr>
<tr>
<td><strong>deprecated.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-3a5d80eac45bff881716e2cf099f90130659576eca017ad238a61781d3341b7e ">+1/-1</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-e67f49865e1cd36775282c55ba0c0579a52ce7542f9d8dde450033f0c66a9f4c ">+2/-2</a>
</td>
</tr>
<tr>
<td><strong>ingest.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-24f250254ecfdbe004af9dc0ea23f8a2c0105ae0898690ed3e565089e5d8eef9 ">+2/-2</a>
</td>
</tr>
<tr>
<td><strong>loki.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-34691872632b619be8aadf255104f0e004fcf701dba98f88e231de5966eed099 ">+1/-1</a>
</td>
</tr>
<tr>
<td><strong>settings.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-1e807254f2faf8fd09b802f71b23412f203eed289033f96bfc62b15dd997db8c ">+5/-5</a>
</td>
</tr>
<tr>
<td><strong>around.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-93b3688de26f8f7c9dc0389763e9999a254bcded15b6ae8c8349f18aec83bd4f ">+9/-9</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-96630a0dbb6dd7b2b0248fa325c7c7041ce31340c32826675a4a70772e9d29f6 ">+19/-20</a>
</td>
</tr>
<tr>
<td><strong>multi_streams.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-c3233cdf1e68cb2455f91457719b9c2f477e735ee16546fdfe111e76768bd054 ">+17/-18</a>
</td>
</tr>
<tr>
<td><strong>search_inspector.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-f818db4ca10b411c3559cc2a2659c49a59a45ecde0e4a22f9ef023da0bf4a009 ">+27/-29</a>
</td>
</tr>
<tr>
<td><strong>search_job.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-d848b311eae318d198558c29a007c8973afb5f8867a31ec056f4e037dcd1b854 ">+4/-4</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-095d70f65b6e69bf7de0180fcbcddcd7c89c4cf00095d7b06f039a07e73916bb ">+2/-3</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-c10dd7b4ea20ffe5840c93a6ae4298aec0c32c8ba1b114192fda6c2bac1c41d1 ">+7/-7</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-ca09c9c9b020049bdbc3382c466bfcca80daac92e98760d9a9a76172abf34049 ">+10/-12</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-619eafba0f0509d905b021f4a16659010aca23c62cffd9cd3e6c2bbe204ce200 ">+14/-18</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-64694c886c809b6efe449e6eba9fbb33d31ce5a3c49cea1a30006fc1d9522665 ">+4/-7</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-56897e42f9436a640bc70fddc25686485ba8ae1223635b32d252a0333309f748 ">+4/-4</a>
</td>
</tr>
<tr>
<td><strong>session.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-6a1a862158668be26ad1575d7050eb9139bca3e469db56b32641b4c46d0f8144 ">+6/-7</a>
</td>
</tr>
<tr>
<td><strong>encoder.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-1825b00f221539b371b9fe8aa89f9cbc1cfb94f3b731f053e64c51fb5c5934cd ">+1/-2</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6 ">+2/-4</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-66207bd699343342402f88cdd331f4c549783754e4921d0c8d43e026e2d8e575 ">+12/-12</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-f8e74805561fa288a434e6e5ca8fb2c9068fb7f54a5a21f0a9bbba5f73249a5f ">+21/-23</a>
</td>
</tr>
<tr>
<td><strong>etcd.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-33bb9257c1a1b9162c8679191b8d8b75c0b7395f72291ce73d77247ff924a66e ">+20/-22</a>
</td>
</tr>
<tr>
<td><strong>postgres.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-1554e69f6d6c29812af88f0dcf093240b24ff85f9e292322c71cb19b0dda808d ">+30/-45</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-90c725d0e77a6e7d4bcc130cf5783848a5cf0efdb304b351cd63cacf8775fc36 ">+1/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-3df833bc826eb12e0819620d1fe316ef1ac352d293c11e57cc3df0da3c031e93 ">+10/-10</a>
</td>
</tr>
<tr>
<td><strong>postgres.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-bb24a30c9c3744eb31fb49e4271a9651d1f9a94a87455ac0074364b879af9651 ">+27/-24</a>
</td>
</tr>
<tr>
<td><strong>sqlite.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-85ff00530831fdaa4e13c05cb649f2e20479cce04a4b65ef78d131cdb89b6c71 ">+36/-35</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-bff7752b2aae0e9cef3ece1b9ddf17b9aa2f52f81fa777e6249039580aefbd69 ">+1/-1</a>
</td>
</tr>
<tr>
<td><strong>nats.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-b05729b03dff9a5e29eaf536d0f90e94a0c27a27d1a3a41b52f25dc6dc380138 ">+8/-23</a>
</td>
</tr>
<tr>
<td><strong>mysql.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-a275fb17137e6e7cd3dbc6432164136b20d1c2d4757eadaadd0b4a983473865a ">+14/-35</a>
</td>
</tr>
<tr>
<td><strong>postgres.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-a281dfd3124defa76bf1100c7e4993def0cb3049cf1ad0b1c513ea0afe522d08 ">+2/-3</a>
</td>
</tr>
<tr>
<td><strong>sqlite.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-07751d2c8c454417335fa9dd06507099d8b1eaea9b04af99857db6d9c619e69f ">+2/-3</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-386fd4fad1e48ab0a90516cab1947b1888b270b2fe912f30d2255cc38e6c519b ">+15/-12</a>
</td>
</tr>
<tr>
<td><strong>accounts.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-7ff63b9d4046282262c347d3458395eae5b6853cb004e7349a37ec2274465fe0 ">+4/-4</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-71f24f3312c19e01c4eedcd6a720893e01e64d176d2afbe1e7c6171a23059716 ">+4/-6</a>
</td>
</tr>
<tr>
<td><strong>remote.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-fea1d32f99821f1913607362236bd9e5b65cb2a5f8ce2c11a9a4956ac646b3d1 ">+1/-1</a>
</td>
</tr>
<tr>
<td><strong>dashboards.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-edd90c6ec720f676311aa31a635f954d559e63582f0b4c39cb9206bfb095737b ">+1/-1</a>
</td>
</tr>
<tr>
<td><strong>m20250109_092400_recreate_tables_with_ksuids.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-c16140f94722b17bff8753f09fa80ee0bbabc2b2f2f24b8800b459ced12fe0e4 ">+1/-1</a>
</td>
</tr>
<tr>
<td><strong>m20250113_144600_create_unique_folder_name_idx.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-1f6dc440bcd651434f5bd6a06a4ec7d3bd88760754edeeb8d38e861208a56cbd ">+1/-1</a>
</td>
</tr>
<tr>
<td><strong>m20250125_132500_populate_templates_table.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-fa1e504b43669656f6fd652e221e954f3f3da7fd2892d524a0e8772b6d738107 ">+1/-1</a>
</td>
</tr>
<tr>
<td><strong>m20250320_000001_remove_alert_name_unique_constraint.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-f4094b2945a877db5709045deafdd2883ff31fd671d0782da058af40e522d559 ">+10/-17</a>
</td>
</tr>
<tr>
<td><strong>m20250611_000003_populate_reports_scheduled_jobs.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-6567d3d0361c1a184ba7877795a8c3c4f73ea9807b3d4c13efe621449dd3f80f ">+8/-12</a>
</td>
</tr>
<tr>
<td><strong>search_jobs.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-b78a126c6e978d43e798fa2bcb56f7bb0132c107c716d4a9891c58b3c5a33ff9 ">+7/-8</a>
</td>
</tr>
<tr>
<td><strong>timed_annotations.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-4bc910d02f5ddc598e394d06172debe456c7db655cc16fef53eab55005880378 ">+14/-21</a>
</td>
</tr>
<tr>
<td><strong>partition.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-441e26dda30cadfa3cb1efbd9569aec96af8fdc779e4916814fb5f9477db3f6c ">+1/-1</a>
</td>
</tr>
<tr>
<td><strong>writer.rs</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-d1a9b1542cc2758afe4fa67abd454dd818718f4e1c7570303519af5a58d6301f ">+5/-5</a>
</td>
</tr>
<tr>
<td><strong>Additional files not shown</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7316/files#diff-2f328e4cd8dbe3ad193e49d92bcf045f47a6b72b1e9487d366f6b8288589b4ca "></a></td>
</tr>
</table></details></td></tr></tr></tbody></table>
___
> <details> <summary> Need help?</summary><li>Type <code>/help how to
...</code> in the comments thread for any questions about PR-Agent
usage.</li><li>Check out the <a
href="https://qodo-merge-docs.qodo.ai/usage-guide/ ">documentation</a>
for more information.</li></details>
2025-07-01 21:25:45 +08:00
40f932198a
feat: separate out ddl client and commands ( #6675 )
...
fixes https://github.com/openobserve/openobserve/issues/6621
This separates the functions which execute ddl statements, and only runs
them if certain entry in meta is mis-matching with what we have in our
code. Introduces a new env var `ZO_META_DDL_DSN` , which is optional and
falls back to the original DSN if not specified. Added a new client to
Postgres and Mysql both which uses this DDL DSN, and that is used only
for the ddl statements.
Also added another command to cli `upgrade-db` which runs the db
initialization functions.
For sqlite, there is no change, as it does not have concept of users.
Primarily tested on Postgres to validate that the behavior with
read-write and admin users is as expected.
`DB_SCHEMA_VERSION` is hardcoded in code, and devs are expected to
manually uptick it everytime we do any changes to db schema, similar to
how we maintain ofga.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com >
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com >
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com >
2025-05-01 16:56:37 +05:30
Hengfei Yang
ab9652e2fa
feat: add file_time strategy for cache gc ( #6489 )
2025-04-10 19:58:42 +08:00
Hengfei Yang
f8c5f012bd
feat: remove configure ZO_USAGE_ORG ( #6142 )
2025-03-01 21:41:39 +08:00
ebcb12c4ea
feat: add cipher keys support for encryption/decryption ( #5683 )
...
This adds support of cipher keys, along with corresponding udfs for
encrypting and decrypting fields from the log.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes: Enterprise Encryption and Key Management Features
- **New Features**
- Added enterprise-level cipher key management functionality.
- Introduced encryption and decryption capabilities for data protection.
- Implemented key storage and retrieval mechanisms.
- New Vue components for adding and managing cipher keys and encryption
mechanisms.
- Added a new settings page for managing cipher keys.
- Integrated advanced key management with multiple provider types.
- Added user-defined functions for encryption and decryption within
DataFusion.
- Introduced a new migration for creating the `cipher_keys` database
table.
- Enhanced HTTP request handling for cipher key operations.
- Added validation for encryption configurations and improved error
handling.
- New asynchronous tasks for managing cipher keys during initialization.
- Enhanced error handling and notifications for cipher key changes.
- **Security Enhancements**
- Added support for managing encryption keys across the platform.
- Implemented secure key storage and rotation mechanisms.
- Enhanced migration logic for cipher keys.
- **User Interface Updates**
- Enhanced user navigation with new tabs for cipher key management.
- Introduced stepper interface for key creation and management.
- Improved localization support for cipher key management.
- **Performance Improvements**
- Optimized query handling for encrypted data.
- Enhanced DataFusion query optimizer with cipher key support.
- **Compatibility**
- Enterprise feature flag controls advanced encryption capabilities.
- Supports multiple key storage and authentication types.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com >
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com >
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com >
Co-authored-by: Bhargav <BJP232004@GMAIL.COM >
Co-authored-by: Subhra264 <chakrabortysubhradeep556@gmail.com >
Co-authored-by: ShyamOOAI <shyam@openobserve.ai >
2025-01-23 18:13:45 +05:30
Patrick Sullivan
a03526eaef
feat: Decouple super_cluster from table via dependency injection ( #5698 )
...
This is the follow-up PR to [this other
PR](https://github.com/openobserve/o2-enterprise/pull/281#issue-2773724675 ).
Please see the other PR description for a detailed explanation of the
changes.
This PR copies the supercluster queue message handling code from
`o2-enterprise` which updates the database when different messages are
received. In this PR these message-handling functions are injected into
`o2-enterprise`'s various `SuperClusterQueue` structures so that the
`SuperClusterQueue`s can execute the logic to update the `openobserve`
database without the `o2-enterprise` crate needing a dependency on the
`table` module.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added comprehensive message processing capabilities for various
enterprise components including alerts, dashboards, distinct values,
meta, pipelines, schemas, search jobs, and short URLs
- Introduced a new super cluster queue system with modular message
handling and initialization
- **Chores**
- Updated library and main application to support enterprise feature
initialization
- Added conditional compilation for enterprise-specific modules
- **Infrastructure**
- Implemented robust error handling and logging for message processing
across different system components
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-01-14 18:04:10 -06:00
Prabhat Sharma and Taiming Liu
9ace66ac75
chore: rename zinc to OpenObserve ( #4807 )
...
rename zinc to OpenObserve
---------
Co-authored-by: Taiming Liu <90490946+taimingl@users.noreply.github.com >
2024-10-16 10:12:34 -07:00
Hengfei Yang
b35692a219
fix: use get_lock for mysql ( #3155 )
2024-04-04 15:08:05 +08:00
Hengfei Yang
e33ae00ea1
feat: merge dev into main ( #3016 )
2024-03-17 22:38:07 +08:00
Ankur Srivastava and oasisk
afc63ad76c
feat: inverted index v1 ( #2821 )
...
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com >
2024-02-29 16:26:27 +05:30
Hengfei Yang
644a475256
feat: compactor supports lock on stream level ( #2664 )
2024-02-09 09:26:18 +08:00
Ankur Srivastava
64c6e3848d
fix: RSS reduction due to lazily loading ua-parser ( #2400 )
...
UserAgent parser is currently used exclusively in RUM related
middleware. Loading it lazily when RUM data is ingested results in
significant RSS reduction at the startup of O2.
2024-01-05 13:21:21 +01:00
baerwang and Hengfei Yang
7b32681617
feat: cli import or export data ( #2071 )
...
feat #2056
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com >
2023-12-16 12:45:09 +08:00
Prabhat Sharma
5db4ca191a
change license to AGPL
2023-11-26 18:34:59 -08:00
Hengfei Yang
681f6e3af9
feat: support mysql for metadata ( #1968 )
2023-11-20 07:42:12 +08:00
Hengfei Yang
8b6d9325b5
feat: add grpc for router ( #1946 )
2023-11-16 09:29:42 +08:00
Hengfei Yang
76fb283191
ci: upgrade rust to nightly-2023-10-24 ( #1804 )
2023-10-30 02:04:43 -05:00
Hengfei Yang
af521fde54
fix: update license ( #1404 )
2023-08-21 15:37:50 +08:00
Ankur Srivastava
e2dd047bf5
refactor: move infra and meta inside common ( #1105 )
...
This is the first step towards splitting the repo into smaller sub
projects.
2023-07-06 12:32:07 +05:30
Ashish Kolhe
cbf2493f02
fix: add unit tests ( #137 )
...
* added UT
* added UT's
* added UT
* added UT's
* added line break
2023-02-22 11:17:22 +05:30
Hengfei Yang
4f9b682d33
style: add License Header ( #48 )
2023-02-13 11:15:23 +08:00
oasisk
147362cb40
zinc-observe initial commit
2023-02-02 09:03:11 +05:30
Hengfei Yang
Simha G
ByteBaker
Huaijin
Patrick Sullivan
Prabhat SharmaandTaiming Liu
Ankur Srivastavaandoasisk
baerwangandHengfei Yang
Ashish Kolhe