openobserve

mirror of https://github.com/openobserve/openobserve.git synced 2026-11-05 16:10:43 +00:00

Author	SHA1	Message	Date
hengfei yang	31d5e6de07	fix: optimize topk	2025-07-13 06:41:47 -07:00
Hengfei Yang	c5be35f071	fix: single node should push down the whole plan (#7528 )	2025-07-11 11:47:03 +08:00
Yashodhan Joshi Yashodhan Joshi Bhargav Taiming Liu Abhay Padamani Subhra264 sai nikhil kethe	9444447498	fix: Free trial rebase (#6922 ) Cherry pick of https://github.com/openobserve/openobserve/pull/6807 after cloud branch was merged into main --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Bhargav <BJP232004@GMAIL.COM> Co-authored-by: Taiming Liu <liutaiming3@gmail.com> Co-authored-by: Abhay Padamani <141122205+ktx-abhay@users.noreply.github.com> Co-authored-by: Subhra264 <chakrabortysubhradeep556@gmail.com> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>	2025-07-10 11:23:41 -07:00
udev	c66b273ee6	fix: remove ws (#7486 ) ### PR Type Enhancement ___ ### Description - Introduce HTTP/2 streaming search pipeline - Implement caching and partitioned query execution - Add delta processing and progress updates - Remove WebSocket support and update configs ___ ### Changes diagram ```mermaid flowchart LR A["process_search_stream_request"] -->\|check cache\| B["handle_cache_responses_and_deltas"] A -->\|no cache\| C["do_partitioned_search"] B --> D["StreamResponses to client"] C --> D ``` ___ ### Changes walkthrough 📝 <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>8 files</summary><table> <tr> <td><strong>execution.rs</strong><dd><code>Add partitioned streaming search execution</code>                              </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-75d432351865224f0b00d7cc819554c2ad5fa1f777e9c645d91a3fd5c1113bc0">+705/-0</a>  </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Initialize streaming search module</code>                                              </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-f236c6c880ed80233fd264e657d5ab658c16c7368bdc70e20f123862b7214374">+535/-0</a>  </td> </tr> <tr> <td><strong>cache.rs</strong><dd><code>Implement caching for streaming search</code>                                      </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-55769b7cb67b550daf45363c6d7d569b39d4a6235f6a68c091beb40f16b6511e">+485/-0</a>  </td> </tr> <tr> <td><strong>utils.rs</strong><dd><code>Add utilities for streaming search</code>                                              </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-20915cbb55c73e652f71128beb9898a02d9333115d15ecbf50b5df775923e0d4">+140/-0</a>  </td> </tr> <tr> <td><strong>sorting.rs</strong><dd><code>Refactor sorting strategy for streaming</code>                                    </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-138cca545e29b77f27da957a19d04bae58ee0c9f84b75509f48c2f0efe5000bb">+46/-48</a>  </td> </tr> <tr> <td><strong>search.rs</strong><dd><code>Add SearchResultType and SortStrategy enums</code>                            </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-a36a6683e327641569d8098ee0d4f13e894e1b9d51230dc3569abe42e87ba310">+26/-0</a>    </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Move search_stream into streaming module</code>                                  </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-c3b85ea662a885fbd39797968928b291cb59a58f53a03f7b7dfd7ba8b6a7985a">+2/-2</a>      </td> </tr> <tr> <td><strong>search_stream.rs</strong><dd><code>Use new streaming request handler</code>                                                </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-642de43ff6b19ae41087a9f1efc02d8b0b873809e0c92cbddfac1a30ef3f1184">+3/-2</a>      </td> </tr> </table></details></td></tr><tr><td><strong>Configuration changes</strong></td><td><details><summary>6 files</summary><table> <tr> <td><strong>config.rs</strong><dd><code>Rename WebSocket to HttpStreaming config</code>                                  </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+15/-30</a>  </td> </tr> <tr> <td><strong>router.rs</strong><dd><code>Remove ws route and adjust querier routes</code>                                </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-5227b263a6c37a040ee1e7482b6903ead2186dd71090ee70bdaf1c3b894487ad">+3/-28</a>    </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Drop websocket route handling</code>                                                        </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-d7987e62dcc7b1ec1bd4aeabef94f85d8c41b2ea33de5ab78660794fa4d51a9e">+1/-56</a>    </td> </tr> <tr> <td><strong>search.rs</strong><dd><code>Use HttpStreaming config chunk size</code>                                            </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-373e83a31e5bbc82fba05d7ae274b0dbd624ed748272232cf7eb8b8c3b2f3afc">+2/-2</a>      </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Switch to HttpStreaming in status endpoint</code>                              </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-ca09c9c9b020049bdbc3382c466bfcca80daac92e98760d9a9a76172abf34049">+1/-1</a>      </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Remove ws service endpoint</code>                                                              </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+1/-2</a>      </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>21 files</summary><table> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-d15434a27e2b8406811f5a707b605203b603dad4a262c0642326be860bd2b568">+0/-6</a>      </td> </tr> <tr> <td><strong>auth_tests.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-2688cbdc0b009b3f0dd829c69ab86c6a7d0778ab3e65e916a410a76aa6ae6480">+0/-31</a>    </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-2aa59ce4464e7dbabdb9367fc8bb8c1b489c914b8690577d4916870463ab12db">+0/-1</a>      </td> </tr> <tr> <td><strong>websocket.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-6ff41099eded8eca4a0e579142d7af4429efb83350917780e0d3ae7dac13847a">+0/-225</a>  </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-bab79e8da2d0d9f903cb1a6187d922c6de63d3e4561e8ce633660b664b4e859f">+0/-1</a>      </td> </tr> <tr> <td><strong>validator.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-f4d4b51574b378a8c372345ea48875834a4eadacfac33fa4d0ceca1e0ef96b35">+0/-5</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-a1cfd40da1ab6cd4db0181fd8664965034298047b3b7c3320c5385897a7c4aed">+0/-1</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-dd04c9099b3f291c7822f6e2adcd4197246c67b1610dbd70b09d225eb3b51e40">+0/-283</a>  </td> </tr> <tr> <td><strong>session.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-6a1a862158668be26ad1575d7050eb9139bca3e469db56b32641b4c46d0f8144">+0/-641</a>  </td> </tr> <tr> <td><strong>connection.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-33c5e604300998724046d4a095c0e45a517b3d68ee279c1f49e5ef827bc6a65e">+0/-631</a>  </td> </tr> <tr> <td><strong>error.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-23a4826e6d2d8afd1df90335d54d1bd31c917e8b5689c0aa0edba957f5308ee1">+0/-150</a>  </td> </tr> <tr> <td><strong>handler.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-eadc17f110edfa7f870ce3b66b2e0d0bf301624c76f3b7ebeede043b073dd020">+0/-753</a>  </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-50311955789888994478438442ded5daf30873021cebf4f55d478d03feba116e">+0/-104</a>  </td> </tr> <tr> <td><strong>pool.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-2b0e614010eda77b040ec03f7c9fc5e5feede7e846ae3e21b644e311b1b32956">+0/-118</a>  </td> </tr> <tr> <td><strong>session.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-cdd86aa2ec02449ae32de4b331178c6079dbbc48f4220b8feecf2d1f8e878b00">+0/-205</a>  </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-1b15316505c31acccb26ba66d920fdb0894567d08d2bf1e3457716d128a67b83">+0/-1</a>      </td> </tr> <tr> <td><strong>search_stream.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-b952a89375f2745a0daa273f1bdefc80c22997759a9b17a8fccbc70ae294e8e4">+0/-1667</a></td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-21232733827570717e2ef8988a4c9753e5006665cfe9a46c7b88261e44f4947c">+0/-23</a>    </td> </tr> <tr> <td><strong>search.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-13476631e2e604b75e2dfdd8fb7b7a91c46dcd85faa50813afa0b87583bcdb26">+0/-1271</a></td> </tr> <tr> <td><strong>utils.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-36caf9ef22ae6e1a217d51e023403ce24a1bd3d10b1bc6d5b86c982d6269bdbe">+0/-333</a>  </td> </tr> <tr> <td><strong>values.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7486/files#diff-d5c53f3f28432f096cb4788fa89c0e9bc8551aefe75c4a62be084a08d9025f97">+0/-326</a>  </td> </tr> </table></details></td></tr></tr></tbody></table> ___ > <details> <summary> Need help?</summary><li>Type <code>/help how to ...</code> in the comments thread for any questions about PR-Agent usage.</li><li>Check out the <a href="https://qodo-merge-docs.qodo.ai/usage-guide/">documentation</a> for more information.</li></details>	2025-07-10 13:43:32 +05:30
Ashish Kolhe	40aa8edd31	chore: clippy fixes (#7513 ) ### PR Type Enhancement ___ ### Description - Replace `{}` with named `{var}` in log messages - Standardize log formatting across modules - Improve readability and consistency - No functional changes ___ ### Changes walkthrough 📝 <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Formatting</strong></td><td><details><summary>20 files</summary><table> <tr> <td><strong>handler.rs</strong><dd><code>Switch log macros to named interpolation</code>                                  </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-eadc17f110edfa7f870ce3b66b2e0d0bf301624c76f3b7ebeede043b073dd020">+32/-67</a>  </td> </tr> <tr> <td><strong>mysql.rs</strong><dd><code>Update MySQL log formatting placeholders</code>                                  </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-06d02c738c6a4dc22ff022c16befef8d46aece6cfaa88bb4a5d3d5f92c2e50a1">+31/-36</a>  </td> </tr> <tr> <td><strong>nats.rs</strong><dd><code>Refactor NATS log interpolation syntax</code>                                      </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-2d6809ea34ea04477afb9a6f8287c93cef488b45ef8f0ebeec2241571b93994c">+16/-32</a>  </td> </tr> <tr> <td><strong>schema.rs</strong><dd><code>Standardize migration log messages</code>                                              </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-503e5382af7a29ec9c2819e32d468f6bbf9499da91cfdf43663def51c77d422b">+19/-34</a>  </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Refine OTLP trace handler logging</code>                                                </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-66f3b4f5e7927092fd6e57738687a1091b6f316638f7c58f8d7be9addb72bc30">+17/-36</a>  </td> </tr> <tr> <td><strong>parquet.rs</strong><dd><code>Update parquet job log interpolation</code>                                          </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-27f710981cfe1b938aeeb53abaed05961596c32e2edf3b8a2a59d3e72b2f542b">+18/-49</a>  </td> </tr> <tr> <td><strong>search.rs</strong><dd><code>Refactor search event log syntax</code>                                                  </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-13476631e2e604b75e2dfdd8fb7b7a91c46dcd85faa50813afa0b87583bcdb26">+15/-47</a>  </td> </tr> <tr> <td><strong>merge.rs</strong><dd><code>Improve compactor merge log formatting</code>                                      </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-0ab016dbd5129c9c285c9c459bf78928147bce97dc92a03aa754da3da71168d7">+14/-54</a>  </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Standardize compactor module logs</code>                                                </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-dce43be4d49b0bf99b61099d5b348bc539433d6e9b1ce6b7025d65e907f00b95">+10/-35</a>  </td> </tr> <tr> <td><strong>disk.rs</strong><dd><code>Refactor disk cache log messages</code>                                                  </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-af63f2c5d85313779df0204c9689e50a6c3916e3aadc24e54e23ec640a584fcb">+9/-9</a>      </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Update request WS handler logs</code>                                                      </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-dd04c9099b3f291c7822f6e2adcd4197246c67b1610dbd70b09d225eb3b51e40">+17/-27</a>  </td> </tr> <tr> <td><strong>postgres.rs</strong><dd><code>Standardize Postgres log interpolation</code>                                      </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-bb24a30c9c3744eb31fb49e4271a9651d1f9a94a87455ac0074364b879af9651">+17/-26</a>  </td> </tr> <tr> <td><strong>mysql.rs</strong><dd><code>Refine scheduler MySQL logs</code>                                                            </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-a275fb17137e6e7cd3dbc6432164136b20d1c2d4757eadaadd0b4a983473865a">+17/-23</a>  </td> </tr> <tr> <td><strong>wal.rs</strong><dd><code>Update WAL redo logging syntax</code>                                                      </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-8ba57e3a90cce57ceb88ea5215024c6d54473732bebdc7bf33847462379ec03e">+13/-22</a>  </td> </tr> <tr> <td><strong>users.rs</strong><dd><code>Refactor user service log messages</code>                                              </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-98da2588a28d62bb2e301d989836125f89f89b76272cd7f75e08ec4deeea5f38">+13/-18</a>  </td> </tr> <tr> <td><strong>jwt.rs</strong><dd><code>Improve JWT auth log interpolation</code>                                              </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-232cab74f4cbf135af424744272e17732fbda8ac005d33c0acb4597e65479d02">+9/-24</a>    </td> </tr> <tr> <td><strong>batch_execution.rs</strong><dd><code>Standardize pipeline debug logs</code>                                                    </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-3f6824a6eddaf1603e5cb0f1a4311ac6a45e1ffb967266ccc1f0bba944625e69">+16/-34</a>  </td> </tr> <tr> <td><strong>alert.rs</strong><dd><code>Refine alert scheduler log syntax</code>                                                </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-58f77a5eb00c01f123b13173c99cf9499af7458115e50811d07def0cf6e2d3a7">+12/-13</a>  </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Update OFGA migration logging</code>                                                        </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-1b8694fad31c8249afeb419d92a9facbd170289db2c89100552883df876bb275">+5/-10</a>    </td> </tr> <tr> <td><strong>schema.rs</strong><dd><code>Standardize schema service logs</code>                                                    </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-4696a44abde2df4ac48fed716f930bfdcc01759bdfcfd8d802821c6dd857bff9">+12/-14</a>  </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>101 files</summary><table> <tr> <td><strong>cli.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-cb1eb3e74723d891353ccf1e2e5e0d246d2e78a5e69dbe4c73006ea8412d65e0">+1/-1</a>      </td> </tr> <tr> <td><strong>etcd.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-37c2c2803d1c12ebb4ef9299bc79c9e6740fc5f19f7f4d84e7cfbb610b9561a9">+4/-4</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-d15434a27e2b8406811f5a707b605203b603dad4a262c0642326be860bd2b568">+6/-6</a>      </td> </tr> <tr> <td><strong>nats.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-901773980aa255942e7865b977fe63811f31511ac40bc512d6bbb14dbb54bece">+8/-8</a>      </td> </tr> <tr> <td><strong>config.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-cda0d9179ed982fa5f0989c9b26f21cfa49dd1de28195a98135650ffcc7c602a">+12/-18</a>  </td> </tr> <tr> <td><strong>alert_folders.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-6ed0a38075313ced0916343ee6e6b86ab639f8ed59e4968df395aa30d72bd300">+10/-10</a>  </td> </tr> <tr> <td><strong>wal.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-4928ae81597b28facdc3644f786ac14c60b0a406e1c7b8eac7232fbafbcbb6e3">+2/-2</a>      </td> </tr> <tr> <td><strong>telemetry.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-3ff026b46d9431e0657828a3bde4ab118fec006b353c2abc80be4b74633c98b4">+3/-3</a>      </td> </tr> <tr> <td><strong>auth.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-9bc2b1841504a435f66ad094f72ddbe7b7fa75e5e1557b8adbfe36f7f0592e1b">+2/-2</a>      </td> </tr> <tr> <td><strong>ider.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-6ea42b8eab8ba0708f93f9a8b44056d9a9e6ac42517ffba12f81420d6b0bd2b0">+1/-1</a>      </td> </tr> <tr> <td><strong>search.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-373e83a31e5bbc82fba05d7ae274b0dbd624ed748272232cf7eb8b8c3b2f3afc">+2/-3</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-2ba3bceb0d80c8008da06e45374c1f744441375e2c28b55e6a55a1be3337e79b">+1/-1</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-9a4dddd253d61f35b8d61d372477e24a4523f419aad2a63b226b19a584b8624f">+4/-7</a>      </td> </tr> <tr> <td><strong>event.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-689769712cbd6bd02beba078c1e1dfc188e2be39ab19a0e232b8bd12bc7f1e78">+6/-6</a>      </td> </tr> <tr> <td><strong>traces.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-38e83bff9d1b99d37475a02c28cf9b2b7fd4d84d366a1eade3928a5d10b471f8">+1/-1</a>      </td> </tr> <tr> <td><strong>token.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-3339b4a964c121766ef87fcec0070f27a2b7085cf81281b53154e53d33cb28cf">+1/-1</a>      </td> </tr> <tr> <td><strong>validator.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-f4d4b51574b378a8c372345ea48875834a4eadacfac33fa4d0ceca1e0ef96b35">+6/-7</a>      </td> </tr> <tr> <td><strong>ai.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-c7e88d14798788e6186cadbb1cef3386dad2e585af6c36fd7b9cab3dbef2e46b">+2/-2</a>      </td> </tr> <tr> <td><strong>reports.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-00435e0d9301342b51cb57ae313f1413e7ed973642e0badf3f3959125d0677d1">+1/-1</a>      </td> </tr> <tr> <td><strong>timed_annotations.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-b6a4ef9157a0dbf25b875111f7cdb230d3d0f2d7a430d335de6c6af82f95b365">+5/-5</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-72fb8ce13ff14a37ab38d48e5b452205b30c8a8b544badf42579fec4fe5f90fd">+2/-2</a>      </td> </tr> <tr> <td><strong>ingest.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-24f250254ecfdbe004af9dc0ea23f8a2c0105ae0898690ed3e565089e5d8eef9">+4/-10</a>    </td> </tr> <tr> <td><strong>loki.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-34691872632b619be8aadf255104f0e004fcf701dba98f88e231de5966eed099">+4/-8</a>      </td> </tr> <tr> <td><strong>org.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-b47c0c1a8a78524196176430268eecdbfaf6f46d6b95873d3cbe70cfd67182b0">+2/-10</a>    </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-0ad5729a1fbce00ff53076728c51822400c13af9918e1d8554f41be56d060fa0">+6/-6</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-8ce79cea5617c3a661e5b64058ad3846abf8652bb44fbb88dfb9117ffcc577fd">+1/-1</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-96630a0dbb6dd7b2b0248fa325c7c7041ce31340c32826675a4a70772e9d29f6">+9/-9</a>      </td> </tr> <tr> <td><strong>multi_streams.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-c3233cdf1e68cb2455f91457719b9c2f477e735ee16546fdfe111e76768bd054">+5/-5</a>      </td> </tr> <tr> <td><strong>search_inspector.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-f818db4ca10b411c3559cc2a2659c49a59a45ecde0e4a22f9ef023da0bf4a009">+2/-2</a>      </td> </tr> <tr> <td><strong>search_job.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-d848b311eae318d198558c29a007c8973afb5f8867a31ec056f4e037dcd1b854">+1/-1</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-ca09c9c9b020049bdbc3382c466bfcca80daac92e98760d9a9a76172abf34049">+1/-1</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-619eafba0f0509d905b021f4a16659010aca23c62cffd9cd3e6c2bbe204ce200">+2/-6</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-64694c886c809b6efe449e6eba9fbb33d31ce5a3c49cea1a30006fc1d9522665">+2/-2</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-56897e42f9436a640bc70fddc25686485ba8ae1223635b32d252a0333309f748">+2/-2</a>      </td> </tr> <tr> <td><strong>service_accounts.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-8d58793d63492cb2b4ec77288a91b0cf995a56cac7ea6e0faa9f006aa105b368">+1/-1</a>      </td> </tr> <tr> <td><strong>session.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-6a1a862158668be26ad1575d7050eb9139bca3e469db56b32641b4c46d0f8144">+6/-16</a>    </td> </tr> <tr> <td><strong>slow_log.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-45522a89e42bfd2029972622703bbf58b40d16fc7a7c4346ac270082728661ec">+1/-7</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-66207bd699343342402f88cdd331f4c549783754e4921d0c8d43e026e2d8e575">+11/-11</a>  </td> </tr> <tr> <td><strong>delete.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-9f484d99b002885e92d01b21c11686f8e99077cf5dd422b673c6877db15f2f8d">+1/-1</a>      </td> </tr> <tr> <td><strong>memory.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-ff00d697a9b223ef2858c5e41111a8918e7cc347dc22891c02bda6982682eee1">+4/-7</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-f8e74805561fa288a434e6e5ca8fb2c9068fb7f54a5a21f0a9bbba5f73249a5f">+2/-8</a>      </td> </tr> <tr> <td><strong>alerts.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-9a2bb454050f39797fc752caa4a72da28f398c76f2a7f54f74d1c293d3b27698">+1/-1</a>      </td> </tr> <tr> <td><strong>etcd.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-33bb9257c1a1b9162c8679191b8d8b75c0b7395f72291ce73d77247ff924a66e">+6/-9</a>      </td> </tr> <tr> <td><strong>mysql.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-412dc407af348f4b54b7d98de6a928876b0333ea501792498a0568850f83d7b7">+10/-10</a>  </td> </tr> <tr> <td><strong>postgres.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-1554e69f6d6c29812af88f0dcf093240b24ff85f9e292322c71cb19b0dda808d">+13/-13</a>  </td> </tr> <tr> <td><strong>sqlite.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-85f17b22460902614e5c28df8966ea3e7073325bd8dd43fa396ed9268b6b4b06">+5/-11</a>    </td> </tr> <tr> <td><strong>grpc.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-211c5cb8fa12eac14c3637a43d74cd114034b8a6f6f8c1ead51e56efb54141cd">+1/-1</a>      </td> </tr> <tr> <td><strong>sqlite.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-85ff00530831fdaa4e13c05cb649f2e20479cce04a4b65ef78d131cdb89b6c71">+4/-9</a>      </td> </tr> <tr> <td><strong>mysql.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-836ef77752dcfeba5e1161c2c42e92ec651d9a9131f12d48fb390fe3d861ab53">+8/-8</a>      </td> </tr> <tr> <td><strong>postgres.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-122e67af3f562870a16eff139f68d5c126841cabe5ebefe2c729530d676e883f">+8/-11</a>    </td> </tr> <tr> <td><strong>sqlite.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-e81d5c9c0ac9af5dc99c66eca0cfa1523f34063a33f75f6b7c1e4fc0d1802ed6">+8/-8</a>      </td> </tr> <tr> <td><strong>postgres.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-a281dfd3124defa76bf1100c7e4993def0cb3049cf1ad0b1c513ea0afe522d08">+9/-9</a>      </td> </tr> <tr> <td><strong>sqlite.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-07751d2c8c454417335fa9dd06507099d8b1eaea9b04af99857db6d9c619e69f">+5/-5</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-386fd4fad1e48ab0a90516cab1947b1888b270b2fe912f30d2255cc38e6c519b">+1/-1</a>      </td> </tr> <tr> <td><strong>local.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-8e49c3ffac3d43c06780a18937b7ecd7943a8872ec81dca39d4497d9f94e33a7">+4/-7</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-71f24f3312c19e01c4eedcd6a720893e01e64d176d2afbe1e7c6171a23059716">+3/-3</a>      </td> </tr> <tr> <td><strong>remote.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-fea1d32f99821f1913607362236bd9e5b65cb2a5f8ce2c11a9a4956ac646b3d1">+6/-13</a>    </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-238b8087501b1743c3c4608469596d431869f021e9bb30e9a058d1294229ee67">+1/-1</a>      </td> </tr> <tr> <td><strong>m20241227_000100_populate_organizations_table.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-53abf79ce8f9c17107749f25c5a544591903422b918e322643068a38eaee6050">+1/-1</a>      </td> </tr> <tr> <td><strong>m20241227_000400_populate_users_table.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-fd66003231079d3fcf3681ab7a57d64b61dea22e55a3c22ebe5527d095a52b5e">+2/-2</a>      </td> </tr> <tr> <td><strong>m20250320_000001_remove_alert_name_unique_constraint.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-f4094b2945a877db5709045deafdd2883ff31fd671d0782da058af40e522d559">+1/-1</a>      </td> </tr> <tr> <td><strong>m20250611_000002_populate_reports_table.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-8247af6d6f8ccb6a5a51daff4cb1e352585be2d1c785d86ae5ebfea1dbc1099c">+2/-2</a>      </td> </tr> <tr> <td><strong>m20250611_000003_populate_reports_scheduled_jobs.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-6567d3d0361c1a184ba7877795a8c3c4f73ea9807b3d4c13efe621449dd3f80f">+3/-6</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-af023301203a27a1ef34f698d5a329ad7bfced65ac65e3829b1c0905a0c74fb2">+1/-2</a>      </td> </tr> <tr> <td><strong>org_users.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-8e62ecba19308fdaaecfdb3d1f23021c926e244a510d9452dfbd71ac8f35ee9f">+1/-1</a>      </td> </tr> <tr> <td><strong>organizations.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-4d87f9fe379a5d161a3e7c3c77f7f320be738e6a70c785aa5ce93cb0830c0147">+1/-1</a>      </td> </tr> <tr> <td><strong>ratelimit.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-231f9b18e0d61409b6e3348196dd4d4456847ebb25e1822eec52e7e09b00b5ef">+1/-1</a>      </td> </tr> <tr> <td><strong>queries.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-144c4572241c5b3f13abe628832674ad8cebb72e51acb1d1b14fae2a8307208a">+1/-1</a>      </td> </tr> <tr> <td><strong>search_job_partitions.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-a72ad2e06cfec317bd6901681b9b7c8a27b4b5736e6340f58d711296222ffee1">+1/-1</a>      </td> </tr> <tr> <td><strong>short_urls.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-5e9b140c2fe41290633080bed471cd7473165119bc1fe84bd69853fec91d7a3a">+1/-1</a>      </td> </tr> <tr> <td><strong>timed_annotations.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-4bc910d02f5ddc598e394d06172debe456c7db655cc16fef53eab55005880378">+1/-1</a>      </td> </tr> <tr> <td><strong>users.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-a8183f71611dc4c9329231bb5083ceb43a19f43011f86be5c5c9e47240d2073f">+1/-1</a>      </td> </tr> <tr> <td><strong>lib.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-603267e2f712eeb4bc46fdff981f1fad1f3a9d3b989a7db4b3c0984fc5b1ed43">+4/-4</a>      </td> </tr> <tr> <td><strong>partition.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-441e26dda30cadfa3cb1efbd9569aec96af8fdc779e4916814fb5f9477db3f6c">+1/-1</a>      </td> </tr> <tr> <td><strong>writer.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-d1a9b1542cc2758afe4fa67abd454dd818718f4e1c7570303519af5a58d6301f">+7/-10</a>    </td> </tr> <tr> <td><strong>alert_manager.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-7e30858abad652a860186eb047347305488f52356112b1495dd53ce740316d7c">+5/-8</a>      </td> </tr> <tr> <td><strong>file_downloader.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-dcd9d92e9718a63ee7341b7239f03d9f55746a0f848537e4982661e9219ea05c">+7/-25</a>    </td> </tr> <tr> <td><strong>broadcast.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-930f8671c8be6478fe4739b46af3ff38994f14a558a2003cc136060a7c790687">+1/-1</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-30f42c3685037c219f0033617f032b23a7f74624ba4ae6c4e3687af401304591">+1/-1</a>      </td> </tr> <tr> <td><strong>metrics.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-ca98d90d1293682c98c33a777a653cf3c1466f6297969ae94a359bb81abca4b1">+3/-3</a>      </td> </tr> <tr> <td><strong>mmdb_downloader.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-d36dee2ad5f79654acc9d5d5c3b95a552a9b397f97c25c2e23a19195ddc7fe86">+4/-12</a>    </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-f2e4c29b8eef22189ec704a832633c4bd135a24444f054864c98064bd0963681">+1/-1</a>      </td> </tr> <tr> <td><strong>promql.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-15414a41d64eb87bc5c0cd00a07213b3f99c31b346e8e2c36178f7276a1f058c">+1/-1</a>      </td> </tr> <tr> <td><strong>stats.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-f3ca67d9d2c68beeafd6e5de947dbc9a648ac2d2699c5dc87458c369887d09b9">+2/-5</a>      </td> </tr> <tr> <td><strong>syslog_server.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-cb7610142582b44885f673583ab16262c3b9ca6de1f59a9def65e503360665a0">+1/-1</a>      </td> </tr> <tr> <td><strong>main.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-42cb6807ad74b3e201c5a7ca98b911c5fa08380e942be6e4ac5807f8377f87fc">+8/-23</a>    </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-eb855be0fa22d1a540379f6142cc4a473d49f89c3e0fe16cfd5078d5a69f9a3b">+1/-1</a>      </td> </tr> <tr> <td><strong>pipeline_func.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-701a2afe5253e4ef729a1b93372e69d79ada9cf9d1f0f5a8556bc4841f46f301">+1/-4</a>      </td> </tr> <tr> <td><strong>report.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-a8d5d0e0fc56e67b38a20a46b91cc2e6bf76ade360a37ad29fce51cc4661a9a6">+1/-2</a>      </td> </tr> <tr> <td><strong>server.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-804074c17dbf0e9757a57ade0cd81a00518a14204762cbbe3fa44bfb43936dc8">+1/-1</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-d7987e62dcc7b1ec1bd4aeabef94f85d8c41b2ea33de5ab78660794fa4d51a9e">+2/-2</a>      </td> </tr> <tr> <td><strong>connection.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-33c5e604300998724046d4a095c0e45a517b3d68ee279c1f49e5ef827bc6a65e">+5/-17</a>    </td> </tr> <tr> <td><strong>resource_extractor.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-6bf05d4ebced31bf58505b67b2362abd833c00422743650218998ed6bc4b45af">+2/-7</a>      </td> </tr> <tr> <td><strong>alert.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-574be3504bf4d855c65fd3d7a10797a970237ff552edac4962b474b3983e094e">+1/-4</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-ba9b65130898444f7815fd6068bdb0dc590612f74cc893f79d91b0afc44cc851">+3/-7</a>      </td> </tr> <tr> <td><strong>handlers.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-5eca8995a88fcecb6d6b8efd3c7c4496f4a96a4b060fb4ed85593574f7cf49eb">+8/-17</a>    </td> </tr> <tr> <td><strong>worker.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-3d37991258d3f26b16ad99bba59b63adfa614bfbc4866c4088544ca9308ae49d">+10/-22</a>  </td> </tr> <tr> <td><strong>deleted.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-4900b3bce3c590ac846c7f12e82e1d1cf7c2de3de9fc385fdbe53deb8a8e50ab">+4/-4</a>      </td> </tr> <tr> <td><strong>flatten.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-93a66ecfe3a7fdfe5509dda69e4d444f02b18c841c34ab3078f1fb314af8e3ca">+1/-5</a>      </td> </tr> <tr> <td><strong>retention.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-6abd048f5ef4e7163cffbb51cb4bee10c07fb88b871469d5b3724f7cd04743dc">+9/-29</a>    </td> </tr> <tr> <td><strong>Additional files not shown</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7513/files#diff-2f328e4cd8dbe3ad193e49d92bcf045f47a6b72b1e9487d366f6b8288589b4ca"></a></td> </tr> </table></details></td></tr></tr></tbody></table> ___ > <details> <summary> Need help?</summary><li>Type <code>/help how to ...</code> in the comments thread for any questions about PR-Agent usage.</li><li>Check out the <a href="https://qodo-merge-docs.qodo.ai/usage-guide/">documentation</a> for more information.</li></details>	2025-07-09 19:16:13 +05:30
Yashodhan JoshiandYashodhan Joshi	babe3b08fb	feat: support path based value extraction in cipher functions (#7483 ) ### User description cherry-pick of https://github.com/openobserve/openobserve/pull/7482 for main ___ ### PR Type Enhancement, Tests ___ ### Description - Add JSON path extraction utility - Extend cipher UDFs with path argument - Rewrite optimizer to handle paths - Add tests for path-based ciphers ___ ### Changes diagram ```mermaid flowchart LR J["get_value_from_path"] --> U["cipher UDFs"] U --> O["optimizer rewrite"] O --> T["path-based cipher tests"] ``` ___ ### Changes walkthrough 📝 <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table> <tr> <td> <details> <summary><strong>auth_tests.rs</strong><dd><code>Remove `batch_size` default in tests</code>                                          </dd></summary> <hr> src/common/utils/auth_tests.rs - Remove deprecated `batch_size` default </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/7483/files#diff-2688cbdc0b009b3f0dd829c69ab86c6a7d0778ab3e65e916a410a76aa6ae6480">+0/-1</a>      </td> </tr> </table></td></tr><tr><td><strong>Enhancement</strong></td><td><table> <tr> <td> <details> <summary><strong>json.rs</strong><dd><code>Implement JSON path extraction utility</code>                                      </dd></summary> <hr> src/config/src/utils/json.rs <li>Import <code>itertools</code> for flattening<br> <li> Add <code>get_value_from_path</code> implementation<br> <li> Handle nested JSON and wildcard paths<br> <li> Add unit tests for JSON path extraction </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/7483/files#diff-f638bd6d218ad53a671681c2fa8e420cfbf8369e6e93269c9559e9711338d52a">+131/-0</a>  </td> </tr> <tr> <td> <details> <summary><strong>cipher.rs</strong><dd><code>Support path argument in optimizer cipher rewrite</code>                </dd></summary> <hr> src/service/search/datafusion/optimizer/cipher.rs <li>Require only two args for rewrite checks<br> <li> Update sanity checks to allow optional path<br> <li> Preserve or default path in <code>CipherKeyRewrite</code><br> <li> Add tests for path-based cipher calls </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/7483/files#diff-1dfdd6163a17d67d426ab189446fa4c43551094aea50ffd82d334abd3ffe4f81">+40/-11</a>  </td> </tr> <tr> <td> <details> <summary><strong>cipher_udf.rs</strong><dd><code>Extend cipher UDFs with JSON path support</code>                                </dd></summary> <hr> src/service/search/datafusion/udf/cipher_udf.rs <li>Extend UDFs to accept three parameters<br> <li> Validate third <code>path</code> argument in UDFs<br> <li> Parse JSON and extract value at <code>path</code><br> <li> Update encryption/decryption logic for arrays </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/7483/files#diff-aca30595e1e899bf83b213e84d91063cc621a8972ce22e67bcf4705af817188f">+175/-19</a></td> </tr> <tr> <td> <details> <summary><strong>sql.rs</strong><dd><code>Require three arguments in SQL cipher parser</code>                          </dd></summary> <hr> src/service/search/sql.rs <li>Update cipher function argument count to three<br> <li> Adjust SQL parser validation for paths </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/7483/files#diff-d9ccf393c53794b0ef2816eef12742c26e23d2788cde27780af7570662aea446">+1/-1</a>      </td> </tr> </table></td></tr></tr></tbody></table> ___ > <details> <summary> Need help?</summary><li>Type <code>/help how to ...</code> in the comments thread for any questions about PR-Agent usage.</li><li>Check out the <a href="https://qodo-merge-docs.qodo.ai/usage-guide/">documentation</a> for more information.</li></details> --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>	2025-07-09 15:51:32 +05:30
ByteBaker	25ce21cc3e	chore: clippy delints (#7446 )	2025-07-06 16:45:54 +08:00
ByteBaker	bdb4338903	chore: add UT for CI (#7464 ) ### PR Type Tests ___ ### Description - Add unit tests for email validation - Add unit tests for OFGA unsupported check - Add tests for OFGA format conversion - Add tests to validate if a user is part of organization - Remove unused invite token stub ___ ### Changes diagram ```mermaid flowchart LR validTest["Test valid emails"] -- "calls" --> isValidEmail["is_valid_email"] ofgaTest["Test OFGA unsupported"] -- "calls" --> isOfgaUnsupported["is_ofga_unsupported"] formatTest["Test OFGA format"] -- "calls" --> intoFormat["into_ofga_supported_format"] userOrgTest["Test user from org"] -- "calls" --> isUserFromOrg["is_user_from_org"] ``` ___ ### Changes walkthrough 📝 <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table> <tr> <td> <details> <summary><strong>auth.rs</strong><dd><code>Add tests and cleanup in auth utils</code>                                            </dd></summary> <hr> src/common/utils/auth.rs <li>Removed stub <code>generate_invite_token</code><br> <li> Added <code>test_valid_emails</code> unit tests<br> <li> Added <code>test_is_ofga_unsupported</code> unit tests<br> <li> Added <code>test_into_ofga_supported_format</code> unit tests<br> <li> Updated <code>println!</code> macros to use inline variables </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/7464/files#diff-9bc2b1841504a435f66ad094f72ddbe7b7fa75e5e1557b8adbfe36f7f0592e1b">+36/-9</a>    </td> </tr> </table></td></tr></tr></tbody></table> ___ > <details> <summary> Need help?</summary><li>Type <code>/help how to ...</code> in the comments thread for any questions about PR-Agent usage.</li><li>Check out the <a href="https://qodo-merge-docs.qodo.ai/usage-guide/">documentation</a> for more information.</li></details>	2025-07-05 10:59:39 +05:30
Hengfei Yang	f88fcb6302	feat: sync rc9 to main(BE) (#7447 )	2025-07-04 11:34:23 +08:00
Subhra264	ab5c296f50	fix: openfga migration for ai (#7434 ) ### User description This pr adds the migration needed for ai RBAC controls. ___ ### PR Type Enhancement ___ ### Description - Add AI chat permissions migration flow - Introduce version v0.0.17→v0.0.18 check - Populate AI ownership tuples per org - Require authentication for `/ai` endpoints ___ ### Changes walkthrough 📝 <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table> <tr> <td> <details> <summary><strong>mod.rs</strong><dd><code>Add AI chat permissions migration logic</code>                                    </dd></summary> <hr> src/common/infra/ofga/mod.rs <li>Added <code>need_ai_chat_permissions_migration</code> flag<br> <li> Defined versions v0.0.17 and v0.0.18<br> <li> Checked meta vs existing model for AI migration<br> <li> Included AI ownership tuple during migration </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/7434/files#diff-1b8694fad31c8249afeb419d92a9facbd170289db2c89100552883df876bb275">+10/-0</a>    </td> </tr> </table></td></tr><tr><td><strong>Bug fix</strong></td><td><table> <tr> <td> <details> <summary><strong>auth.rs</strong><dd><code>Enforce authentication on AI endpoints</code>                                      </dd></summary> <hr> src/common/utils/auth.rs <li>Removed skip-auth condition for AI paths<br> <li> Enforced authentication on <code>/ai</code> routes </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/7434/files#diff-9bc2b1841504a435f66ad094f72ddbe7b7fa75e5e1557b8adbfe36f7f0592e1b">+0/-1</a>      </td> </tr> </table></td></tr></tr></tbody></table> ___ > <details> <summary> Need help?</summary><li>Type <code>/help how to ...</code> in the comments thread for any questions about PR-Agent usage.</li><li>Check out the <a href="https://qodo-merge-docs.qodo.ai/usage-guide/">documentation</a> for more information.</li></details>	2025-07-03 13:20:03 +05:30
Huaijin	5883edc5d3	feat: upgrade datafusion to v47.0.0 (#7421 )	2025-07-02 18:48:58 +08:00
Huaijin	b141504cf8	chore: upgrade rust to 2025-05-20 (#7316 ) ### PR Type Enhancement ___ ### Description - Use `if let ... && let ...` patterns - Inline named args in `format!` calls - Simplify visitor `visit` calls with `_ =` - Replace string interpolation in logging macros ___ ### Changes walkthrough 📝 <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>5 files</summary><table> <tr> <td><strong>sql.rs</strong><dd><code>Refactor SQL visitor and limit logic</code>                                          </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-d9ccf393c53794b0ef2816eef12742c26e23d2788cde27780af7570662aea446">+249/-255</a></td> </tr> <tr> <td><strong>mysql.rs</strong><dd><code>Refactor time_range early-return patterns</code>                                </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-06d02c738c6a4dc22ff022c16befef8d46aece6cfaa88bb4a5d3d5f92c2e50a1">+51/-51</a>  </td> </tr> <tr> <td><strong>sql.rs</strong><dd><code>Expand UDF list and simplify visitors</code>                                        </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-a0a09c66a20d878ec7683ae222c919fbe092b8001871ade7f455b89573ce0911">+43/-39</a>  </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Refactor error mapping with `std::io::Error::other`</code>            </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-8ce79cea5617c3a661e5b64058ad3846abf8652bb44fbb88dfb9117ffcc577fd">+20/-36</a>  </td> </tr> <tr> <td><strong>engine.rs</strong><dd><code>Simplify unary ops and format messages</code>                                      </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-a98ac5b55a4c9440c09c4e2dfa3b6f7f24f9e58fcb6bf8b10577a0b4f123ed41">+49/-63</a>  </td> </tr> </table></details></td></tr><tr><td><strong>Formatting</strong></td><td><details><summary>4 files</summary><table> <tr> <td><strong>mysql.rs</strong><dd><code>Inline format! and logging braces</code>                                                </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-412dc407af348f4b54b7d98de6a928876b0333ea501792498a0568850f83d7b7">+59/-76</a>  </td> </tr> <tr> <td><strong>nats.rs</strong><dd><code>Simplify error mapping and formatting</code>                                        </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-2d6809ea34ea04477afb9a6f8287c93cef488b45ef8f0ebeec2241571b93994c">+49/-54</a>  </td> </tr> <tr> <td><strong>sqlite.rs</strong><dd><code>Use brace shorthand in SQL strings</code>                                              </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-85f17b22460902614e5c28df8966ea3e7073325bd8dd43fa396ed9268b6b4b06">+45/-56</a>  </td> </tr> <tr> <td><strong>action.rs</strong><dd><code>Use brace shorthand in Display impls</code>                                          </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-fc87c75a7df9539d7e7ee6184dfab02d4a8932323e81f95d57c05363aa44752d">+4/-4</a>      </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>101 files</summary><table> <tr> <td><strong>Cargo.toml</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542">+1/-1</a>      </td> </tr> <tr> <td><strong>clippy.toml</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-f3dbfb2563c3490394f44c26b51837018e79a79e75fd31e89b19e943a38317ea">+1/-0</a>      </td> </tr> <tr> <td><strong>rust-toolchain.toml</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-2b1bde2cf3a858b7bf7424cb8bcbf01f35b94dc80b925d9432cbab3319ca9b4e">+1/-1</a>      </td> </tr> <tr> <td><strong>cli.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-cb1eb3e74723d891353ccf1e2e5e0d246d2e78a5e69dbe4c73006ea8412d65e0">+8/-8</a>      </td> </tr> <tr> <td><strong>http.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-790b7780a51e8eecd4754b623af8d15ad87baaaf29455ba646e2e410f83f5498">+1/-1</a>      </td> </tr> <tr> <td><strong>load.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-bfc7e9b23a8fa100bcad8490d697a3090f139440fcd2ce02fb539c1791100d50">+4/-4</a>      </td> </tr> <tr> <td><strong>export.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-b9ca11eac019064e747b0c75157c280c8e8fa6d75edb83ca9ae46777e9668cc0">+2/-5</a>      </td> </tr> <tr> <td><strong>import.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-e4415a147f0fe6e557c9fee2a5cad9058a46c30cfb79fe131d13802c0c8bb804">+2/-2</a>      </td> </tr> <tr> <td><strong>etcd.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-37c2c2803d1c12ebb4ef9299bc79c9e6740fc5f19f7f4d84e7cfbb610b9561a9">+4/-4</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-d15434a27e2b8406811f5a707b605203b603dad4a262c0642326be860bd2b568">+4/-4</a>      </td> </tr> <tr> <td><strong>nats.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-901773980aa255942e7865b977fe63811f31511ac40bc512d6bbb14dbb54bece">+4/-4</a>      </td> </tr> <tr> <td><strong>alert_folders.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-6ed0a38075313ced0916343ee6e6b86ab639f8ed59e4968df395aa30d72bd300">+3/-3</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-1b8694fad31c8249afeb419d92a9facbd170289db2c89100552883df876bb275">+4/-4</a>      </td> </tr> <tr> <td><strong>wal.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-4928ae81597b28facdc3644f786ac14c60b0a406e1c7b8eac7232fbafbcbb6e3">+5/-5</a>      </td> </tr> <tr> <td><strong>loki.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-121c166af5eab8b9cb83716afa7b65fa7856c006c809b3d910ced184c52ec587">+8/-8</a>      </td> </tr> <tr> <td><strong>telemetry.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-3ff026b46d9431e0657828a3bde4ab118fec006b353c2abc80be4b74633c98b4">+2/-2</a>      </td> </tr> <tr> <td><strong>auth.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-9bc2b1841504a435f66ad094f72ddbe7b7fa75e5e1557b8adbfe36f7f0592e1b">+39/-43</a>  </td> </tr> <tr> <td><strong>auth_tests.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-2688cbdc0b009b3f0dd829c69ab86c6a7d0778ab3e65e916a410a76aa6ae6480">+1/-0</a>      </td> </tr> <tr> <td><strong>functions.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-f4df68dbcf485d6b98c0e673c12a19be2edab416e5fc22c7d08396d375cd9126">+1/-1</a>      </td> </tr> <tr> <td><strong>http.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-3b50a69c1ec2f63f8eebb5f9880e17826463a586768aed3e3e15242833609d6b">+2/-3</a>      </td> </tr> <tr> <td><strong>jwt.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-ad8a7b809caff90b4c49885a9b5d49ae03ca3fe00bef37a72d61b08b59df449c">+5/-5</a>      </td> </tr> <tr> <td><strong>redirect_response.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-f0415572330f30fceb16f230517e07f2c186e834cb8ca634795e5f92c1c57717">+3/-3</a>      </td> </tr> <tr> <td><strong>stream.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-2a60b2ae71e0a5c9dcceb588f262738f99df65fc2cb5b7002f2c37f87639e3b6">+2/-3</a>      </td> </tr> <tr> <td><strong>websocket.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-6ff41099eded8eca4a0e579142d7af4429efb83350917780e0d3ae7dac13847a">+16/-17</a>  </td> </tr> <tr> <td><strong>ider.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-6ea42b8eab8ba0708f93f9a8b44056d9a9e6ac42517ffba12f81420d6b0bd2b0">+1/-4</a>      </td> </tr> <tr> <td><strong>alert.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-45c7eaf1a764ab52f3e014fe0555c03dc59d77c5ce974684cdee4ed90463a234">+2/-5</a>      </td> </tr> <tr> <td><strong>destinations.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-e0ba33d95ef7f0eac13bdac7e750462907a0308f01c027ea206d058e282c977b">+1/-1</a>      </td> </tr> <tr> <td><strong>inverted_index.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-8bf9d9403f05f53f0c6fd5e6a6dbf0204ea80465b233567552cb7395d08d1c3c">+1/-1</a>      </td> </tr> <tr> <td><strong>components.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-e2d647c422ae05ac91abc683c7b11a3fb7da61ee30968ca6e84cdba40221bbb9">+2/-0</a>      </td> </tr> <tr> <td><strong>promql.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-0893731b3f0345378627894e3ea4a9f8e8735b38622b2b96ab55e81760d188db">+1/-1</a>      </td> </tr> <tr> <td><strong>ratelimit.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-fa7dc03fb526c09d5eb58021872d821fcd72eeb4d6a670224f5286012485bc3d">+1/-4</a>      </td> </tr> <tr> <td><strong>search.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-373e83a31e5bbc82fba05d7ae274b0dbd624ed748272232cf7eb8b8c3b2f3afc">+22/-22</a>  </td> </tr> <tr> <td><strong>sql.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-845dcf22be9d370a278c081df83a72bb8dc4b729f95c201839787088c82b57e7">+39/-39</a>  </td> </tr> <tr> <td><strong>stream.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-2438d04c3fe97bc5452e5b74a303163b8723d93455cda32c95f0525435ad46bd">+1/-1</a>      </td> </tr> <tr> <td><strong>timed_annotations.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-f94b4d6d12d68e7054ad33601e666b08e4594586013dc34384fa21b1df5fe6e8">+4/-4</a>      </td> </tr> <tr> <td><strong>async_file.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-8275f369cab85b5e9a97ed11d236de3364850b6989698936f19e70a0144e6a0d">+19/-23</a>  </td> </tr> <tr> <td><strong>download_utils.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-990aff76afde0d775c850302bf6399998aee73da05c76e719b5fbad6e7ed4b27">+3/-3</a>      </td> </tr> <tr> <td><strong>file.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-405c4fb06d186c2423f78d42f3d195b12961877b091feff032f7df500450a72a">+6/-9</a>      </td> </tr> <tr> <td><strong>flatten.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-d13aed536f8b484d0223d7158e45e3a51ea181846965f73be42dedae975afcc7">+1/-1</a>      </td> </tr> <tr> <td><strong>md5.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-f4d317c045b3df075e418dd6a7cd51f275befd1f2d15622afcb41c20024a8ee5">+1/-1</a>      </td> </tr> <tr> <td><strong>prom_json_encoder.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-5c6b4c5bb9f3859855269fe2b06a05167469ebc2ac04b84fb45d8d92aba6b825">+1/-1</a>      </td> </tr> <tr> <td><strong>query_select_utils.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-2396baca05d001b681c6cc302c2acc901d4dc1e7acf32dd1be6fa9a759578a46">+30/-30</a>  </td> </tr> <tr> <td><strong>record_batch_ext.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-f84aa2842a164d6e85aa05a718159e87a325cac139c59e561f6f59313c6d7c5d">+1/-2</a>      </td> </tr> <tr> <td><strong>schema.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-2431bc157b9e51be2f16836a995b5ed52f41dded921a404bf2eebd506b49fdfb">+1/-2</a>      </td> </tr> <tr> <td><strong>cgroup.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-167dbe688715e2a299b86aa662fd5170b68e54ec0f6a04e0ebfa0492b3ec6b00">+11/-9</a>    </td> </tr> <tr> <td><strong>net.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-f95051aabe608a9348aecbcc7f6866576f683f0e72cb4c84f7ed3e06c7e78b67">+7/-8</a>      </td> </tr> <tr> <td><strong>time.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-7b7e5972c321c99c5147af56a71afc1136a7c75844b8442e207de60bc86b7ddf">+4/-4</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-9a4dddd253d61f35b8d61d372477e24a4523f419aad2a63b226b19a584b8624f">+2/-5</a>      </td> </tr> <tr> <td><strong>event.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-689769712cbd6bd02beba078c1e1dfc188e2be39ab19a0e232b8bd12bc7f1e78">+11/-11</a>  </td> </tr> <tr> <td><strong>ingest.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-27ffe683f384ff836c6aac746240d755bdf17f3c6e0a65d8ec1a1ff82aaf7d5c">+7/-12</a>    </td> </tr> <tr> <td><strong>jwt.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-232cab74f4cbf135af424744272e17732fbda8ac005d33c0acb4597e65479d02">+61/-62</a>  </td> </tr> <tr> <td><strong>validator.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-f4d4b51574b378a8c372345ea48875834a4eadacfac33fa4d0ceca1e0ef96b35">+17/-18</a>  </td> </tr> <tr> <td><strong>dashboards.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-250eeebeec9d4ac0ac271466ed83cd33fea3f121080491b340dc95d9af287259">+4/-4</a>      </td> </tr> <tr> <td><strong>action.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-616df3014c553f763ea57e925f1bff7bb47f9467c632940d93a45b7b1e4c3a07">+5/-6</a>      </td> </tr> <tr> <td><strong>deprecated.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-3a5d80eac45bff881716e2cf099f90130659576eca017ad238a61781d3341b7e">+1/-1</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-e67f49865e1cd36775282c55ba0c0579a52ce7542f9d8dde450033f0c66a9f4c">+2/-2</a>      </td> </tr> <tr> <td><strong>ingest.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-24f250254ecfdbe004af9dc0ea23f8a2c0105ae0898690ed3e565089e5d8eef9">+2/-2</a>      </td> </tr> <tr> <td><strong>loki.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-34691872632b619be8aadf255104f0e004fcf701dba98f88e231de5966eed099">+1/-1</a>      </td> </tr> <tr> <td><strong>settings.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-1e807254f2faf8fd09b802f71b23412f203eed289033f96bfc62b15dd997db8c">+5/-5</a>      </td> </tr> <tr> <td><strong>around.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-93b3688de26f8f7c9dc0389763e9999a254bcded15b6ae8c8349f18aec83bd4f">+9/-9</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-96630a0dbb6dd7b2b0248fa325c7c7041ce31340c32826675a4a70772e9d29f6">+19/-20</a>  </td> </tr> <tr> <td><strong>multi_streams.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-c3233cdf1e68cb2455f91457719b9c2f477e735ee16546fdfe111e76768bd054">+17/-18</a>  </td> </tr> <tr> <td><strong>search_inspector.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-f818db4ca10b411c3559cc2a2659c49a59a45ecde0e4a22f9ef023da0bf4a009">+27/-29</a>  </td> </tr> <tr> <td><strong>search_job.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-d848b311eae318d198558c29a007c8973afb5f8867a31ec056f4e037dcd1b854">+4/-4</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-095d70f65b6e69bf7de0180fcbcddcd7c89c4cf00095d7b06f039a07e73916bb">+2/-3</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-c10dd7b4ea20ffe5840c93a6ae4298aec0c32c8ba1b114192fda6c2bac1c41d1">+7/-7</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-ca09c9c9b020049bdbc3382c466bfcca80daac92e98760d9a9a76172abf34049">+10/-12</a>  </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-619eafba0f0509d905b021f4a16659010aca23c62cffd9cd3e6c2bbe204ce200">+14/-18</a>  </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-64694c886c809b6efe449e6eba9fbb33d31ce5a3c49cea1a30006fc1d9522665">+4/-7</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-56897e42f9436a640bc70fddc25686485ba8ae1223635b32d252a0333309f748">+4/-4</a>      </td> </tr> <tr> <td><strong>session.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-6a1a862158668be26ad1575d7050eb9139bca3e469db56b32641b4c46d0f8144">+6/-7</a>      </td> </tr> <tr> <td><strong>encoder.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-1825b00f221539b371b9fe8aa89f9cbc1cfb94f3b731f053e64c51fb5c5934cd">+1/-2</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+2/-4</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-66207bd699343342402f88cdd331f4c549783754e4921d0c8d43e026e2d8e575">+12/-12</a>  </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-f8e74805561fa288a434e6e5ca8fb2c9068fb7f54a5a21f0a9bbba5f73249a5f">+21/-23</a>  </td> </tr> <tr> <td><strong>etcd.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-33bb9257c1a1b9162c8679191b8d8b75c0b7395f72291ce73d77247ff924a66e">+20/-22</a>  </td> </tr> <tr> <td><strong>postgres.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-1554e69f6d6c29812af88f0dcf093240b24ff85f9e292322c71cb19b0dda808d">+30/-45</a>  </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-90c725d0e77a6e7d4bcc130cf5783848a5cf0efdb304b351cd63cacf8775fc36">+1/-0</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-3df833bc826eb12e0819620d1fe316ef1ac352d293c11e57cc3df0da3c031e93">+10/-10</a>  </td> </tr> <tr> <td><strong>postgres.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-bb24a30c9c3744eb31fb49e4271a9651d1f9a94a87455ac0074364b879af9651">+27/-24</a>  </td> </tr> <tr> <td><strong>sqlite.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-85ff00530831fdaa4e13c05cb649f2e20479cce04a4b65ef78d131cdb89b6c71">+36/-35</a>  </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-bff7752b2aae0e9cef3ece1b9ddf17b9aa2f52f81fa777e6249039580aefbd69">+1/-1</a>      </td> </tr> <tr> <td><strong>nats.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-b05729b03dff9a5e29eaf536d0f90e94a0c27a27d1a3a41b52f25dc6dc380138">+8/-23</a>    </td> </tr> <tr> <td><strong>mysql.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-a275fb17137e6e7cd3dbc6432164136b20d1c2d4757eadaadd0b4a983473865a">+14/-35</a>  </td> </tr> <tr> <td><strong>postgres.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-a281dfd3124defa76bf1100c7e4993def0cb3049cf1ad0b1c513ea0afe522d08">+2/-3</a>      </td> </tr> <tr> <td><strong>sqlite.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-07751d2c8c454417335fa9dd06507099d8b1eaea9b04af99857db6d9c619e69f">+2/-3</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-386fd4fad1e48ab0a90516cab1947b1888b270b2fe912f30d2255cc38e6c519b">+15/-12</a>  </td> </tr> <tr> <td><strong>accounts.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-7ff63b9d4046282262c347d3458395eae5b6853cb004e7349a37ec2274465fe0">+4/-4</a>      </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-71f24f3312c19e01c4eedcd6a720893e01e64d176d2afbe1e7c6171a23059716">+4/-6</a>      </td> </tr> <tr> <td><strong>remote.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-fea1d32f99821f1913607362236bd9e5b65cb2a5f8ce2c11a9a4956ac646b3d1">+1/-1</a>      </td> </tr> <tr> <td><strong>dashboards.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-edd90c6ec720f676311aa31a635f954d559e63582f0b4c39cb9206bfb095737b">+1/-1</a>      </td> </tr> <tr> <td><strong>m20250109_092400_recreate_tables_with_ksuids.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-c16140f94722b17bff8753f09fa80ee0bbabc2b2f2f24b8800b459ced12fe0e4">+1/-1</a>      </td> </tr> <tr> <td><strong>m20250113_144600_create_unique_folder_name_idx.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-1f6dc440bcd651434f5bd6a06a4ec7d3bd88760754edeeb8d38e861208a56cbd">+1/-1</a>      </td> </tr> <tr> <td><strong>m20250125_132500_populate_templates_table.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-fa1e504b43669656f6fd652e221e954f3f3da7fd2892d524a0e8772b6d738107">+1/-1</a>      </td> </tr> <tr> <td><strong>m20250320_000001_remove_alert_name_unique_constraint.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-f4094b2945a877db5709045deafdd2883ff31fd671d0782da058af40e522d559">+10/-17</a>  </td> </tr> <tr> <td><strong>m20250611_000003_populate_reports_scheduled_jobs.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-6567d3d0361c1a184ba7877795a8c3c4f73ea9807b3d4c13efe621449dd3f80f">+8/-12</a>    </td> </tr> <tr> <td><strong>search_jobs.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-b78a126c6e978d43e798fa2bcb56f7bb0132c107c716d4a9891c58b3c5a33ff9">+7/-8</a>      </td> </tr> <tr> <td><strong>timed_annotations.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-4bc910d02f5ddc598e394d06172debe456c7db655cc16fef53eab55005880378">+14/-21</a>  </td> </tr> <tr> <td><strong>partition.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-441e26dda30cadfa3cb1efbd9569aec96af8fdc779e4916814fb5f9477db3f6c">+1/-1</a>      </td> </tr> <tr> <td><strong>writer.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-d1a9b1542cc2758afe4fa67abd454dd818718f4e1c7570303519af5a58d6301f">+5/-5</a>      </td> </tr> <tr> <td><strong>Additional files not shown</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7316/files#diff-2f328e4cd8dbe3ad193e49d92bcf045f47a6b72b1e9487d366f6b8288589b4ca"></a></td> </tr> </table></details></td></tr></tr></tbody></table> ___ > <details> <summary> Need help?</summary><li>Type <code>/help how to ...</code> in the comments thread for any questions about PR-Agent usage.</li><li>Check out the <a href="https://qodo-merge-docs.qodo.ai/usage-guide/">documentation</a> for more information.</li></details>	2025-07-01 21:25:45 +08:00
ByteBaker	3aad96721d	chore: maintenance and cleanup (#7289 )	2025-06-30 16:26:09 +08:00
Loakesh IndiranandHengfei Yang	600c0d3b7e	feat: aggregation cache (#6923 ) Cache Intermediate record batches for aggregate sql queries ## TODO - [ ] Add in memory cache file index - [ ] Modify exiting delete stream results api to also delete `aggregations` cache - [ ] use bounded Chanel for `CACHE_QUEUE` and and an env for queue size --------- Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>	2025-06-17 21:08:48 +05:30
Hengfei Yang	42541ad53a	fix: add memory circuit breaker for ingester (#7171 ) ### User description This PR add circuit breaker for ingester to protect OOM. To enable this feature we need to config: ``` ZO_MEMORY_CIRCUIT_BREAKER_ENABLED=true ZO_MEMORY_CIRCUIT_BREAKER_RATIO=90 ``` If the system memory usage over `90%` , we will return `503` with error `MemoryCircuitBreakerError`. Default is `false`, disabled. ___ ### PR Type Enhancement ___ ### Description - Add memory circuit breaker config and defaults - Implement breaker check in ingester writer logic - Integrate ingestion resource check across services - Introduce new error variants for circuit breaker ___ ### Changes walkthrough 📝 <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Configuration changes</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>config.rs</strong><dd><code>Define memory circuit breaker configuration defaults</code>          </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7171/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+1/-1</a>      </td> </tr> <tr> <td><strong>loki.proto</strong><dd><code>Add Java proto generation options</code>                                                </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7171/files#diff-c6860d35cb793f55cbf91f6dc9e9366bf45a42c3b6385c6c07e813f22d18ccfb">+10/-14</a>  </td> </tr> </table></details></td></tr><tr><td><strong>Error handling</strong></td><td><details><summary>3 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Add IngestionError error variant</code>                                                  </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7171/files#diff-90c725d0e77a6e7d4bcc130cf5783848a5cf0efdb304b351cd63cacf8775fc36">+2/-0</a>      </td> </tr> <tr> <td><strong>errors.rs</strong><dd><code>Introduce MemoryCircuitBreakerError variant</code>                            </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7171/files#diff-bf45932466265b732f13d0cd9b9cffbaefbe37c5861ace9315449eeca9250713">+1/-0</a>      </td> </tr> <tr> <td><strong>syslog.rs</strong><dd><code>Handle circuit breaker error logging</code>                                          </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7171/files#diff-4bc46f36fad79f5df8c42b8a8d5d8c4a70b1320b966bcff3421deeb96a1981fa">+2/-1</a>      </td> </tr> </table></details></td></tr><tr><td><strong>Miscellaneous</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>lib.rs</strong><dd><code>Export check_memory_circuit_breaker function</code>                          </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7171/files#diff-603267e2f712eeb4bc46fdff981f1fad1f3a9d3b989a7db4b3c0984fc5b1ed43">+4/-1</a>      </td> </tr> <tr> <td><strong>hec.rs</strong><dd><code>Remove pre-ingest resource check</code>                                                  </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7171/files#diff-5434f454898e66d4a982801761a349ac7e6862836e15eefaf643b31d0d2933dd">+1/-9</a>      </td> </tr> </table></details></td></tr><tr><td><strong>Enhancement</strong></td><td><details><summary>12 files</summary><table> <tr> <td><strong>writer.rs</strong><dd><code>Implement memory circuit breaker logic</code>                                      </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7171/files#diff-d1a9b1542cc2758afe4fa67abd454dd818718f4e1c7570303519af5a58d6301f">+16/-2</a>    </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Extend check_ingestion_allowed signature and call breaker</code></dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7171/files#diff-7c7b5bca85f0ced652965cae66540c96f56dfcca4f4c16b77bf7b45aac203d1b">+9/-2</a>      </td> </tr> <tr> <td><strong>bulk.rs</strong><dd><code>Pass stream type to ingestion check</code>                                            </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7171/files#diff-2017dbe9076c78a7e7919ed5c8738729123d630038803cef0ce2c356d6fa2c48">+1/-1</a>      </td> </tr> <tr> <td><strong>ingest.rs</strong><dd><code>Include circuit breaker in log ingest checks</code>                          </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7171/files#diff-e62118e45be183266d39ffa08491f255017954235ff0f3d6937683e314da406f">+3/-1</a>      </td> </tr> <tr> <td><strong>otlp.rs</strong><dd><code>Add breaker check for OTLP logs</code>                                                    </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7171/files#diff-183498d98b17ddf344cbab4da6b2d9fe89ac81112db51655c142a89023b46ef3">+1/-1</a>      </td> </tr> <tr> <td><strong>json.rs</strong><dd><code>Use unified ingestion check for metrics</code>                                    </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7171/files#diff-506987e61095a54e47f7f208dad89f9f1bce595fa6bf1ba8a7d746f2b2ff6c4a">+10/-20</a>  </td> </tr> <tr> <td><strong>otlp.rs</strong><dd><code>Add resource check for metrics OTLP</code>                                            </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7171/files#diff-a24391bef75a4678c50724e8b5f305ec0ee8c2c74c68be9d677765de68b5d8dd">+4/-23</a>    </td> </tr> <tr> <td><strong>prom.rs</strong><dd><code>Invoke ingestion check in Prom remote_write</code>                            </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7171/files#diff-c4e9e1d49a2bd0fefe67684b3768c4792be63cebcf727d71a48141f5b5957b8c">+4/-18</a>    </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Use new error variant for circuit breaker</code>                                </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7171/files#diff-124b08e44cb8740f8cb0dde3342ccd8c1b3b4e3c6a0edb12c151e060c54a0bd0">+5/-1</a>      </td> </tr> <tr> <td><strong>storage.rs</strong><dd><code>Always apply memory circuit breaker check</code>                                </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7171/files#diff-ab2d867e1e2e3dd8307505c2e130039628a84f8131a9b38cd2e7b4e1c141e094">+2/-3</a>      </td> </tr> <tr> <td><strong>wal.rs</strong><dd><code>Enforce breaker and release files on error</code>                              </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7171/files#diff-48aea52fd870678e577e03f29fc9769920b7f74761b1447242835dbf1b8ac8df">+5/-6</a>      </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add breaker check for traces ingestion</code>                                      </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7171/files#diff-66f3b4f5e7927092fd6e57738687a1091b6f316638f7c58f8d7be9addb72bc30">+14/-58</a>  </td> </tr> </table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>loki.rs</strong><dd><code>Remove obsolete comment in StreamAdapter</code>                                  </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7171/files#diff-519e5b0c03a8916b62712d1c892dab46282acb6c3f9942c2001169a0e2ea2f3c">+0/-1</a>      </td> </tr> </table></details></td></tr><tr><td><strong>Formatting</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>search_stream.rs</strong><dd><code>Reformat long comment lines for clarity</code>                                    </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7171/files#diff-b952a89375f2745a0daa273f1bdefc80c22997759a9b17a8fccbc70ae294e8e4">+9/-6</a>      </td> </tr> </table></details></td></tr><tr><td><strong>Tests</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>write.rs</strong><dd><code>Adjust Writer initializer tuple unpack</code>                                      </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7171/files#diff-791f2d9a83e5cd9620a9649156f12988a8cf1846337d6953bfc07bea9a46b71a">+1/-1</a>      </td> </tr> </table></details></td></tr></tr></tbody></table> ___ > <details> <summary> Need help?</summary><li>Type <code>/help how to ...</code> in the comments thread for any questions about PR-Agent usage.</li><li>Check out the <a href="https://qodo-merge-docs.qodo.ai/usage-guide/">documentation</a> for more information.</li></details>	2025-06-14 16:25:07 +08:00
Loakesh Indiranandudev	c29bc97f0a	fix: histogram interval update subquery for HTTP2 (#7170 ) (#7174 ) ### User description fixes #7169 for HTTP2 streaming cherry-picked from: https://github.com/openobserve/openobserve/pull/7170 ___ ### PR Type Bug fix, Enhancement ___ ### Description - Replace manual AST walk with VisitorMut for histogram - Inject missing interval arg into `histogram` calls - Remove deprecated `visit_statements_mut` logic - Reflow comments in search_stream.rs ___ ### Changes walkthrough 📝 <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table> <tr> <td> <details> <summary><strong>websocket.rs</strong><dd><code>Use VisitorMut for histogram interval update</code>                          </dd></summary> <hr> src/common/utils/websocket.rs <li>Introduce <code>HistogramIntervalVisitorMut</code> struct<br> <li> Implement <code>VisitorMut</code> to append missing interval args<br> <li> Update <code>update_histogram_interval_in_query</code> to use visitor<br> <li> Remove old <code>visit_statements_mut</code> & helper function </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/7174/files#diff-6ff41099eded8eca4a0e579142d7af4429efb83350917780e0d3ae7dac13847a">+39/-44</a>  </td> </tr> </table></td></tr><tr><td><strong>Formatting</strong></td><td><table> <tr> <td> <details> <summary><strong>search_stream.rs</strong><dd><code>Reflow comments in partitioned search</code>                                        </dd></summary> <hr> src/service/search/search_stream.rs <li>Reflow and wrap multi-line comments<br> <li> Adjust comment formatting around skip logic </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/7174/files#diff-b952a89375f2745a0daa273f1bdefc80c22997759a9b17a8fccbc70ae294e8e4">+9/-6</a>      </td> </tr> </table></td></tr></tr></tbody></table> ___ > <details> <summary> Need help?</summary><li>Type <code>/help how to ...</code> in the comments thread for any questions about PR-Agent usage.</li><li>Check out the <a href="https://qodo-merge-docs.qodo.ai/usage-guide/">documentation</a> for more information.</li></details> Co-authored-by: udev <uddhav@openobserve.ai>	2025-06-13 23:19:04 +05:30
Subhra264	c4e1c6e363	fix: _around api rbac returns 403 error (#7042 ) For rbac for `_around` api, the method needs to be converted into `GET` method instead of `POST`.	2025-06-09 18:31:14 +05:30
Hengfei Yang	94a0241e3f	ci: improve unit test coverage (#7068 ) Now the coverage is: ✅ Functions coverage: 46.83% ✅ Lines coverage: 46.35% ✅ Regions coverage: 34.09%	2025-06-06 20:37:41 +08:00
udevandomkark06	2bcae056ac	fix: redirect short url (#7026 ) (#7033 ) with this POC we will let FE handle all short url redirections. We keep the old routes intact for backwards compatibility --------- --------- Co-authored-by: omkark06 <omkar@zinclabs.io>	2025-06-06 17:33:16 +05:30
udevandloaki07	3f7582e0db	fix: increase test case coverage (#7054 ) Co-authored-by: loaki07 <rustyloaki@gmail.com>	2025-06-06 13:19:24 +05:30
Taiming Liu	961976f8df	fix: ent build (#6957 )	2025-05-28 09:43:42 -07:00
Hengfei Yang	fa6454267c	feat: add search scheduler (#6950 ) impl #6948 - [x] When we run query, we get node list and check the node CPU, drop at most 20% querier nodes. - [x] When download file, we first try to download from original node( by consistent hashing), then download from object store. - [ ] ? should also apply this logic for ingester node ? - [x] Fixed metrics keep increase issue of DownloadManager	2025-05-28 23:31:57 +08:00
Yashodhan JoshiandYashodhan Joshi	7540824633	fix: use _all_ in org related api call perm checks (#6920 ) Fixes permission check for organization related api. In ofga when we set a role with org perms, it sets to `org:_all_{org}` but in the check , we check for permission on `org:{org}` which causes users with role having permission on org, not be able to rename org. This fixes it by updating it to correct check. --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>	2025-05-28 11:02:01 +05:30
Sai Nikhil Subhradeep Chakraborty Taiming Liu	31703cc9d2	feat: o2 AI Agent (#6899 ) this PR introduces o2 ai agent with this user able to ask o2 ai about sql query , vrl query and logs --------- Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com> Co-authored-by: Taiming Liu <liutaiming3@gmail.com>	2025-05-27 15:09:19 +05:30
+2 Subhra264 Bhargav Taiming Liu hengfei yang harshiv-7 Huaijin Zhao.wei ShyamOOAI oasisk sai nikhil kethe Yashodhan Joshi Yashodhan Joshi	0d9f47966b	feat: Organization revamp (#5579 ) Fixes #5131 Addresses #5408 ### Revamp - Created a new table to store organizations in the `organizations` table. Earlier, we never stored organizations in any table, rather we used to store the organizations in the meta table as part of the `user` json. - The users and organization membership relations are now moved to its own separate tables - `users` and `org_users`. ### New changes API changes - - `/organizations` - `POST` api to create an org and save it in the meta table. In non-enterprise version, currently only the root user can create the org. For enterprise version with rbac enabled, anyone with `CREATE` access for organizations can create the organization and becomes an admin. Root user can always create an organization. - `/{org_id}/rename` - `PUT` api to rename the existing org. Only root user and the admins of the org can access this API. The payload for following APIs are changed - - `POST api/{org_id}/roles` - `{"name":"new_role"}` is changed to `{"role":"new_role"} - `POST api/{org_id}/users/{user}` - `{"role":"viewer","custom_role":["role_name"]}` is the new payload where `custom_role` is optional and is array of custom roles (only works in enterprise with rbac enabled) which will be assigned to the user. The value of `role` can be standard role or as well as any custom role (when it is custom role, the standard role assigned to the user is decided based on the enterprise env variable - `O2_DEX_DEFAULT_ROLE`). - `POST api/{org_id}/users`- the payload - ``` {"email":"user@user.com","role":"viewer","custom_role":["create_org_permission"],"password":"12345678", "first_name": "", "last_name":""} ``` - `PUT api//{org_id}/users/{email_id}` the payload - ``` {"first_name":"Hello","last_name":"j","role":"viewer","isExternal":false,"change_password":false,"password":"","custom_role":["create_org_permission"]} ``` A new ENV variable is introduced - `ZO_CREATE_ORG_THROUGH_INGESTION` (default `false`) which if true, an org will automatically created when ingesting data to a new org. If false, user can not create org by ingesting data to new org, they need to first call the `POST /organizations` API. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes New Features - Added a `cloud` feature for enhanced organizational management capabilities. - Introduced new methods for managing organizational users and asynchronous message processing. - Enhanced user invitation and membership features, including improved role handling. - Implemented configurable ingestion URL support across various components. - Added new functionality for user management and organization handling in the super cluster architecture. Improvements - Refined user authentication and authorization processes, including error handling. - Updated database schema to support multi-organization architecture. - Enhanced pagination settings and UI components for better user experience. - Improved handling of organization names and user roles in notifications and alerts. - Streamlined service account token management with visibility enhancements. - Enhanced the `AddUpdateOrganization` component with improved user input handling and validation. - Updated the `AppOrganizations` component to dynamically load the appropriate organization management interface. - Improved the `ServiceAccountsList` component with updated pagination options and token handling logic. Bug Fixes - Resolved issues with user role assignments and organization management inconsistencies. - Corrected routing and navigation logic for login and logout processes. - Addressed various error handling scenarios in user and organization processes. - Fixed issues with the visibility of service account options in the `EditGroup` and `EditRole` components based on the cloud configuration. Breaking Changes - Significant restructuring of user and organization data models. - Changes to authentication workflows and configuration handling. - Removal of certain components related to user and organization management. - Updated the logic for determining the ingestion URL across various components. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Bhargav <BJP232004@GMAIL.COM> Co-authored-by: Taiming Liu <liutaiming3@gmail.com> Co-authored-by: hengfei yang <hengfei.yang@gmail.com> Co-authored-by: harshiv-7 <harshiv@openobserve.ai> Co-authored-by: Huaijin <haohuaijin@gmail.com> Co-authored-by: Zhao.wei <weizhao029@gmail.com> Co-authored-by: ShyamOOAI <shyam@openobserve.ai> Co-authored-by: oasisk <ashish.j.kolhe@gmail.com> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> Co-authored-by: Yashodhan Joshi <yashodhan@openobserve.ai> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>	2025-05-25 21:53:55 -07:00
Taiming Liu	f680073307	feat: add _timestamp to tantivy index to optimize histogram queries (#6801 )	2025-05-24 11:15:46 +08:00
Loakesh Indiran	1ad3f8368e	fix: chery-pick fixes (#6853 ) - fix: auth for _values_stream (#6849) - fix: simple distinct sql for http api (#6836) - feat: add env to enable use_stream_setting for partitions (#6866) # closes - https://github.com/openobserve/openobserve/issues/6863 New Env ``` ZO_USE_STREAM_SETTINGS_FOR_PARTITIONS_ENABLED # default false ```	2025-05-21 17:12:52 +05:30
udev	da2a769762	feat: Add HTTP2 Streaming support (#6721 ) (#6827 )	2025-05-20 00:06:05 +08:00
Subhra264andsai nikhil kethe	330cf14ec0	fix: ZO_DEFAULT_MAX_QUERY_RANGE_DAYS env should be returned by streams list api (#6762 ) Fixes #6753 --------- Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>	2025-05-09 14:22:58 +05:30
Ashish Kolhe	1f47ac2614	fix: base64 encode cookies (#6596 ) fixes : [6564](https://github.com/openobserve/openobserve/issues/6564)	2025-04-28 18:08:48 +05:30
Sai NikhilandSubhradeep Chakraborty	a5716d88e0	feat: bulk move and export dashboard (#6507 ) 1. this PR implements bulk move and export dashboards --------- Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>	2025-04-16 22:21:05 +05:30
Loakesh Indiran	9b8b75c5c7	feat: add mini search partitions (#6494 ) # Mini Partitions for Faster Initial Search Results ## Summary of Changes This PR introduces "mini partitions" to improve user experience for search queries by providing faster initial results, especially for large time range queries. ## Old Partition Logic - Created partitions of roughly equal size based on time range and data volume - Optimized for balanced workload distribution across processing nodes - Users had to wait until a full partition was processed before seeing any results - For large time ranges (e.g., 7 days), the first result could take long time to appear ## New Partition Logic - Creates small "mini partitions" at the beginning of the time range - Followed by regular-sized partitions for the remaining time range - Preserves all the original constraints and optimizations for regular partitions - Users see initial results within seconds, even for large time range queries ## New Env ```sh ZO_MINI_SEARCH_PARTITION_DURATION_SECS: Duration of each mini partition in seconds (default: 60) ```	2025-04-15 08:17:15 +05:30
Hengfei Yang	3ba66e5f0b	feat: add ZO_INGEST_ALLOWED_IN_FUTURE (#6544 ) Now we have two ENV for control ingestion time: - ZO_INGEST_ALLOWED_UPTO: old than x hours will discard - ZO_INGEST_ALLOWED_IN_FUTURE: future than x hours will discard ``` ZO_INGEST_ALLOWED_UPTO=5 ZO_INGEST_ALLOWED_IN_FUTURE=24 ``` Default is `5` hours and `24` hours.	2025-04-14 10:59:42 -07:00
Hengfei Yang	a2dfb20f9c	fix: trace_id for search (#6533 )	2025-04-14 22:56:25 +08:00
Hengfei Yang	00e93c126f	feat: add query parquet cache metrics (#6526 )	2025-04-14 11:38:35 +08:00
Loakesh Indiran	b598735a48	fix: ws `Router-Querier` conn reset & remove ws v1 (#6499 ) - [x] add `jitter` to `ping_interval` for heartbeats - [x] deprecate `is_connected` in `Connection` trait - [x] remove `maintain_connection`, instead the `health_check` tokio tasks for each querier will do the similar functionality by manual heart beats - [x] add `clean_up` for `QuerrierConnectionPool` & `QuerierConnection` with graceful shutdown - [x] use `ZO_INSTANCE_NAME` as the `req_id` for the connection between the `Rotuer-Querier` - [x] remove ws v1	2025-04-09 15:04:48 -07:00
Subhra264	9f8647c59d	fix: rbac migration for service accounts for existing organizations (#6462 ) When introducing service accounts rbac, we didn't do a rbac openfga migration to save openfga tuples (`_all_{org_id}`) for existing organizations. Hence as a result, viewer and editor are not able to view/edit service accounts by default. Currently, they can do it only after given service accounts explicitly through the roles edit UI page. This pr fixes this by adding a migration which saves `_all_{org_id}` tuples for every organizations.	2025-04-07 10:34:41 +05:30
Hengfei Yang	83a3fcb7f6	fix: improve tokio task consume (#6474 ) ## What is changed - [x] Try to improve task starving issue - [x] Update mmdb frequency - [x] Improve slow log detection ## MMDB - Removed ENV `ZO_MMDB_UPDATE_DURATION` because is in seconds - Add new ENV `ZO_MMDB_UPDATE_DURATION_DAYS` is in days ## Tokio task We try to use `task::coop` to improve task starving issue in Tokio. <img width="1496" alt="image" src="https://github.com/user-attachments/assets/1666488a-146b-49d7-b460-c629528e06f9" /> The second chart is for this PR, it is a big slow but more stable.	2025-04-07 12:03:56 +08:00
Zhao.wei	871805a32c	feat: impl ratelimit feature (#6072 )	2025-04-03 23:47:18 +08:00
Hengfei Yang	871f954fe2	feat: add new ENV to set token for root user (#6438 )	2025-04-01 12:27:11 +08:00
Ashish KolheandSubhra264	5f9f304379	fix: improve error handling (#6432 ) Co-authored-by: Subhra264 <chakrabortysubhradeep556@gmail.com>	2025-03-31 18:13:31 +05:30
Ashish Kolhe	e9b13ee424	fix: service account search range restriction (#6409 )	2025-03-27 08:34:11 +05:30
+2 Loakesh Indiran Taiming Liu Hengfei Yang Abhay Padamani Ashish Kolhe udev omkark06 ktx-abhay Omkar Kesarkhane udev Subhra264 ktx-vaidehi	8cd0683a0b	feat: ws v2 (#6404 ) # WebSocket v2 Architecture ## NOTE: Open Issues: - [ ] `session_drain` from server is experimental and not ready for prod. Instead, the client maintains ongoing `trace_id`'s and when cookie expires it will retry search for all the ongoing searches. ## Overview Implements websocket v2 with the main goal to establish `Client-Router` connections `per tab` and `Router-Querier` `persistent` connections. ## New Env (Default): ```sh ZO_WEBSOCKET_CHECK_COOKIE_EXPIRY=true ZO_WEBSOCKET_HEALTH_CHECK_INTERVAL_SECS=60 ZO_WEBSOCKET_IS_SESSION_DRAIN_ENABLED=false # This is experimental, not meant for prod ``` ## High-Level Flow 1. Client establishes WebSocket connection with the router. 2. Router creates a session. 3. Router maintains persistent connections to all queriers. 4. Client requests are routed to the appropriate querier using consistent hashing. 5. Responses are routed back using `trace_id` mappings. 6. Health checks maintain connection validity. 7. Failed connections trigger automatic cleanup and reconnection. 8. Internal Communication Validation Using: - 8a. Internal gRPC tokens - 8b. IP validation - 8c. WebSocket upgrade headers ## Connection Lifecycle ```mermaid sequenceDiagram participant Client participant Router participant ConnectionPool participant QuerierConn participant Querier participant Cluster Client->>Router: WS Connection Request Router->>SessionManager: register_client(client_id) Note over Router,SessionManager: Creates channels:<br/>response_tx/rx<br/>disconnect_tx/rx par Handle Incoming Messages Router->>Router: spawn handle_incoming task and Handle Outgoing Messages Router->>Router: spawn handle_outgoing task end Client->>Router: WS Client Event Router->>SessionManager: get_querier_for_trace(trace_id) alt No Existing Mapping Router->>Cluster: get_node_from_consistent_hash(trace_id) Router->>ConnectionPool: get_or_create_connection(querier) alt No Existing Connection ConnectionPool->>QuerierConn: create_connection QuerierConn->>Querier: WS Connection Request par Connection Tasks QuerierConn->>QuerierConn: spawn listen_to_querier_response QuerierConn->>QuerierConn: spawn health_check end Querier-->>QuerierConn: Connection Established end Router->>SessionManager: set_querier_for_trace(trace_id, querier) end Router->>QuerierConn: register_request(trace_id) Router->>QuerierConn: send_message(event) Querier-->>QuerierConn: Response QuerierConn->>Router: Route Response via response_tx Router->>Client: Forward Response Note over Router,Client: On Disconnect/Error:<br/>1. Send disconnect signal<br/>2. Clean up sessions<br/>3. Drain if needed ``` ## Connections ```mermaid graph TB subgraph "Client Layer" UI1[UI Client 1] UI2[UI Client 2] UI3[UI Client 3] end subgraph "Router Layer" R1[Router 1] R2[Router 2] subgraph "Router 1 Connection Pool" CP1[Connection Pool] QC11[Querier1 Conn] QC12[Querier2 Conn] QC13[Querier3 Conn] CP1 --- QC11 CP1 --- QC12 CP1 --- QC13 end subgraph "Router 2 Connection Pool" CP2[Connection Pool] QC21[Querier1 Conn] QC22[Querier2 Conn] QC23[Querier3 Conn] CP2 --- QC21 CP2 --- QC22 CP2 --- QC23 end R1 --- CP1 R2 --- CP2 end subgraph "Querier Layer" Q1[Querier 1] Q2[Querier 2] Q3[Querier 3] end UI1 -.->\|WS\| R1 UI2 -.->\|WS\| R1 UI3 -.->\|WS\| R2 QC11 ===\|Persistent WS\| Q1 QC12 ===\|Persistent WS\| Q2 QC13 ===\|Persistent WS\| Q3 QC21 ===\|Persistent WS\| Q1 QC22 ===\|Persistent WS\| Q2 QC23 ===\|Persistent WS\| Q3 classDef client fill:#f9f,stroke:#333,stroke-width:2px classDef router fill:#bbf,stroke:#333,stroke-width:2px classDef querier fill:#bfb,stroke:#333,stroke-width:2px classDef pool fill:#ffd,stroke:#333,stroke-width:2px class UI1,UI2,UI3 client class R1,R2 router class Q1,Q2,Q3 querier class CP1,CP2 pool ``` ## Components ```mermaid classDiagram class WsHandler { +session_manager: Arc<SessionManager> +connection_pool: Arc<QuerierConnectionPool> +handle_connection() +remove_querier_connection() +_shutdown() } class SessionManager { +sessions: RwAHashMap<ClientId, SessionInfo> +mapped_queriers: RwAHashMap<QuerierName, HashSet<TraceId>> +register_client() +update_session_activity() +remove_trace_id() +unregister_client() +set_querier_for_trace() +get_querier_for_trace() } class QuerierConnectionPool { +connections: RwAHashMap<QuerierName, Arc<QuerierConnection>> +get_or_create_connection() +remove_querier_connection() +maintain_connections() +get_active_connection() } class QuerierConnection { +querier_name: QuerierName +write: Arc<Mutex<Option<SplitSink>>> +response_router: Arc<ResponseRouter> +is_connected: Arc<AtomicBool> +connect() +disconnect() +send_message() +register_request() +unregister_request() } class ResponseRouter { +routes: RwAHashMap<TraceId, Sender<WsServerEvents>> +route_response() +flush() +spawn_cleanup_task() } WsHandler --> SessionManager WsHandler --> QuerierConnectionPool QuerierConnectionPool --> QuerierConnection QuerierConnection --> ResponseRouter ``` Related Issues: #6396 --------- Co-authored-by: Taiming Liu <liutaiming3@gmail.com> Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com> Co-authored-by: Abhay Padamani <141122205+ktx-abhay@users.noreply.github.com> Co-authored-by: Ashish Kolhe <35160958+oasisk@users.noreply.github.com> Co-authored-by: udev <uddhav@openobserve.ai> Co-authored-by: omkark06 <omkar@zinclabs.io> Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech> Co-authored-by: Omkar Kesarkhane <127078473+omkarK06@users.noreply.github.com> Co-authored-by: udev <uddhavdave1@gmail.com> Co-authored-by: Subhra264 <chakrabortysubhradeep556@gmail.com> Co-authored-by: ktx-vaidehi <134508096+ktx-vaidehi@users.noreply.github.com>	2025-03-26 11:43:40 -07:00
Hengfei Yang	afedd1907e	fix: duplicated env for enrichment table (#6384 )	2025-03-25 15:30:44 +08:00
udev	9f7521c74f	fix: Actions bugs (#6367 ) rename token env and disallow changing execution type	2025-03-23 23:47:16 +05:30
Hengfei Yang	2c039bfbb9	feat: support db replicas (#6358 )	2025-03-21 20:19:33 +08:00
Subhra264	6926f67e08	fix: alert move rbac issues (#6352 ) - Fixes 403 error when alert move api is called - 403 error when trying to create alert in default folder even after having permission - Fixes # Regular user without `GET` user permission should not be able to get list of all users	2025-03-21 00:19:00 +05:30
Subhra264	caf798e5c2	fix: Alert folder rbac migration issues (#6338 )	2025-03-20 11:48:47 +05:30
udev greptile-apps[bot]Subhradeep Chakraborty hengfei yang	57df492b62	fix: alert v2 cache (#6302 ) - Added trigger conditions in list alerts response - RBAC for alert folders, and RBAC migration for existing alerts --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com> Co-authored-by: hengfei yang <hengfei.yang@gmail.com>	2025-03-19 12:02:24 +05:30
Subhra264	bcd5bf30ce	fix: remove semaphore in alert manager (#6295 ) - Fixes #6289 - Also, Removes use of semaphore inside the scheduler. Currently it only uses the mpsc channel and waits if the mpsc channel is full. Apart from this, it uses db locks in postgres and mysql when pulling scheduled_jobs.	2025-03-18 23:04:33 +05:30

1 2 3 4 5