mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-06 00:09:22 +00:00
404a588ae4af26ffa5eb4e8ea65a0e9c77f8039a
714
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Huaijin
|
9190a9d650 | feat: upgrade datafusion to 51 (#9202) | ||
 Harsh Mahajan
|
1f06d4a46a | feat:adding configuration ENV for colour theme (#9165) | ||
 Ashish Kolhe
|
9059afdb2b | feat: sso claim parser (#9156) | ||
 Simha G
|
6ca371ec99 | fix: Concurrency control for file locking has multiple anti-concurrent design patterns in the way (#8523) | ||
 Hengfei Yang
|
b950b1cec1 |
fix: debug metrics stream keep update (#9141)
I saw a lot of logs like this:
```
DEBUG openobserve::service::db::schema: [Schema:watch] Received event: Put(EventData { key: "/schema/default/metrics/http_server_duration", value: Some(b""), start_dt: None
```
This PR optimized the metadata check logic also try to debug this case.
|
||
 Sai Nikhil
|
803434bd30 | fix: theme customization (#9077) | ||
 
|
85a8fc1648 |
feat: license check (#8522)
### **User description** This adds license check job ___ ### **PR Type** Enhancement, Bug fix ___ ### **Description** - Add enterprise license HTTP APIs and DB watch - Enforce quota checks on search and ingestion - Always enable usage reporting; simplify modes - Expose license expiry in config and UI ___ ### Diagram Walkthrough ```mermaid flowchart LR Backend["Backend: License APIs + quota checks"] DB["DB: License watch + dist lock"] Config["Config: usage/reporting & expiry"] UI["UI: Settings License page + banner"] Search["Search/PromQL: quota enforcement"] Backend -- "watch updates" --> DB Backend -- "expose expiry" --> Config UI -- "GET/POST /api/license" --> Backend Search -- "check_search_allowed" --> Backend ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>21 files</summary><table> <tr> <td><strong>config.rs</strong><dd><code>Validate usage mode and publish interval bounds</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+16/-4</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add license GET/POST endpoints with validation</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-d569000673e415b222a20a6f4f50a93c278c189a7d6e43231d5ab8807e93021f">+75/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Register license routes for enterprise builds</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-a1cfd40da1ab6cd4db0181fd8664965034298047b3b7c3320c5385897a7c4aed">+2/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Enforce search quota in PromQL endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-0ad5729a1fbce00ff53076728c51822400c13af9918e1d8554f41be56d060fa0">+24/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add enterprise search quota checks in endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-96630a0dbb6dd7b2b0248fa325c7c7041ce31340c32826675a4a70772e9d29f6">+51/-1</a> </td> </tr> <tr> <td><strong>search_stream.rs</strong><dd><code>Enforce quota in streaming search endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-642de43ff6b19ae41087a9f1efc02d8b0b873809e0c92cbddfac1a30ef3f1184">+21/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Expose license expiry; always enable usage flag</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-ca09c9c9b020049bdbc3382c466bfcca80daac92e98760d9a9a76172abf34049">+8/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Wire license endpoints into router</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+3/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add license check locking and startup watchers</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-f2e4c29b8eef22189ec704a832633c4bd135a24444f054864c98064bd0963681">+57/-0</a> </td> </tr> <tr> <td><strong>license.rs</strong><dd><code>Implement license update broadcast and watch</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-43bb2d7d70d03657337dfabb078079de71fae8a7f58962a10ac68c03594e0f81">+70/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Export license DB module for enterprise</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-b7e3bf7f0ac23dd084c0c817a310a4d5e7a64ff532746d182986ef940fa03fd0">+2/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Enforce ingestion quota for enterprise</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-7c7b5bca85f0ced652965cae66540c96f56dfcca4f4c16b77bf7b45aac203d1b">+10/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Provide search quota check helper</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-c3b85ea662a885fbd39797968928b291cb59a58f53a03f7b7dfd7ba8b6a7985a">+18/-0</a> </td> </tr> <tr> <td><strong>ingestion.rs</strong><dd><code>Simplify reporting mode; always requeue on error</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-3edd5f4ed1eed0f765b7b450ef0fbf587632e6b22302006a5e397049e297057f">+19/-50</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Always enable usage queue and publishing</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-c6255010f7a87d6acdb3ad04954fc9f2c27350ae319c53a1857b120aa06408a3">+2/-25</a> </td> </tr> <tr> <td><strong>useManagementRoutes.ts</strong><dd><code>Add license management route</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-b71a1d39b1fab0749f0668d5f16382536fbda6e2f0c92c56a972fb5d8b5835f5">+8/-0</a> </td> </tr> <tr> <td><strong>license_server.ts</strong><dd><code>Add client for license APIs</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-3ac3fe8faca5f5a6aebd9d5e8885837f1511834eb7c78d0830af8f00c70523cb">+16/-0</a> </td> </tr> <tr> <td><strong>License.vue</strong><dd><code>New license management UI with usage</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-614c7b48f6f2ab1e8041d20c7583b6864285eacf9db032c27508b76f6de8467f">+469/-0</a> </td> </tr> <tr> <td><strong>index.vue</strong><dd><code>Add License tab and routing guards</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-699c63979f851c48ad440a078e93282fd797a18fc065e5331b2e41122ddb8b69">+27/-0</a> </td> </tr> <tr> <td><strong>LicensePeriod.vue</strong><dd><code>License expiry banner component</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-14f02914aaa78ef3e82717173fa95455b7f1092faff2d0dbd51e243663850ad4">+111/-0</a> </td> </tr> <tr> <td><strong>HomeView.vue</strong><dd><code>Show license expiry banner and navigate</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-9af441af191d3254ef5f9db603dca75486de7638cda41a3c050e7b3c33d68280">+9/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Bug fix</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>schema.rs</strong><dd><code>Remove usage_enabled dependency in org creation</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-4696a44abde2df4ac48fed716f930bfdcc01759bdfcfd8d802821c6dd857bff9">+1/-3</a> </td> </tr> </table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>en.json</strong><dd><code>Add i18n key for License tab label</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8522/files#diff-892aa3d75ab6e129ef9116aa21bec1b9a15ba69ce5366d2a19128078c690d824">+2/-1</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com> Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> |
||
 Loakesh Indiran
|
6b76fbf785 | feat: clear cache main (#8919) | ||
 ByteBaker
|
1593db0e22 |
feat: skip aud validation for MCP-invoked requests
- Add `x-o2-mcp` header support for MCP request identification in token validator - Wrap OpenAPI schema types with `inline()` for better documentation generation - Update AI module imports to use new agent namespace - fix compilation for `cloud` feature, also return early if user isn't in trial - Enable service graph by default |
||
 Ashish Kolheandomkark06
|
196512057c |
feat: log patterns (#8927)
PR introduces below features: - [x] Reporting triggers to source org - [x] Alerts Insights for detailed visibility in alerts Enterprise features: - [x] Patterns extraction during query - UI and API - [x] Patterns extraction as part of ingestion process (No Sampling today) - [x] Settings to enable pattern extraction --------- Co-authored-by: omkark06 <omkar@zinclabs.io> |
||
|
Hengfei Yang
|
f56b2b8f17 | fix: metrics can't search parquet in wal (#8896) | ||
|
Loakesh Indiran
|
875d1f3005 |
feat: cache hist query with non ts col order by (#8837)
### **User description** ref: #8731 ___ ### **PR Type** Enhancement, Bug fix ___ ### **Description** - Cache support for histogram with non-TS ORDER BY - Scan all hits to compute time range - Preserve ORDER BY metadata in cache writes - Refactor utils for ORDER BY timestamp checks ___ ### Diagram Walkthrough ```mermaid flowchart LR SQL["SQL parse (histogram, ORDER BY)"] Check["check_cache: detect non-TS histogram order"] Req["CacheQueryRequest + is_histogram_non_ts_order"] Utils["result_utils: is_timestamp_field, has_non_timestamp_ordering, extract_timestamp_range"] Write["write_results: scan hits for min/max when needed"] WS["streaming cache: propagate flag"] Mod["search mod: cache write integration"] SQL -- "order_by, ts_column" --> Check Check -- "set flag" --> Req Check -- "use timestamp detection" --> Utils Req -- "persist flag" --> Write Mod -- "compute is_histogram_non_ts_order" --> Write WS -- "compute and pass flag" --> Write Write -- "min/max via Utils" --> Utils ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table> <tr> <td> <details> <summary><strong>search.rs</strong><dd><code>Add non-TS histogram order flag to request</code> </dd></summary> <hr> src/common/meta/search.rs <ul><li>Add <code>is_histogram_non_ts_order</code> to <code>CacheQueryRequest</code><br> <li> Document behavior requiring full-hit scan</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/8837/files#diff-a36a6683e327641569d8098ee0d4f13e894e1b9d51230dc3569abe42e87ba310">+5/-0</a> </td> </tr> <tr> <td> <details> <summary><strong>cacher.rs</strong><dd><code>Detect and flag non-TS ordered histograms</code> </dd></summary> <hr> src/service/search/cache/cacher.rs <ul><li>Import new utils for ORDER BY checks<br> <li> Allow histogram caching without TS ORDER BY<br> <li> Determine <code>is_histogram_non_ts_order</code><br> <li> Propagate flag in cache write paths</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/8837/files#diff-8c387b22b2c0a282600ee41ca702b3db60f5a284d507c0feb7418608ff062c6c">+33/-3</a> </td> </tr> <tr> <td> <details> <summary><strong>mod.rs</strong><dd><code>Compute and pass histogram order flag to writer</code> </dd></summary> <hr> src/service/search/cache/mod.rs <ul><li>Import <code>extract_timestamp_range</code><br> <li> Compute <code>is_histogram_non_ts_order</code> on write<br> <li> Pass flag to <code>write_results</code></ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/8837/files#diff-f07bfacc0501b9c234e64b16c1dd8eb0ae8fcbe231f90c81fed72bcc94946f74">+18/-5</a> </td> </tr> <tr> <td> <details> <summary><strong>result_utils.rs</strong><dd><code>Utilities for timestamp/order checks and range extraction</code></dd></summary> <hr> src/service/search/cache/result_utils.rs <ul><li>Add <code>is_timestamp_field</code> helper<br> <li> Add <code>has_non_timestamp_ordering</code><br> <li> Add <code>extract_timestamp_range</code> with scan fallback</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/8837/files#diff-fb601e2f37657029b6a4dcb82d16aeff5916e13660d284e7fb00db6002dc0f73">+59/-0</a> </td> </tr> <tr> <td> <details> <summary><strong>cache.rs</strong><dd><code>Streaming cache supports non-TS histogram ordering</code> </dd></summary> <hr> src/service/search/streaming/cache.rs <ul><li>Detect non-TS histogram order in streaming path<br> <li> Pass flag to cache <code>write_results</code></ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/8837/files#diff-55769b7cb67b550daf45363c6d7d569b39d4a6235f6a68c091beb40f16b6511e">+10/-0</a> </td> </tr> </table></td></tr><tr><td><strong>Miscellaneous</strong></td><td><table> <tr> <td> <details> <summary><strong>mod.rs</strong><dd><code>Hardcode partition timing and count</code> </dd></summary> <hr> src/service/search/mod.rs <ul><li>Force <code>total_secs = 4</code> (debug change)<br> <li> Force <code>part_num = 10</code> (debug change)</ul> </details> </td> <td><a href="https://github.com/openobserve/openobserve/pull/8837/files#diff-c3b85ea662a885fbd39797968928b291cb59a58f53a03f7b7dfd7ba8b6a7985a">+2/-0</a> </td> </tr> </table></td></tr></tr></tbody></table> </details> ___ --------- Signed-off-by: loaki07 <rustyloaki@gmail.com> |
||
|
Hengfei Yang
|
e850ea2f29 | fix: read user session from db if the cache is missing (#8846) | ||
|
ByteBaker
|
bb725e94ee |
feat: add MCP OAuth 2.0 support and auth improvements
- Implement OAuth 2.0 Authorization Server Metadata endpoint (RFC 8414) - Add JWT token validation for MCP endpoints with dynamic clients - Add `WWW-Authenticate` header support for unauthorized responses - Add `AuthTokensExt::has_expired()` method for token validation - Refactor base64 decode to accept generic byte inputs - Update authentication flow to allow MCP users without DB records - Clippy delinting |
||
|
Hengfei Yang
|
5dee87a8d5 | feat: add update fields api (#8826) | ||
 Loakesh Indiranandktx-abhay
|
826cbbd276 |
feat: partial err warning if results exceeds query limit only for (#8802)
### **User description** - show warning in dashboards if the ZO_QUERY_DEFAULT_LIMIT is reached only dashboards - truncate the results exactly to the query limit on error - remove query result error if search_type is ui ___ ### **PR Type** Enhancement, Bug fix ___ ### **Description** - Add capped results warning on default limit - Hide cachable function errors for UI - Truncate results when default limit exceeded - Fix partition function naming in cluster paths ___ ### Diagram Walkthrough ```mermaid flowchart LR DF["DataFusion execution"] -- "row count" --> CL["check_query_default_limit_exceeded"] CL -- "set partial_err with capped msg" --> PR["Partial response propagation"] CL -- "truncate rows" --> HC["HTTP cluster response"] UI["UI search requests"] -- "filter function_error" --> HF["HTTP search handlers"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>9 files</summary><table> <tr> <td><strong>search.rs</strong><dd><code>Add capped results warning constant</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8802/files#diff-a36a6683e327641569d8098ee0d4f13e894e1b9d51230dc3569abe42e87ba310">+2/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Hide cachable function errors for UI</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8802/files#diff-96630a0dbb6dd7b2b0248fa325c7c7041ce31340c32826675a4a70772e9d29f6">+9/-1</a> </td> </tr> <tr> <td><strong>search_stream.rs</strong><dd><code>Filter cachable errors in streaming UI responses</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8802/files#diff-642de43ff6b19ae41087a9f1efc02d8b0b873809e0c92cbddfac1a30ef3f1184">+15/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Standardize partial error message usage and mark partial on fn errors</code></dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8802/files#diff-f07bfacc0501b9c234e64b16c1dd8eb0ae8fcbe231f90c81fed72bcc94946f74">+3/-2</a> </td> </tr> <tr> <td><strong>flight.rs</strong><dd><code>Add default-limit check and rename partition function</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8802/files#diff-dbffb1a03b65f6a9163b18c5bb1241ba2966735657be39bbe9f4d0e3260d8c1d">+12/-5</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Over-fetch default limit by 5 to detect cap</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8802/files#diff-3b1c9f0a84041235ca3765361e8eadfd7e9309a52319da882dee0540c826eb3d">+2/-1</a> </td> </tr> <tr> <td><strong>execution.rs</strong><dd><code>Simplify partial error handling logic</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8802/files#diff-75d432351865224f0b00d7cc819554c2ad5fa1f777e9c645d91a3fd5c1113bc0">+0/-2</a> </td> </tr> <tr> <td><strong>leader.rs</strong><dd><code>Inject default-limit partial warning via DF path</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8802/files#diff-21a32af318adcd10e62fb0d45944993f623fb3d2d6df4bb70d882dfc8f677fdc">+10/-3</a> </td> </tr> <tr> <td><strong>utils.rs</strong><dd><code>Add limit-exceeded checks and cachable-error helper</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8802/files#diff-bfdb4ff6a6e6bebf219b885f74dcbd3058ada7d2a7aa4c306b2e604aa1471474">+80/-1</a> </td> </tr> </table></details></td></tr><tr><td><strong>Bug fix</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>http.rs</strong><dd><code>Truncate results when default limit exceeded</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8802/files#diff-fe0a1bbec7ef6418120544b589a531e5144b28cd2650f99135ed8a70ac17ac03">+12/-2</a> </td> </tr> <tr> <td><strong>follower.rs</strong><dd><code>Use corrected partition function name</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8802/files#diff-59e0cc386582cf8baf201977475fd9a7e479fd8ed000bbaa6746a51ecb28632f">+2/-2</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Signed-off-by: loaki07 <rustyloaki@gmail.com> Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech> |
||
 Simha GandOpenObserve
|
78efd236ed |
feat: Introduce query optimization recommendation service to background jobs (#8791)
### **User description**
# PR Type
Enhancement
# Description
This PR introduces a configurable query recommendation service / back
ground job that will take org wise usage data from metadata and generate
recommendations for the same and reingests this data into _meta org's
query_recommendation stream.
___
### **PR Type**
Enhancement
___
### **Description**
- Add enterprise query recommendation background job
- Expose config for schedule and limits
- Map meta stream to enterprise type
- Wire job init under enterprise feature
___
### Diagram Walkthrough
```mermaid
flowchart LR
CFG["Config limits: recommendation params"]
JOB["Background job: QueryRecommendationService"]
CTX["OptimizerContext (engine impl)"]
ORGS["List org IDs"]
STREAMS["Get stream settings"]
SEARCH["Run usage search"]
INGEST["Ingest recommendations to _meta/query_recommendations"]
CFG -- interval/duration/top_k --> JOB
JOB -- uses --> CTX
CTX -- fetch --> ORGS
CTX -- fetch --> STREAMS
CTX -- analyze --> SEARCH
CTX -- publish --> INGEST
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>stream.rs</strong><dd><code>Add enterprise Stream
conversion for recommendations</code>
</dd></summary>
<hr>
src/common/meta/stream.rs
<ul><li>Gate enterprise import behind feature flag<br> <li> Implement
From<Stream> to enterprise meta Stream</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8791/files#diff-6f5ee19bbcc9cc7fb699fc1c0f43fc96d42698ae1168a2827b5c65f7948d5dbc">+14/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>mod.rs</strong><dd><code>Wire query recommendation job
into init</code>
</dd></summary>
<hr>
src/job/mod.rs
<ul><li>Add enterprise-only module include<br> <li> Spawn recommendation
job at init</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8791/files#diff-f2e4c29b8eef22189ec704a832633c4bd135a24444f054864c98064bd0963681">+5/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>query_optimization_recommendation.rs</strong><dd><code>Implement
enterprise query recommendation service job</code>
</dd></summary>
<hr>
src/job/query_optimization_recommendation.rs
<ul><li>Add enterprise-only background job implementation<br> <li>
Implement engine trait with org/streams/search<br> <li> Ingest
recommendations into meta stream<br> <li> Run service using configurable
schedule</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8791/files#diff-85c42193130f71a11dc1d54d1a80a5888f1311857d5899b48f8c92973c21ae21">+124/-0</a>
</td>
</tr>
</table></td></tr><tr><td><strong>Configuration
changes</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>config.rs</strong><dd><code>Configure query
recommendation scheduling and limits</code>
</dd></summary>
<hr>
src/config/src/config.rs
<ul><li>Add recommendation duration config<br> <li> Add analysis
interval config<br> <li> Add top-k limit config<br> <li> Set sensible
defaults via env vars</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8791/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+7/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
Co-authored-by: OpenObserve <o2@OpenObserves-MacBook-Pro.local>
|
||
|
ByteBaker
|
c868605736 | chore: add tests, fmt, clippy | ||
|
ByteBaker
|
da96d301ab |
refactor: extract reusable Headers extractor for actix-web handlers
This commit: - Creates a new extractors.rs module with a generic Headers<T> extractor - Implements FromRequest trait for type-safe header deserialization - Centralizes header parsing logic to reduce code duplication across HTTP handlers |
||
 udev
|
c0e8efe084 |
fix: node health api (#8744)
### **PR Type**
Bug fix
___
### **Description**
- Initialize health map on node add
- Remove from health map on node delete
- Fail when online node missing health entry
- Fix heartbeat comment typo
___
### Diagram Walkthrough
```mermaid
flowchart LR
add["Node added (watch_node_list)"] -- "insert" --> healthMap["NODES_HEALTH_CHECK[uuid]=0"]
del["Node deleted (watch_node_list)"] -- "remove" --> healthMap
check["check_nodes_status"] -- "missing entry" --> err["Return error + warn log"]
check -- "present entry" --> incr["Increment failure counter"]
note["Comment fix"] -- "typo" --> hb["heartbeat"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Bug
fix</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>mod.rs</strong><dd><code>Health check map lifecycle and
validation fixes</code>
</dd></summary>
<hr>
src/common/infra/cluster/mod.rs
<ul><li>Initialize <code>NODES_HEALTH_CHECK</code> on node add.<br> <li>
Remove <code>NODES_HEALTH_CHECK</code> entry on node delete.<br> <li>
Validate presence in health map; error if missing.<br> <li> Fix comment
typo: heartbeat.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8744/files#diff-d15434a27e2b8406811f5a707b605203b603dad4a262c0642326be860bd2b568">+16/-4</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
 Subhra264
|
41dc18866a |
feat: use cache for scheduled pipeline (#8707)
### **User description**
Currently we call the db when we process a scheduled pipeline and hence
increases db calls from the alert manager. This pr maintains a cache for
scheduled pipeline, so that alert manager does not make unnecessary db
calls.
___
### **PR Type**
Enhancement, Tests
___
### **Description**
- Add cache for scheduled pipelines
- Use cache in scheduler handler
- Extend pipeline cache/watch logic
- Add tests for cache behavior
___
### Diagram Walkthrough
```mermaid
flowchart LR
Scheduler["Scheduler handler"] -- "get/cache pipeline" --> PipelineDB["DB pipeline module"]
PipelineDB -- "reads/writes" --> ScheduledCache["Scheduled pipelines cache"]
RealtimeCache["Realtime exec cache"] -- "unchanged usage" --> Scheduler
Watcher["Pipelines watcher"] -- "update caches" --> ScheduledCache
Watcher -- "update caches" --> RealtimeCache
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>config.rs</strong><dd><code>Introduce global scheduled
pipelines cache</code>
</dd></summary>
<hr>
src/common/infra/config.rs
- Import `Pipeline` type.
- Add `SCHEDULED_PIPELINES` global cache.
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8707/files#diff-cda0d9179ed982fa5f0989c9b26f21cfa49dd1de28195a98135650ffcc7c602a">+3/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>handlers.rs</strong><dd><code>Scheduler uses cache for
pipeline lookups</code>
</dd></summary>
<hr>
src/service/alerts/scheduler/handlers.rs
<ul><li>Use scheduled pipeline cache before DB.<br> <li> Cache
DB-fetched pipelines on miss.<br> <li> Preserve existing error handling
path.<br> <li> Add debug logs for cache hits/misses.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8707/files#diff-5eca8995a88fcecb6d6b8efd3c7c4496f4a96a4b060fb4ed85593574f7cf49eb">+64/-40</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>pipeline.rs</strong><dd><code>Implement scheduled
pipelines cache and watcher integration</code></dd></summary>
<hr>
src/service/db/pipeline.rs
<ul><li>Add cache APIs: get/cache/remove/size/clear/stats.<br> <li>
Populate scheduled cache during full cache load.<br> <li> Extend watcher
to maintain scheduled cache.<br> <li> Improve logging and delete
handling for both caches.<br> <li> Add unit tests for scheduled cache
behavior.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8707/files#diff-0856951eadc646b4d4d3f0417317150cdbdc37b46c2f2a25a3c2bae15601dc50">+229/-57</a></td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
|
ByteBaker
|
33237447c4 |
chore: bump crate versions
- `async-walkdir`: v1.0.0 -> v2.1.0 - `maxminddb`: v0.25 -> v0.26 - `pprof`: v0.14 -> v0.15.0 - `prometheus`: v0.13 -> v0.14 - `promql-parser`: v0.4 -> v0.6 - `rcgen`: v0.13 -> v0.14.4 - `snafu`: v0.7.5 -> v0.8.9 - `tantivy`: v0.24 -> v0.25 - `vector-enrichment`: 063cabb -> 2003d61 - `vrl`: v0.22 -> v0.27.0 Removed `tokio-tungstenite` from dependencies. Also updated `vrl` to use git source from main branch as `vector-enrichment` needed syncing Changes to enrichment tables: - return early for empty conditions, avoiding computation - avoid vector collection before iterating again - added support for `FromDate` & `ToDate` condition types - simplified data processing using iterator chains - update `vector-enrichment` trait impl (add `wildcard` param) |
||
|
Huaijin
|
ade98b64d1 |
perf: improve load speed for enrichment table in EnrichmentExec (#8679)
|
||
|
Hengfei Yang
|
fafd4444ef | fix: search partition alignment (#8581) | ||
|
51d9782e5d |
feat: add separate telemetry url (#8564)
### **PR Type**
Enhancement
___
### **Description**
- Add keyevent telemetry support and API
- Introduce `ZO_KEYEVENT_TELEMETRY_URL` config
- Send both standard and keyevent telemetry
- Update handlers to use new API
___
### Diagram Walkthrough
```mermaid
flowchart LR
Cfg["Config: add ZO_KEYEVENT_TELEMETRY_URL"]
Tel["Telemetry: send_keyevent_track_event()"]
JWT["JWT auth: new user registered"]
Bill["Billing: new subscription started"]
Mkt["Marketing: new user attribution"]
Job["Job: no ingestion after creation"]
Cfg -- "provides URL" --> Tel
JWT -- "send standard + keyevent" --> Tel
Bill -- "send standard + keyevent" --> Tel
Mkt -- "send standard + keyevent" --> Tel
Job -- "send standard + keyevent" --> Tel
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>telemetry.rs</strong><dd><code>Add keyevent telemetry
send method</code>
</dd></summary>
<hr>
src/common/meta/telemetry.rs
<ul><li>Add <code>send_keyevent_track_event</code> method.<br> <li> Use
separate keyevent telemetry URL.<br> <li> Respect telemetry enabled flag
(non-cloud).<br> <li> Robust error/status logging for keyevent
sends.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8564/files#diff-3ff026b46d9431e0657828a3bde4ab118fec006b353c2abc80be4b74633c98b4">+60/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>jwt.rs</strong><dd><code>Emit keyevent telemetry on
user registration</code>
</dd></summary>
<hr>
src/handler/http/auth/jwt.rs
<ul><li>Instantiate telemetry once per flow.<br> <li> Send both standard
and keyevent events for registration.<br> <li> Duplicate payload reuse
for both sends.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8564/files#diff-232cab74f4cbf135af424744272e17732fbda8ac005d33c0acb4597e65479d02">+11/-1</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>billings.rs</strong><dd><code>Add keyevent telemetry
for subscription start</code>
</dd></summary>
<hr>
src/handler/http/request/cloud/billings.rs
<ul><li>Send standard and keyevent for new subscription.<br> <li> Reuse
event data across sends.<br> <li> Single telemetry instance per
request.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8564/files#diff-725936e3d6e955175e6100e8ca6f4437f334a2b554d6f11fa349ee9cbac70425">+11/-1</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>marketing.rs</strong><dd><code>Add keyevent telemetry
for attribution events</code>
</dd></summary>
<hr>
src/handler/http/request/cloud/marketing.rs
<ul><li>Send standard and keyevent for attribution.<br> <li> Share
segment event data between sends.<br> <li> Use single telemetry
instance.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8564/files#diff-424975d70871fd725536b17e4aa2131af8fd29e3c9d30ded551526e8ae71cfcc">+11/-1</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>cloud.rs</strong><dd><code>Add keyevent telemetry to
no-ingestion job</code>
</dd></summary>
<hr>
src/job/cloud.rs
<ul><li>Send standard and keyevent for no-ingestion report.<br> <li>
Reuse event data; single telemetry instance.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8564/files#diff-7fc16d008216c106d63f78f75c867fbefe7102433fcaa8893c314f0b7ea8d0e6">+11/-1</a>
</td>
</tr>
</table></td></tr><tr><td><strong>Configuration
changes</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>config.rs</strong><dd><code>Introduce keyevent
telemetry configuration</code>
</dd></summary>
<hr>
src/config/src/config.rs
<ul><li>Add <code>keyevent_telemetry_url</code> to
<code>Common</code>.<br> <li> Map to
<code>ZO_KEYEVENT_TELEMETRY_URL</code> env var.<br> <li> Default empty
string disables keyevent sending.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8564/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+2/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Prabhat Sharma <hi.prabhat@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
|
||
|
Hengfei Yang
|
24bc37a80e | feat: remove grpc call for get result cache also improve RW logic (#8246) | ||
|
|
6e75dc0e62 |
feat: add bulk enable/disable for alert and pipelines (#8438)
### **User description**
This adds two apis, one for bulk enable/disable of pipeline and one for
bulk enable/disable of alerts.
___
### **PR Type**
Enhancement
___
### **Description**
- Add bulk enable/disable for alerts
- Add bulk enable/disable for pipelines
- Introduce enterprise permission checks helper
- Update auth bypass for bulk endpoints
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["New models: Bulk request/response (alerts, pipelines)"]
B["Alerts: POST /v2/{org_id}/alerts/bulk/enable"]
C["Pipelines: POST /{org_id}/pipelines/bulk/enable"]
D["Enterprise permission check: check_resource_permissions"]
E["Router: register bulk endpoints"]
F["Auth: bypass for /bulk/enable paths"]
A -- "used by" --> B
A -- "used by" --> C
D -- "enforced in" --> B
D -- "enforced in" --> C
E -- "exposes" --> B
E -- "exposes" --> C
F -- "allows unauth extraction for" --> B
F -- "allows unauth extraction for" --> C
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>6
files</summary><table>
<tr>
<td><strong>auth.rs</strong><dd><code>Allow auth extractor bypass for
bulk enable endpoints</code> </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8438/files#diff-9bc2b1841504a435f66ad094f72ddbe7b7fa75e5e1557b8adbfe36f7f0592e1b">+1/-0</a>
</td>
</tr>
<tr>
<td><strong>responses.rs</strong><dd><code>Add alerts bulk enable
request/response models</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8438/files#diff-224027f90893122eea28e35778467acf1a4b5572ce612fc9bc9ec44ca8ebd9fb">+12/-0</a>
</td>
</tr>
<tr>
<td><strong>pipelines.rs</strong><dd><code>Add pipelines bulk enable
request/response models</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8438/files#diff-61bb28d7091130297c917a0530d96202fcb4b073b514410531f9fe71acfc814f">+12/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Implement alerts bulk enable
endpoint with permissions</code> </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8438/files#diff-83431801d27626ef27482e67b62661f900043544a45ddf958bd027cf2fa3bd8d">+91/-2</a>
</td>
</tr>
<tr>
<td><strong>pipeline.rs</strong><dd><code>Implement pipelines bulk
enable endpoint with permissions</code></dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8438/files#diff-c69f4f94eee97ce226caddc497b2adf89e26869f2d72b0d0206dc31ab2714daf">+89/-1</a>
</td>
</tr>
<tr>
<td><strong>utils.rs</strong><dd><code>Add generic enterprise resource
permission checker</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8438/files#diff-96bb9f4a8f547c821f26c957e8505d0437095b08d2b36687531f71565d287bd6">+38/-0</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Configuration
changes</strong></td><td><details><summary>1 files</summary><table>
<tr>
<td><strong>mod.rs</strong><dd><code>Register bulk enable endpoints in
service routes</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8438/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+2/-0</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>1
files</summary><table>
<tr>
<td><strong>openapi.rs</strong><dd><code>Expose bulk enable endpoints in
OpenAPI</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8438/files#diff-edb8ce232c0cd08366879f642788a75d99f6bab64b4c0e77391281fdb9241e90">+2/-0</a>
</td>
</tr>
</table></details></td></tr></tr></tbody></table>
</details>
___
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
|
||
|
Hengfei Yang
|
215cf796cd | feat: remove etcd (#8466) | ||
|
Hengfei Yang
|
97d09eff1e | fix: support prefix star match_all (#8451) | ||
|
Hengfei Yang
|
237c757c7c | fix: improve nats queue with limit policy (#8436) | ||
|
Ashish Kolhe
|
34a7341e87 |
chore: node list & user session refresh apis (#8198) (#8200)
### **PR Type** Enhancement ___ ### **Description** - Add CLI commands to refresh caches - Expose HTTP endpoints for refresh actions - Cache node list building in cluster module - Replace retention flag with mode enum ___ ### Diagram Walkthrough ```mermaid flowchart LR CLI["CLI subcommands"] -- triggers --> HTTP["HTTP endpoints"] HTTP -- calls --> Cluster["cluster::cache_node_list"] HTTP -- calls --> Sessions["db::session::cache()"] Config["Config: file_list_deleted_mode"] -- controls --> Retention["Compactor retention logic"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>8 files</summary><table> <tr> <td><strong>cli.rs</strong><dd><code>New CLI subcommands for cache refresh</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8200/files#diff-cb1eb3e74723d891353ccf1e2e5e0d246d2e78a5e69dbe4c73006ea8412d65e0">+6/-0</a> </td> </tr> <tr> <td><strong>http.rs</strong><dd><code>HTTP client helpers for refresh endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8200/files#diff-790b7780a51e8eecd4754b623af8d15ad87baaaf29455ba646e2e410f83f5498">+22/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Cache node list and update consistent hash</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8200/files#diff-d15434a27e2b8406811f5a707b605203b603dad4a262c0642326be860bd2b568">+33/-0</a> </td> </tr> <tr> <td><strong>config.rs</strong><dd><code>Replace retention flag with mode string config</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8200/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+3/-2</a> </td> </tr> <tr> <td><strong>stream.rs</strong><dd><code>Add FileListBookKeepMode enum and conversions</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8200/files#diff-2438d04c3fe97bc5452e5b74a303163b8723d93455cda32c95f0525435ad46bd">+29/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add refresh nodes and user sessions endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8200/files#diff-ca09c9c9b020049bdbc3382c466bfcca80daac92e98760d9a9a76172abf34049">+20/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Wire refresh endpoints into basic routes</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8200/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+3/-1</a> </td> </tr> <tr> <td><strong>retention.rs</strong><dd><code>Use retention mode for history/deleted behavior</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8200/files#diff-6abd048f5ef4e7163cffbb51cb4bee10c07fb88b871469d5b3724f7cd04743dc">+16/-6</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ |
||
|
Subhra264
|
6f40d2ffc0 | feat: use nats queue for cluster coordination (#8413) | ||
|
Hengfei Yang
|
b72e85e0fe | feat: support nats ttl marker (#8377) | ||
|
0730d5f7a4 |
fix: landing page / Home page revamp (#6729)
### **User description**
1. This PR implements the latest UI of landing page / Home page with 2
charts for showing alerts and pipelines
2.
___
### **PR Type**
Enhancement
___
### **Description**
- Redesigned Home page layout and tiles
- Added alerts and pipelines charts
- Implemented theme-aware icons and styles
- Added loading state for summary fetch
___
### Diagram Walkthrough
```mermaid
flowchart LR
HomeView["HomeView.vue"]
SummaryAPI["Summary fetch & parsing"]
Charts["CustomChartRenderer charts"]
Tiles["Responsive metric tiles"]
Theming["Dark/Light computed icons & styles"]
Loading["Loading spinner state"]
HomeView -- "fetches org summary" --> SummaryAPI
SummaryAPI -- "sets metrics & statuses" --> Tiles
SummaryAPI -- "feeds alert/pipeline data" --> Charts
HomeView -- "applies theme classes" --> Theming
HomeView -- "shows while fetching" --> Loading
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>HomeView.vue</strong><dd><code>Home page redesign with
charts and theme-aware UI</code>
</dd></summary>
<hr>
web/src/views/HomeView.vue
<ul><li>Rebuilt template with responsive sections and tiles.<br> <li>
Added alerts pie and pipelines bar charts via CustomChartRenderer.<br>
<li> Introduced loading state, improved summary parsing, and theme-aware
<br>icons.<br> <li> Added extensive SCSS for dark/light theming and
layout.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/6729/files#diff-9af441af191d3254ef5f9db603dca75486de7638cda41a3c050e7b3c33d68280">+568/-229</a></td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: ByteBaker <42913098+ByteBaker@users.noreply.github.com>
|
||
|
Hengfei Yang
|
55a0c62d3f | fix: root user password reset (#8348) | ||
|
sktrpathi
|
f8cc585408 |
feat: store ratelimit info in vendor extension (#8259)
### **PR Type** Enhancement, Documentation ___ ### **Description** - Replace ratelimit comments with OpenAPI extensions - Add `x-o2-ratelimit` metadata to endpoints - Import `serde_json::json` where needed - Clean up obsolete inline ratelimit markers |
||
|
8af3495611 |
fix: cloud sync (#8222)
### **User description**
This is for syncing the changes done in cloud to main branch. No changes
for non-cloud code.
___
### **PR Type**
Bug fix, Enhancement
___
### **Description**
- Add invite listing and decline APIs
- Bypass auth for user-scoped invite routes
- Improve trial-expiry handling and logging
- Add org cache and invalidation hooks
___
### Diagram Walkthrough
```mermaid
flowchart LR
FE["Login & Invitations UI"] -- "list/accept/decline" --> API["/api/invites*, /{org}/member_subscription"]
API -- "auth bypass for user-scoped invites" --> Auth["AuthExtractor/TokenValidator"]
Auth -- "pending invites flag" --> LoginRedir["/config/redirect builds URL params"]
OrgSvc["Organization Service"] -- "generate/list/decline/delete" --> InvitesDB["Org Invites"]
OrgSvc -- "invalidate cache" --> OrgCache["Org table in-memory cache"]
Ingest["Ingestion services"] -- "trial expired -> 429" --> Clients["Ingestion Clients"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>25
files</summary><table>
<tr>
<td><strong>user.rs</strong><dd><code>Expand invite payload with org and
inviter</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-e13b3a981fd70182b9738a6b0ecfdee915c5b36757703a87741e1eddf3f6e786">+2/-0</a>
</td>
</tr>
<tr>
<td><strong>auth.rs</strong><dd><code>Auth bypass and key handling for
invite routes</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-9bc2b1841504a435f66ad094f72ddbe7b7fa75e5e1557b8adbfe36f7f0592e1b">+36/-0</a>
</td>
</tr>
<tr>
<td><strong>jwt.rs</strong><dd><code>Return new user and pending invite
flags; defer org creation</code></dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-232cab74f4cbf135af424744272e17732fbda8ac005d33c0acb4597e65479d02">+22/-4</a>
</td>
</tr>
<tr>
<td><strong>token.rs</strong><dd><code>Allow invite/member routes
without org user; header injection</code></dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-3339b4a964c121766ef87fcec0070f27a2b7085cf81281b53154e53d33cb28cf">+34/-0</a>
</td>
</tr>
<tr>
<td><strong>ingest.rs</strong><dd><code>Suppress logging for trial
expiry errors</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-24f250254ecfdbe004af9dc0ea23f8a2c0105ae0898690ed3e565089e5d8eef9">+30/-9</a>
</td>
</tr>
<tr>
<td><strong>org.rs</strong><dd><code>Pending-only invites; delete invite
API; trial extend cache poke</code></dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-b47c0c1a8a78524196176430268eecdbfaf6f46d6b95873d3cbe70cfd67182b0">+40/-4</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Include pending_invites flag in
login redirect</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-ca09c9c9b020049bdbc3382c466bfcca80daac92e98760d9a9a76172abf34049">+4/-4</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>List pending invites and decline
invite endpoint</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-56897e42f9436a640bc70fddc25686485ba8ae1223635b32d252a0333309f748">+36/-1</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Wire new invite endpoints into
router</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+2/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Add TrialPeriodExpired error
variant</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-90c725d0e77a6e7d4bcc130cf5783848a5cf0efdb304b351cd63cacf8775fc36">+2/-0</a>
</td>
</tr>
<tr>
<td><strong>organizations.rs</strong><dd><code>Add in-memory org cache
with invalidation</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-4d87f9fe379a5d161a3e7c3c77f7f320be738e6a70c785aa5ce93cb0830c0147">+77/-7</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Start cloud billing watcher on
cloud builds</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-f2e4c29b8eef22189ec704a832633c4bd135a24444f054864c98064bd0963681">+3/-0</a>
</td>
</tr>
<tr>
<td><strong>organization.rs</strong><dd><code>Invalidate org cache on
watch and mutations</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-ce65cea926a09721a8fea14d07fe188bc2b887218e95d74a8bffefb54001da50">+4/-0</a>
</td>
</tr>
<tr>
<td><strong>user.rs</strong><dd><code>Delete invites for user
helper</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-6ee6b0fc9ed03224d4ddf0b7b631c6f81ff7d1d579bdb98d2c5d4453852eb5d6">+4/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Return trial expiry as dedicated
error</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-7c7b5bca85f0ced652965cae66540c96f56dfcca4f4c16b77bf7b45aac203d1b">+1/-3</a>
</td>
</tr>
<tr>
<td><strong>loki.rs</strong><dd><code>Silence trial expiry logs for Loki
ingestion</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-a970c8c4785d4e4d878218615b17eec0cf4b4976121766b002c2b2f76371bfe6">+4/-1</a>
</td>
</tr>
<tr>
<td><strong>json.rs</strong><dd><code>Map trial expiry to HTTP 429 in
JSON ingest</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-506987e61095a54e47f7f208dad89f9f1bce595fa6bf1ba8a7d746f2b2ff6c4a">+15/-6</a>
</td>
</tr>
<tr>
<td><strong>otlp.rs</strong><dd><code>Return 429 for trial expiry;
reduce noise</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-a24391bef75a4678c50724e8b5f305ec0ee8c2c74c68be9d677765de68b5d8dd">+17/-7</a>
</td>
</tr>
<tr>
<td><strong>organization.rs</strong><dd><code>Invite generation
validation; delete/decline invite flows</code></dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-6b3c3c2b027c1158c0d41302cc69dfcd20ab68a6a4fa365cda403f838940bbb4">+48/-1</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Handle trial expiry with 429;
logging tweaks</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-66f3b4f5e7927092fd6e57738687a1091b6f316638f7c58f8d7be9addb72bc30">+34/-14</a>
</td>
</tr>
<tr>
<td><strong>users.rs</strong><dd><code>Enrich invite list; optional
pending filter; cleanup on removal</code></dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-98da2588a28d62bb2e301d989836125f89f89b76272cd7f75e08ec4deeea5f38">+50/-6</a>
</td>
</tr>
<tr>
<td><strong>organizations.ts</strong><dd><code>Add decline subscription
API client</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-2c3bad4c7a9fcedd7a653599381149b82f9c350b177de9d72431013807d4c36f">+10/-7</a>
</td>
</tr>
<tr>
<td><strong>users.ts</strong><dd><code>Add pending/accept/reject invite
APIs</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-a334d8d570d9bdcaa3d4c8c2a993eff009a1ec8eb499c8a3d1a292d9224159d0">+12/-4</a>
</td>
</tr>
<tr>
<td><strong>InvitationList.vue</strong><dd><code>New UI to accept or
reject invites</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-ee97546ff1aba36cc0b24cb20782502c7c88279850cc3d540ea5685209f10702">+409/-0</a>
</td>
</tr>
<tr>
<td><strong>Login.vue</strong><dd><code>First-login invite handling and
logo tweaks</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-a135c379ef6ac8f92199ea5f7ff19a227971a9b0a87ccccf87c0268740522c2c">+178/-58</a></td>
</tr>
</table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>1
files</summary><table>
<tr>
<td><strong>en.json</strong><dd><code>Add invitation-related
translations</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-892aa3d75ab6e129ef9116aa21bec1b9a15ba69ce5366d2a19128078c690d824">+18/-0</a>
</td>
</tr>
</table></details></td></tr></tr></tbody></table>
</details>
___
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Subhra264 <chakrabortysubhradeep556@gmail.com>
Co-authored-by: udev <uddhav@openobserve.ai>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: udev <uddhavdave1@gmail.com>
|
||
|
sktrpathi
|
d87d346bba |
fix: OpenAPI descriptions for endpoints (#8231)
### **User description** So far the OpenAPI descriptions for endpoint handlers was directly derived from the doc comments, leaving little control. This also leaked any dev-specific information into API spec. Now the summary and description are explicitly populated for all the endpoints, detaching them from doc comments. Also update response body annotations wherever necessary Also bump - utoipa: 4.0.0 -> 5.0.0 - utoipa-swagger-ui: 4.0.0 -> 9.0.0 ___ ### **PR Type** Documentation, Enhancement ___ ### **Description** - Add explicit OpenAPI summaries/descriptions - Correct response body schemas to Object/() - Update params/types for accuracy - Improve endpoint examples and metadata |
||
|
Subhra264
|
c49df079c7 |
test: add unit tests for alerts and self reporting (#8228)
### **PR Type**
Tests
___
### **Description**
- Add comprehensive unit tests for alerts
- Add extensive self-reporting queue tests
- Add serialization and model tests for usage
- Validate batching and timing behaviors
___
### Diagram Walkthrough
```mermaid
flowchart LR
Alerts["Alert model"] -- "unit tests" --> AlertsTests["alerts::tests"]
Reporting["Self-reporting queue/runner"] -- "unit tests" --> ReportingTests["self_reporting::tests"]
Usage["Usage/Trigger types"] -- "serde, traits tests" --> UsageTests["usage::tests"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>alert.rs</strong><dd><code>Add comprehensive tests for
Alert and params</code>
</dd></summary>
<hr>
src/config/src/meta/alerts/alert.rs
<ul><li>Add tests for <code>Alert</code> defaults and equality<br> <li>
Validate unique key logic and timestamps getters/setters<br> <li> Test
<code>ListAlertsParams</code> builder methods and chaining<br> <li>
Cover ancillary fields: destinations, context, owner/editor, flags</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8228/files#diff-45c7eaf1a764ab52f3e014fe0555c03dc59d77c5ce974684cdee4ed90463a234">+415/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>mod.rs</strong><dd><code>Add tests for self-reporting
queue and runner</code>
</dd></summary>
<hr>
src/config/src/meta/self_reporting/mod.rs
<ul><li>Add async tests for <code>ReportingQueue</code>
enqueue/start/shutdown<br> <li> Test batching logic in
<code>ReportingRunner</code><br> <li> Validate debug formatting and
mixed data handling<br> <li> Add ignored test for channel-full
behavior</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8228/files#diff-f4101fa8423f7a25f311dd82dab6796ed4ac8fef68449b5f018157dc33ca7de3">+674/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>usage.rs</strong><dd><code>Add serialization and model
tests for usage types</code>
</dd></summary>
<hr>
src/config/src/meta/self_reporting/usage.rs
<ul><li>Add serde tests for <code>UsageData</code>,
<code>TriggerData</code>, enums<br> <li> Test display helpers and
classification methods<br> <li> Validate <code>RequestStats</code> and
<code>Stats</code> defaults/from/serde<br> <li> Test hashing/equality
for grouping keys and enums</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8228/files#diff-b15a6adcb163dc27993c83528105a6726c44c86d8a84c5cf5d1d382c6ec8cc29">+684/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
|
Hengfei Yang
|
1989432ba7 |
feat: remove syslog (#8227)
### **User description** close #8186 ___ ### **PR Type** Enhancement, Documentation ___ ### **Description** - Remove syslog feature across codebase - Clean config and routing references - Update metadata prefixes and SBOM files - Refresh i18n and OpenAPI assets ___ ### Diagram Walkthrough ```mermaid flowchart LR SYS["Syslog feature"] -- "deprecated & removed" --> CODE["Code paths"] CODE -- "delete flags/routes" --> CFG["Configuration"] CODE -- "remove storage keys" --> META["Metadata prefixes"] CODE -- "update specs" --> SBOM["SBOM/XML assets"] CODE -- "update docs/i18n" --> WEB["Web locales & OpenAPI"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>13 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Remove syslog config import from job module</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-f2e4c29b8eef22189ec704a832633c4bd135a24444f054864c98064bd0963681">+3/-26</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Stop glob importing config; drop syslog usage</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-ca09c9c9b020049bdbc3382c466bfcca80daac92e98760d9a9a76172abf34049">+1/-3</a> </td> </tr> <tr> <td><strong>meta.rs</strong><dd><code>Drop `/syslog` from metadata item prefixes</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-1797798cd07803457a6273ef793d56568d8c91e743f6c4649c6db733325fb0af">+1/-2</a> </td> </tr> <tr> <td><strong>config.rs</strong><dd><code>Remove syslog-related configuration options</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+0/-34</a> </td> </tr> <tr> <td><strong>config.rs</strong><dd><code>Purge syslog feature flag and defaults</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-cda0d9179ed982fa5f0989c9b26f21cfa49dd1de28195a98135650ffcc7c602a">+0/-25</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Clean references tied to syslog ingestion</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-5061e662a8899b7e00e5744bd72f808b28ac479a8921ac519f7409bffdc4438a">+0/-60</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Delete syslog router endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+0/-5</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Detach syslog handler wiring</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-99620cff6123acf0598ed2ec9d1c06f02cb0ddece1bef898047eb303d0b71fc2">+0/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Remove syslog-specific log ingestion paths</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-ce72aea3fa4c44904cbd9ec07bd844bec71b49c9676d667b14056aa5798f96b2">+0/-2</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Clean DB interactions for syslog keys</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-b7e3bf7f0ac23dd084c0c817a310a4d5e7a64ff532746d182986ef940fa03fd0">+0/-2</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Adjust request handling without syslog</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-a1cfd40da1ab6cd4db0181fd8664965034298047b3b7c3320c5385897a7c4aed">+0/-2</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Unexport syslog services</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-1b15316505c31acccb26ba66d920fdb0894567d08d2bf1e3457716d128a67b83">+0/-2</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Remove syslog metadata definitions</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-fbf797574839e69491ea0429141c038c0ff96297bdf3d8c60d60cdc4acec74b1">+0/-2</a> </td> </tr> </table></details></td></tr><tr><td><strong>Tests</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>integration_test.rs</strong><dd><code>Update tests to exclude syslog paths</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-7ff815b68943f08bef79f083c09e55ca8c18db8e25033034b12fa31f1d43dab4">+0/-57</a> </td> </tr> <tr> <td><strong>useIngestionRoutes.spec.ts</strong><dd><code>Update UI tests to remove syslog routes</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-3b7ebacf9481f4643a2b56ffe53ab161fc89d416e280d45b48568866a4dee40c">+0/-1</a> </td> </tr> </table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>20 files</summary><table> <tr> <td><strong>openapi.rs</strong><dd><code>Remove syslog routes from OpenAPI</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-edb8ce232c0cd08366879f642788a75d99f6bab64b4c0e77391281fdb9241e90">+0/-5</a> </td> </tr> <tr> <td><strong>config.cdx.xml</strong><dd><code>Update config component index (remove syslog)</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-41ee11320f37e20fc96f754d35a890df5068af5072fd5388169d1cf305a1ed3b">+15011/-4611</a></td> </tr> <tr> <td><strong>wal.cdx.xml</strong><dd><code>Update WAL component index references</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-3d72e8b949a91b0bd78e97aa3ebf69e2887e2851223ea3d10fedeb0125c35a19">+19476/-358</a></td> </tr> <tr> <td><strong>openobserve.cdx.xml</strong><dd><code>Refresh SBOM root; remove syslog entries</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-454f38f03cf69f6b5358f5a528be4214ec2d33081b9e98ebe5d92c8bef47dc48">+14015/-7932</a></td> </tr> <tr> <td><strong>infra.cdx.xml</strong><dd><code>Update infra component index, no syslog</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-e51bf18198bc65a604b53ee18152ad9f2453ab7870d5be949022058e054ada73">+14127/-6086</a></td> </tr> <tr> <td><strong>proto.cdx.xml</strong><dd><code>Update proto index; purge syslog mentions</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-a15fa99b53917ea85b4e268bb7612d43ee881dbe4ecf70a6682bb919b1f2f219">+8000/-1513</a></td> </tr> <tr> <td><strong>ingester.cdx.xml</strong><dd><code>Adjust ingester index excluding syslog</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-e72a9be4ba2aaff1a6b837a7f7149489603f8fe720766757c4291082e98b5a65">+15845/-4615</a></td> </tr> <tr> <td><strong>flight.cdx.xml</strong><dd><code>Add/refresh flight component index</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-f41393a0191b0560da893a5e24b2e4529f7c9bb34fc71231a532b8790630cba2">+19900/-0</a></td> </tr> <tr> <td><strong>hi.json</strong><dd><code>Remove syslog strings from Hindi locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-1e7b52b5c4e762db2c923d0d3efb6d693702825ed3a1a112d3069f26b0d5f605">+0/-13</a> </td> </tr> <tr> <td><strong>fr.json</strong><dd><code>Remove syslog strings from French locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-db80e762b88ddf183820c050cdc6b215e6b02b365b5e73f5152af46b0d26f1a2">+0/-13</a> </td> </tr> <tr> <td><strong>tr.json</strong><dd><code>Remove syslog strings from Turkish locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-6f0f131e9e2591480c4c5d1eaffc72d02dc31cbd8d3f90d8f012f08cabac8818">+0/-13</a> </td> </tr> <tr> <td><strong>de.json</strong><dd><code>Remove syslog strings from German locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-78f24ecb2dbcfc38fdf3ec22b2559a5b40d236ea689d1c55f5d3e87168310264">+0/-13</a> </td> </tr> <tr> <td><strong>ko.json</strong><dd><code>Remove syslog strings from Korean locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-50e27ca1b4825a5026f3a251bfcf79c97cab07cfade2ebd512d332453d707c48">+0/-13</a> </td> </tr> <tr> <td><strong>it.json</strong><dd><code>Remove syslog strings from Italian locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-2de78074a5f887afddce60d694ead9322e6ab223eb70251c13819f3028bdc70d">+0/-13</a> </td> </tr> <tr> <td><strong>nl.json</strong><dd><code>Remove syslog strings from Dutch locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-a1706021687ae5d8a1c006c49eab3946174fecd895f9b70a89c2e76d97250906">+0/-13</a> </td> </tr> <tr> <td><strong>es.json</strong><dd><code>Remove syslog strings from Spanish locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-ac2a2167b8b973e3f776f9fe3b43bef4576470d8ad33f3d7cfaff6edff0e5778">+0/-13</a> </td> </tr> <tr> <td><strong>pt.json</strong><dd><code>Remove syslog strings from Portuguese locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-38e804f64cb5e0e451849fc84e9b46e53c954a4a4263d51c18665ad45f4b4815">+0/-13</a> </td> </tr> <tr> <td><strong>zh.json</strong><dd><code>Remove syslog strings from Chinese locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-8818055db58f83230aeea4ebfd4cec31d924e8c47b6f91298e356959ab25339e">+0/-13</a> </td> </tr> <tr> <td><strong>ja.json</strong><dd><code>Remove syslog strings from Japanese locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-092e49c5f9fbc1b539ab9c2866d108d0ce1d314022b4c294eb4611b285e7bbd3">+0/-13</a> </td> </tr> <tr> <td><strong>en.json</strong><dd><code>Remove syslog strings from English locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-892aa3d75ab6e129ef9116aa21bec1b9a15ba69ce5366d2a19128078c690d824">+0/-13</a> </td> </tr> </table></details></td></tr><tr><td><strong>Dependencies</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>Cargo.toml</strong><dd><code>Update dependencies/features after syslog removal</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542">+0/-3</a> </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>10 files</summary><table> <tr> <td><strong>syslog.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-dc6813e31e32c23aae5760091460d2ec4023d37ae73e49d29052d9f4e9f2e1fa">+0/-43</a> </td> </tr> <tr> <td><strong>websocket.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-48f05dcd4b6f1d8345cb5707e1a3c34eedd76d94a3c3904eeabd799c7820789d">+0/-90</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-58010c6e78f69dc57c353890a807371cd43c2b4cceeeb1e9378130f26d898397">+0/-143</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-66207bd699343342402f88cdd331f4c549783754e4921d0c8d43e026e2d8e575">+0/-135</a> </td> </tr> <tr> <td><strong>syslog_server.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-cb7610142582b44885f673583ab16262c3b9ca6de1f59a9def65e503360665a0">+0/-137</a> </td> </tr> <tr> <td><strong>report_server.cdx.xml</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-d7a2fd1b635919119af9187ef44188c384bd01e444e74a9d15eaa21b0840b7ed">+20472/-0</a></td> </tr> <tr> <td><strong>syslog.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-4c0e9282e3d2352f9a382099901fe034bae2961eed01d764f16ad52f27f55db4">+0/-180</a> </td> </tr> <tr> <td><strong>syslog.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-4bc46f36fad79f5df8c42b8a8d5d8c4a70b1320b966bcff3421deeb96a1981fa">+0/-559</a> </td> </tr> <tr> <td><strong>syslogs_route.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-325df35568c993fc0727bfb0499b682e6bd30af781687706306b3bdb429c49e2">+0/-191</a> </td> </tr> <tr> <td><strong>syslog.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-69ec0272ddac65f2cc96c79d84ca3933fc335a8d21ab5dd6909d71ea02a46486">+0/-37</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ |
||
 Hengfei YangandBhargav
|
2c0395124f |
feat: deprecated syslog (#8187)
### **User description** impl #8186 ___ ### **PR Type** Enhancement, Tests ___ ### **Description** - Mark syslog modules as deprecated - Add allow(deprecated) to callers - Block enabling syslog setting - Remove syslog-related tests and OpenAPI types ___ ### Diagram Walkthrough ```mermaid flowchart LR meta["meta::syslog module"] -- "mark deprecated" --> callers["call sites"] callers -- "add allow(deprecated)" --> compile_ok["compiles with warnings suppressed"] db_set["db::syslog::toggle_syslog_setting"] -- "reject enable=true" --> error["returns deprecation error"] openapi["OpenAPI schema"] -- "remove syslog types" --> spec["updated API spec"] tests["syslog tests"] -- "removed" --> leaner["reduced test surface"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>8 files</summary><table> <tr> <td><strong>config.rs</strong><dd><code>Suppress deprecation warnings for syslog globals</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-cda0d9179ed982fa5f0989c9b26f21cfa49dd1de28195a98135650ffcc7c602a">+2/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Allow deprecated syslog handlers; annotate endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-58010c6e78f69dc57c353890a807371cd43c2b4cceeeb1e9378130f26d898397">+6/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Suppress deprecation in API scope setup</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+1/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Allow deprecated syslog server wiring</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-66207bd699343342402f88cdd331f4c549783754e4921d0c8d43e026e2d8e575">+3/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Allow deprecated syslog cache/watch tasks</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-f2e4c29b8eef22189ec704a832633c4bd135a24444f054864c98064bd0963681">+4/-0</a> </td> </tr> <tr> <td><strong>syslog_server.rs</strong><dd><code>Allow deprecated syslog server run linkage</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-cb7610142582b44885f673583ab16262c3b9ca6de1f59a9def65e503360665a0">+2/-0</a> </td> </tr> <tr> <td><strong>syslog.rs</strong><dd><code>Allow deprecated syslog ingest; remove tests</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-4bc46f36fad79f5df8c42b8a8d5d8c4a70b1320b966bcff3421deeb96a1981fa">+3/-14</a> </td> </tr> <tr> <td><strong>syslogs_route.rs</strong><dd><code>Allow deprecated route endpoints; remove tests</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-325df35568c993fc0727bfb0499b682e6bd30af781687706306b3bdb429c49e2">+7/-52</a> </td> </tr> </table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>6 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Mark syslog meta module as deprecated</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-fbf797574839e69491ea0429141c038c0ff96297bdf3d8c60d60cdc4acec74b1">+1/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Mark syslog HTTP request module deprecated</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-a1cfd40da1ab6cd4db0181fd8664965034298047b3b7c3320c5385897a7c4aed">+1/-0</a> </td> </tr> <tr> <td><strong>openapi.rs</strong><dd><code>Remove syslog types from OpenAPI imports</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-edb8ce232c0cd08366879f642788a75d99f6bab64b4c0e77391281fdb9241e90">+0/-2</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Deprecate syslog DB module</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-b7e3bf7f0ac23dd084c0c817a310a4d5e7a64ff532746d182986ef940fa03fd0">+1/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Deprecate syslog logs module</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-ce72aea3fa4c44904cbd9ec07bd844bec71b49c9676d667b14056aa5798f96b2">+1/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Deprecate syslogs_route service module</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-1b15316505c31acccb26ba66d920fdb0894567d08d2bf1e3457716d128a67b83">+1/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Tests</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>syslog.rs</strong><dd><code>Remove unit tests for syslog meta types</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-dc6813e31e32c23aae5760091460d2ec4023d37ae73e49d29052d9f4e9f2e1fa">+0/-127</a> </td> </tr> </table></details></td></tr><tr><td><strong>Error handling</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>syslog.rs</strong><dd><code>Block enabling syslog; allow deprecated routes/cache</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-4c0e9282e3d2352f9a382099901fe034bae2961eed01d764f16ad52f27f55db4">+11/-0</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Co-authored-by: Bhargav <BJP232004@GMAIL.COM> |
||
|
sktrpathi
|
75fc81cecf |
chore: remove bogus unit tests (#8172)
### **User description** A significant chunk of API unit tests were not testing any actual business logic, but performing trivial actions such as struct-field-access, derive trait validation such as `Eq`, `PartialEq`, `Clone`, `Serialize` etc, and many of the tests were also just comparing constant values. They added no value to the test suite and have now been removed from the codebase, making the tests more focused. Additionally, - A lot of flaky tests have been corrected. These used to fail due to occasional race conditions when ran locally. - Some mock setup and teardown functions have also been cleaned up, which makes tests a lot more consistent. ___ ### **PR Type** Tests, Enhancement ___ ### **Description** - Remove trivial, non-value unit tests - Relax flaky assertions and panics handling - Improve test isolation via DB/cache resets - Minor test output formatting improvements ___ ### Diagram Walkthrough ```mermaid flowchart LR tests["Remove trivial/boilerplate tests"] -- cleaner suite --> stability["Stabilize assertions and panics"] stability -- reduce flakiness --> isolation["DB/Cache resets in tests"] isolation -- consistent runs --> outputs["Improve test log formatting"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><details><summary>11 files</summary><table> <tr> <td><strong>organization.rs</strong><dd><code>Remove struct-field and default-value trivial tests</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-4f089715a7ac4f78b45039d773f2fccbb4fbc6a409e0ab9ceee1fcf7911bc44b">+4/-325</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Drop JSON roundtrip serialization tests; tidy imports</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-c4bd542e4e07ed7706b0643086f5a7d4edbb3647c7ceedbe4bea516c5274d8bd">+1/-183</a> </td> </tr> <tr> <td><strong>auth.rs</strong><dd><code>Remove Debug/PartialEq trivial tests; format strings</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-9bc2b1841504a435f66ad094f72ddbe7b7fa75e5e1557b8adbfe36f7f0592e1b">+2/-104</a> </td> </tr> <tr> <td><strong>main.rs</strong><dd><code>Make tracing test tolerant; ignore enterprise tests</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-42cb6807ad74b3e201c5a7ca98b911c5fa08380e942be6e4ac5807f8377f87fc">+23/-60</a> </td> </tr> <tr> <td><strong>event.rs</strong><dd><code>Remove constants and simple struct creation tests</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-689769712cbd6bd02beba078c1e1dfc188e2be39ab19a0e232b8bd12bc7f1e78">+1/-95</a> </td> </tr> <tr> <td><strong>validator.rs</strong><dd><code>Ensure test isolation with ORM init and cache clears</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-f4d4b51574b378a8c372345ea48875834a4eadacfac33fa4d0ceca1e0ef96b35">+28/-22</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Make base URI tests robust to env errors</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-d7987e62dcc7b1ec1bd4aeabef94f85d8c41b2ea33de5ab78660794fa4d51a9e">+25/-4</a> </td> </tr> <tr> <td><strong>users.rs</strong><dd><code>Use best-effort cleanup; clear caches; improve logs</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-98da2588a28d62bb2e301d989836125f89f89b76272cd7f75e08ec4deeea5f38">+12/-9</a> </td> </tr> <tr> <td><strong>integration_test.rs</strong><dd><code>Compact println formatting and panic messages</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-7ff815b68943f08bef79f083c09e55ca8c18db8e25033034b12fa31f1d43dab4">+3/-3</a> </td> </tr> <tr> <td><strong>config.rs</strong><dd><code>Remove strict zero-length assertions on globals</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-cda0d9179ed982fa5f0989c9b26f21cfa49dd1de28195a98135650ffcc7c602a">+1/-13</a> </td> </tr> <tr> <td><strong>schema.rs</strong><dd><code>Remove unused constant import in tests</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-a5ea87a7766cb24619f64a6100f385f4b42c47af7d33b8e2f060229c536d7418">+1/-8</a> </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>26 files</summary><table> <tr> <td><strong>authz.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-28a815d1c4728aff854a9996de0ba3d3dafe1bea6bfb0cf2a9f4186c40f985b6">+0/-52</a> </td> </tr> <tr> <td><strong>loki.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-121c166af5eab8b9cb83716afa7b65fa7856c006c809b3d910ced184c52ec587">+0/-7</a> </td> </tr> <tr> <td><strong>proxy.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-7f12a4f2483e737671ac41f9c35b411e7323e0ff2731ad92a3b309e144d1316a">+0/-46</a> </td> </tr> <tr> <td><strong>service.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-61d456e7217d19bcec3920a4d1c17fd396a257c81e5394655db3632a057bdd9e">+0/-7</a> </td> </tr> <tr> <td><strong>service_account.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-878ad520e1c84500cc21ff6fc4088c8da64ff2febd79e60c0d4cc8c695c49798">+0/-80</a> </td> </tr> <tr> <td><strong>traces.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-bda8182c1f17a8360337c769edade0cb9478ff1443dd6aedce7f526afbe97afe">+0/-106</a> </td> </tr> <tr> <td><strong>user.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-e13b3a981fd70182b9738a6b0ecfdee915c5b36757703a87741e1eddf3f6e786">+0/-101</a> </td> </tr> <tr> <td><strong>auth_tests.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-2688cbdc0b009b3f0dd829c69ab86c6a7d0778ab3e65e916a410a76aa6ae6480">+0/-2961</a></td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-2aa59ce4464e7dbabdb9367fc8bb8c1b489c914b8690577d4916870463ab12db">+0/-1</a> </td> </tr> <tr> <td><strong>action.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-fc87c75a7df9539d7e7ee6184dfab02d4a8932323e81f95d57c05363aa44752d">+0/-18</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-5bf716db7e1d91dfbd427a2e320ce407c4f0c92d55d68f8e636daed1b02abd56">+0/-13</a> </td> </tr> <tr> <td><strong>reports.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-d2117a3d67d4ec47b0017861845c92143baf917e10e6746df5dd97f925f81eeb">+0/-10</a> </td> </tr> <tr> <td><strong>destinations.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-e0ba33d95ef7f0eac13bdac7e750462907a0308f01c027ea206d058e282c977b">+0/-103</a> </td> </tr> <tr> <td><strong>inverted_index.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-8bf9d9403f05f53f0c6fd5e6a6dbf0204ea80465b233567552cb7395d08d1c3c">+0/-22</a> </td> </tr> <tr> <td><strong>meta_store.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-45018c838da377fa9ded9305a1ee752adde06acdb607addf26c95f5747537615">+0/-6</a> </td> </tr> <tr> <td><strong>ratelimit.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-fa7dc03fb526c09d5eb58021872d821fcd72eeb4d6a670224f5286012485bc3d">+0/-10</a> </td> </tr> <tr> <td><strong>timed_annotations.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-f94b4d6d12d68e7054ad33601e666b08e4594586013dc34384fa21b1df5fe6e8">+0/-12</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-ca09c9c9b020049bdbc3382c466bfcca80daac92e98760d9a9a76172abf34049">+0/-34</a> </td> </tr> <tr> <td><strong>m20250611_000002_populate_reports_table.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-8247af6d6f8ccb6a5a51daff4cb1e352585be2d1c785d86ae5ebfea1dbc1099c">+0/-1</a> </td> </tr> <tr> <td><strong>folders.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-402482c22fda5dff8f33382ac11d0908f693975dec7c08b37b1c5fe24a20486a">+0/-1</a> </td> </tr> <tr> <td><strong>bulk.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-2017dbe9076c78a7e7919ed5c8738729123d630038803cef0ce2c356d6fa2c48">+0/-9</a> </td> </tr> <tr> <td><strong>ingestion.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-3edd5f4ed1eed0f765b7b450ef0fbf587632e6b22302006a5e397049e297057f">+0/-7</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-2f7f0bac01afc6e37908521d1d33d05fcd650b7a08792062976e1e7465b4a1f9">+0/-42</a> </td> </tr> <tr> <td><strong>footer_cache.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-f9278e53ed7131c953fc30f8b15cda801b74b81fb2d570fe7f80a3b7fc1de872">+0/-7</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-1602f85399080c1e2838d1ef1d62af23d96f353379e49a7770026e892c954018">+0/-20</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-66f3b4f5e7927092fd6e57738687a1091b6f316638f7c58f8d7be9addb72bc30">+0/-31</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ |
||
|
udev
|
b4eac8f607 |
fix: Inc UT coverage (#8128)
increase UT coverage improve the email regex to catch `doubledotsequence` in email |
||
|
Hengfei Yang
|
48115e1ab9 |
fix: enhance machine id check (#8160)
### **User description** - Add duplicated machine id check - Add 4 random chars for generate parquet filename - Use Atomic replace unsafe code ___ ### **PR Type** Bug fix, Enhancement ___ ### **Description** - Centralize node ID generation logic - Replace unsafe globals with atomics - Detect and resolve duplicate node IDs - Truncate long Snowflake IDs when parsing ___ ### Diagram Walkthrough ```mermaid flowchart LR A["Cluster register (etcd/nats)"] -- "collect node ids" --> B["generate_node_id()"] B -- "new id" --> C["LOCAL_NODE_ID (AtomicI32)"] C -- "reload if changed" --> D["Snowflake ID generator"] E["set_status()"] -- "detect duplicate id" --> B F["CLI parse-id"] -- "truncate to 19 digits" --> G["parse i64"] H["Parquet filename"] -- "append 4-hex random" --> I["unique file path"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Bug fix</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>cli.rs</strong><dd><code>Truncate IDs longer than 19 digits before parse</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8160/files#diff-cb1eb3e74723d891353ccf1e2e5e0d246d2e78a5e69dbe4c73006ea8412d65e0">+8/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Enhancement</strong></td><td><details><summary>10 files</summary><table> <tr> <td><strong>etcd.rs</strong><dd><code>Use centralized node-id generation and atomics</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8160/files#diff-37c2c2803d1c12ebb4ef9299bc79c9e6740fc5f19f7f4d84e7cfbb610b9561a9">+3/-17</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add reusable generate_node_id utility</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8160/files#diff-d15434a27e2b8406811f5a707b605203b603dad4a262c0642326be860bd2b568">+17/-0</a> </td> </tr> <tr> <td><strong>nats.rs</strong><dd><code>Centralize id generation; detect duplicates; reload</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8160/files#diff-901773980aa255942e7865b977fe63811f31511ac40bc512d6bbb14dbb54bece">+20/-17</a> </td> </tr> <tr> <td><strong>cluster.rs</strong><dd><code>Replace unsafe globals with AtomicI32</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8160/files#diff-68d7195ab7cccf22579c4944a37c8a2f592392c30371ce907fadf62fca6f97dd">+3/-3</a> </td> </tr> <tr> <td><strong>ider.rs</strong><dd><code>Read machine id atomically; support reload</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8160/files#diff-6ea42b8eab8ba0708f93f9a8b44056d9a9e6ac42517ffba12f81420d6b0bd2b0">+10/-2</a> </td> </tr> <tr> <td><strong>cluster.rs</strong><dd><code>Node equality includes id; NodeStatus i32</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8160/files#diff-a2667c1be522ce095796a7066e02631a7e01acc85a46573e1e3a157d2aefb7c3">+4/-3</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add 4-hex random suffix to filenames</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8160/files#diff-30f42c3685037c219f0033617f032b23a7f74624ba4ae6c4e3687af401304591">+2/-1</a> </td> </tr> <tr> <td><strong>schema.rs</strong><dd><code>Initialize generators using atomic node id</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8160/files#diff-4696a44abde2df4ac48fed716f930bfdcc01759bdfcfd8d802821c6dd857bff9">+7/-3</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Generate record ids using atomic node id</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8160/files#diff-7c7b5bca85f0ced652965cae66540c96f56dfcca4f4c16b77bf7b45aac203d1b">+2/-2</a> </td> </tr> <tr> <td><strong>schema.rs</strong><dd><code>Use atomic node id for stream generators</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8160/files#diff-d9596182a0c7a46a6a7ff5cee822554cdae05ceec312f86c93b538db9f73bbfd">+7/-2</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ |
||
|
udev
|
77e50caea5 |
fix: make config reactive to env changes (#7996) (#8092)
### **User description** fix #7996 sync to main ___ ### **PR Type** Bug fix, Enhancement ___ ### **Description** - Make periodic jobs react to config changes - Introduce reusable `spawn_pausable_job!` macro - Fix env var name `ZO_SEARCH_JOB_RUN_TIMEOUT` - Replace fixed intervals with dynamic getters ___ ### Diagram Walkthrough ```mermaid flowchart LR CFG["get_config() (dynamic)"] MACRO["spawn_pausable_job! macro"] JOBS["Background jobs (GC, compactor, alerts, stats, etc.)"] ENV["Env vars (e.g., ZO_SEARCH_JOB_RUN_TIMEOUT)"] ENV -- "drives" --> CFG CFG -- "intervals/pause_if" --> MACRO MACRO -- "spawns/controls" --> JOBS ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Bug fix</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Heartbeat interval now read each loop</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-d15434a27e2b8406811f5a707b605203b603dad4a262c0642326be860bd2b568">+2/-1</a> </td> </tr> <tr> <td><strong>config.rs</strong><dd><code>Fix search job timeout env var name</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+1/-1</a> </td> </tr> </table></details></td></tr><tr><td><strong>Enhancement</strong></td><td><details><summary>16 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Export pausable_job utilities module</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-f5df40ca59915d9d58ad04ea2440e6ebf29388afaeb8cb0d6c63d2a979df0c17">+1/-0</a> </td> </tr> <tr> <td><strong>pausable_job.rs</strong><dd><code>Add pausable job spawning macro and helpers</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-7e2dce7a3942789804bae659939c214c4eeb68d89315e23a08139214c339ea24">+143/-0</a> </td> </tr> <tr> <td><strong>disk.rs</strong><dd><code>Use pausable job for disk cache GC</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-af63f2c5d85313779df0204c9689e50a6c3916e3aadc24e54e23ec640a584fcb">+4/-10</a> </td> </tr> <tr> <td><strong>memory.rs</strong><dd><code>Use pausable job for memory cache GC</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-ff00d697a9b223ef2858c5e41111a8918e7cc347dc22891c02bda6982682eee1">+4/-16</a> </td> </tr> <tr> <td><strong>lib.rs</strong><dd><code>Mem persist uses dynamic sleep interval</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-603267e2f712eeb4bc46fdff981f1fad1f3a9d3b989a7db4b3c0984fc5b1ed43">+4/-5</a> </td> </tr> <tr> <td><strong>alert_manager.rs</strong><dd><code>Make alert/search jobs pausable and reactive</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-7e30858abad652a860186eb047347305488f52356112b1495dd53ce740316d7c">+61/-104</a></td> </tr> <tr> <td><strong>compactor.rs</strong><dd><code>Refactor compactor jobs to pausable macro</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-550629b7ebf6460b36fdd46cdfea466c3d3f1e30023c71ee54b87e7998efc500">+134/-188</a></td> </tr> <tr> <td><strong>parquet.rs</strong><dd><code>Use dynamic interval for file push loop</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-27f710981cfe1b938aeeb53abaed05961596c32e2edf3b8a2a59d3e72b2f542b">+1/-1</a> </td> </tr> <tr> <td><strong>flatten_compactor.rs</strong><dd><code>Flatten generator via pausable macro</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-dd30b917380ba55bd17aadeaee6aa2c87a67e861257a6a14ba2113a9b3015030">+15/-20</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Wire jobs with reactive spawning; telemetry</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-f2e4c29b8eef22189ec704a832633c4bd135a24444f054864c98064bd0963681">+21/-13</a> </td> </tr> <tr> <td><strong>promql.rs</strong><dd><code>Make metrics leader push pausable/optional</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-15414a41d64eb87bc5c0cd00a07213b3f99c31b346e8e2c36178f7276a1f058c">+26/-31</a> </td> </tr> <tr> <td><strong>stats.rs</strong><dd><code>Stats jobs via pausable macro; conditional start</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-f3ca67d9d2c68beeafd6e5de947dbc9a648ac2d2699c5dc87458c369887d09b9">+51/-42</a> </td> </tr> <tr> <td><strong>telemetry.rs</strong><dd><code>Remove legacy telemetry loop (replaced)</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-e82cf6d4c890bfaa5e711293b599d7f7a742bb96180639815ede33958b1bc1d9">+0/-36</a> </td> </tr> <tr> <td><strong>storage.rs</strong><dd><code>Make enrichment merge job pausable</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-335bb1eea9193dc398da6838c6a6a11dd6f09e984a1ed4a2e611618a427754f0">+42/-35</a> </td> </tr> <tr> <td><strong>distinct_values.rs</strong><dd><code>Periodic flush via pausable macro</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-3fb2320197c6e8e900658f5a24158e922610eadcbf081ab0df230f770ef27e07">+11/-18</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Audit publish via pausable macro, optional run</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-c6255010f7a87d6acdb3ad04954fc9f2c27350ae319c53a1857b120aa06408a3">+17/-15</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ |
||
|
sktrpathi
|
9a11a680ae |
chore: add test cases (#8095)
### **User description** - also clear over 400 clippy issues ___ ### **PR Type** Tests, Enhancement ___ ### **Description** Add extensive unit tests for enrichment storage Expose helpers for testability in storage module Modernize tests: clippy-friendly string formatting Normalize constants in time-based tests |
||
|
|
75f2235d04 |
feat: create stream with field and data type (#7851)
### **User description** closes #6660 ___ ### **PR Type** Enhancement, Bug fix ___ ### **Description** - Add field data types and UDS support - Validate field names with strict regex - Store defined schema as objects, not strings - Backward-compatible parsing and UI updates ___ ### Diagram Walkthrough ```mermaid flowchart LR API["API: Stream settings/schema"] --> Parse["Parse defined_schema_fields (strings|objects)"] Parse -- "normalize to DataField{name,type}" --> Settings["StreamSettings.defined_schema_fields: DataField[]"] Settings -- "new stream" --> SchemaAdd["Add fields to Arrow schema with types"] Settings -- "existing + UDS" --> UDSUpdate["Add new fields to UDS + persist"] Types["DataType enum (Utf8, Int64, Uint64, Float64, Boolean)"] --> Arrow["Arrow DataType mapping"] Validate["Regex field name validation"] --> Save["save/update settings endpoints"] UI["UI: Add/Edit stream fields"] --> ObjectFmt["Send/handle {name,type} objects"] ObjectFmt -- "compat" --> Compat["Store original + derive string list for display"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>10 files</summary><table> <tr> <td><strong>stream.rs</strong><dd><code>Rename stream property field to raw type key</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7851/files#diff-6f5ee19bbcc9cc7fb699fc1c0f43fc96d42698ae1168a2827b5c65f7948d5dbc">+5/-6</a> </td> </tr> <tr> <td><strong>stream.rs</strong><dd><code>Introduce DataType and DataField; settings parsing</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7851/files#diff-2438d04c3fe97bc5452e5b74a303163b8723d93455cda32c95f0525435ad46bd">+99/-13</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Optimize settings fetch; adapt defined fields helpers</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7851/files#diff-386fd4fad1e48ab0a90516cab1947b1888b270b2fe912f30d2255cc38e6c519b">+13/-6</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Use DataField names when building UDS sets</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7851/files#diff-7c7b5bca85f0ced652965cae66540c96f56dfcca4f4c16b77bf7b45aac203d1b">+6/-1</a> </td> </tr> <tr> <td><strong>schema.rs</strong><dd><code>Add DataField flows; new/UDS logic and conversions</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7851/files#diff-d9596182a0c7a46a6a7ff5cee822554cdae05ceec312f86c93b538db9f73bbfd">+270/-18</a></td> </tr> <tr> <td><strong>stream.rs</strong><dd><code>Validate field names; resolve defined field types</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7851/files#diff-41452a7fb29b4634437174fd6aecf6a9050dbfdef41b9077a21d59af3d79e9ed">+126/-7</a> </td> </tr> <tr> <td><strong>AddStream.vue</strong><dd><code>Send defined fields as objects with type</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7851/files#diff-78b8c84a558e60d942bacffee365633d074042e464be465eb6984b0ee5df82b9">+6/-2</a> </td> </tr> <tr> <td><strong>StreamFieldInputs.vue</strong><dd><code>UI for field data type and UX tweaks</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7851/files#diff-c8173413939f320f197b0f8790e619da18666f79009345f72b3325e89bbb7566">+90/-27</a> </td> </tr> <tr> <td><strong>schema.vue</strong><dd><code>Support object-based defined fields; sync logic</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7851/files#diff-9196be2ccae4054a00b0b1aa9d1e87b366c10920d6eaf9ef8c6e6293d50bcf46">+119/-25</a></td> </tr> <tr> <td><strong>useStreams.ts</strong><dd><code>Compare/sync object-based defined fields</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7851/files#diff-a083369344f022cdf8d4f536719179b1d920369e33e2e26234e70dd36d151a7c">+50/-2</a> </td> </tr> </table></details></td></tr><tr><td><strong>Bug fix</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Map UDS fields by object name not string</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7851/files#diff-619eafba0f0509d905b021f4a16659010aca23c62cffd9cd3e6c2bbe204ce200">+1/-1</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com> |
||
|
Subhra264andudev
|
c46fe2bf8c |
fix: new sso admin/viewer/editor should be able to enjoy given permissions (#8002)
Currently, openfga returns error when new sso user (admin/editor/viewer) tries to login into openobserve. This pr fixes that. --------- Co-authored-by: udev <uddhav@openobserve.ai> |
||
|
Huaijin
|
3eb271620a | refactor: try move remotescan, stream aggregate, topk from rewrite to optimize rule (#7998) | ||
 
|
1d0490875b |
fix: is_multi_stream_search and enable_align_histogram (#8003)
### **User description** Fixes: - https://github.com/openobserve/openobserve/issues/7986 ### **PR Type** Enhancement, Bug fix ___ ### **Description** - Add multi-stream search histogram support - Gate histogram alignment via query flag - Remove unused search_type plumbing - Propagate multi-stream flag through services ___ ### Diagram Walkthrough ```mermaid flowchart LR HTTP["HTTP handlers"] GRPC["gRPC handlers"] Utils["HTTP query utils"] SQL["SQL utils: histogram eligibility"] Hist["SQL: histogram query builder"] Svc["Search service"] Cache["Search cache"] Stream["Streaming pipeline"] Utils -- "parse is_multi_stream_search / enable_align_histogram" --> HTTP Utils -- "parse flags" --> GRPC HTTP -- "pass flags" --> Svc GRPC -- "pass flags" --> Svc Svc -- "eligibility check (multi/unified)" --> SQL Svc -- "partitioning uses align flag" --> Svc Svc -- "search() with multi-stream flag" --> Cache Cache -- "eligibility(query, multi)" --> SQL HTTP -- "convert_to_histogram_query(..., multi)" --> Hist Stream -- "do_search(..., multi)" --> Cache Hist -- "build single/multi-stream histogram query" --> Svc ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>15 files</summary><table> <tr> <td><strong>http.rs</strong><dd><code>Parse multi-stream and align histogram flags</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-3b50a69c1ec2f63f8eebb5f9880e17826463a586768aed3e3e15242833609d6b">+20/-0</a> </td> </tr> <tr> <td><strong>search.rs</strong><dd><code>Remove SearchPartitionRequest.search_type field</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-373e83a31e5bbc82fba05d7ae274b0dbd624ed748272232cf7eb8b8c3b2f3afc">+0/-3</a> </td> </tr> <tr> <td><strong>sql.rs</strong><dd><code>Add multi-stream histogram eligibility (UNION ALL)</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-a0a09c66a20d878ec7683ae222c919fbe092b8001871ade7f455b89573ce0911">+134/-38</a></td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Pass multi-stream flag into search service</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-8dcdb10340635be8bb58d45a815569026877f6fdbb3b69a773a30698b91674b0">+3/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Wire flags; histogram conversion uses multi-stream</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-96630a0dbb6dd7b2b0248fa325c7c7041ce31340c32826675a4a70772e9d29f6">+11/-7</a> </td> </tr> <tr> <td><strong>multi_streams.rs</strong><dd><code>Partition multi: add align flag; remove search_type</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-c3233cdf1e68cb2455f91457719b9c2f477e735ee16546fdfe111e76768bd054">+10/-9</a> </td> </tr> <tr> <td><strong>search_inspector.rs</strong><dd><code>Pass multi-stream flag in search profile</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-f818db4ca10b411c3559cc2a2659c49a59a45ecde0e4a22f9ef023da0bf4a009">+1/-0</a> </td> </tr> <tr> <td><strong>search_stream.rs</strong><dd><code>Stream API supports multi-stream histogram and flag</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-642de43ff6b19ae41087a9f1efc02d8b0b873809e0c92cbddfac1a30ef3f1184">+9/-3</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Propagate multi-stream flag in traces search</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-64694c886c809b6efe449e6eba9fbb33d31ce5a3c49cea1a30006fc1d9522665">+2/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Cache search considers multi-stream eligibility</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-f07bfacc0501b9c234e64b16c1dd8eb0ae8fcbe231f90c81fed72bcc94946f74">+2/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Align partitions via flag; multi partition API updates</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-c3b85ea662a885fbd39797968928b291cb59a58f53a03f7b7dfd7ba8b6a7985a">+22/-12</a> </td> </tr> <tr> <td><strong>histogram.rs</strong><dd><code>Build histogram for multi-stream queries (UNION ALL)</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-a5fc4cd11b3ec569b4cedf278ef0b14f540504abfbb23e29eb896d5b2a6cc2b2">+64/-8</a> </td> </tr> <tr> <td><strong>cache.rs</strong><dd><code>Streaming cache handles multi-stream flag</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-55769b7cb67b550daf45363c6d7d569b39d4a6235f6a68c091beb40f16b6511e">+3/-0</a> </td> </tr> <tr> <td><strong>execution.rs</strong><dd><code>Execution path passes multi-stream flag to searches</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-75d432351865224f0b00d7cc819554c2ad5fa1f777e9c645d91a3fd5c1113bc0">+25/-4</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Stream request processing threads multi-stream flag</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-f236c6c880ed80233fd264e657d5ab658c16c7368bdc70e20f123862b7214374">+4/-0</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ ___ ### **PR Type** Enhancement, Bug fix ___ ### **Description** - Add multi-stream histogram query support - Align histogram via query flag - Remove search_type plumbing in partitions - Propagate multi-stream flag end-to-end ___ ### Diagram Walkthrough ```mermaid flowchart LR Http["HTTP handlers"] Grpc["gRPC handlers"] Utils["HTTP query utils"] Front["Web client"] Service["Search service"] Cache["Search cache"] SQLU["SQL utils: histogram eligibility"] Hist["Histogram query builder"] Front -- "is_multi_stream, align flag" --> Http Utils -- "parse is_multi_stream/align" --> Http Http -- "enable_align_histogram" --> Service Http -- "is_multi_stream_search" --> Cache Grpc -- "flags default false" --> Service Cache -- "eligibility(sql, multi)" --> SQLU Http -- "convert_to_histogram_query(..., multi)" --> Hist Service -- "partition(gen interval if enabled)" --> Service ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>19 files</summary><table> <tr> <td><strong>http.rs</strong><dd><code>Parse multi-stream and align-histogram flags</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-3b50a69c1ec2f63f8eebb5f9880e17826463a586768aed3e3e15242833609d6b">+20/-0</a> </td> </tr> <tr> <td><strong>search.rs</strong><dd><code>Remove search_type from partition request</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-373e83a31e5bbc82fba05d7ae274b0dbd624ed748272232cf7eb8b8c3b2f3afc">+0/-4</a> </td> </tr> <tr> <td><strong>sql.rs</strong><dd><code>Histogram eligibility with multi-stream UNION ALL</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-a0a09c66a20d878ec7683ae222c919fbe092b8001871ade7f455b89573ce0911">+134/-38</a></td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Pass default multi-stream flag to service</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-8dcdb10340635be8bb58d45a815569026877f6fdbb3b69a773a30698b91674b0">+3/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add flags to HTTP search; align histogram handling</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-96630a0dbb6dd7b2b0248fa325c7c7041ce31340c32826675a4a70772e9d29f6">+11/-7</a> </td> </tr> <tr> <td><strong>multi_streams.rs</strong><dd><code>Partition multi endpoint supports align flag</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-c3233cdf1e68cb2455f91457719b9c2f477e735ee16546fdfe111e76768bd054">+10/-9</a> </td> </tr> <tr> <td><strong>search_inspector.rs</strong><dd><code>Forward multi-stream flag in inspector search</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-f818db4ca10b411c3559cc2a2659c49a59a45ecde0e4a22f9ef023da0bf4a009">+1/-0</a> </td> </tr> <tr> <td><strong>search_stream.rs</strong><dd><code>Streaming search supports multi-stream histogram</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-642de43ff6b19ae41087a9f1efc02d8b0b873809e0c92cbddfac1a30ef3f1184">+9/-3</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Forward multi-stream flag in traces helpers</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-64694c886c809b6efe449e6eba9fbb33d31ce5a3c49cea1a30006fc1d9522665">+2/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Eligibility check aware of multi-stream</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-f07bfacc0501b9c234e64b16c1dd8eb0ae8fcbe231f90c81fed72bcc94946f74">+2/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Align histogram in partition; multi aggregate</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-c3b85ea662a885fbd39797968928b291cb59a58f53a03f7b7dfd7ba8b6a7985a">+22/-12</a> </td> </tr> <tr> <td><strong>histogram.rs</strong><dd><code>Build single/multi-stream histogram SQL</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-a5fc4cd11b3ec569b4cedf278ef0b14f540504abfbb23e29eb896d5b2a6cc2b2">+64/-8</a> </td> </tr> <tr> <td><strong>cache.rs</strong><dd><code>Propagate multi-stream flag in streaming cache</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-55769b7cb67b550daf45363c6d7d569b39d4a6235f6a68c091beb40f16b6511e">+3/-0</a> </td> </tr> <tr> <td><strong>execution.rs</strong><dd><code>Pass multi flag through streaming execution</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-75d432351865224f0b00d7cc819554c2ad5fa1f777e9c645d91a3fd5c1113bc0">+25/-4</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Wire multi-stream flag across streaming path</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-f236c6c880ed80233fd264e657d5ab658c16c7368bdc70e20f123862b7214374">+4/-0</a> </td> </tr> <tr> <td><strong>IndexList.vue</strong><dd><code>Use align flag in partition requests</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-08f058f883b659e713e606991a7ea37ae1ff016b7cccef970489ab5840dc3d09">+1/-1</a> </td> </tr> <tr> <td><strong>usePanelDataLoader.ts</strong><dd><code>Toggle align histogram for dashboards</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-5a42e9493a7bcd722b88e61fd38883c4f61001bc9cafa399f65f8eafda3aace9">+3/-1</a> </td> </tr> <tr> <td><strong>useLogs.ts</strong><dd><code>Multi-stream histogram flow and params</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-1040f61354f92711e0e77ed4e9bfa15586d45a94c7e2ab06cc37372068030e7b">+3/-2</a> </td> </tr> <tr> <td><strong>search.ts</strong><dd><code>API params for is_multi_stream and align flag</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-885ab872dbc6e7b20264df815e3ef4727f0abc86ce90d2fb47e4aff572814c75">+7/-5</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Co-authored-by: Bhargav <BJP232004@GMAIL.COM> Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech> Co-authored-by: Shrinath Rao <shnath@openobserve.ai> Co-authored-by: omkark06 <omkar@zinclabs.io> Co-authored-by: ktx-akshay <akshay.patil@kiara.tech> |