mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-06 00:09:22 +00:00
78efd236ed7685fa111e8a33ba18ff01a7a502d4
698
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Simha GandOpenObserve
|
78efd236ed |
feat: Introduce query optimization recommendation service to background jobs (#8791)
### **User description**
# PR Type
Enhancement
# Description
This PR introduces a configurable query recommendation service / back
ground job that will take org wise usage data from metadata and generate
recommendations for the same and reingests this data into _meta org's
query_recommendation stream.
___
### **PR Type**
Enhancement
___
### **Description**
- Add enterprise query recommendation background job
- Expose config for schedule and limits
- Map meta stream to enterprise type
- Wire job init under enterprise feature
___
### Diagram Walkthrough
```mermaid
flowchart LR
CFG["Config limits: recommendation params"]
JOB["Background job: QueryRecommendationService"]
CTX["OptimizerContext (engine impl)"]
ORGS["List org IDs"]
STREAMS["Get stream settings"]
SEARCH["Run usage search"]
INGEST["Ingest recommendations to _meta/query_recommendations"]
CFG -- interval/duration/top_k --> JOB
JOB -- uses --> CTX
CTX -- fetch --> ORGS
CTX -- fetch --> STREAMS
CTX -- analyze --> SEARCH
CTX -- publish --> INGEST
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>stream.rs</strong><dd><code>Add enterprise Stream
conversion for recommendations</code>
</dd></summary>
<hr>
src/common/meta/stream.rs
<ul><li>Gate enterprise import behind feature flag<br> <li> Implement
From<Stream> to enterprise meta Stream</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8791/files#diff-6f5ee19bbcc9cc7fb699fc1c0f43fc96d42698ae1168a2827b5c65f7948d5dbc">+14/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>mod.rs</strong><dd><code>Wire query recommendation job
into init</code>
</dd></summary>
<hr>
src/job/mod.rs
<ul><li>Add enterprise-only module include<br> <li> Spawn recommendation
job at init</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8791/files#diff-f2e4c29b8eef22189ec704a832633c4bd135a24444f054864c98064bd0963681">+5/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>query_optimization_recommendation.rs</strong><dd><code>Implement
enterprise query recommendation service job</code>
</dd></summary>
<hr>
src/job/query_optimization_recommendation.rs
<ul><li>Add enterprise-only background job implementation<br> <li>
Implement engine trait with org/streams/search<br> <li> Ingest
recommendations into meta stream<br> <li> Run service using configurable
schedule</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8791/files#diff-85c42193130f71a11dc1d54d1a80a5888f1311857d5899b48f8c92973c21ae21">+124/-0</a>
</td>
</tr>
</table></td></tr><tr><td><strong>Configuration
changes</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>config.rs</strong><dd><code>Configure query
recommendation scheduling and limits</code>
</dd></summary>
<hr>
src/config/src/config.rs
<ul><li>Add recommendation duration config<br> <li> Add analysis
interval config<br> <li> Add top-k limit config<br> <li> Set sensible
defaults via env vars</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8791/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+7/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
Co-authored-by: OpenObserve <o2@OpenObserves-MacBook-Pro.local>
|
||
 ByteBaker
|
c868605736 | chore: add tests, fmt, clippy | ||
|
ByteBaker
|
da96d301ab |
refactor: extract reusable Headers extractor for actix-web handlers
This commit: - Creates a new extractors.rs module with a generic Headers<T> extractor - Implements FromRequest trait for type-safe header deserialization - Centralizes header parsing logic to reduce code duplication across HTTP handlers |
||
 udev
|
c0e8efe084 |
fix: node health api (#8744)
### **PR Type**
Bug fix
___
### **Description**
- Initialize health map on node add
- Remove from health map on node delete
- Fail when online node missing health entry
- Fix heartbeat comment typo
___
### Diagram Walkthrough
```mermaid
flowchart LR
add["Node added (watch_node_list)"] -- "insert" --> healthMap["NODES_HEALTH_CHECK[uuid]=0"]
del["Node deleted (watch_node_list)"] -- "remove" --> healthMap
check["check_nodes_status"] -- "missing entry" --> err["Return error + warn log"]
check -- "present entry" --> incr["Increment failure counter"]
note["Comment fix"] -- "typo" --> hb["heartbeat"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Bug
fix</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>mod.rs</strong><dd><code>Health check map lifecycle and
validation fixes</code>
</dd></summary>
<hr>
src/common/infra/cluster/mod.rs
<ul><li>Initialize <code>NODES_HEALTH_CHECK</code> on node add.<br> <li>
Remove <code>NODES_HEALTH_CHECK</code> entry on node delete.<br> <li>
Validate presence in health map; error if missing.<br> <li> Fix comment
typo: heartbeat.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8744/files#diff-d15434a27e2b8406811f5a707b605203b603dad4a262c0642326be860bd2b568">+16/-4</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
 Subhra264
|
41dc18866a |
feat: use cache for scheduled pipeline (#8707)
### **User description**
Currently we call the db when we process a scheduled pipeline and hence
increases db calls from the alert manager. This pr maintains a cache for
scheduled pipeline, so that alert manager does not make unnecessary db
calls.
___
### **PR Type**
Enhancement, Tests
___
### **Description**
- Add cache for scheduled pipelines
- Use cache in scheduler handler
- Extend pipeline cache/watch logic
- Add tests for cache behavior
___
### Diagram Walkthrough
```mermaid
flowchart LR
Scheduler["Scheduler handler"] -- "get/cache pipeline" --> PipelineDB["DB pipeline module"]
PipelineDB -- "reads/writes" --> ScheduledCache["Scheduled pipelines cache"]
RealtimeCache["Realtime exec cache"] -- "unchanged usage" --> Scheduler
Watcher["Pipelines watcher"] -- "update caches" --> ScheduledCache
Watcher -- "update caches" --> RealtimeCache
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>config.rs</strong><dd><code>Introduce global scheduled
pipelines cache</code>
</dd></summary>
<hr>
src/common/infra/config.rs
- Import `Pipeline` type.
- Add `SCHEDULED_PIPELINES` global cache.
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8707/files#diff-cda0d9179ed982fa5f0989c9b26f21cfa49dd1de28195a98135650ffcc7c602a">+3/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>handlers.rs</strong><dd><code>Scheduler uses cache for
pipeline lookups</code>
</dd></summary>
<hr>
src/service/alerts/scheduler/handlers.rs
<ul><li>Use scheduled pipeline cache before DB.<br> <li> Cache
DB-fetched pipelines on miss.<br> <li> Preserve existing error handling
path.<br> <li> Add debug logs for cache hits/misses.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8707/files#diff-5eca8995a88fcecb6d6b8efd3c7c4496f4a96a4b060fb4ed85593574f7cf49eb">+64/-40</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>pipeline.rs</strong><dd><code>Implement scheduled
pipelines cache and watcher integration</code></dd></summary>
<hr>
src/service/db/pipeline.rs
<ul><li>Add cache APIs: get/cache/remove/size/clear/stats.<br> <li>
Populate scheduled cache during full cache load.<br> <li> Extend watcher
to maintain scheduled cache.<br> <li> Improve logging and delete
handling for both caches.<br> <li> Add unit tests for scheduled cache
behavior.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8707/files#diff-0856951eadc646b4d4d3f0417317150cdbdc37b46c2f2a25a3c2bae15601dc50">+229/-57</a></td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
|
ByteBaker
|
33237447c4 |
chore: bump crate versions
- `async-walkdir`: v1.0.0 -> v2.1.0 - `maxminddb`: v0.25 -> v0.26 - `pprof`: v0.14 -> v0.15.0 - `prometheus`: v0.13 -> v0.14 - `promql-parser`: v0.4 -> v0.6 - `rcgen`: v0.13 -> v0.14.4 - `snafu`: v0.7.5 -> v0.8.9 - `tantivy`: v0.24 -> v0.25 - `vector-enrichment`: 063cabb -> 2003d61 - `vrl`: v0.22 -> v0.27.0 Removed `tokio-tungstenite` from dependencies. Also updated `vrl` to use git source from main branch as `vector-enrichment` needed syncing Changes to enrichment tables: - return early for empty conditions, avoiding computation - avoid vector collection before iterating again - added support for `FromDate` & `ToDate` condition types - simplified data processing using iterator chains - update `vector-enrichment` trait impl (add `wildcard` param) |
||
 Huaijin
|
ade98b64d1 |
perf: improve load speed for enrichment table in EnrichmentExec (#8679)
|
||
 Hengfei Yang
|
fafd4444ef | fix: search partition alignment (#8581) | ||
  
|
51d9782e5d |
feat: add separate telemetry url (#8564)
### **PR Type**
Enhancement
___
### **Description**
- Add keyevent telemetry support and API
- Introduce `ZO_KEYEVENT_TELEMETRY_URL` config
- Send both standard and keyevent telemetry
- Update handlers to use new API
___
### Diagram Walkthrough
```mermaid
flowchart LR
Cfg["Config: add ZO_KEYEVENT_TELEMETRY_URL"]
Tel["Telemetry: send_keyevent_track_event()"]
JWT["JWT auth: new user registered"]
Bill["Billing: new subscription started"]
Mkt["Marketing: new user attribution"]
Job["Job: no ingestion after creation"]
Cfg -- "provides URL" --> Tel
JWT -- "send standard + keyevent" --> Tel
Bill -- "send standard + keyevent" --> Tel
Mkt -- "send standard + keyevent" --> Tel
Job -- "send standard + keyevent" --> Tel
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>telemetry.rs</strong><dd><code>Add keyevent telemetry
send method</code>
</dd></summary>
<hr>
src/common/meta/telemetry.rs
<ul><li>Add <code>send_keyevent_track_event</code> method.<br> <li> Use
separate keyevent telemetry URL.<br> <li> Respect telemetry enabled flag
(non-cloud).<br> <li> Robust error/status logging for keyevent
sends.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8564/files#diff-3ff026b46d9431e0657828a3bde4ab118fec006b353c2abc80be4b74633c98b4">+60/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>jwt.rs</strong><dd><code>Emit keyevent telemetry on
user registration</code>
</dd></summary>
<hr>
src/handler/http/auth/jwt.rs
<ul><li>Instantiate telemetry once per flow.<br> <li> Send both standard
and keyevent events for registration.<br> <li> Duplicate payload reuse
for both sends.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8564/files#diff-232cab74f4cbf135af424744272e17732fbda8ac005d33c0acb4597e65479d02">+11/-1</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>billings.rs</strong><dd><code>Add keyevent telemetry
for subscription start</code>
</dd></summary>
<hr>
src/handler/http/request/cloud/billings.rs
<ul><li>Send standard and keyevent for new subscription.<br> <li> Reuse
event data across sends.<br> <li> Single telemetry instance per
request.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8564/files#diff-725936e3d6e955175e6100e8ca6f4437f334a2b554d6f11fa349ee9cbac70425">+11/-1</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>marketing.rs</strong><dd><code>Add keyevent telemetry
for attribution events</code>
</dd></summary>
<hr>
src/handler/http/request/cloud/marketing.rs
<ul><li>Send standard and keyevent for attribution.<br> <li> Share
segment event data between sends.<br> <li> Use single telemetry
instance.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8564/files#diff-424975d70871fd725536b17e4aa2131af8fd29e3c9d30ded551526e8ae71cfcc">+11/-1</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>cloud.rs</strong><dd><code>Add keyevent telemetry to
no-ingestion job</code>
</dd></summary>
<hr>
src/job/cloud.rs
<ul><li>Send standard and keyevent for no-ingestion report.<br> <li>
Reuse event data; single telemetry instance.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8564/files#diff-7fc16d008216c106d63f78f75c867fbefe7102433fcaa8893c314f0b7ea8d0e6">+11/-1</a>
</td>
</tr>
</table></td></tr><tr><td><strong>Configuration
changes</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>config.rs</strong><dd><code>Introduce keyevent
telemetry configuration</code>
</dd></summary>
<hr>
src/config/src/config.rs
<ul><li>Add <code>keyevent_telemetry_url</code> to
<code>Common</code>.<br> <li> Map to
<code>ZO_KEYEVENT_TELEMETRY_URL</code> env var.<br> <li> Default empty
string disables keyevent sending.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8564/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+2/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Prabhat Sharma <hi.prabhat@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
|
||
|
Hengfei Yang
|
24bc37a80e | feat: remove grpc call for get result cache also improve RW logic (#8246) | ||
|
6e75dc0e62 |
feat: add bulk enable/disable for alert and pipelines (#8438)
### **User description**
This adds two apis, one for bulk enable/disable of pipeline and one for
bulk enable/disable of alerts.
___
### **PR Type**
Enhancement
___
### **Description**
- Add bulk enable/disable for alerts
- Add bulk enable/disable for pipelines
- Introduce enterprise permission checks helper
- Update auth bypass for bulk endpoints
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["New models: Bulk request/response (alerts, pipelines)"]
B["Alerts: POST /v2/{org_id}/alerts/bulk/enable"]
C["Pipelines: POST /{org_id}/pipelines/bulk/enable"]
D["Enterprise permission check: check_resource_permissions"]
E["Router: register bulk endpoints"]
F["Auth: bypass for /bulk/enable paths"]
A -- "used by" --> B
A -- "used by" --> C
D -- "enforced in" --> B
D -- "enforced in" --> C
E -- "exposes" --> B
E -- "exposes" --> C
F -- "allows unauth extraction for" --> B
F -- "allows unauth extraction for" --> C
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>6
files</summary><table>
<tr>
<td><strong>auth.rs</strong><dd><code>Allow auth extractor bypass for
bulk enable endpoints</code> </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8438/files#diff-9bc2b1841504a435f66ad094f72ddbe7b7fa75e5e1557b8adbfe36f7f0592e1b">+1/-0</a>
</td>
</tr>
<tr>
<td><strong>responses.rs</strong><dd><code>Add alerts bulk enable
request/response models</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8438/files#diff-224027f90893122eea28e35778467acf1a4b5572ce612fc9bc9ec44ca8ebd9fb">+12/-0</a>
</td>
</tr>
<tr>
<td><strong>pipelines.rs</strong><dd><code>Add pipelines bulk enable
request/response models</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8438/files#diff-61bb28d7091130297c917a0530d96202fcb4b073b514410531f9fe71acfc814f">+12/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Implement alerts bulk enable
endpoint with permissions</code> </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8438/files#diff-83431801d27626ef27482e67b62661f900043544a45ddf958bd027cf2fa3bd8d">+91/-2</a>
</td>
</tr>
<tr>
<td><strong>pipeline.rs</strong><dd><code>Implement pipelines bulk
enable endpoint with permissions</code></dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8438/files#diff-c69f4f94eee97ce226caddc497b2adf89e26869f2d72b0d0206dc31ab2714daf">+89/-1</a>
</td>
</tr>
<tr>
<td><strong>utils.rs</strong><dd><code>Add generic enterprise resource
permission checker</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8438/files#diff-96bb9f4a8f547c821f26c957e8505d0437095b08d2b36687531f71565d287bd6">+38/-0</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Configuration
changes</strong></td><td><details><summary>1 files</summary><table>
<tr>
<td><strong>mod.rs</strong><dd><code>Register bulk enable endpoints in
service routes</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8438/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+2/-0</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>1
files</summary><table>
<tr>
<td><strong>openapi.rs</strong><dd><code>Expose bulk enable endpoints in
OpenAPI</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8438/files#diff-edb8ce232c0cd08366879f642788a75d99f6bab64b4c0e77391281fdb9241e90">+2/-0</a>
</td>
</tr>
</table></details></td></tr></tr></tbody></table>
</details>
___
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
|
||
|
Hengfei Yang
|
215cf796cd | feat: remove etcd (#8466) | ||
|
Hengfei Yang
|
97d09eff1e | fix: support prefix star match_all (#8451) | ||
|
Hengfei Yang
|
237c757c7c | fix: improve nats queue with limit policy (#8436) | ||
 Ashish Kolhe
|
34a7341e87 |
chore: node list & user session refresh apis (#8198) (#8200)
### **PR Type** Enhancement ___ ### **Description** - Add CLI commands to refresh caches - Expose HTTP endpoints for refresh actions - Cache node list building in cluster module - Replace retention flag with mode enum ___ ### Diagram Walkthrough ```mermaid flowchart LR CLI["CLI subcommands"] -- triggers --> HTTP["HTTP endpoints"] HTTP -- calls --> Cluster["cluster::cache_node_list"] HTTP -- calls --> Sessions["db::session::cache()"] Config["Config: file_list_deleted_mode"] -- controls --> Retention["Compactor retention logic"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>8 files</summary><table> <tr> <td><strong>cli.rs</strong><dd><code>New CLI subcommands for cache refresh</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8200/files#diff-cb1eb3e74723d891353ccf1e2e5e0d246d2e78a5e69dbe4c73006ea8412d65e0">+6/-0</a> </td> </tr> <tr> <td><strong>http.rs</strong><dd><code>HTTP client helpers for refresh endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8200/files#diff-790b7780a51e8eecd4754b623af8d15ad87baaaf29455ba646e2e410f83f5498">+22/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Cache node list and update consistent hash</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8200/files#diff-d15434a27e2b8406811f5a707b605203b603dad4a262c0642326be860bd2b568">+33/-0</a> </td> </tr> <tr> <td><strong>config.rs</strong><dd><code>Replace retention flag with mode string config</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8200/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+3/-2</a> </td> </tr> <tr> <td><strong>stream.rs</strong><dd><code>Add FileListBookKeepMode enum and conversions</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8200/files#diff-2438d04c3fe97bc5452e5b74a303163b8723d93455cda32c95f0525435ad46bd">+29/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add refresh nodes and user sessions endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8200/files#diff-ca09c9c9b020049bdbc3382c466bfcca80daac92e98760d9a9a76172abf34049">+20/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Wire refresh endpoints into basic routes</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8200/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+3/-1</a> </td> </tr> <tr> <td><strong>retention.rs</strong><dd><code>Use retention mode for history/deleted behavior</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8200/files#diff-6abd048f5ef4e7163cffbb51cb4bee10c07fb88b871469d5b3724f7cd04743dc">+16/-6</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ |
||
|
Subhra264
|
6f40d2ffc0 | feat: use nats queue for cluster coordination (#8413) | ||
|
Hengfei Yang
|
b72e85e0fe | feat: support nats ttl marker (#8377) | ||
|
0730d5f7a4 |
fix: landing page / Home page revamp (#6729)
### **User description**
1. This PR implements the latest UI of landing page / Home page with 2
charts for showing alerts and pipelines
2.
___
### **PR Type**
Enhancement
___
### **Description**
- Redesigned Home page layout and tiles
- Added alerts and pipelines charts
- Implemented theme-aware icons and styles
- Added loading state for summary fetch
___
### Diagram Walkthrough
```mermaid
flowchart LR
HomeView["HomeView.vue"]
SummaryAPI["Summary fetch & parsing"]
Charts["CustomChartRenderer charts"]
Tiles["Responsive metric tiles"]
Theming["Dark/Light computed icons & styles"]
Loading["Loading spinner state"]
HomeView -- "fetches org summary" --> SummaryAPI
SummaryAPI -- "sets metrics & statuses" --> Tiles
SummaryAPI -- "feeds alert/pipeline data" --> Charts
HomeView -- "applies theme classes" --> Theming
HomeView -- "shows while fetching" --> Loading
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>HomeView.vue</strong><dd><code>Home page redesign with
charts and theme-aware UI</code>
</dd></summary>
<hr>
web/src/views/HomeView.vue
<ul><li>Rebuilt template with responsive sections and tiles.<br> <li>
Added alerts pie and pipelines bar charts via CustomChartRenderer.<br>
<li> Introduced loading state, improved summary parsing, and theme-aware
<br>icons.<br> <li> Added extensive SCSS for dark/light theming and
layout.</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/6729/files#diff-9af441af191d3254ef5f9db603dca75486de7638cda41a3c050e7b3c33d68280">+568/-229</a></td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: ByteBaker <42913098+ByteBaker@users.noreply.github.com>
|
||
|
Hengfei Yang
|
55a0c62d3f | fix: root user password reset (#8348) | ||
|
sktrpathi
|
f8cc585408 |
feat: store ratelimit info in vendor extension (#8259)
### **PR Type** Enhancement, Documentation ___ ### **Description** - Replace ratelimit comments with OpenAPI extensions - Add `x-o2-ratelimit` metadata to endpoints - Import `serde_json::json` where needed - Clean up obsolete inline ratelimit markers |
||
 
|
8af3495611 |
fix: cloud sync (#8222)
### **User description**
This is for syncing the changes done in cloud to main branch. No changes
for non-cloud code.
___
### **PR Type**
Bug fix, Enhancement
___
### **Description**
- Add invite listing and decline APIs
- Bypass auth for user-scoped invite routes
- Improve trial-expiry handling and logging
- Add org cache and invalidation hooks
___
### Diagram Walkthrough
```mermaid
flowchart LR
FE["Login & Invitations UI"] -- "list/accept/decline" --> API["/api/invites*, /{org}/member_subscription"]
API -- "auth bypass for user-scoped invites" --> Auth["AuthExtractor/TokenValidator"]
Auth -- "pending invites flag" --> LoginRedir["/config/redirect builds URL params"]
OrgSvc["Organization Service"] -- "generate/list/decline/delete" --> InvitesDB["Org Invites"]
OrgSvc -- "invalidate cache" --> OrgCache["Org table in-memory cache"]
Ingest["Ingestion services"] -- "trial expired -> 429" --> Clients["Ingestion Clients"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>25
files</summary><table>
<tr>
<td><strong>user.rs</strong><dd><code>Expand invite payload with org and
inviter</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-e13b3a981fd70182b9738a6b0ecfdee915c5b36757703a87741e1eddf3f6e786">+2/-0</a>
</td>
</tr>
<tr>
<td><strong>auth.rs</strong><dd><code>Auth bypass and key handling for
invite routes</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-9bc2b1841504a435f66ad094f72ddbe7b7fa75e5e1557b8adbfe36f7f0592e1b">+36/-0</a>
</td>
</tr>
<tr>
<td><strong>jwt.rs</strong><dd><code>Return new user and pending invite
flags; defer org creation</code></dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-232cab74f4cbf135af424744272e17732fbda8ac005d33c0acb4597e65479d02">+22/-4</a>
</td>
</tr>
<tr>
<td><strong>token.rs</strong><dd><code>Allow invite/member routes
without org user; header injection</code></dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-3339b4a964c121766ef87fcec0070f27a2b7085cf81281b53154e53d33cb28cf">+34/-0</a>
</td>
</tr>
<tr>
<td><strong>ingest.rs</strong><dd><code>Suppress logging for trial
expiry errors</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-24f250254ecfdbe004af9dc0ea23f8a2c0105ae0898690ed3e565089e5d8eef9">+30/-9</a>
</td>
</tr>
<tr>
<td><strong>org.rs</strong><dd><code>Pending-only invites; delete invite
API; trial extend cache poke</code></dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-b47c0c1a8a78524196176430268eecdbfaf6f46d6b95873d3cbe70cfd67182b0">+40/-4</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Include pending_invites flag in
login redirect</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-ca09c9c9b020049bdbc3382c466bfcca80daac92e98760d9a9a76172abf34049">+4/-4</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>List pending invites and decline
invite endpoint</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-56897e42f9436a640bc70fddc25686485ba8ae1223635b32d252a0333309f748">+36/-1</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Wire new invite endpoints into
router</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+2/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Add TrialPeriodExpired error
variant</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-90c725d0e77a6e7d4bcc130cf5783848a5cf0efdb304b351cd63cacf8775fc36">+2/-0</a>
</td>
</tr>
<tr>
<td><strong>organizations.rs</strong><dd><code>Add in-memory org cache
with invalidation</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-4d87f9fe379a5d161a3e7c3c77f7f320be738e6a70c785aa5ce93cb0830c0147">+77/-7</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Start cloud billing watcher on
cloud builds</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-f2e4c29b8eef22189ec704a832633c4bd135a24444f054864c98064bd0963681">+3/-0</a>
</td>
</tr>
<tr>
<td><strong>organization.rs</strong><dd><code>Invalidate org cache on
watch and mutations</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-ce65cea926a09721a8fea14d07fe188bc2b887218e95d74a8bffefb54001da50">+4/-0</a>
</td>
</tr>
<tr>
<td><strong>user.rs</strong><dd><code>Delete invites for user
helper</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-6ee6b0fc9ed03224d4ddf0b7b631c6f81ff7d1d579bdb98d2c5d4453852eb5d6">+4/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Return trial expiry as dedicated
error</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-7c7b5bca85f0ced652965cae66540c96f56dfcca4f4c16b77bf7b45aac203d1b">+1/-3</a>
</td>
</tr>
<tr>
<td><strong>loki.rs</strong><dd><code>Silence trial expiry logs for Loki
ingestion</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-a970c8c4785d4e4d878218615b17eec0cf4b4976121766b002c2b2f76371bfe6">+4/-1</a>
</td>
</tr>
<tr>
<td><strong>json.rs</strong><dd><code>Map trial expiry to HTTP 429 in
JSON ingest</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-506987e61095a54e47f7f208dad89f9f1bce595fa6bf1ba8a7d746f2b2ff6c4a">+15/-6</a>
</td>
</tr>
<tr>
<td><strong>otlp.rs</strong><dd><code>Return 429 for trial expiry;
reduce noise</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-a24391bef75a4678c50724e8b5f305ec0ee8c2c74c68be9d677765de68b5d8dd">+17/-7</a>
</td>
</tr>
<tr>
<td><strong>organization.rs</strong><dd><code>Invite generation
validation; delete/decline invite flows</code></dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-6b3c3c2b027c1158c0d41302cc69dfcd20ab68a6a4fa365cda403f838940bbb4">+48/-1</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Handle trial expiry with 429;
logging tweaks</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-66f3b4f5e7927092fd6e57738687a1091b6f316638f7c58f8d7be9addb72bc30">+34/-14</a>
</td>
</tr>
<tr>
<td><strong>users.rs</strong><dd><code>Enrich invite list; optional
pending filter; cleanup on removal</code></dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-98da2588a28d62bb2e301d989836125f89f89b76272cd7f75e08ec4deeea5f38">+50/-6</a>
</td>
</tr>
<tr>
<td><strong>organizations.ts</strong><dd><code>Add decline subscription
API client</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-2c3bad4c7a9fcedd7a653599381149b82f9c350b177de9d72431013807d4c36f">+10/-7</a>
</td>
</tr>
<tr>
<td><strong>users.ts</strong><dd><code>Add pending/accept/reject invite
APIs</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-a334d8d570d9bdcaa3d4c8c2a993eff009a1ec8eb499c8a3d1a292d9224159d0">+12/-4</a>
</td>
</tr>
<tr>
<td><strong>InvitationList.vue</strong><dd><code>New UI to accept or
reject invites</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-ee97546ff1aba36cc0b24cb20782502c7c88279850cc3d540ea5685209f10702">+409/-0</a>
</td>
</tr>
<tr>
<td><strong>Login.vue</strong><dd><code>First-login invite handling and
logo tweaks</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-a135c379ef6ac8f92199ea5f7ff19a227971a9b0a87ccccf87c0268740522c2c">+178/-58</a></td>
</tr>
</table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>1
files</summary><table>
<tr>
<td><strong>en.json</strong><dd><code>Add invitation-related
translations</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8222/files#diff-892aa3d75ab6e129ef9116aa21bec1b9a15ba69ce5366d2a19128078c690d824">+18/-0</a>
</td>
</tr>
</table></details></td></tr></tr></tbody></table>
</details>
___
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Subhra264 <chakrabortysubhradeep556@gmail.com>
Co-authored-by: udev <uddhav@openobserve.ai>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: udev <uddhavdave1@gmail.com>
|
||
|
sktrpathi
|
d87d346bba |
fix: OpenAPI descriptions for endpoints (#8231)
### **User description** So far the OpenAPI descriptions for endpoint handlers was directly derived from the doc comments, leaving little control. This also leaked any dev-specific information into API spec. Now the summary and description are explicitly populated for all the endpoints, detaching them from doc comments. Also update response body annotations wherever necessary Also bump - utoipa: 4.0.0 -> 5.0.0 - utoipa-swagger-ui: 4.0.0 -> 9.0.0 ___ ### **PR Type** Documentation, Enhancement ___ ### **Description** - Add explicit OpenAPI summaries/descriptions - Correct response body schemas to Object/() - Update params/types for accuracy - Improve endpoint examples and metadata |
||
|
Subhra264
|
c49df079c7 |
test: add unit tests for alerts and self reporting (#8228)
### **PR Type**
Tests
___
### **Description**
- Add comprehensive unit tests for alerts
- Add extensive self-reporting queue tests
- Add serialization and model tests for usage
- Validate batching and timing behaviors
___
### Diagram Walkthrough
```mermaid
flowchart LR
Alerts["Alert model"] -- "unit tests" --> AlertsTests["alerts::tests"]
Reporting["Self-reporting queue/runner"] -- "unit tests" --> ReportingTests["self_reporting::tests"]
Usage["Usage/Trigger types"] -- "serde, traits tests" --> UsageTests["usage::tests"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>alert.rs</strong><dd><code>Add comprehensive tests for
Alert and params</code>
</dd></summary>
<hr>
src/config/src/meta/alerts/alert.rs
<ul><li>Add tests for <code>Alert</code> defaults and equality<br> <li>
Validate unique key logic and timestamps getters/setters<br> <li> Test
<code>ListAlertsParams</code> builder methods and chaining<br> <li>
Cover ancillary fields: destinations, context, owner/editor, flags</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8228/files#diff-45c7eaf1a764ab52f3e014fe0555c03dc59d77c5ce974684cdee4ed90463a234">+415/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>mod.rs</strong><dd><code>Add tests for self-reporting
queue and runner</code>
</dd></summary>
<hr>
src/config/src/meta/self_reporting/mod.rs
<ul><li>Add async tests for <code>ReportingQueue</code>
enqueue/start/shutdown<br> <li> Test batching logic in
<code>ReportingRunner</code><br> <li> Validate debug formatting and
mixed data handling<br> <li> Add ignored test for channel-full
behavior</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8228/files#diff-f4101fa8423f7a25f311dd82dab6796ed4ac8fef68449b5f018157dc33ca7de3">+674/-0</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>usage.rs</strong><dd><code>Add serialization and model
tests for usage types</code>
</dd></summary>
<hr>
src/config/src/meta/self_reporting/usage.rs
<ul><li>Add serde tests for <code>UsageData</code>,
<code>TriggerData</code>, enums<br> <li> Test display helpers and
classification methods<br> <li> Validate <code>RequestStats</code> and
<code>Stats</code> defaults/from/serde<br> <li> Test hashing/equality
for grouping keys and enums</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/8228/files#diff-b15a6adcb163dc27993c83528105a6726c44c86d8a84c5cf5d1d382c6ec8cc29">+684/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
|
Hengfei Yang
|
1989432ba7 |
feat: remove syslog (#8227)
### **User description** close #8186 ___ ### **PR Type** Enhancement, Documentation ___ ### **Description** - Remove syslog feature across codebase - Clean config and routing references - Update metadata prefixes and SBOM files - Refresh i18n and OpenAPI assets ___ ### Diagram Walkthrough ```mermaid flowchart LR SYS["Syslog feature"] -- "deprecated & removed" --> CODE["Code paths"] CODE -- "delete flags/routes" --> CFG["Configuration"] CODE -- "remove storage keys" --> META["Metadata prefixes"] CODE -- "update specs" --> SBOM["SBOM/XML assets"] CODE -- "update docs/i18n" --> WEB["Web locales & OpenAPI"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>13 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Remove syslog config import from job module</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-f2e4c29b8eef22189ec704a832633c4bd135a24444f054864c98064bd0963681">+3/-26</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Stop glob importing config; drop syslog usage</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-ca09c9c9b020049bdbc3382c466bfcca80daac92e98760d9a9a76172abf34049">+1/-3</a> </td> </tr> <tr> <td><strong>meta.rs</strong><dd><code>Drop `/syslog` from metadata item prefixes</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-1797798cd07803457a6273ef793d56568d8c91e743f6c4649c6db733325fb0af">+1/-2</a> </td> </tr> <tr> <td><strong>config.rs</strong><dd><code>Remove syslog-related configuration options</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+0/-34</a> </td> </tr> <tr> <td><strong>config.rs</strong><dd><code>Purge syslog feature flag and defaults</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-cda0d9179ed982fa5f0989c9b26f21cfa49dd1de28195a98135650ffcc7c602a">+0/-25</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Clean references tied to syslog ingestion</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-5061e662a8899b7e00e5744bd72f808b28ac479a8921ac519f7409bffdc4438a">+0/-60</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Delete syslog router endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+0/-5</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Detach syslog handler wiring</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-99620cff6123acf0598ed2ec9d1c06f02cb0ddece1bef898047eb303d0b71fc2">+0/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Remove syslog-specific log ingestion paths</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-ce72aea3fa4c44904cbd9ec07bd844bec71b49c9676d667b14056aa5798f96b2">+0/-2</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Clean DB interactions for syslog keys</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-b7e3bf7f0ac23dd084c0c817a310a4d5e7a64ff532746d182986ef940fa03fd0">+0/-2</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Adjust request handling without syslog</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-a1cfd40da1ab6cd4db0181fd8664965034298047b3b7c3320c5385897a7c4aed">+0/-2</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Unexport syslog services</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-1b15316505c31acccb26ba66d920fdb0894567d08d2bf1e3457716d128a67b83">+0/-2</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Remove syslog metadata definitions</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-fbf797574839e69491ea0429141c038c0ff96297bdf3d8c60d60cdc4acec74b1">+0/-2</a> </td> </tr> </table></details></td></tr><tr><td><strong>Tests</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>integration_test.rs</strong><dd><code>Update tests to exclude syslog paths</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-7ff815b68943f08bef79f083c09e55ca8c18db8e25033034b12fa31f1d43dab4">+0/-57</a> </td> </tr> <tr> <td><strong>useIngestionRoutes.spec.ts</strong><dd><code>Update UI tests to remove syslog routes</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-3b7ebacf9481f4643a2b56ffe53ab161fc89d416e280d45b48568866a4dee40c">+0/-1</a> </td> </tr> </table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>20 files</summary><table> <tr> <td><strong>openapi.rs</strong><dd><code>Remove syslog routes from OpenAPI</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-edb8ce232c0cd08366879f642788a75d99f6bab64b4c0e77391281fdb9241e90">+0/-5</a> </td> </tr> <tr> <td><strong>config.cdx.xml</strong><dd><code>Update config component index (remove syslog)</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-41ee11320f37e20fc96f754d35a890df5068af5072fd5388169d1cf305a1ed3b">+15011/-4611</a></td> </tr> <tr> <td><strong>wal.cdx.xml</strong><dd><code>Update WAL component index references</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-3d72e8b949a91b0bd78e97aa3ebf69e2887e2851223ea3d10fedeb0125c35a19">+19476/-358</a></td> </tr> <tr> <td><strong>openobserve.cdx.xml</strong><dd><code>Refresh SBOM root; remove syslog entries</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-454f38f03cf69f6b5358f5a528be4214ec2d33081b9e98ebe5d92c8bef47dc48">+14015/-7932</a></td> </tr> <tr> <td><strong>infra.cdx.xml</strong><dd><code>Update infra component index, no syslog</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-e51bf18198bc65a604b53ee18152ad9f2453ab7870d5be949022058e054ada73">+14127/-6086</a></td> </tr> <tr> <td><strong>proto.cdx.xml</strong><dd><code>Update proto index; purge syslog mentions</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-a15fa99b53917ea85b4e268bb7612d43ee881dbe4ecf70a6682bb919b1f2f219">+8000/-1513</a></td> </tr> <tr> <td><strong>ingester.cdx.xml</strong><dd><code>Adjust ingester index excluding syslog</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-e72a9be4ba2aaff1a6b837a7f7149489603f8fe720766757c4291082e98b5a65">+15845/-4615</a></td> </tr> <tr> <td><strong>flight.cdx.xml</strong><dd><code>Add/refresh flight component index</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-f41393a0191b0560da893a5e24b2e4529f7c9bb34fc71231a532b8790630cba2">+19900/-0</a></td> </tr> <tr> <td><strong>hi.json</strong><dd><code>Remove syslog strings from Hindi locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-1e7b52b5c4e762db2c923d0d3efb6d693702825ed3a1a112d3069f26b0d5f605">+0/-13</a> </td> </tr> <tr> <td><strong>fr.json</strong><dd><code>Remove syslog strings from French locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-db80e762b88ddf183820c050cdc6b215e6b02b365b5e73f5152af46b0d26f1a2">+0/-13</a> </td> </tr> <tr> <td><strong>tr.json</strong><dd><code>Remove syslog strings from Turkish locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-6f0f131e9e2591480c4c5d1eaffc72d02dc31cbd8d3f90d8f012f08cabac8818">+0/-13</a> </td> </tr> <tr> <td><strong>de.json</strong><dd><code>Remove syslog strings from German locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-78f24ecb2dbcfc38fdf3ec22b2559a5b40d236ea689d1c55f5d3e87168310264">+0/-13</a> </td> </tr> <tr> <td><strong>ko.json</strong><dd><code>Remove syslog strings from Korean locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-50e27ca1b4825a5026f3a251bfcf79c97cab07cfade2ebd512d332453d707c48">+0/-13</a> </td> </tr> <tr> <td><strong>it.json</strong><dd><code>Remove syslog strings from Italian locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-2de78074a5f887afddce60d694ead9322e6ab223eb70251c13819f3028bdc70d">+0/-13</a> </td> </tr> <tr> <td><strong>nl.json</strong><dd><code>Remove syslog strings from Dutch locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-a1706021687ae5d8a1c006c49eab3946174fecd895f9b70a89c2e76d97250906">+0/-13</a> </td> </tr> <tr> <td><strong>es.json</strong><dd><code>Remove syslog strings from Spanish locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-ac2a2167b8b973e3f776f9fe3b43bef4576470d8ad33f3d7cfaff6edff0e5778">+0/-13</a> </td> </tr> <tr> <td><strong>pt.json</strong><dd><code>Remove syslog strings from Portuguese locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-38e804f64cb5e0e451849fc84e9b46e53c954a4a4263d51c18665ad45f4b4815">+0/-13</a> </td> </tr> <tr> <td><strong>zh.json</strong><dd><code>Remove syslog strings from Chinese locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-8818055db58f83230aeea4ebfd4cec31d924e8c47b6f91298e356959ab25339e">+0/-13</a> </td> </tr> <tr> <td><strong>ja.json</strong><dd><code>Remove syslog strings from Japanese locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-092e49c5f9fbc1b539ab9c2866d108d0ce1d314022b4c294eb4611b285e7bbd3">+0/-13</a> </td> </tr> <tr> <td><strong>en.json</strong><dd><code>Remove syslog strings from English locale</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-892aa3d75ab6e129ef9116aa21bec1b9a15ba69ce5366d2a19128078c690d824">+0/-13</a> </td> </tr> </table></details></td></tr><tr><td><strong>Dependencies</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>Cargo.toml</strong><dd><code>Update dependencies/features after syslog removal</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-2e9d962a08321605940b5a657135052fbcef87b5e360662bb527c96d9a615542">+0/-3</a> </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>10 files</summary><table> <tr> <td><strong>syslog.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-dc6813e31e32c23aae5760091460d2ec4023d37ae73e49d29052d9f4e9f2e1fa">+0/-43</a> </td> </tr> <tr> <td><strong>websocket.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-48f05dcd4b6f1d8345cb5707e1a3c34eedd76d94a3c3904eeabd799c7820789d">+0/-90</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-58010c6e78f69dc57c353890a807371cd43c2b4cceeeb1e9378130f26d898397">+0/-143</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-66207bd699343342402f88cdd331f4c549783754e4921d0c8d43e026e2d8e575">+0/-135</a> </td> </tr> <tr> <td><strong>syslog_server.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-cb7610142582b44885f673583ab16262c3b9ca6de1f59a9def65e503360665a0">+0/-137</a> </td> </tr> <tr> <td><strong>report_server.cdx.xml</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-d7a2fd1b635919119af9187ef44188c384bd01e444e74a9d15eaa21b0840b7ed">+20472/-0</a></td> </tr> <tr> <td><strong>syslog.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-4c0e9282e3d2352f9a382099901fe034bae2961eed01d764f16ad52f27f55db4">+0/-180</a> </td> </tr> <tr> <td><strong>syslog.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-4bc46f36fad79f5df8c42b8a8d5d8c4a70b1320b966bcff3421deeb96a1981fa">+0/-559</a> </td> </tr> <tr> <td><strong>syslogs_route.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-325df35568c993fc0727bfb0499b682e6bd30af781687706306b3bdb429c49e2">+0/-191</a> </td> </tr> <tr> <td><strong>syslog.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8227/files#diff-69ec0272ddac65f2cc96c79d84ca3933fc335a8d21ab5dd6909d71ea02a46486">+0/-37</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ |
||
 Hengfei YangandBhargav
|
2c0395124f |
feat: deprecated syslog (#8187)
### **User description** impl #8186 ___ ### **PR Type** Enhancement, Tests ___ ### **Description** - Mark syslog modules as deprecated - Add allow(deprecated) to callers - Block enabling syslog setting - Remove syslog-related tests and OpenAPI types ___ ### Diagram Walkthrough ```mermaid flowchart LR meta["meta::syslog module"] -- "mark deprecated" --> callers["call sites"] callers -- "add allow(deprecated)" --> compile_ok["compiles with warnings suppressed"] db_set["db::syslog::toggle_syslog_setting"] -- "reject enable=true" --> error["returns deprecation error"] openapi["OpenAPI schema"] -- "remove syslog types" --> spec["updated API spec"] tests["syslog tests"] -- "removed" --> leaner["reduced test surface"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>8 files</summary><table> <tr> <td><strong>config.rs</strong><dd><code>Suppress deprecation warnings for syslog globals</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-cda0d9179ed982fa5f0989c9b26f21cfa49dd1de28195a98135650ffcc7c602a">+2/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Allow deprecated syslog handlers; annotate endpoints</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-58010c6e78f69dc57c353890a807371cd43c2b4cceeeb1e9378130f26d898397">+6/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Suppress deprecation in API scope setup</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+1/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Allow deprecated syslog server wiring</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-66207bd699343342402f88cdd331f4c549783754e4921d0c8d43e026e2d8e575">+3/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Allow deprecated syslog cache/watch tasks</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-f2e4c29b8eef22189ec704a832633c4bd135a24444f054864c98064bd0963681">+4/-0</a> </td> </tr> <tr> <td><strong>syslog_server.rs</strong><dd><code>Allow deprecated syslog server run linkage</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-cb7610142582b44885f673583ab16262c3b9ca6de1f59a9def65e503360665a0">+2/-0</a> </td> </tr> <tr> <td><strong>syslog.rs</strong><dd><code>Allow deprecated syslog ingest; remove tests</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-4bc46f36fad79f5df8c42b8a8d5d8c4a70b1320b966bcff3421deeb96a1981fa">+3/-14</a> </td> </tr> <tr> <td><strong>syslogs_route.rs</strong><dd><code>Allow deprecated route endpoints; remove tests</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-325df35568c993fc0727bfb0499b682e6bd30af781687706306b3bdb429c49e2">+7/-52</a> </td> </tr> </table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>6 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Mark syslog meta module as deprecated</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-fbf797574839e69491ea0429141c038c0ff96297bdf3d8c60d60cdc4acec74b1">+1/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Mark syslog HTTP request module deprecated</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-a1cfd40da1ab6cd4db0181fd8664965034298047b3b7c3320c5385897a7c4aed">+1/-0</a> </td> </tr> <tr> <td><strong>openapi.rs</strong><dd><code>Remove syslog types from OpenAPI imports</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-edb8ce232c0cd08366879f642788a75d99f6bab64b4c0e77391281fdb9241e90">+0/-2</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Deprecate syslog DB module</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-b7e3bf7f0ac23dd084c0c817a310a4d5e7a64ff532746d182986ef940fa03fd0">+1/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Deprecate syslog logs module</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-ce72aea3fa4c44904cbd9ec07bd844bec71b49c9676d667b14056aa5798f96b2">+1/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Deprecate syslogs_route service module</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-1b15316505c31acccb26ba66d920fdb0894567d08d2bf1e3457716d128a67b83">+1/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Tests</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>syslog.rs</strong><dd><code>Remove unit tests for syslog meta types</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-dc6813e31e32c23aae5760091460d2ec4023d37ae73e49d29052d9f4e9f2e1fa">+0/-127</a> </td> </tr> </table></details></td></tr><tr><td><strong>Error handling</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>syslog.rs</strong><dd><code>Block enabling syslog; allow deprecated routes/cache</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8187/files#diff-4c0e9282e3d2352f9a382099901fe034bae2961eed01d764f16ad52f27f55db4">+11/-0</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Co-authored-by: Bhargav <BJP232004@GMAIL.COM> |
||
|
sktrpathi
|
75fc81cecf |
chore: remove bogus unit tests (#8172)
### **User description** A significant chunk of API unit tests were not testing any actual business logic, but performing trivial actions such as struct-field-access, derive trait validation such as `Eq`, `PartialEq`, `Clone`, `Serialize` etc, and many of the tests were also just comparing constant values. They added no value to the test suite and have now been removed from the codebase, making the tests more focused. Additionally, - A lot of flaky tests have been corrected. These used to fail due to occasional race conditions when ran locally. - Some mock setup and teardown functions have also been cleaned up, which makes tests a lot more consistent. ___ ### **PR Type** Tests, Enhancement ___ ### **Description** - Remove trivial, non-value unit tests - Relax flaky assertions and panics handling - Improve test isolation via DB/cache resets - Minor test output formatting improvements ___ ### Diagram Walkthrough ```mermaid flowchart LR tests["Remove trivial/boilerplate tests"] -- cleaner suite --> stability["Stabilize assertions and panics"] stability -- reduce flakiness --> isolation["DB/Cache resets in tests"] isolation -- consistent runs --> outputs["Improve test log formatting"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Tests</strong></td><td><details><summary>11 files</summary><table> <tr> <td><strong>organization.rs</strong><dd><code>Remove struct-field and default-value trivial tests</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-4f089715a7ac4f78b45039d773f2fccbb4fbc6a409e0ab9ceee1fcf7911bc44b">+4/-325</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Drop JSON roundtrip serialization tests; tidy imports</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-c4bd542e4e07ed7706b0643086f5a7d4edbb3647c7ceedbe4bea516c5274d8bd">+1/-183</a> </td> </tr> <tr> <td><strong>auth.rs</strong><dd><code>Remove Debug/PartialEq trivial tests; format strings</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-9bc2b1841504a435f66ad094f72ddbe7b7fa75e5e1557b8adbfe36f7f0592e1b">+2/-104</a> </td> </tr> <tr> <td><strong>main.rs</strong><dd><code>Make tracing test tolerant; ignore enterprise tests</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-42cb6807ad74b3e201c5a7ca98b911c5fa08380e942be6e4ac5807f8377f87fc">+23/-60</a> </td> </tr> <tr> <td><strong>event.rs</strong><dd><code>Remove constants and simple struct creation tests</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-689769712cbd6bd02beba078c1e1dfc188e2be39ab19a0e232b8bd12bc7f1e78">+1/-95</a> </td> </tr> <tr> <td><strong>validator.rs</strong><dd><code>Ensure test isolation with ORM init and cache clears</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-f4d4b51574b378a8c372345ea48875834a4eadacfac33fa4d0ceca1e0ef96b35">+28/-22</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Make base URI tests robust to env errors</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-d7987e62dcc7b1ec1bd4aeabef94f85d8c41b2ea33de5ab78660794fa4d51a9e">+25/-4</a> </td> </tr> <tr> <td><strong>users.rs</strong><dd><code>Use best-effort cleanup; clear caches; improve logs</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-98da2588a28d62bb2e301d989836125f89f89b76272cd7f75e08ec4deeea5f38">+12/-9</a> </td> </tr> <tr> <td><strong>integration_test.rs</strong><dd><code>Compact println formatting and panic messages</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-7ff815b68943f08bef79f083c09e55ca8c18db8e25033034b12fa31f1d43dab4">+3/-3</a> </td> </tr> <tr> <td><strong>config.rs</strong><dd><code>Remove strict zero-length assertions on globals</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-cda0d9179ed982fa5f0989c9b26f21cfa49dd1de28195a98135650ffcc7c602a">+1/-13</a> </td> </tr> <tr> <td><strong>schema.rs</strong><dd><code>Remove unused constant import in tests</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-a5ea87a7766cb24619f64a6100f385f4b42c47af7d33b8e2f060229c536d7418">+1/-8</a> </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>26 files</summary><table> <tr> <td><strong>authz.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-28a815d1c4728aff854a9996de0ba3d3dafe1bea6bfb0cf2a9f4186c40f985b6">+0/-52</a> </td> </tr> <tr> <td><strong>loki.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-121c166af5eab8b9cb83716afa7b65fa7856c006c809b3d910ced184c52ec587">+0/-7</a> </td> </tr> <tr> <td><strong>proxy.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-7f12a4f2483e737671ac41f9c35b411e7323e0ff2731ad92a3b309e144d1316a">+0/-46</a> </td> </tr> <tr> <td><strong>service.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-61d456e7217d19bcec3920a4d1c17fd396a257c81e5394655db3632a057bdd9e">+0/-7</a> </td> </tr> <tr> <td><strong>service_account.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-878ad520e1c84500cc21ff6fc4088c8da64ff2febd79e60c0d4cc8c695c49798">+0/-80</a> </td> </tr> <tr> <td><strong>traces.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-bda8182c1f17a8360337c769edade0cb9478ff1443dd6aedce7f526afbe97afe">+0/-106</a> </td> </tr> <tr> <td><strong>user.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-e13b3a981fd70182b9738a6b0ecfdee915c5b36757703a87741e1eddf3f6e786">+0/-101</a> </td> </tr> <tr> <td><strong>auth_tests.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-2688cbdc0b009b3f0dd829c69ab86c6a7d0778ab3e65e916a410a76aa6ae6480">+0/-2961</a></td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-2aa59ce4464e7dbabdb9367fc8bb8c1b489c914b8690577d4916870463ab12db">+0/-1</a> </td> </tr> <tr> <td><strong>action.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-fc87c75a7df9539d7e7ee6184dfab02d4a8932323e81f95d57c05363aa44752d">+0/-18</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-5bf716db7e1d91dfbd427a2e320ce407c4f0c92d55d68f8e636daed1b02abd56">+0/-13</a> </td> </tr> <tr> <td><strong>reports.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-d2117a3d67d4ec47b0017861845c92143baf917e10e6746df5dd97f925f81eeb">+0/-10</a> </td> </tr> <tr> <td><strong>destinations.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-e0ba33d95ef7f0eac13bdac7e750462907a0308f01c027ea206d058e282c977b">+0/-103</a> </td> </tr> <tr> <td><strong>inverted_index.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-8bf9d9403f05f53f0c6fd5e6a6dbf0204ea80465b233567552cb7395d08d1c3c">+0/-22</a> </td> </tr> <tr> <td><strong>meta_store.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-45018c838da377fa9ded9305a1ee752adde06acdb607addf26c95f5747537615">+0/-6</a> </td> </tr> <tr> <td><strong>ratelimit.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-fa7dc03fb526c09d5eb58021872d821fcd72eeb4d6a670224f5286012485bc3d">+0/-10</a> </td> </tr> <tr> <td><strong>timed_annotations.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-f94b4d6d12d68e7054ad33601e666b08e4594586013dc34384fa21b1df5fe6e8">+0/-12</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-ca09c9c9b020049bdbc3382c466bfcca80daac92e98760d9a9a76172abf34049">+0/-34</a> </td> </tr> <tr> <td><strong>m20250611_000002_populate_reports_table.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-8247af6d6f8ccb6a5a51daff4cb1e352585be2d1c785d86ae5ebfea1dbc1099c">+0/-1</a> </td> </tr> <tr> <td><strong>folders.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-402482c22fda5dff8f33382ac11d0908f693975dec7c08b37b1c5fe24a20486a">+0/-1</a> </td> </tr> <tr> <td><strong>bulk.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-2017dbe9076c78a7e7919ed5c8738729123d630038803cef0ce2c356d6fa2c48">+0/-9</a> </td> </tr> <tr> <td><strong>ingestion.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-3edd5f4ed1eed0f765b7b450ef0fbf587632e6b22302006a5e397049e297057f">+0/-7</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-2f7f0bac01afc6e37908521d1d33d05fcd650b7a08792062976e1e7465b4a1f9">+0/-42</a> </td> </tr> <tr> <td><strong>footer_cache.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-f9278e53ed7131c953fc30f8b15cda801b74b81fb2d570fe7f80a3b7fc1de872">+0/-7</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-1602f85399080c1e2838d1ef1d62af23d96f353379e49a7770026e892c954018">+0/-20</a> </td> </tr> <tr> <td><strong>mod.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/8172/files#diff-66f3b4f5e7927092fd6e57738687a1091b6f316638f7c58f8d7be9addb72bc30">+0/-31</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ |
||
|
udev
|
b4eac8f607 |
fix: Inc UT coverage (#8128)
increase UT coverage improve the email regex to catch `doubledotsequence` in email |
||
|
Hengfei Yang
|
48115e1ab9 |
fix: enhance machine id check (#8160)
### **User description** - Add duplicated machine id check - Add 4 random chars for generate parquet filename - Use Atomic replace unsafe code ___ ### **PR Type** Bug fix, Enhancement ___ ### **Description** - Centralize node ID generation logic - Replace unsafe globals with atomics - Detect and resolve duplicate node IDs - Truncate long Snowflake IDs when parsing ___ ### Diagram Walkthrough ```mermaid flowchart LR A["Cluster register (etcd/nats)"] -- "collect node ids" --> B["generate_node_id()"] B -- "new id" --> C["LOCAL_NODE_ID (AtomicI32)"] C -- "reload if changed" --> D["Snowflake ID generator"] E["set_status()"] -- "detect duplicate id" --> B F["CLI parse-id"] -- "truncate to 19 digits" --> G["parse i64"] H["Parquet filename"] -- "append 4-hex random" --> I["unique file path"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Bug fix</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>cli.rs</strong><dd><code>Truncate IDs longer than 19 digits before parse</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8160/files#diff-cb1eb3e74723d891353ccf1e2e5e0d246d2e78a5e69dbe4c73006ea8412d65e0">+8/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Enhancement</strong></td><td><details><summary>10 files</summary><table> <tr> <td><strong>etcd.rs</strong><dd><code>Use centralized node-id generation and atomics</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8160/files#diff-37c2c2803d1c12ebb4ef9299bc79c9e6740fc5f19f7f4d84e7cfbb610b9561a9">+3/-17</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add reusable generate_node_id utility</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8160/files#diff-d15434a27e2b8406811f5a707b605203b603dad4a262c0642326be860bd2b568">+17/-0</a> </td> </tr> <tr> <td><strong>nats.rs</strong><dd><code>Centralize id generation; detect duplicates; reload</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8160/files#diff-901773980aa255942e7865b977fe63811f31511ac40bc512d6bbb14dbb54bece">+20/-17</a> </td> </tr> <tr> <td><strong>cluster.rs</strong><dd><code>Replace unsafe globals with AtomicI32</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8160/files#diff-68d7195ab7cccf22579c4944a37c8a2f592392c30371ce907fadf62fca6f97dd">+3/-3</a> </td> </tr> <tr> <td><strong>ider.rs</strong><dd><code>Read machine id atomically; support reload</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8160/files#diff-6ea42b8eab8ba0708f93f9a8b44056d9a9e6ac42517ffba12f81420d6b0bd2b0">+10/-2</a> </td> </tr> <tr> <td><strong>cluster.rs</strong><dd><code>Node equality includes id; NodeStatus i32</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8160/files#diff-a2667c1be522ce095796a7066e02631a7e01acc85a46573e1e3a157d2aefb7c3">+4/-3</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add 4-hex random suffix to filenames</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8160/files#diff-30f42c3685037c219f0033617f032b23a7f74624ba4ae6c4e3687af401304591">+2/-1</a> </td> </tr> <tr> <td><strong>schema.rs</strong><dd><code>Initialize generators using atomic node id</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8160/files#diff-4696a44abde2df4ac48fed716f930bfdcc01759bdfcfd8d802821c6dd857bff9">+7/-3</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Generate record ids using atomic node id</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8160/files#diff-7c7b5bca85f0ced652965cae66540c96f56dfcca4f4c16b77bf7b45aac203d1b">+2/-2</a> </td> </tr> <tr> <td><strong>schema.rs</strong><dd><code>Use atomic node id for stream generators</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8160/files#diff-d9596182a0c7a46a6a7ff5cee822554cdae05ceec312f86c93b538db9f73bbfd">+7/-2</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ |
||
|
udev
|
77e50caea5 |
fix: make config reactive to env changes (#7996) (#8092)
### **User description** fix #7996 sync to main ___ ### **PR Type** Bug fix, Enhancement ___ ### **Description** - Make periodic jobs react to config changes - Introduce reusable `spawn_pausable_job!` macro - Fix env var name `ZO_SEARCH_JOB_RUN_TIMEOUT` - Replace fixed intervals with dynamic getters ___ ### Diagram Walkthrough ```mermaid flowchart LR CFG["get_config() (dynamic)"] MACRO["spawn_pausable_job! macro"] JOBS["Background jobs (GC, compactor, alerts, stats, etc.)"] ENV["Env vars (e.g., ZO_SEARCH_JOB_RUN_TIMEOUT)"] ENV -- "drives" --> CFG CFG -- "intervals/pause_if" --> MACRO MACRO -- "spawns/controls" --> JOBS ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Bug fix</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Heartbeat interval now read each loop</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-d15434a27e2b8406811f5a707b605203b603dad4a262c0642326be860bd2b568">+2/-1</a> </td> </tr> <tr> <td><strong>config.rs</strong><dd><code>Fix search job timeout env var name</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+1/-1</a> </td> </tr> </table></details></td></tr><tr><td><strong>Enhancement</strong></td><td><details><summary>16 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Export pausable_job utilities module</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-f5df40ca59915d9d58ad04ea2440e6ebf29388afaeb8cb0d6c63d2a979df0c17">+1/-0</a> </td> </tr> <tr> <td><strong>pausable_job.rs</strong><dd><code>Add pausable job spawning macro and helpers</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-7e2dce7a3942789804bae659939c214c4eeb68d89315e23a08139214c339ea24">+143/-0</a> </td> </tr> <tr> <td><strong>disk.rs</strong><dd><code>Use pausable job for disk cache GC</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-af63f2c5d85313779df0204c9689e50a6c3916e3aadc24e54e23ec640a584fcb">+4/-10</a> </td> </tr> <tr> <td><strong>memory.rs</strong><dd><code>Use pausable job for memory cache GC</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-ff00d697a9b223ef2858c5e41111a8918e7cc347dc22891c02bda6982682eee1">+4/-16</a> </td> </tr> <tr> <td><strong>lib.rs</strong><dd><code>Mem persist uses dynamic sleep interval</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-603267e2f712eeb4bc46fdff981f1fad1f3a9d3b989a7db4b3c0984fc5b1ed43">+4/-5</a> </td> </tr> <tr> <td><strong>alert_manager.rs</strong><dd><code>Make alert/search jobs pausable and reactive</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-7e30858abad652a860186eb047347305488f52356112b1495dd53ce740316d7c">+61/-104</a></td> </tr> <tr> <td><strong>compactor.rs</strong><dd><code>Refactor compactor jobs to pausable macro</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-550629b7ebf6460b36fdd46cdfea466c3d3f1e30023c71ee54b87e7998efc500">+134/-188</a></td> </tr> <tr> <td><strong>parquet.rs</strong><dd><code>Use dynamic interval for file push loop</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-27f710981cfe1b938aeeb53abaed05961596c32e2edf3b8a2a59d3e72b2f542b">+1/-1</a> </td> </tr> <tr> <td><strong>flatten_compactor.rs</strong><dd><code>Flatten generator via pausable macro</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-dd30b917380ba55bd17aadeaee6aa2c87a67e861257a6a14ba2113a9b3015030">+15/-20</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Wire jobs with reactive spawning; telemetry</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-f2e4c29b8eef22189ec704a832633c4bd135a24444f054864c98064bd0963681">+21/-13</a> </td> </tr> <tr> <td><strong>promql.rs</strong><dd><code>Make metrics leader push pausable/optional</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-15414a41d64eb87bc5c0cd00a07213b3f99c31b346e8e2c36178f7276a1f058c">+26/-31</a> </td> </tr> <tr> <td><strong>stats.rs</strong><dd><code>Stats jobs via pausable macro; conditional start</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-f3ca67d9d2c68beeafd6e5de947dbc9a648ac2d2699c5dc87458c369887d09b9">+51/-42</a> </td> </tr> <tr> <td><strong>telemetry.rs</strong><dd><code>Remove legacy telemetry loop (replaced)</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-e82cf6d4c890bfaa5e711293b599d7f7a742bb96180639815ede33958b1bc1d9">+0/-36</a> </td> </tr> <tr> <td><strong>storage.rs</strong><dd><code>Make enrichment merge job pausable</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-335bb1eea9193dc398da6838c6a6a11dd6f09e984a1ed4a2e611618a427754f0">+42/-35</a> </td> </tr> <tr> <td><strong>distinct_values.rs</strong><dd><code>Periodic flush via pausable macro</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-3fb2320197c6e8e900658f5a24158e922610eadcbf081ab0df230f770ef27e07">+11/-18</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Audit publish via pausable macro, optional run</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8092/files#diff-c6255010f7a87d6acdb3ad04954fc9f2c27350ae319c53a1857b120aa06408a3">+17/-15</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ |
||
|
sktrpathi
|
9a11a680ae |
chore: add test cases (#8095)
### **User description** - also clear over 400 clippy issues ___ ### **PR Type** Tests, Enhancement ___ ### **Description** Add extensive unit tests for enrichment storage Expose helpers for testability in storage module Modernize tests: clippy-friendly string formatting Normalize constants in time-based tests |
||
|
|
75f2235d04 |
feat: create stream with field and data type (#7851)
### **User description** closes #6660 ___ ### **PR Type** Enhancement, Bug fix ___ ### **Description** - Add field data types and UDS support - Validate field names with strict regex - Store defined schema as objects, not strings - Backward-compatible parsing and UI updates ___ ### Diagram Walkthrough ```mermaid flowchart LR API["API: Stream settings/schema"] --> Parse["Parse defined_schema_fields (strings|objects)"] Parse -- "normalize to DataField{name,type}" --> Settings["StreamSettings.defined_schema_fields: DataField[]"] Settings -- "new stream" --> SchemaAdd["Add fields to Arrow schema with types"] Settings -- "existing + UDS" --> UDSUpdate["Add new fields to UDS + persist"] Types["DataType enum (Utf8, Int64, Uint64, Float64, Boolean)"] --> Arrow["Arrow DataType mapping"] Validate["Regex field name validation"] --> Save["save/update settings endpoints"] UI["UI: Add/Edit stream fields"] --> ObjectFmt["Send/handle {name,type} objects"] ObjectFmt -- "compat" --> Compat["Store original + derive string list for display"] ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>10 files</summary><table> <tr> <td><strong>stream.rs</strong><dd><code>Rename stream property field to raw type key</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7851/files#diff-6f5ee19bbcc9cc7fb699fc1c0f43fc96d42698ae1168a2827b5c65f7948d5dbc">+5/-6</a> </td> </tr> <tr> <td><strong>stream.rs</strong><dd><code>Introduce DataType and DataField; settings parsing</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7851/files#diff-2438d04c3fe97bc5452e5b74a303163b8723d93455cda32c95f0525435ad46bd">+99/-13</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Optimize settings fetch; adapt defined fields helpers</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7851/files#diff-386fd4fad1e48ab0a90516cab1947b1888b270b2fe912f30d2255cc38e6c519b">+13/-6</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Use DataField names when building UDS sets</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7851/files#diff-7c7b5bca85f0ced652965cae66540c96f56dfcca4f4c16b77bf7b45aac203d1b">+6/-1</a> </td> </tr> <tr> <td><strong>schema.rs</strong><dd><code>Add DataField flows; new/UDS logic and conversions</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7851/files#diff-d9596182a0c7a46a6a7ff5cee822554cdae05ceec312f86c93b538db9f73bbfd">+270/-18</a></td> </tr> <tr> <td><strong>stream.rs</strong><dd><code>Validate field names; resolve defined field types</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7851/files#diff-41452a7fb29b4634437174fd6aecf6a9050dbfdef41b9077a21d59af3d79e9ed">+126/-7</a> </td> </tr> <tr> <td><strong>AddStream.vue</strong><dd><code>Send defined fields as objects with type</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7851/files#diff-78b8c84a558e60d942bacffee365633d074042e464be465eb6984b0ee5df82b9">+6/-2</a> </td> </tr> <tr> <td><strong>StreamFieldInputs.vue</strong><dd><code>UI for field data type and UX tweaks</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7851/files#diff-c8173413939f320f197b0f8790e619da18666f79009345f72b3325e89bbb7566">+90/-27</a> </td> </tr> <tr> <td><strong>schema.vue</strong><dd><code>Support object-based defined fields; sync logic</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7851/files#diff-9196be2ccae4054a00b0b1aa9d1e87b366c10920d6eaf9ef8c6e6293d50bcf46">+119/-25</a></td> </tr> <tr> <td><strong>useStreams.ts</strong><dd><code>Compare/sync object-based defined fields</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7851/files#diff-a083369344f022cdf8d4f536719179b1d920369e33e2e26234e70dd36d151a7c">+50/-2</a> </td> </tr> </table></details></td></tr><tr><td><strong>Bug fix</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Map UDS fields by object name not string</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7851/files#diff-619eafba0f0509d905b021f4a16659010aca23c62cffd9cd3e6c2bbe204ce200">+1/-1</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com> |
||
|
Subhra264andudev
|
c46fe2bf8c |
fix: new sso admin/viewer/editor should be able to enjoy given permissions (#8002)
Currently, openfga returns error when new sso user (admin/editor/viewer) tries to login into openobserve. This pr fixes that. --------- Co-authored-by: udev <uddhav@openobserve.ai> |
||
|
Huaijin
|
3eb271620a | refactor: try move remotescan, stream aggregate, topk from rewrite to optimize rule (#7998) | ||
   
|
1d0490875b |
fix: is_multi_stream_search and enable_align_histogram (#8003)
### **User description** Fixes: - https://github.com/openobserve/openobserve/issues/7986 ### **PR Type** Enhancement, Bug fix ___ ### **Description** - Add multi-stream search histogram support - Gate histogram alignment via query flag - Remove unused search_type plumbing - Propagate multi-stream flag through services ___ ### Diagram Walkthrough ```mermaid flowchart LR HTTP["HTTP handlers"] GRPC["gRPC handlers"] Utils["HTTP query utils"] SQL["SQL utils: histogram eligibility"] Hist["SQL: histogram query builder"] Svc["Search service"] Cache["Search cache"] Stream["Streaming pipeline"] Utils -- "parse is_multi_stream_search / enable_align_histogram" --> HTTP Utils -- "parse flags" --> GRPC HTTP -- "pass flags" --> Svc GRPC -- "pass flags" --> Svc Svc -- "eligibility check (multi/unified)" --> SQL Svc -- "partitioning uses align flag" --> Svc Svc -- "search() with multi-stream flag" --> Cache Cache -- "eligibility(query, multi)" --> SQL HTTP -- "convert_to_histogram_query(..., multi)" --> Hist Stream -- "do_search(..., multi)" --> Cache Hist -- "build single/multi-stream histogram query" --> Svc ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>15 files</summary><table> <tr> <td><strong>http.rs</strong><dd><code>Parse multi-stream and align histogram flags</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-3b50a69c1ec2f63f8eebb5f9880e17826463a586768aed3e3e15242833609d6b">+20/-0</a> </td> </tr> <tr> <td><strong>search.rs</strong><dd><code>Remove SearchPartitionRequest.search_type field</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-373e83a31e5bbc82fba05d7ae274b0dbd624ed748272232cf7eb8b8c3b2f3afc">+0/-3</a> </td> </tr> <tr> <td><strong>sql.rs</strong><dd><code>Add multi-stream histogram eligibility (UNION ALL)</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-a0a09c66a20d878ec7683ae222c919fbe092b8001871ade7f455b89573ce0911">+134/-38</a></td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Pass multi-stream flag into search service</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-8dcdb10340635be8bb58d45a815569026877f6fdbb3b69a773a30698b91674b0">+3/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Wire flags; histogram conversion uses multi-stream</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-96630a0dbb6dd7b2b0248fa325c7c7041ce31340c32826675a4a70772e9d29f6">+11/-7</a> </td> </tr> <tr> <td><strong>multi_streams.rs</strong><dd><code>Partition multi: add align flag; remove search_type</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-c3233cdf1e68cb2455f91457719b9c2f477e735ee16546fdfe111e76768bd054">+10/-9</a> </td> </tr> <tr> <td><strong>search_inspector.rs</strong><dd><code>Pass multi-stream flag in search profile</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-f818db4ca10b411c3559cc2a2659c49a59a45ecde0e4a22f9ef023da0bf4a009">+1/-0</a> </td> </tr> <tr> <td><strong>search_stream.rs</strong><dd><code>Stream API supports multi-stream histogram and flag</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-642de43ff6b19ae41087a9f1efc02d8b0b873809e0c92cbddfac1a30ef3f1184">+9/-3</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Propagate multi-stream flag in traces search</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-64694c886c809b6efe449e6eba9fbb33d31ce5a3c49cea1a30006fc1d9522665">+2/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Cache search considers multi-stream eligibility</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-f07bfacc0501b9c234e64b16c1dd8eb0ae8fcbe231f90c81fed72bcc94946f74">+2/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Align partitions via flag; multi partition API updates</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-c3b85ea662a885fbd39797968928b291cb59a58f53a03f7b7dfd7ba8b6a7985a">+22/-12</a> </td> </tr> <tr> <td><strong>histogram.rs</strong><dd><code>Build histogram for multi-stream queries (UNION ALL)</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-a5fc4cd11b3ec569b4cedf278ef0b14f540504abfbb23e29eb896d5b2a6cc2b2">+64/-8</a> </td> </tr> <tr> <td><strong>cache.rs</strong><dd><code>Streaming cache handles multi-stream flag</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-55769b7cb67b550daf45363c6d7d569b39d4a6235f6a68c091beb40f16b6511e">+3/-0</a> </td> </tr> <tr> <td><strong>execution.rs</strong><dd><code>Execution path passes multi-stream flag to searches</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-75d432351865224f0b00d7cc819554c2ad5fa1f777e9c645d91a3fd5c1113bc0">+25/-4</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Stream request processing threads multi-stream flag</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-f236c6c880ed80233fd264e657d5ab658c16c7368bdc70e20f123862b7214374">+4/-0</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ ___ ### **PR Type** Enhancement, Bug fix ___ ### **Description** - Add multi-stream histogram query support - Align histogram via query flag - Remove search_type plumbing in partitions - Propagate multi-stream flag end-to-end ___ ### Diagram Walkthrough ```mermaid flowchart LR Http["HTTP handlers"] Grpc["gRPC handlers"] Utils["HTTP query utils"] Front["Web client"] Service["Search service"] Cache["Search cache"] SQLU["SQL utils: histogram eligibility"] Hist["Histogram query builder"] Front -- "is_multi_stream, align flag" --> Http Utils -- "parse is_multi_stream/align" --> Http Http -- "enable_align_histogram" --> Service Http -- "is_multi_stream_search" --> Cache Grpc -- "flags default false" --> Service Cache -- "eligibility(sql, multi)" --> SQLU Http -- "convert_to_histogram_query(..., multi)" --> Hist Service -- "partition(gen interval if enabled)" --> Service ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>19 files</summary><table> <tr> <td><strong>http.rs</strong><dd><code>Parse multi-stream and align-histogram flags</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-3b50a69c1ec2f63f8eebb5f9880e17826463a586768aed3e3e15242833609d6b">+20/-0</a> </td> </tr> <tr> <td><strong>search.rs</strong><dd><code>Remove search_type from partition request</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-373e83a31e5bbc82fba05d7ae274b0dbd624ed748272232cf7eb8b8c3b2f3afc">+0/-4</a> </td> </tr> <tr> <td><strong>sql.rs</strong><dd><code>Histogram eligibility with multi-stream UNION ALL</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-a0a09c66a20d878ec7683ae222c919fbe092b8001871ade7f455b89573ce0911">+134/-38</a></td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Pass default multi-stream flag to service</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-8dcdb10340635be8bb58d45a815569026877f6fdbb3b69a773a30698b91674b0">+3/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Add flags to HTTP search; align histogram handling</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-96630a0dbb6dd7b2b0248fa325c7c7041ce31340c32826675a4a70772e9d29f6">+11/-7</a> </td> </tr> <tr> <td><strong>multi_streams.rs</strong><dd><code>Partition multi endpoint supports align flag</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-c3233cdf1e68cb2455f91457719b9c2f477e735ee16546fdfe111e76768bd054">+10/-9</a> </td> </tr> <tr> <td><strong>search_inspector.rs</strong><dd><code>Forward multi-stream flag in inspector search</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-f818db4ca10b411c3559cc2a2659c49a59a45ecde0e4a22f9ef023da0bf4a009">+1/-0</a> </td> </tr> <tr> <td><strong>search_stream.rs</strong><dd><code>Streaming search supports multi-stream histogram</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-642de43ff6b19ae41087a9f1efc02d8b0b873809e0c92cbddfac1a30ef3f1184">+9/-3</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Forward multi-stream flag in traces helpers</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-64694c886c809b6efe449e6eba9fbb33d31ce5a3c49cea1a30006fc1d9522665">+2/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Eligibility check aware of multi-stream</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-f07bfacc0501b9c234e64b16c1dd8eb0ae8fcbe231f90c81fed72bcc94946f74">+2/-1</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Align histogram in partition; multi aggregate</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-c3b85ea662a885fbd39797968928b291cb59a58f53a03f7b7dfd7ba8b6a7985a">+22/-12</a> </td> </tr> <tr> <td><strong>histogram.rs</strong><dd><code>Build single/multi-stream histogram SQL</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-a5fc4cd11b3ec569b4cedf278ef0b14f540504abfbb23e29eb896d5b2a6cc2b2">+64/-8</a> </td> </tr> <tr> <td><strong>cache.rs</strong><dd><code>Propagate multi-stream flag in streaming cache</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-55769b7cb67b550daf45363c6d7d569b39d4a6235f6a68c091beb40f16b6511e">+3/-0</a> </td> </tr> <tr> <td><strong>execution.rs</strong><dd><code>Pass multi flag through streaming execution</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-75d432351865224f0b00d7cc819554c2ad5fa1f777e9c645d91a3fd5c1113bc0">+25/-4</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Wire multi-stream flag across streaming path</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-f236c6c880ed80233fd264e657d5ab658c16c7368bdc70e20f123862b7214374">+4/-0</a> </td> </tr> <tr> <td><strong>IndexList.vue</strong><dd><code>Use align flag in partition requests</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-08f058f883b659e713e606991a7ea37ae1ff016b7cccef970489ab5840dc3d09">+1/-1</a> </td> </tr> <tr> <td><strong>usePanelDataLoader.ts</strong><dd><code>Toggle align histogram for dashboards</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-5a42e9493a7bcd722b88e61fd38883c4f61001bc9cafa399f65f8eafda3aace9">+3/-1</a> </td> </tr> <tr> <td><strong>useLogs.ts</strong><dd><code>Multi-stream histogram flow and params</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-1040f61354f92711e0e77ed4e9bfa15586d45a94c7e2ab06cc37372068030e7b">+3/-2</a> </td> </tr> <tr> <td><strong>search.ts</strong><dd><code>API params for is_multi_stream and align flag</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/8003/files#diff-885ab872dbc6e7b20264df815e3ef4727f0abc86ce90d2fb47e4aff572814c75">+7/-5</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Co-authored-by: Bhargav <BJP232004@GMAIL.COM> Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech> Co-authored-by: Shrinath Rao <shnath@openobserve.ai> Co-authored-by: omkark06 <omkar@zinclabs.io> Co-authored-by: ktx-akshay <akshay.patil@kiara.tech> |
||
|
Hengfei Yang
|
1f15b7ed58 | ci: bump version (#7992) | ||
   
|
748cca3ecb |
feat: visualization enhancement (#7456)
### **User description**
### Which query will be used?
- If query is empty will use histogram query
- Else query will be available:
- If query has group by clause :- will use same query
- Else will use histogram query
___
### **PR Type**
Enhancement, Tests, Bug fix
___
### **Description**
- Add server API to extract result schema
- Implement DataFusion plan visitor for projections
- Integrate visualization flow with schema and histogram
- Numerous UI fixes and UX improvements
___
### Diagram Walkthrough
```mermaid
flowchart LR
DF["DataFusion Plan Visitor (Rust)"]
API["HTTP API /result_schema (Rust)"]
FE["Logs Visualize UI (Vue)"]
Loader["Panel Data Loader (TS)"]
Cache["Histogram Handler & Cache (Rust)"]
DF -- "extract projections/group_by/timeseries" --> API
API -- "JSON ResultSchemaResponse" --> FE
FE -- "build config, set fields, decide histogram" --> Loader
Loader -- "inject searchResponse / is_ui_histogram" --> FE
Cache -- "normalize histogram()" --> DF
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>14
files</summary><table>
<tr>
<td><strong>projections.rs</strong><dd><code>Add DataFusion logical plan
visitor to extract schema</code> </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-7a0420d3649da44b80be36d7f1043d6ff81d88395c961266882c3c700f7edff7">+651/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Add /result_schema endpoint and
wire parsing/security</code> </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-96630a0dbb6dd7b2b0248fa325c7c7041ce31340c32826675a4a70772e9d29f6">+224/-2</a>
</td>
</tr>
<tr>
<td><strong>cacher.rs</strong><dd><code>Improve histogram() handling and
make public</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-8c387b22b2c0a282600ee41ca702b3db60f5a284d507c0feb7418608ff062c6c">+22/-11</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Export projections
module</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-70bb941c2eef1365e5ead1a59b65c5397457b9197a4e22596e18d17db7eaf922">+1/-0</a>
</td>
</tr>
<tr>
<td><strong>Index.vue</strong><dd><code>Integrate visualization with
schema API and histogram</code> </dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-0323c3da69b369843c5d72bf4df5b00ca6b5147192a213baf331659ecf92af62">+564/-154</a></td>
</tr>
<tr>
<td><strong>VisualizeLogsQuery.vue</strong><dd><code>Field list UX,
props for searchResponse/histogram</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-979b01d943d58c5d70faeeac5948b7f02f0b2392a12b84ef07ba7f7393ab0261">+236/-174</a></td>
</tr>
<tr>
<td><strong>SearchBar.vue</strong><dd><code>Visualize toggle gating,
SQL/VRL disabled states</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-81893de4edf186b0556f610acb280c87baa2ffe67d3b48a60a866094c7635baf">+102/-52</a></td>
</tr>
<tr>
<td><strong>FieldList.vue</strong><dd><code>Readonly/visibility options
for logs visualize</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-a071c06e3e26629cb4ae4751450acc39c61fa49b07a1cc4be3711697829668d4">+13/-1</a>
</td>
</tr>
<tr>
<td><strong>TransformSelector.vue</strong><dd><code>Disable transform
controls in visualize mode</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-0037d8dd9cfbacc26309b26e98ae953e7ccbd0891ac2fd01c6314ebd7eed9d2c">+11/-4</a>
</td>
</tr>
<tr>
<td><strong>ChartSelection.vue</strong><dd><code>Allow restricting
selectable chart types</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-384ce178717731e2c85f1312f12a54b42ceabd0b86f7e2befb0e421ef568a6a3">+14/-11</a>
</td>
</tr>
<tr>
<td><strong>PanelSchemaRenderer.vue</strong><dd><code>Accept
searchResponse and is_ui_histogram props</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-e5376e7e77bf3aaf6590c2522d8c2dbffcc95a8006ae0d6dc0040a37825367d6">+13/-0</a>
</td>
</tr>
<tr>
<td><strong>usePanelDataLoader.ts</strong><dd><code>Support injected
searchResponse and ui histogram</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-5a42e9493a7bcd722b88e61fd38883c4f61001bc9cafa399f65f8eafda3aace9">+30/-5</a>
</td>
</tr>
<tr>
<td><strong>visualizationUtils.ts</strong><dd><code>Add SQL alias
validator for visualization</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-c2c391afcaa0b44c33d3f4fa531a4f31f044ae036f0baa182c8c88e83de066c5">+80/-0</a>
</td>
</tr>
<tr>
<td><strong>query.ts</strong><dd><code>Extend response metadata for
visualization</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-12dd5b3bb9741bbeaa348c0efbbd745d0a582899678738159c6927d34a35191d">+2/-0</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Tests</strong></td><td><details><summary>1
files</summary><table>
<tr>
<td><strong>mod.rs</strong><dd><code>Add test for compact delete value
check</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-b7e3bf7f0ac23dd084c0c817a310a4d5e7a64ff532746d182986ef940fa03fd0">+17/-0</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Configuration
changes</strong></td><td><details><summary>2 files</summary><table>
<tr>
<td><strong>auth.rs</strong><dd><code>Allow auth bypass for
result_schema route</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-9bc2b1841504a435f66ad094f72ddbe7b7fa75e5e1557b8adbfe36f7f0592e1b">+3/-0</a>
</td>
</tr>
<tr>
<td><strong>mod.rs</strong><dd><code>Register result_schema HTTP
route</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-02556e3f515441b5d2b6bbea7829df488beac03c4cc41309a6638ae3c0e0aec6">+1/-0</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>1
files</summary><table>
<tr>
<td><strong>search.rs</strong><dd><code>Add ResultSchemaResponse
contract</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-373e83a31e5bbc82fba05d7ae274b0dbd624ed748272232cf7eb8b8c3b2f3afc">+7/-0</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Bug
fix</strong></td><td><details><summary>1 files</summary><table>
<tr>
<td><strong>TableRenderer.vue</strong><dd><code>Render null/undefined as
empty string</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-794706ea9da3a70012a0327c0464f9fd16802f78ece26055fe951716c2d73d59">+1/-1</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Formatting</strong></td><td><details><summary>1
files</summary><table>
<tr>
<td><strong>sqlUtils.ts</strong><dd><code>Minor formatting
fix</code>
</dd></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-dde40e8fcb5c37b06c011ede08922c44a3c025682fbc341aee97ba046b73a0e3">+1/-1</a>
</td>
</tr>
</table></details></td></tr><tr><td><strong>Additional
files</strong></td><td><details><summary>11 files</summary><table>
<tr>
<td><strong>playwright.yml</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-7afcd2d8f7b49bda74843f209eefb7b2da45f7e7803bf2e4bd636699b76aa2d3">+1/-1</a>
</td>
</tr>
<tr>
<td><strong>visualise.js</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-c14ac1db3e4aa86190d7525e8e38c410bcbbae7f46b01890e74937cc11be67e6">+1/-5</a>
</td>
</tr>
<tr>
<td><strong>logsPage.js</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-8045b343491d8943869c1af74cdafab204dc49c02a70294417e73f47d35e0b59">+2/-2</a>
</td>
</tr>
<tr>
<td><strong>visualize.spec.js</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-d144f23552ec150db375c89e52c3a53ede9ff3fa5dcd6092694fbe0a308d22ec">+225/-183</a></td>
</tr>
<tr>
<td><strong>useDashboardPanel.ts</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-fbba1700750053b9eaf38ddd30dab84531bc44131c795ebb713811e4930cd7cb">+420/-6</a>
</td>
</tr>
<tr>
<td><strong>useLogs.ts</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-1040f61354f92711e0e77ed4e9bfa15586d45a94c7e2ab06cc37372068030e7b">+151/-19</a></td>
</tr>
<tr>
<td><strong>useNotifications.ts</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-32ff5072df9badfebc0a72c401fe337de3324f76b186a52d202eb6cc2fa71e3a">+25/-0</a>
</td>
</tr>
<tr>
<td><strong>en.json</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-892aa3d75ab6e129ef9116aa21bec1b9a15ba69ce5366d2a19128078c690d824">+2/-0</a>
</td>
</tr>
<tr>
<td><strong>search.ts</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-885ab872dbc6e7b20264df815e3ef4727f0abc86ce90d2fb47e4aff572814c75">+43/-0</a>
</td>
</tr>
<tr>
<td><strong>logs.ts</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-637857fa918f47ffb02ebfc8a35b124001ef4f65fd180c467f0f153382d2dc76">+1/-0</a>
</td>
</tr>
<tr>
<td><strong>convertDataIntoUnitValue.ts</strong></td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7456/files#diff-14c6fe442ef231566ec411f9e0262cdd24ec912fc67b9154eec743282ab7211a">+120/-29</a></td>
</tr>
</table></details></td></tr></tr></tbody></table>
</details>
___
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: ktx-akshay <akshay.patil@kiara.tech>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: loaki07 <rustyloaki@gmail.com>
Co-authored-by: Loakesh Indiran <66478092+Loaki07@users.noreply.github.com>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
Co-authored-by: ktx-vaidehi <vaidehi.akhani@kiara.tech>
Co-authored-by: ktx-vaidehi <134508096+ktx-vaidehi@users.noreply.github.com>
Co-authored-by: Yashodhan Joshi <yashodhan@openobserve.ai>
|
||
|
Loakesh Indiranandsai nikhil kethe
|
e46f6d5b82 |
feat: custom histogram interval & aligned partitions (#7909)
- `is_ui_histogram` query param for search api, the backend will convert the sql to histogram query and respond accordingly - `search_type` for search partition query, and if search type is `ui` only then we generate partitions with mini partition that is aligned with histogram interval ### **PR Type** Enhancement, Bug fix, Tests, Documentation ___ ### **Description** - Add UI histogram query conversion and flag - Align partitions to histogram with mini - Stream empty metadata+hints consistently - Expose histogram eligibility and interval ___ ### Diagram Walkthrough ```mermaid flowchart LR FE["Web UI logs & services"] -- "is_ui_histogram, searchType" --> HTTP["HTTP search endpoints"] HTTP -- "convert to histogram SQL\n(resolve streams, eligibility)" --> SQL["Histogram SQL converter"] HTTP -- "pass search_type, flags" --> SVC["Search service"] SVC -- "is_eligible, gen interval" --> PART["PartitionGenerator"] PART -- "aligned partitions\n(+mini optional)" --> STREAM["Streaming execution"] STREAM -- "SearchResponse + metadata\n(empty hits emitted)" --> FE SVC -- "Response: histogram_interval, is_histogram_eligible, converted_query" --> FE ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>16 files</summary><table> <tr> <td><strong>partition.rs</strong><dd><code>Histogram-aligned partitions with optional mini support</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-348fa23375d0cea3e5fb039cc6b381d79aa1d5121e6b9792acc048d3fd02e37c">+798/-269</a></td> </tr> <tr> <td><strong>execution.rs</strong><dd><code>Always emit responses; accumulate and metadata tweaks</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-75d432351865224f0b00d7cc819554c2ad5fa1f777e9c645d91a3fd5c1113bc0">+148/-152</a></td> </tr> <tr> <td><strong>search_stream.rs</strong><dd><code>HTTP2 stream: UI histogram conversion and interval handling</code></dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-642de43ff6b19ae41087a9f1efc02d8b0b873809e0c92cbddfac1a30ef3f1184">+149/-69</a></td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>Partition API: histogram eligibility, interval, UI paths</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-c3b85ea662a885fbd39797968928b291cb59a58f53a03f7b7dfd7ba8b6a7985a">+71/-5</a> </td> </tr> <tr> <td><strong>histogram.rs</strong><dd><code>Convert original SQL to histogram-safe query</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-a5fc4cd11b3ec569b4cedf278ef0b14f540504abfbb23e29eb896d5b2a6cc2b2">+140/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>HTTP search: UI histogram conversion and response enrichment</code></dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-96630a0dbb6dd7b2b0248fa325c7c7041ce31340c32826675a4a70772e9d29f6">+38/-3</a> </td> </tr> <tr> <td><strong>search.rs</strong><dd><code>Response/request fields for histogram meta and streaming chunks</code></dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-373e83a31e5bbc82fba05d7ae274b0dbd624ed748272232cf7eb8b8c3b2f3afc">+45/-0</a> </td> </tr> <tr> <td><strong>mod.rs</strong><dd><code>New error: SearchHistogramNotAvailable</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-90c725d0e77a6e7d4bcc130cf5783848a5cf0efdb304b351cd63cacf8775fc36">+7/-0</a> </td> </tr> <tr> <td><strong>multi_streams.rs</strong><dd><code>Multi-stream partition search accepts search_type</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-c3233cdf1e68cb2455f91457719b9c2f477e735ee16546fdfe111e76768bd054">+16/-4</a> </td> </tr> <tr> <td><strong>sql.rs</strong><dd><code>Detect histogram-eligible queries and subqueries</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-a0a09c66a20d878ec7683ae222c919fbe092b8001871ade7f455b89573ce0911">+27/-0</a> </td> </tr> <tr> <td><strong>http.rs</strong><dd><code>Add query flag parser for is_ui_histogram</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-3b50a69c1ec2f63f8eebb5f9880e17826463a586768aed3e3e15242833609d6b">+8/-0</a> </td> </tr> <tr> <td><strong>Index.vue</strong><dd><code>UI: histogram streaming flow, timeouts cleanup</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-0323c3da69b369843c5d72bf4df5b00ca6b5147192a213baf331659ecf92af62">+45/-12</a> </td> </tr> <tr> <td><strong>IndexList.vue</strong><dd><code>Pass searchType=ui to partition API</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-08f058f883b659e713e606991a7ea37ae1ff016b7cccef970489ab5840dc3d09">+1/-0</a> </td> </tr> <tr> <td><strong>search.ts</strong><dd><code>API: support is_ui_histogram, searchType plumbed</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-885ab872dbc6e7b20264df815e3ef4727f0abc86ce90d2fb47e4aff572814c75">+7/-2</a> </td> </tr> <tr> <td><strong>useStreamingSearch.ts</strong><dd><code>Include is_ui_histogram in streaming URL</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-8fce3e25a161cd458e734bd9b1f98aadefe3482da7b6529a5b1f4fa20ec115e2">+1/-0</a> </td> </tr> <tr> <td><strong>usePanelDataLoader.ts</strong><dd><code>Dashboard partition calls pass searchType</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-5a42e9493a7bcd722b88e61fd38883c4f61001bc9cafa399f65f8eafda3aace9">+2/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Tests</strong></td><td><details><summary>3 files</summary><table> <tr> <td><strong>cacher.rs</strong><dd><code>Adjust tests for new response fields</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-8c387b22b2c0a282600ee41ca702b3db60f5a284d507c0feb7418608ff062c6c">+6/-0</a> </td> </tr> <tr> <td><strong>SearchResult.spec.ts</strong><dd><code>Skip flaky resetPlotChart unit test</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-8ff7d14fc3cdac80eeec5550ecd89a77f015b4ebe928da9d4a76a9585d64d788">+1/-1</a> </td> </tr> <tr> <td><strong>useLogs.spec.ts</strong><dd><code>Skip failing query partitions test</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-842307ae8db03aa51891ddde98cfdbbabc712db1e1614c9eee99a59add395f70">+1/-1</a> </td> </tr> </table></details></td></tr><tr><td><strong>Error handling</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>error_utils.rs</strong><dd><code>Map histogram unavailability to 400 with code</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-d4de63bc9275a19c868f6967de0f7835bd2d7bda74ff55ce97b8fb3f9af54da9">+3/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Miscellaneous</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Export histogram SQL module</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-dc0ab53d0ff551cf3cd12027c73d0da5e66680c0bf2d1830778639a33bac3464">+1/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Bug fix</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>SearchResult.vue</strong><dd><code>UI: robust empty-histogram handling and reset watcher</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-5e65533aa4754bb6d0074fda18ffc1ecacded3b9ca6bc28a161c3f6b12f20195">+17/-15</a> </td> </tr> <tr> <td><strong>SearchBar.vue</strong><dd><code>Cancel running query before applying saved view</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-81893de4edf186b0556f610acb280c87baa2ffe67d3b48a60a866094c7635baf">+2/-1</a> </td> </tr> </table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>query.ts</strong><dd><code>Types: histogram interval and eligibility fields</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-12dd5b3bb9741bbeaa348c0efbbd745d0a582899678738159c6927d34a35191d">+2/-0</a> </td> </tr> </table></details></td></tr><tr><td><strong>Additional files</strong></td><td><details><summary>2 files</summary><table> <tr> <td><strong>fga.rs</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-e7c99f37f3abbb5e0507d2760bbf67df010d9218704b38dce84c5477acecd22e">+0/-2</a> </td> </tr> <tr> <td><strong>useLogs.ts</strong></td> <td><a href="https://github.com/openobserve/openobserve/pull/7909/files#diff-1040f61354f92711e0e77ed4e9bfa15586d45a94c7e2ab06cc37372068030e7b">+154/-168</a></td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ --------- Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai> |
||
|
Hengfei Yang
|
41d0d4acb6 |
fix: add more logs for flower (#7944)
### **User description** - [x] Add more logs for flower - [x] Optimize parquet footer cache calculate ___ ### **PR Type** Enhancement, Bug fix ___ ### **Description** - Switch query partition strategy to enum - Improve file stats cache size accuracy - Add follower leader work-group and partition logs - Minor consistent-hash refactor for clarity ___ ### Diagram Walkthrough ```mermaid flowchart LR CFG["Config Common.feature_query_partition_strategy (enum)"] STREAM["meta/stream QueryPartitionStrategy FromStr"] FLIGHT["search/cluster/flight uses enum directly"] FOLL["super_cluster/follower adds detailed logs"] HASH["cluster get_node_from_consistent_hash minor reorder"] CACHE["file_statistics_cache counts min/max/sum sizes"] STREAM -- "provides parser" --> CFG CFG -- "typed value" --> FLIGHT FLIGHT -- "strategy choice" --> FOLL CACHE -- "more accurate memory estimate" --> FLIGHT HASH -- "clarify flow" --> FLIGHT ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>6 files</summary><table> <tr> <td><strong>mod.rs</strong><dd><code>Precompute hash before node-set access</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7944/files#diff-d15434a27e2b8406811f5a707b605203b603dad4a262c0642326be860bd2b568">+1/-1</a> </td> </tr> <tr> <td><strong>config.rs</strong><dd><code>Make query partition strategy a parsed enum</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7944/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+7/-3</a> </td> </tr> <tr> <td><strong>stream.rs</strong><dd><code>Add FromStr and default for strategy enum</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7944/files#diff-2438d04c3fe97bc5452e5b74a303163b8723d93455cda32c95f0525435ad46bd">+8/-5</a> </td> </tr> <tr> <td><strong>flight.rs</strong><dd><code>Use enum-config strategy directly</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7944/files#diff-dbffb1a03b65f6a9163b18c5bb1241ba2966735657be39bbe9f4d0e3260d8c1d">+1/-2</a> </td> </tr> <tr> <td><strong>file_statistics_cache.rs</strong><dd><code>Include min/max/sum sizes in cache estimate</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7944/files#diff-5b3f948f2ad04a6babe72c15365b95a284d0533720419fa5dc793aa379783a7d">+5/-1</a> </td> </tr> <tr> <td><strong>follower.rs</strong><dd><code>Add work-group wait and partition logs</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7944/files#diff-59e0cc386582cf8baf201977475fd9a7e479fd8ed000bbaa6746a51ecb28632f">+8/-1</a> </td> </tr> </table></details></td></tr><tr><td><strong>Miscellaneous</strong></td><td><details><summary>1 files</summary><table> <tr> <td><strong>lib.rs</strong><dd><code>Remove unused async_drop feature gate</code> </dd></td> <td><a href="https://github.com/openobserve/openobserve/pull/7944/files#diff-b1a35a68f14e696205874893c07fd24fdb88882b47c23cc0e0c80a30c7d53759">+0/-1</a> </td> </tr> </table></details></td></tr></tr></tbody></table> </details> ___ |
||
|
Loakesh Indiran
|
1ebd0679db | feat: api to delete stream by time period (#7600) | ||
 ktx-vaidehiandAshish Kolhe
|
45bac67dec |
feat: add run ID management and panel ID for dashboard refresh (#7760)
### **PR Type** Enhancement ___ ### **Description** - Introduce DashboardInfo struct with run/panel/tab IDs - Extract dashboard_info from HTTP requests - Propagate dashboard_info in backend search & stats - Frontend: manage runId state and include IDs in requests ___ ### Diagram Walkthrough ```mermaid flowchart LR FE["Frontend components"] HTTPRequest["HTTP Request"] ExtractInfo["get_dashboard_info_from_request"] SearchService["SearchService::search"] UsageStats["report_request_usage_stats"] FE -- "include run/panel/tab IDs" --> HTTPRequest HTTPRequest -- "extract dashboard info" --> ExtractInfo ExtractInfo -- "pass DashboardInfo" --> SearchService SearchService -- "attach stats with info" --> UsageStats ``` <details> <summary><h3> File Walkthrough</h3></summary> <table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody></tr></tbody></table> </details> ___ --------- Co-authored-by: Ashish Kolhe <ashish.j.kolhe@gmail.com> |
||
|
Huaijin
|
682033437e | chore: refactor sql.rs (#7846) | ||
|
udev
|
a5af8cea73 |
ci: update cargo deny (#7791)
### **User description**
add libbz2 to exceptions
dependency from datafusion
___
### **PR Type**
Enhancement
___
### **Description**
- Add libbz2-rs-sys license exception
- Update cargo-deny CI configuration
___
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Configuration
changes</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>deny.toml</strong><dd><code>Add libbz2-rs-sys
BSD-2-Clause exception</code>
</dd></summary>
<hr>
deny.toml
<ul><li>Added <code>libbz2-rs-sys</code> exception entry<br> <li>
Allowed <code>BSD-2-Clause</code> license</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7791/files#diff-1040309c64844eb1b6b63d8fd67938adbf9461f1b3c61f12cf738f064a02d3de">+1/-0</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|
||
|
sktrpathi
|
7e08f5ac7a | chore: add UTs, run fmt and clipyy (#7753) | ||
|
Huaijin
|
b41bb71779 | feat: upgrade datafusion to v49 (#7649) | ||
|
sktrpathiandHengfei Yang
|
2da35485f5 |
chore: add UTs, run fmt and clippy (#7720)
### **Description** - Add unit tests for cipher, registry, saved_view, hash utils - Refactor `verify_decode_token` logic with algorithm conversion - Remove async from JWT calls in HTTP handlers - Simplify code in registry, job/cipher, wal modules Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com> |
||
|
ByteBaker
|
90fbcca801 | chore: cleanup auth tests | ||
|
ByteBaker
|
b7e3b331b1 | chore: add UTs, run fmt and clipyy | ||
|
Yashodhan Joshi
|
b10b234079 | fix: rbac migration for re_patterns (#7701) | ||
|
sktrpathi
|
f0851c3c7e | chore: add UTs, run fmt and clipyy (#7691) | ||
|
Hengfei Yang
|
65540f8e79 |
perf: improve concurrency for index search (#7695)
### **User description**
Add new env
```
ZO_QUERY_INDEX_THREAD_NUM = 0
```
Default value is `CPU * 4`.
___
### **PR Type**
Enhancement, Tests
___
### **Description**
- Add env var for index search concurrency
- Configure default query_index thread counts
- Use new thread count in search filter
- Refactor file grouping and update tests
___
### Diagram Walkthrough
```mermaid
flowchart LR
A["Load ZO_QUERY_INDEX_THREAD_NUM env"] --> B["Set query_index_thread_num default"]
B --> C["filter_file_list_by_tantivy_index uses new count"]
C --> D["Refactor group & repartition logic"]
D --> E["Update grouping tests"]
```
<details> <summary><h3> File Walkthrough</h3></summary>
<table><thead><tr><th></th><th align="left">Relevant
files</th></tr></thead><tbody><tr><td><strong>Configuration
changes</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>config.rs</strong><dd><code>Add query index thread
configuration</code>
</dd></summary>
<hr>
src/config/src/config.rs
<ul><li>Introduce <code>ZO_QUERY_INDEX_THREAD_NUM</code> env var<br>
<li> Set default based on local_mode vs production<br> <li> Apply new
var in <code>check_limit_config</code></ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7695/files#diff-9aeee4a74c0520daa40f9b81c7016f5b6b3254375b968dd7598e745630980d62">+9/-0</a>
</td>
</tr>
</table></td></tr><tr><td><strong>Formatting</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>record_batch_ext.rs</strong><dd><code>Cleanup imports
and whitespace in tests</code>
</dd></summary>
<hr>
src/config/src/utils/record_batch_ext.rs
<ul><li>Reformat import statements for arrow util<br> <li> Remove
trailing whitespace in tests</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7695/files#diff-f84aa2842a164d6e85aa05a718159e87a325cac139c59e561f6f59313c6d7c5d">+7/-4</a>
</td>
</tr>
</table></td></tr><tr><td><strong>Enhancement</strong></td><td><table>
<tr>
<td>
<details>
<summary><strong>storage.rs</strong><dd><code>Enhance storage search
partitioning logic</code>
</dd></summary>
<hr>
src/service/search/grpc/storage.rs
<ul><li>Use <code>query_index_thread_num</code> for
target_partitions<br> <li> Log target_partitions, group_num,
max_group_len<br> <li> Compute <code>expect_group_elements</code> in
grouping<br> <li> Simplify <code>repartition_sorted_groups</code>
logic<br> <li> Remove unused <code>find_max_group_index</code><br> <li>
Update tests to use <code>expect_group_elements</code></ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7695/files#diff-ab2d867e1e2e3dd8307505c2e130039628a84f8131a9b38cd2e7b4e1c141e094">+27/-62</a>
</td>
</tr>
<tr>
<td>
<details>
<summary><strong>mod.rs</strong><dd><code>Simplify search stream
logging</code>
</dd></summary>
<hr>
src/service/search/streaming/mod.rs
<ul><li>Simplify log message placeholders<br> <li> Remove unused
variables in cache log</ul>
</details>
</td>
<td><a
href="https://github.com/openobserve/openobserve/pull/7695/files#diff-f236c6c880ed80233fd264e657d5ab658c16c7368bdc70e20f123862b7214374">+1/-5</a>
</td>
</tr>
</table></td></tr></tr></tbody></table>
</details>
___
|