Commit Graph
1513 Commits
Author SHA1 Message Date
Sai Nikhil 8291ded0b3 fix: multiple api calls issue while creating pipelines , alerts (#6531)
1. this PR fixes the issue #6530
2025-04-14 14:53:54 +05:30
dependabot[bot] 012230a078 chore(deps-dev): bump vite from 6.2.5 to 6.2.6 in /web (#6521) 2025-04-13 16:36:17 +08:00
Sai Nikhil 064d029939 fix: schema page tab improvements (#6524)
1. This PR adds 
**previous approach** 
1. we were showing both All Fields and UDS tabs even user doesnot have
UDS enabled
2. UDS is shown as second tab


**current approach**
1. only show UDS if enabled otherwise show only All fields 
3. if UDS enabled show it as first tab and instead of All Fields(when
UDS enabled) renamed it to Other Fields and also added tooltip with some
explanatory text after Other Fields with a info icon as
**_

> Other fields show only the schema fields that existed before the
stream was configured to use a user-defined schema.

_**
2025-04-12 18:05:36 +05:30
Sai Nikhil 7e58382519 fix: race condition of alerts list handled (#6523)
1. This PR handles the issue of alerts of being overridden when user
tries to shift the folders rapidly
example:
folder1 -> 100alerts
folder2 -> 2alerts
folder3 -> 0 alerts
so first users goes to folder2 and fetches the alerts meanwhile user
goes to folder1 and immediately switches to folder3 , lets assume the
folder3 alerts are already fetched and they are in store then it will
show folder1 alerts because api takes time to resolve, overrides the
current alerts after resolving so tries to show the wrong folder alerts
2025-04-12 16:10:24 +05:30
Sai Nikhil 0aa8969a77 fix: logs page issues (#6518)
this PR fixes
1. histogram loader should while both search and histogram request as
well as no data found should only show when there is no actual data not
at the time of loading
2. in the detail table when we add a field to the table it again shows
add a field to the table instead should show remove from the table
3. when user try to add fields using quick mode with sql mode off there
is one console error
2025-04-11 20:20:26 +05:30
Sai Nikhil 63d17eb1a9 fix: show total events in distinct query when histogram is on (#6506)
1. this PR fixes the issue that is related to when user tries to run
distinct query when histogram is on
eg:
`SELECT DISTINCT body FROM "default"`
currently it will show total events as if 50 records per page it will
show
1 to 50 out of 50 events
2025-04-11 15:02:41 +05:30
Loakesh Indiranandomkark06 471c0c88b3 fix: ws fixes (#6504)
- [x] `Router-Client` if sending `pong` from `router` fails force close
the ws connection
- [ ] `Router-Querier` conn if `pong` is not received from querier for
certain time (ping_interval + 10 secs) then force close the connection
- [x] ws behavior should be such that if one querier disconnects from
the connection pool the entire `Router-Client` connection should not be
disconnected, searches/events should continue with other queriers in the
pool. Also, if one of the querier disconnects send `should_client_retry`
as `true`

---------

Co-authored-by: omkark06 <omkar@zinclabs.io>
2025-04-10 19:48:08 +05:30
Sai Nikhil a8c82017fa fix: ignore re render of histogram if pagination call (#6497)
This PR fixes the issue that is there related to histogram unnecessarily
re rendering in pagination calls happens
2025-04-09 17:04:32 +05:30
Sai Nikhil db07a9236f fix: additional resources in data source (#6496)
This PR adds some additional resources in data sources page that are
defined in the below issue
#6479
2025-04-09 12:12:20 +05:30
Ashish KolheandHengfei Yang abf47e958a fix: avoid saving invalid query for alerts (#6493)
Fixes: 

- avoid saving invalid query for alerts 
- https://github.com/openobserve/openobserve/issues/6490

---------

Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
2025-04-08 19:24:02 +05:30
Sai Nikhil 6a61a4e199 fix: dynamic stream name should be taken for search job (#6482)
1. This PR fixes the issue that for search jobs stream type should be
taken from the response not static stream type
2025-04-07 20:54:59 +05:30
Sai Nikhil 3c456b4d8c fix: reset histogram only when user clicks run query (#6478) 2025-04-07 17:02:21 +05:30
dependabot[bot] 6b62dfe3b6 chore(deps-dev): bump vite from 6.2.4 to 6.2.5 in /web (#6469) 2025-04-06 18:26:52 +08:00
Abhay Padamani 8305945250 fix: validate filters config while dashboard panel save (#6460)
- #6464
2025-04-04 16:01:54 +05:30
ktx-vaidehi 25dc49dafc feat: dashboard filter value based on type (#6457) 2025-04-04 11:40:40 +05:30
Zhao.wei 871805a32c feat: impl ratelimit feature (#6072) 2025-04-03 23:47:18 +08:00
Sai Nikhil 595d9b3f4f fix: node list changes (#6449)
1. this PR will fix the issues in #6444
2025-04-03 19:07:26 +05:30
Sai Nikhil bdfb248006 fix: alerts owner should be logged in user (#6447)
1. This PR makes sure that cloned alert and imported alert owners should
be logged in users only
2025-04-02 13:37:19 +05:30
Sai Nikhil ebfcdb34b5 fix: added folders in cloning of alerts (#6441)
1. This PR will allow cloning of alerts to different folders
2025-04-01 15:15:48 +05:30
Bhargav 78f6549e38 fix: fixed auto filter issue on TCP , CPU and memory usage (#6440) 2025-04-01 14:27:39 +05:30
Omkar Kesarkhane 4a165c7981 fix: Removed dead code from websockets (#6439) 2025-04-01 11:01:46 +05:30
dependabot[bot] 98e4ea21c1 chore(deps-dev): bump vite from 6.2.2 to 6.2.4 in /web (#6437) 2025-04-01 11:53:15 +08:00
a092bc7502 fix: ws session drain (#6430)
- [x] fix: `session_drain` i.e. when a cookie expires no new `search`
events will be accepted by the router/server while the ongoing searches
will be drained to completion or allowed up to the
`ZO_WEBSOCKET_SESSION_IDLE_TIMEOUT_SECS` (default is 5 mins) after which
the session will be closed by the server.
- [x] track search duration for every `trace_id` and log it

---------

Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech>
2025-03-31 17:43:46 +05:30
Sai Nikhil 0aa8115658 fix: introduced Alert tabs (#6426)
1.this PR allows alert tabs 
All, Scheduled, Real time
2025-03-31 14:35:56 +05:30
Bhargav 58341f6f3f feat: nodes page implementation (#6427) 2025-03-31 12:18:37 +05:30
Sai Nikhil ab75eb800e fix: histogram issue (#6419)
1. This PR fixes the histogram no data for histogram found only when
there is no data
2025-03-30 07:57:54 +05:30
Hengfei YangandBhargav e89dc0a80c feat: add separate ingestion url (#6423)
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
2025-03-29 14:21:28 +08:00
Bhargav 0285b0fe1a fix: ui fixed for datasources page (#6424) 2025-03-29 11:11:41 +05:30
ktx-vaidehi 16134fbbc0 fix: dashboard color option changed based on theme (#6421) 2025-03-28 15:27:57 +05:30
5dd59cd495 fix: alerts folders UI improvements (#6398)
1. code refactoring for alert list

---------

Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: udev <uddhavdave1@gmail.com>
2025-03-27 07:46:17 +05:30
Omkar Kesarkhane 700b9e59c4 fix: fixed reset pagecount search on error in web-socket (#6408) 2025-03-26 14:13:14 -07:00
+2 8cd0683a0b feat: ws v2 (#6404)
# WebSocket v2 Architecture

## NOTE: Open Issues:
- [ ] `session_drain` from server is experimental and not ready for
prod. Instead, the client maintains ongoing `trace_id`'s and when cookie
expires it will retry search for all the ongoing searches.

## Overview
Implements websocket v2 with the main goal to establish `Client-Router`
connections `per tab` and `Router-Querier` `persistent` connections.

## New Env (Default):
```sh
ZO_WEBSOCKET_CHECK_COOKIE_EXPIRY=true
ZO_WEBSOCKET_HEALTH_CHECK_INTERVAL_SECS=60
ZO_WEBSOCKET_IS_SESSION_DRAIN_ENABLED=false # This is experimental, not meant for prod
```

## High-Level Flow

1. Client establishes WebSocket connection with the router.  
2. Router creates a session.  
3. Router maintains persistent connections to all queriers.  
4. Client requests are routed to the appropriate querier using
consistent hashing.
5. Responses are routed back using `trace_id` mappings.  
6. Health checks maintain connection validity.  
7. Failed connections trigger automatic cleanup and reconnection.  
8. **Internal Communication Validation Using:**  
   - **8a.** Internal gRPC tokens  
   - **8b.** IP validation  
   - **8c.** WebSocket upgrade headers  

## Connection Lifecycle
```mermaid
sequenceDiagram
    participant Client
    participant Router
    participant ConnectionPool
    participant QuerierConn
    participant Querier
    participant Cluster

    Client->>Router: WS Connection Request
    Router->>SessionManager: register_client(client_id)
    
    Note over Router,SessionManager: Creates channels:<br/>response_tx/rx<br/>disconnect_tx/rx

    par Handle Incoming Messages
        Router->>Router: spawn handle_incoming task
    and Handle Outgoing Messages
        Router->>Router: spawn handle_outgoing task
    end

    Client->>Router: WS Client Event
    Router->>SessionManager: get_querier_for_trace(trace_id)
    
    alt No Existing Mapping
        Router->>Cluster: get_node_from_consistent_hash(trace_id)
        Router->>ConnectionPool: get_or_create_connection(querier)
        
        alt No Existing Connection
            ConnectionPool->>QuerierConn: create_connection
            QuerierConn->>Querier: WS Connection Request
            
            par Connection Tasks
                QuerierConn->>QuerierConn: spawn listen_to_querier_response
                QuerierConn->>QuerierConn: spawn health_check
            end
            
            Querier-->>QuerierConn: Connection Established
        end
        
        Router->>SessionManager: set_querier_for_trace(trace_id, querier)
    end

    Router->>QuerierConn: register_request(trace_id)
    Router->>QuerierConn: send_message(event)
    Querier-->>QuerierConn: Response
    QuerierConn->>Router: Route Response via response_tx
    Router->>Client: Forward Response

    Note over Router,Client: On Disconnect/Error:<br/>1. Send disconnect signal<br/>2. Clean up sessions<br/>3. Drain if needed
```

## Connections
```mermaid
graph TB
    subgraph "Client Layer"
        UI1[UI Client 1]
        UI2[UI Client 2]
        UI3[UI Client 3]
    end

    subgraph "Router Layer"
        R1[Router 1]
        R2[Router 2]
        
        subgraph "Router 1 Connection Pool"
            CP1[Connection Pool]
            QC11[Querier1 Conn]
            QC12[Querier2 Conn]
            QC13[Querier3 Conn]
            CP1 --- QC11
            CP1 --- QC12
            CP1 --- QC13
        end
        
        subgraph "Router 2 Connection Pool"
            CP2[Connection Pool]
            QC21[Querier1 Conn]
            QC22[Querier2 Conn]
            QC23[Querier3 Conn]
            CP2 --- QC21
            CP2 --- QC22
            CP2 --- QC23
        end
        
        R1 --- CP1
        R2 --- CP2
    end

    subgraph "Querier Layer"
        Q1[Querier 1]
        Q2[Querier 2]
        Q3[Querier 3]
    end

    UI1 -.->|WS| R1
    UI2 -.->|WS| R1
    UI3 -.->|WS| R2

    QC11 ===|Persistent WS| Q1
    QC12 ===|Persistent WS| Q2
    QC13 ===|Persistent WS| Q3
    QC21 ===|Persistent WS| Q1
    QC22 ===|Persistent WS| Q2
    QC23 ===|Persistent WS| Q3

    classDef client fill:#f9f,stroke:#333,stroke-width:2px
    classDef router fill:#bbf,stroke:#333,stroke-width:2px
    classDef querier fill:#bfb,stroke:#333,stroke-width:2px
    classDef pool fill:#ffd,stroke:#333,stroke-width:2px
    
    class UI1,UI2,UI3 client
    class R1,R2 router
    class Q1,Q2,Q3 querier
    class CP1,CP2 pool
```

## Components
```mermaid
classDiagram
    class WsHandler {
        +session_manager: Arc<SessionManager>
        +connection_pool: Arc<QuerierConnectionPool>
        +handle_connection()
        +remove_querier_connection()
        +_shutdown()
    }

    class SessionManager {
        +sessions: RwAHashMap<ClientId, SessionInfo>
        +mapped_queriers: RwAHashMap<QuerierName, HashSet<TraceId>>
        +register_client()
        +update_session_activity()
        +remove_trace_id()
        +unregister_client()
        +set_querier_for_trace()
        +get_querier_for_trace()
    }

    class QuerierConnectionPool {
        +connections: RwAHashMap<QuerierName, Arc<QuerierConnection>>
        +get_or_create_connection()
        +remove_querier_connection()
        +maintain_connections()
        +get_active_connection()
    }

    class QuerierConnection {
        +querier_name: QuerierName
        +write: Arc<Mutex<Option<SplitSink>>>
        +response_router: Arc<ResponseRouter>
        +is_connected: Arc<AtomicBool>
        +connect()
        +disconnect()
        +send_message()
        +register_request()
        +unregister_request()
    }

    class ResponseRouter {
        +routes: RwAHashMap<TraceId, Sender<WsServerEvents>>
        +route_response()
        +flush()
        +spawn_cleanup_task()
    }

    WsHandler --> SessionManager
    WsHandler --> QuerierConnectionPool
    QuerierConnectionPool --> QuerierConnection
    QuerierConnection --> ResponseRouter
```


Related Issues: #6396

---------

Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
Co-authored-by: Abhay Padamani <141122205+ktx-abhay@users.noreply.github.com>
Co-authored-by: Ashish Kolhe <35160958+oasisk@users.noreply.github.com>
Co-authored-by: udev <uddhav@openobserve.ai>
Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech>
Co-authored-by: Omkar Kesarkhane <127078473+omkarK06@users.noreply.github.com>
Co-authored-by: udev <uddhavdave1@gmail.com>
Co-authored-by: Subhra264 <chakrabortysubhradeep556@gmail.com>
Co-authored-by: ktx-vaidehi <134508096+ktx-vaidehi@users.noreply.github.com>
2025-03-26 11:43:40 -07:00
ktx-vaidehi 64bfaa10e3 feat: dashboard background color option for metric chart (#6397) 2025-03-26 20:18:28 +05:30
Bhargav dace1fef23 fix: UI changes on datasource page (#6390) 2025-03-25 22:08:29 +05:30
Sai Nikhil b006800b37 fix: alert enable or disable is not updated in the UI when filter applied (#6388)
1. This PR fixes that alert enable is not updated in the UI when filter
is applied
2. This PR also fixes the alert enable disable in search across folders
3. This PR also removes the id from exported alert
2025-03-25 16:07:58 +05:30
Subhra264 5b7536d7fc feat: add variables for trigger time in seconds, millis for alert template (#6387)
This pr adds two more variables to alert templates -
`alert_trigger_time_millis` (the alert trigger occurance time in epoch
milliseconds) and `alert_trigger_time_seconds` (the alert trigger
occurance time in epoch seconds). Note that the `alert_trigger_time` is
alert trigger occurance time in UTC epoch microseconds.
2025-03-25 15:03:34 +05:30
ktx-vaidehiandKirtan Thakkar b435ce798f fix: passing variables of dashboard drilldown to same dashboard (#6385)
Co-authored-by: Kirtan Thakkar <kirtan@kiara.tech>
2025-03-25 13:58:35 +05:30
Omkar Kesarkhane f18f265e12 fix: Updated actions execution type label (#6374) 2025-03-25 09:00:38 +05:30
c3759a4231 fix: Migration for scheduled_jobs table to use alert id as module_key (#6365)
Co-authored-by: udev <uddhavdave1@gmail.com>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2025-03-24 15:10:44 +05:30
Omkar Kesarkhane b7af94f487 fix: Actions UI improvements (#6372)
1. Remove action_id from Histogram and Page-count search payload
2. Rename Action type `Service` to `Real Time`
3. Added validation for service account selection while creating action
2025-03-24 14:33:16 +05:30
Sai Nikhil f7d3e5aee0 fix: added tooltip for all icons in icon bar (#6371)
1. added tooltip for every icons in icon bar
2025-03-24 12:32:27 +05:30
06920b203b fix: previous histogram getting display even after there is no data (#6346)
Co-authored-by: ktx-vaidehi <vaidehi.akhani@kiara.tech>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
2025-03-24 11:27:20 +05:30
ktx-vaidehi 986c01a4a2 fix: dashboard error message getting removed (#6366) 2025-03-23 11:47:52 +05:30
Sai Nikhil c5be09fbf4 fix: changes for icon bar in logs page (#6362)
This PR includes changes 
1. introduced icons for histogram , sql mode, quickmode and moved search
history , download logs , search scheduled jobs options into menu that
will be there at right end
2. this will make the UI to accommodate upto 1200px with datetime as
relative , 1350 when it is absolute
3. introducing icons saved space of ~200px in total for both sides
2025-03-23 10:16:34 +05:30
ktx-vaidehi 2af8b83699 fix: dashboard panel name and folder text changes (#6364) 2025-03-22 18:38:57 +05:30
Hengfei Yang a09d626d42 fix: disable add stream & add field when user doesn't enable UDS (#6361) 2025-03-22 10:54:35 +08:00
Abhay Padamani 2c51f7fde3 fix: dashboard empty error issue (#6348)
- #3044
2025-03-21 17:01:57 +05:30
71b32480ab feat: alerts folder implementation (UI) (#6228)
implemented alerts folders in the UI

1. search across folders
2. alert listing page moved condition and description column to the
expand so whenever user clicks on any row it will show respective
condition and description if there
3. provided fixed width for columns so that all columns are in view port
visibility expect name column as it can be varied depending upon name
given by user
4. search across all folders implemented in this PR only
Import alert add ons
1 additional things done to import alert as well like introducing the
folders in import page

---------

Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
2025-03-21 13:31:54 +05:30
ktx-vaidehi 4ad2865ddb fix: dashboard variables dependency not working properly (#6350) 2025-03-20 18:19:51 +05:30
ktx-vaidehi 901ad1c5dc fix: dashboard breakdown value not present chart not getting plot (#6344)
- #6310
2025-03-20 16:37:29 +05:30