Commit Graph
3411 Commits
Author SHA1 Message Date
5dd59cd495 fix: alerts folders UI improvements (#6398)
1. code refactoring for alert list

---------

Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: udev <uddhavdave1@gmail.com>
2025-03-27 07:46:17 +05:30
Omkar Kesarkhane 700b9e59c4 fix: fixed reset pagecount search on error in web-socket (#6408) 2025-03-26 14:13:14 -07:00
+2 8cd0683a0b feat: ws v2 (#6404)
# WebSocket v2 Architecture

## NOTE: Open Issues:
- [ ] `session_drain` from server is experimental and not ready for
prod. Instead, the client maintains ongoing `trace_id`'s and when cookie
expires it will retry search for all the ongoing searches.

## Overview
Implements websocket v2 with the main goal to establish `Client-Router`
connections `per tab` and `Router-Querier` `persistent` connections.

## New Env (Default):
```sh
ZO_WEBSOCKET_CHECK_COOKIE_EXPIRY=true
ZO_WEBSOCKET_HEALTH_CHECK_INTERVAL_SECS=60
ZO_WEBSOCKET_IS_SESSION_DRAIN_ENABLED=false # This is experimental, not meant for prod
```

## High-Level Flow

1. Client establishes WebSocket connection with the router.  
2. Router creates a session.  
3. Router maintains persistent connections to all queriers.  
4. Client requests are routed to the appropriate querier using
consistent hashing.
5. Responses are routed back using `trace_id` mappings.  
6. Health checks maintain connection validity.  
7. Failed connections trigger automatic cleanup and reconnection.  
8. **Internal Communication Validation Using:**  
   - **8a.** Internal gRPC tokens  
   - **8b.** IP validation  
   - **8c.** WebSocket upgrade headers  

## Connection Lifecycle
```mermaid
sequenceDiagram
    participant Client
    participant Router
    participant ConnectionPool
    participant QuerierConn
    participant Querier
    participant Cluster

    Client->>Router: WS Connection Request
    Router->>SessionManager: register_client(client_id)
    
    Note over Router,SessionManager: Creates channels:<br/>response_tx/rx<br/>disconnect_tx/rx

    par Handle Incoming Messages
        Router->>Router: spawn handle_incoming task
    and Handle Outgoing Messages
        Router->>Router: spawn handle_outgoing task
    end

    Client->>Router: WS Client Event
    Router->>SessionManager: get_querier_for_trace(trace_id)
    
    alt No Existing Mapping
        Router->>Cluster: get_node_from_consistent_hash(trace_id)
        Router->>ConnectionPool: get_or_create_connection(querier)
        
        alt No Existing Connection
            ConnectionPool->>QuerierConn: create_connection
            QuerierConn->>Querier: WS Connection Request
            
            par Connection Tasks
                QuerierConn->>QuerierConn: spawn listen_to_querier_response
                QuerierConn->>QuerierConn: spawn health_check
            end
            
            Querier-->>QuerierConn: Connection Established
        end
        
        Router->>SessionManager: set_querier_for_trace(trace_id, querier)
    end

    Router->>QuerierConn: register_request(trace_id)
    Router->>QuerierConn: send_message(event)
    Querier-->>QuerierConn: Response
    QuerierConn->>Router: Route Response via response_tx
    Router->>Client: Forward Response

    Note over Router,Client: On Disconnect/Error:<br/>1. Send disconnect signal<br/>2. Clean up sessions<br/>3. Drain if needed
```

## Connections
```mermaid
graph TB
    subgraph "Client Layer"
        UI1[UI Client 1]
        UI2[UI Client 2]
        UI3[UI Client 3]
    end

    subgraph "Router Layer"
        R1[Router 1]
        R2[Router 2]
        
        subgraph "Router 1 Connection Pool"
            CP1[Connection Pool]
            QC11[Querier1 Conn]
            QC12[Querier2 Conn]
            QC13[Querier3 Conn]
            CP1 --- QC11
            CP1 --- QC12
            CP1 --- QC13
        end
        
        subgraph "Router 2 Connection Pool"
            CP2[Connection Pool]
            QC21[Querier1 Conn]
            QC22[Querier2 Conn]
            QC23[Querier3 Conn]
            CP2 --- QC21
            CP2 --- QC22
            CP2 --- QC23
        end
        
        R1 --- CP1
        R2 --- CP2
    end

    subgraph "Querier Layer"
        Q1[Querier 1]
        Q2[Querier 2]
        Q3[Querier 3]
    end

    UI1 -.->|WS| R1
    UI2 -.->|WS| R1
    UI3 -.->|WS| R2

    QC11 ===|Persistent WS| Q1
    QC12 ===|Persistent WS| Q2
    QC13 ===|Persistent WS| Q3
    QC21 ===|Persistent WS| Q1
    QC22 ===|Persistent WS| Q2
    QC23 ===|Persistent WS| Q3

    classDef client fill:#f9f,stroke:#333,stroke-width:2px
    classDef router fill:#bbf,stroke:#333,stroke-width:2px
    classDef querier fill:#bfb,stroke:#333,stroke-width:2px
    classDef pool fill:#ffd,stroke:#333,stroke-width:2px
    
    class UI1,UI2,UI3 client
    class R1,R2 router
    class Q1,Q2,Q3 querier
    class CP1,CP2 pool
```

## Components
```mermaid
classDiagram
    class WsHandler {
        +session_manager: Arc<SessionManager>
        +connection_pool: Arc<QuerierConnectionPool>
        +handle_connection()
        +remove_querier_connection()
        +_shutdown()
    }

    class SessionManager {
        +sessions: RwAHashMap<ClientId, SessionInfo>
        +mapped_queriers: RwAHashMap<QuerierName, HashSet<TraceId>>
        +register_client()
        +update_session_activity()
        +remove_trace_id()
        +unregister_client()
        +set_querier_for_trace()
        +get_querier_for_trace()
    }

    class QuerierConnectionPool {
        +connections: RwAHashMap<QuerierName, Arc<QuerierConnection>>
        +get_or_create_connection()
        +remove_querier_connection()
        +maintain_connections()
        +get_active_connection()
    }

    class QuerierConnection {
        +querier_name: QuerierName
        +write: Arc<Mutex<Option<SplitSink>>>
        +response_router: Arc<ResponseRouter>
        +is_connected: Arc<AtomicBool>
        +connect()
        +disconnect()
        +send_message()
        +register_request()
        +unregister_request()
    }

    class ResponseRouter {
        +routes: RwAHashMap<TraceId, Sender<WsServerEvents>>
        +route_response()
        +flush()
        +spawn_cleanup_task()
    }

    WsHandler --> SessionManager
    WsHandler --> QuerierConnectionPool
    QuerierConnectionPool --> QuerierConnection
    QuerierConnection --> ResponseRouter
```


Related Issues: #6396

---------

Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
Co-authored-by: Abhay Padamani <141122205+ktx-abhay@users.noreply.github.com>
Co-authored-by: Ashish Kolhe <35160958+oasisk@users.noreply.github.com>
Co-authored-by: udev <uddhav@openobserve.ai>
Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech>
Co-authored-by: Omkar Kesarkhane <127078473+omkarK06@users.noreply.github.com>
Co-authored-by: udev <uddhavdave1@gmail.com>
Co-authored-by: Subhra264 <chakrabortysubhradeep556@gmail.com>
Co-authored-by: ktx-vaidehi <134508096+ktx-vaidehi@users.noreply.github.com>
2025-03-26 11:43:40 -07:00
Subhra264andTaiming Liu 0045f0b21e fix: remove clean_complete job in scheduler (#6406)
The `clean_complete` scheduler job is not really required as in the
handler job we can check if the job has exceeded max retries, if it does
exceed, we will skip to the next scheduled trigger.


---------

Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
2025-03-26 10:47:58 -07:00
ktx-vaidehi 64bfaa10e3 feat: dashboard background color option for metric chart (#6397) 2025-03-26 20:18:28 +05:30
Taiming Liu ac215f4847 fix: pipeline dynamic stream (#6395)
- avoid creating empty stream with dynamic stream regex
- lower case all stream names created by pipeline, dynamic or static


fix #6383 and #6375
2025-03-26 06:19:00 -07:00
udev ea51d37ac6 fix: escape control characters in templates (#6400) 2025-03-26 17:46:18 +05:30
Shyam Panjiyar b0ac97c523 test: no histogram displays if no data available playwright (#6391) 2025-03-26 13:33:37 +05:30
Bhargav dace1fef23 fix: UI changes on datasource page (#6390) 2025-03-25 22:08:29 +05:30
Sai Nikhil b006800b37 fix: alert enable or disable is not updated in the UI when filter applied (#6388)
1. This PR fixes that alert enable is not updated in the UI when filter
is applied
2. This PR also fixes the alert enable disable in search across folders
3. This PR also removes the id from exported alert
2025-03-25 16:07:58 +05:30
udev 6b256eb255 fix: Action as a destination input format (#6389)
This fix allows users to define any valid json in the alert template.
Alert module will take care of converting the msg to action request.
They need not adhere to a format.
2025-03-25 15:36:11 +05:30
Subhra264 5b7536d7fc feat: add variables for trigger time in seconds, millis for alert template (#6387)
This pr adds two more variables to alert templates -
`alert_trigger_time_millis` (the alert trigger occurance time in epoch
milliseconds) and `alert_trigger_time_seconds` (the alert trigger
occurance time in epoch seconds). Note that the `alert_trigger_time` is
alert trigger occurance time in UTC epoch microseconds.
2025-03-25 15:03:34 +05:30
ktx-vaidehiandKirtan Thakkar b435ce798f fix: passing variables of dashboard drilldown to same dashboard (#6385)
Co-authored-by: Kirtan Thakkar <kirtan@kiara.tech>
2025-03-25 13:58:35 +05:30
Hengfei Yang afedd1907e fix: duplicated env for enrichment table (#6384) 2025-03-25 15:30:44 +08:00
Omkar Kesarkhane f18f265e12 fix: Updated actions execution type label (#6374) 2025-03-25 09:00:38 +05:30
Hengfei Yang 18c781ab34 fix: add unit test for is_wal_file (#6382) 2025-03-25 10:36:59 +08:00
Hengfei Yang b64c8565f5 fix: pending delete on single node (#6378)
This PR fixed the `PENDING_DELETED_FILE` optimizer on single node. it
report a lot of `[INGESTER:JOB] the file was released, delete it` by
mistake.
2025-03-24 09:40:03 -07:00
Loakesh Indiran 14886c1a18 fix: m20250320_000001_remove_alert_name_unique_constraint migration (#6379) 2025-03-24 21:17:38 +05:30
Taiming Liu bb3dfd5bef chore: remove unnecessary logging (#6377) 2025-03-24 22:14:37 +08:00
Subhra264 6352357ea2 fix: nobody can update root except root user (#6376) 2025-03-24 17:36:15 +05:30
Shyam Panjiyar cac2ded2cf test: Create Objects and Refactoring POM PyTest (#6357) 2025-03-24 16:50:30 +05:30
c3759a4231 fix: Migration for scheduled_jobs table to use alert id as module_key (#6365)
Co-authored-by: udev <uddhavdave1@gmail.com>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2025-03-24 15:10:44 +05:30
Omkar Kesarkhane b7af94f487 fix: Actions UI improvements (#6372)
1. Remove action_id from Histogram and Page-count search payload
2. Rename Action type `Service` to `Real Time`
3. Added validation for service account selection while creating action
2025-03-24 14:33:16 +05:30
Sai Nikhil f7d3e5aee0 fix: added tooltip for all icons in icon bar (#6371)
1. added tooltip for every icons in icon bar
2025-03-24 12:32:27 +05:30
06920b203b fix: previous histogram getting display even after there is no data (#6346)
Co-authored-by: ktx-vaidehi <vaidehi.akhani@kiara.tech>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
2025-03-24 11:27:20 +05:30
udev 9f7521c74f fix: Actions bugs (#6367)
rename token env and disallow changing execution type
2025-03-23 23:47:16 +05:30
ktx-vaidehi 986c01a4a2 fix: dashboard error message getting removed (#6366) 2025-03-23 11:47:52 +05:30
Sai Nikhil c5be09fbf4 fix: changes for icon bar in logs page (#6362)
This PR includes changes 
1. introduced icons for histogram , sql mode, quickmode and moved search
history , download logs , search scheduled jobs options into menu that
will be there at right end
2. this will make the UI to accommodate upto 1200px with datetime as
relative , 1350 when it is absolute
3. introducing icons saved space of ~200px in total for both sides
2025-03-23 10:16:34 +05:30
ktx-vaidehi 2af8b83699 fix: dashboard panel name and folder text changes (#6364) 2025-03-22 18:38:57 +05:30
Hengfei Yang a09d626d42 fix: disable add stream & add field when user doesn't enable UDS (#6361) 2025-03-22 10:54:35 +08:00
Hengfei Yang 2c039bfbb9 feat: support db replicas (#6358) 2025-03-21 20:19:33 +08:00
Abhay Padamani 2c51f7fde3 fix: dashboard empty error issue (#6348)
- #3044
2025-03-21 17:01:57 +05:30
71b32480ab feat: alerts folder implementation (UI) (#6228)
implemented alerts folders in the UI

1. search across folders
2. alert listing page moved condition and description column to the
expand so whenever user clicks on any row it will show respective
condition and description if there
3. provided fixed width for columns so that all columns are in view port
visibility expect name column as it can be varied depending upon name
given by user
4. search across all folders implemented in this PR only
Import alert add ons
1 additional things done to import alert as well like introducing the
folders in import page

---------

Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
v0.14.5-rc5
2025-03-21 13:31:54 +05:30
Hengfei Yang 1f75e670c2 ci: update object_store (#6356) 2025-03-21 14:58:22 +08:00
Subhra264 6926f67e08 fix: alert move rbac issues (#6352)
- Fixes 403 error when alert move api is called
- 403 error when trying to create alert in default folder even after
having permission
- Fixes # Regular user without `GET` user permission should not be able
to get list of all users
2025-03-21 00:19:00 +05:30
ktx-vaidehi 4ad2865ddb fix: dashboard variables dependency not working properly (#6350) 2025-03-20 18:19:51 +05:30
ktx-vaidehi 901ad1c5dc fix: dashboard breakdown value not present chart not getting plot (#6344)
- #6310
2025-03-20 16:37:29 +05:30
Subhra264 347ffeb787 fix: when dashboard owner not present in creation payload, use currently logged in user (#6343)
Currently in the main, when creating a dashboard, we check if the
`owner` field is there in the backend. If it is not present, it returns
Missing Owner error. This PR fixes this by checking first if the `owner`
field is non-empty, otherwise it uses the currently logged in user as
`owner`.
2025-03-20 15:43:13 +05:30
Omkar Kesarkhane ab0b3a64f8 fix: added errors when actions are not enabled while creating destination (#6342)
1. If actions are disabled and user tries to import action in
destination, Unknown Error was shown instead of 'Action type not
supported'
2025-03-20 13:47:41 +05:30
Subhra264 caf798e5c2 fix: Alert folder rbac migration issues (#6338) 2025-03-20 11:48:47 +05:30
Omkar Kesarkhaneandudev 28106fc564 fix: Actions UI issues (#6325)
1. Action Issues
2. #6288

---------

Co-authored-by: udev <uddhavdave1@gmail.com>
2025-03-20 09:39:54 +08:00
Hengfei Yang f3fe0d4ba9 fix: ofga version compare (#6339) 2025-03-20 00:37:28 +08:00
Hengfei Yang 3f215526f2 fix: ofga for enterprise build (#6337) 2025-03-19 22:08:40 +08:00
udev 6273125f8d fix: allow patch requests on router (#6336) 2025-03-19 19:28:08 +05:30
Abhay Padamani 0698b9f4e8 fix: with query fields validation issue (#6334)
- #6332
2025-03-19 18:10:32 +05:30
Hengfei Yang 9f33b16feb fix: histogram stream sync (#6316)
fixed #6315
2025-03-19 19:30:10 +08:00
Subhra264 d220d134ea fix: when pipeline reaches max retries by watch_timeout, skip to next run (#6329)
When a pipeline takes more than the scheduler timeout to complete the
task, the scheduler background job `watch_timeout` increases the
`retries` field by 1 and again updates the scheduled jobs status back to
`waiting` from `processing`. If this happens three times, the `retries`
field value go beyond the max retries and hence is never pulled by
scheduler again.

With this change, whenever we process a derived stream we first check if
the current `retries` field matches with the max retries. If it matches
then we simply make the `retries` `0` and again run it after the
scheduled frequency. The start_time for the query is not changed though,
so there should not be any data loss when quering.
2025-03-19 16:33:19 +05:30
Zhao.wei 625717250a fix: cancel init cache stat (#6327) 2025-03-19 17:44:19 +08:00
Sai Nikhil 4289ec0a68 fix: alert and pipeline frequency warning message change (#6318)
This PR fixes couple of issues 
1. changed the warning message to minimum frequency should be -- minutes
when user tries to give less than minimum frequency interval in alert
and pipeline
2. other is made period in sync with frequency so whenever user toggles
the cron so in pipelines we are auto selecting the period according the
frequency so toggling not picking up the latest values of freq to period
2025-03-19 15:09:13 +05:30
Subhra264andoasisk a47d70460a fix: Partial error fix main (#6312)
The search response should be cached only when it is not a partial error
or if it is so, only for range errors we should cache the result. This
is a hacky fix which caches the result when the above condition is
satisfied.

Partly addresses #6311 (does not fix the root cause where is_partial is
false even when there is function error).

---------

Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
2025-03-19 12:36:57 +05:30