Commit Graph
325 Commits
Author SHA1 Message Date
Shyam Panjiyar 04b83127e7 test: sql vrl satisfied and no satisfied alert pytest (#6529) 2025-04-14 10:59:45 +05:30
Sai Nikhil 064d029939 fix: schema page tab improvements (#6524)
1. This PR adds 
**previous approach** 
1. we were showing both All Fields and UDS tabs even user doesnot have
UDS enabled
2. UDS is shown as second tab


**current approach**
1. only show UDS if enabled otherwise show only All fields 
3. if UDS enabled show it as first tab and instead of All Fields(when
UDS enabled) renamed it to Other Fields and also added tooltip with some
explanatory text after Other Fields with a info icon as
**_

> Other fields show only the schema fields that existed before the
stream was configured to use a user-defined schema.

_**
2025-04-12 18:05:36 +05:30
18d42d93e4 test(dashboard): add test cases for dashboard folder (#6500)
Co-authored-by: ktx-kirtan <124270523+ktx-kirtan@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2025-04-10 15:58:24 +05:30
Shyam Panjiyar 6bed6730ed test: histogram reappears when enabled and data exists playwright (#6455) 2025-04-03 16:08:14 +05:30
Ashish KolheandSubhra264 5f9f304379 fix: improve error handling (#6432)
Co-authored-by: Subhra264 <chakrabortysubhradeep556@gmail.com>
2025-03-31 18:13:31 +05:30
Shyam Panjiyar f21dfd871d test: root and admin search automation using pytest (#6433) 2025-03-31 16:16:13 +05:30
ktx-akshay f9a4b65641 test(dashboards): add import test cases (#6399) 2025-03-27 18:05:56 +05:30
+2 8cd0683a0b feat: ws v2 (#6404)
# WebSocket v2 Architecture

## NOTE: Open Issues:
- [ ] `session_drain` from server is experimental and not ready for
prod. Instead, the client maintains ongoing `trace_id`'s and when cookie
expires it will retry search for all the ongoing searches.

## Overview
Implements websocket v2 with the main goal to establish `Client-Router`
connections `per tab` and `Router-Querier` `persistent` connections.

## New Env (Default):
```sh
ZO_WEBSOCKET_CHECK_COOKIE_EXPIRY=true
ZO_WEBSOCKET_HEALTH_CHECK_INTERVAL_SECS=60
ZO_WEBSOCKET_IS_SESSION_DRAIN_ENABLED=false # This is experimental, not meant for prod
```

## High-Level Flow

1. Client establishes WebSocket connection with the router.  
2. Router creates a session.  
3. Router maintains persistent connections to all queriers.  
4. Client requests are routed to the appropriate querier using
consistent hashing.
5. Responses are routed back using `trace_id` mappings.  
6. Health checks maintain connection validity.  
7. Failed connections trigger automatic cleanup and reconnection.  
8. **Internal Communication Validation Using:**  
   - **8a.** Internal gRPC tokens  
   - **8b.** IP validation  
   - **8c.** WebSocket upgrade headers  

## Connection Lifecycle
```mermaid
sequenceDiagram
    participant Client
    participant Router
    participant ConnectionPool
    participant QuerierConn
    participant Querier
    participant Cluster

    Client->>Router: WS Connection Request
    Router->>SessionManager: register_client(client_id)
    
    Note over Router,SessionManager: Creates channels:<br/>response_tx/rx<br/>disconnect_tx/rx

    par Handle Incoming Messages
        Router->>Router: spawn handle_incoming task
    and Handle Outgoing Messages
        Router->>Router: spawn handle_outgoing task
    end

    Client->>Router: WS Client Event
    Router->>SessionManager: get_querier_for_trace(trace_id)
    
    alt No Existing Mapping
        Router->>Cluster: get_node_from_consistent_hash(trace_id)
        Router->>ConnectionPool: get_or_create_connection(querier)
        
        alt No Existing Connection
            ConnectionPool->>QuerierConn: create_connection
            QuerierConn->>Querier: WS Connection Request
            
            par Connection Tasks
                QuerierConn->>QuerierConn: spawn listen_to_querier_response
                QuerierConn->>QuerierConn: spawn health_check
            end
            
            Querier-->>QuerierConn: Connection Established
        end
        
        Router->>SessionManager: set_querier_for_trace(trace_id, querier)
    end

    Router->>QuerierConn: register_request(trace_id)
    Router->>QuerierConn: send_message(event)
    Querier-->>QuerierConn: Response
    QuerierConn->>Router: Route Response via response_tx
    Router->>Client: Forward Response

    Note over Router,Client: On Disconnect/Error:<br/>1. Send disconnect signal<br/>2. Clean up sessions<br/>3. Drain if needed
```

## Connections
```mermaid
graph TB
    subgraph "Client Layer"
        UI1[UI Client 1]
        UI2[UI Client 2]
        UI3[UI Client 3]
    end

    subgraph "Router Layer"
        R1[Router 1]
        R2[Router 2]
        
        subgraph "Router 1 Connection Pool"
            CP1[Connection Pool]
            QC11[Querier1 Conn]
            QC12[Querier2 Conn]
            QC13[Querier3 Conn]
            CP1 --- QC11
            CP1 --- QC12
            CP1 --- QC13
        end
        
        subgraph "Router 2 Connection Pool"
            CP2[Connection Pool]
            QC21[Querier1 Conn]
            QC22[Querier2 Conn]
            QC23[Querier3 Conn]
            CP2 --- QC21
            CP2 --- QC22
            CP2 --- QC23
        end
        
        R1 --- CP1
        R2 --- CP2
    end

    subgraph "Querier Layer"
        Q1[Querier 1]
        Q2[Querier 2]
        Q3[Querier 3]
    end

    UI1 -.->|WS| R1
    UI2 -.->|WS| R1
    UI3 -.->|WS| R2

    QC11 ===|Persistent WS| Q1
    QC12 ===|Persistent WS| Q2
    QC13 ===|Persistent WS| Q3
    QC21 ===|Persistent WS| Q1
    QC22 ===|Persistent WS| Q2
    QC23 ===|Persistent WS| Q3

    classDef client fill:#f9f,stroke:#333,stroke-width:2px
    classDef router fill:#bbf,stroke:#333,stroke-width:2px
    classDef querier fill:#bfb,stroke:#333,stroke-width:2px
    classDef pool fill:#ffd,stroke:#333,stroke-width:2px
    
    class UI1,UI2,UI3 client
    class R1,R2 router
    class Q1,Q2,Q3 querier
    class CP1,CP2 pool
```

## Components
```mermaid
classDiagram
    class WsHandler {
        +session_manager: Arc<SessionManager>
        +connection_pool: Arc<QuerierConnectionPool>
        +handle_connection()
        +remove_querier_connection()
        +_shutdown()
    }

    class SessionManager {
        +sessions: RwAHashMap<ClientId, SessionInfo>
        +mapped_queriers: RwAHashMap<QuerierName, HashSet<TraceId>>
        +register_client()
        +update_session_activity()
        +remove_trace_id()
        +unregister_client()
        +set_querier_for_trace()
        +get_querier_for_trace()
    }

    class QuerierConnectionPool {
        +connections: RwAHashMap<QuerierName, Arc<QuerierConnection>>
        +get_or_create_connection()
        +remove_querier_connection()
        +maintain_connections()
        +get_active_connection()
    }

    class QuerierConnection {
        +querier_name: QuerierName
        +write: Arc<Mutex<Option<SplitSink>>>
        +response_router: Arc<ResponseRouter>
        +is_connected: Arc<AtomicBool>
        +connect()
        +disconnect()
        +send_message()
        +register_request()
        +unregister_request()
    }

    class ResponseRouter {
        +routes: RwAHashMap<TraceId, Sender<WsServerEvents>>
        +route_response()
        +flush()
        +spawn_cleanup_task()
    }

    WsHandler --> SessionManager
    WsHandler --> QuerierConnectionPool
    QuerierConnectionPool --> QuerierConnection
    QuerierConnection --> ResponseRouter
```


Related Issues: #6396

---------

Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
Co-authored-by: Abhay Padamani <141122205+ktx-abhay@users.noreply.github.com>
Co-authored-by: Ashish Kolhe <35160958+oasisk@users.noreply.github.com>
Co-authored-by: udev <uddhav@openobserve.ai>
Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech>
Co-authored-by: Omkar Kesarkhane <127078473+omkarK06@users.noreply.github.com>
Co-authored-by: udev <uddhavdave1@gmail.com>
Co-authored-by: Subhra264 <chakrabortysubhradeep556@gmail.com>
Co-authored-by: ktx-vaidehi <134508096+ktx-vaidehi@users.noreply.github.com>
2025-03-26 11:43:40 -07:00
Shyam Panjiyar b0ac97c523 test: no histogram displays if no data available playwright (#6391) 2025-03-26 13:33:37 +05:30
Shyam Panjiyar cac2ded2cf test: Create Objects and Refactoring POM PyTest (#6357) 2025-03-24 16:50:30 +05:30
c3759a4231 fix: Migration for scheduled_jobs table to use alert id as module_key (#6365)
Co-authored-by: udev <uddhavdave1@gmail.com>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2025-03-24 15:10:44 +05:30
06920b203b fix: previous histogram getting display even after there is no data (#6346)
Co-authored-by: ktx-vaidehi <vaidehi.akhani@kiara.tech>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
2025-03-24 11:27:20 +05:30
Sai Nikhil c5be09fbf4 fix: changes for icon bar in logs page (#6362)
This PR includes changes 
1. introduced icons for histogram , sql mode, quickmode and moved search
history , download logs , search scheduled jobs options into menu that
will be there at right end
2. this will make the UI to accommodate upto 1200px with datetime as
relative , 1350 when it is absolute
3. introducing icons saved space of ~200px in total for both sides
2025-03-23 10:16:34 +05:30
71b32480ab feat: alerts folder implementation (UI) (#6228)
implemented alerts folders in the UI

1. search across folders
2. alert listing page moved condition and description column to the
expand so whenever user clicks on any row it will show respective
condition and description if there
3. provided fixed width for columns so that all columns are in view port
visibility expect name column as it can be varied depending upon name
given by user
4. search across all folders implemented in this PR only
Import alert add ons
1 additional things done to import alert as well like introducing the
folders in import page

---------

Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
2025-03-21 13:31:54 +05:30
Omkar Kesarkhane 205965f854 fix: fixed histogram toggle playwright test (#6305) 2025-03-18 12:05:54 -07:00
Taiming Liu f931ac00fc fix: playwrite test selecting quick mode (#6304) 2025-03-18 11:29:22 -07:00
Omkar Kesarkhaneandudev bb4f30f21d feat: Actions in Logs and Alert Destinations (#6129)
Co-authored-by: udev <uddhavdave1@gmail.com>
2025-03-18 14:42:30 +05:30
ktx-kirtanandSubhra264 022506462d test: dashboard filters remove failed test case (#6286)
Co-authored-by: Subhra264 <chakrabortysubhradeep556@gmail.com>
2025-03-17 19:42:39 +05:30
73b49ea686 test: dashboard filter testcases (#6284)
Co-authored-by: ktx-kirtan <124270523+ktx-kirtan@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Kirtan Thakkar <kirtan@kiara.tech>
2025-03-17 15:13:49 +05:30
Bhargav 0fc1e65fc7 fix: added debounce in stream list filter (#6253) 2025-03-13 22:57:32 +05:30
77a70ef4fa fix: search type to lowercase (#6197)
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
2025-03-13 21:43:05 +05:30
ktx-akshayandHengfei Yang d7b90a48e0 test: dashboard timeout testcases (#6184)
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
2025-03-07 10:46:54 +05:30
Hengfei Yang 595490ea33 ci: update rust to nightly-2025-03-02 (#6145) 2025-03-02 17:27:10 +08:00
Subhra264 4a7784b409 feat: Scheduler redesign with mpsc channels (#6121)
Uses mpsc channels to process pulled scheduled jobs so that it does not
block the entire alert manager to pull other jobs when there is
capacity. Makes use of semaphores so that we pull the same number of
jobs as the number of available workers.

### The new design is -
- We spawn the same number of worker threads as the configured max concurreny env.
- After pulling jobs, it sends to each worker through the mpsc channel
- Each worker processes the received trigger
- After the scheduler interval, the puller job checks how many workers are still available i.e. how many workers are done with processing triggers.
- It only pulls the same number of jobs from db as the number of available workers at that particular time.
The cycle continues.

So, the scheduler pulls at max the "max concurrency" number of jobs at a time depending on the availability of worker threads.
2025-03-01 23:25:46 +05:30
Hengfei Yang f8c5f012bd feat: remove configure ZO_USAGE_ORG (#6142) 2025-03-01 21:41:39 +08:00
Patrick Sullivan 3f4074b8cf fix: Misc V2 alerts routes fixes (#6137)
Makes the following fixes to these routes:
- `ListAlerts`
  - adds `enabled` to the response body
  - adds `name_substring` to the request body
- `MoveAlerts`
- changes the method type to `PATCH` since this is a non-idempotent
update
- changing to `PATCH` coincidentally happens to fix a routing issue
where requests to `MoveAlerts` were being routed to the `UpdateAlerts`
handler
- `EnableAlert`
- changes the method type to `PATCH` since this is a non-idempotent
update
- `TriggerAlert`
- changes the method type to `PATCH` since this is a non-idempotent
update
2025-02-28 17:19:22 -06:00
Sai Nikhil 2bea89307c fix: search jobs UI (#5706)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

## Release Notes

- **New Features**
  - Introduced search job scheduling functionality.
- Added ability to create, list, cancel, retry, and delete scheduled
searches.
- Enhanced user interface for managing scheduled search jobs, including
a new component for displaying scheduled jobs.
- Implemented conditional rendering for search history and search
scheduler components.
- Added new methods for handling scheduled searches within the service
layer.

- **Improvements**
  - Refined search bar interface with new scheduling options.
  - Updated pagination handling for job-related data.
- Enhanced overall functionality related to job management and
pagination.
- Improved loading indicators and user experience during data retrieval.

- **Bug Fixes**
  - Corrected conditional rendering of search history components.
  - Optimized job data retrieval and pagination logic.
  - Simplified sorting logic within the search history component.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-02-27 16:07:45 +05:30
Sai NikhilandTaiming Liu 85e8af977d fix: Pipeline improvements (#6078)
1. this PR introduces new design for query node 
2. dynamic stream name for destination node instead of create new stream
it allows user to input any non existing stream name as destination and
at the time of ingestion it will be created dynamically
3. Fixes issues of #5886

---------

Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
2025-02-26 14:05:11 -08:00
Shyam Panjiyar b897f5439e test: websocket- Undefined Partitions Error When Histogram is disabled and run query and enabled again (#6085) 2025-02-26 14:57:41 +05:30
Subhra264andhengfei yang be2348747c feat: add source_node to triggers stream (#6097)
This PR adds a `source_node` field to the `triggers` stream to indicate
which alert manager processed the particular trigger. This is the name
of the node.

---------

Co-authored-by: hengfei yang <hengfei.yang@gmail.com>
2025-02-26 12:22:20 +05:30
a3407fe12d fix: ws connection optimizations cherry-pick (#6047)
- fix: close frame to-from client #6033
- fix: ws connection optimizations #6038
- fix: ws connection optimizations v3 #6058 
- fix: ws conn #6080 
- fix: only close sink to backend in ws_proxy

New Env:
```sh
ZO_WEBSOCKET_ENABLED=true
ZO_WEBSOCKET_CLOSE_FRAME_DELAY=50 // microseconds
ZO_WEBSOCKET_SESSION_IDLE_TIMEOUT_SECS=300
ZO_WEBSOCKET_SESSION_MAX_LIFETIME_SECS=3600
ZO_WEBSOCKET_SESSION_GC_INTERVAL_SECS=60
ZO_WEBSOCKET_PING_INTERVAL_SECS=15
```

---------

Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: ShyamOOAI <shyam@openobserve.ai>
2025-02-25 18:28:16 +05:30
Neha P 50daaec8dc test: e2e-string-matchignorecase-fix (#6083) 2025-02-24 20:55:35 +05:30
Neha Pandsai nikhil kethe e3f685898b test: e2e-custom-charts (#6007)
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
2025-02-24 19:21:07 +05:30
Shyam Panjiyar b9eb81ffb8 test: Websocket Playwright (#6061) 2025-02-24 12:05:52 +05:30
ktx-akshay ad2d8882f2 test: dashboard login test cases (#5920) 2025-02-22 10:23:19 +05:30
Shyam Panjiyar acb434b455 test: Updated URL variable for UI Super Cluster (#6022) 2025-02-19 13:37:45 +05:30
Subhra264 e315bcc73a feat: move last_satisfied_at and last_triggered_at to scheduled_job table (#5968)
Fixes https://github.com/openobserve/openobserve/issues/5975
2025-02-18 19:24:51 +05:30
dependabot[bot] 0ded1b4a65 chore(deps): bump serialize-javascript and mocha in /tests/ui-testing (#5930) 2025-02-18 20:55:54 +08:00
Bhargav 19d9f6ffe0 feat: reduced rows per page to 50 (#5978) 2025-02-17 16:30:39 +05:30
Neha P 3bac86c507 test: e2e-fix-flaky-tests (#5980) 2025-02-14 18:47:14 +05:30
Shyam PanjiyarandLoakesh Indiran 477446cb1f test: Websocket PyTest Super Cluster (#5878)
Co-authored-by: Loakesh Indiran <66478092+Loaki07@users.noreply.github.com>
2025-02-14 17:22:44 +05:30
Abhay PadamaniandKirtan Thakkar 5e3297903b fix: show warning and allow data parsing if there is a function error (#5961)
Co-authored-by: Kirtan Thakkar <kirtan@kiara.tech>
2025-02-14 16:21:36 +05:30
Neha P f6fb0e0605 test: e2e-logs-loadtest (#5971)
**Tests covered** 
- Ingest 79k fields with 100 user defined schema
- Reset stream selection while reloading home page and navigating to
logs
- Selecting stream and checking schema and values
- Go to streams page and enable store original data 
- Go to stresstest stream via stream explorer 
- Check if results are visible
2025-02-14 13:40:26 +05:30
Shyam Panjiyar 221821dbb9 test: Summary PyTest in super cluster (#5938) 2025-02-13 12:54:27 +05:30
Shyam Panjiyar 9570a93be2 test: Updated logged out steps for reports refactoring playwright (#5927) 2025-02-11 19:08:20 +05:30
Shyam Panjiyar 082255a704 test: internal user for login and super cluster steps refactoring (#5922)
internal user for login and super cluster steps refactoring
2025-02-11 17:29:19 +05:30
Shyam Panjiyar 9ca79d95ef test: Reports playwright two parts with added test cases in Super Cluster (#5906)
Reports playwright two parts with added test cases in Super Cluster
2025-02-10 10:18:56 +05:30
Neha P 527fcbc76e test: fix stream-selection flakiness (#5898) 2025-02-07 13:48:44 +05:30
Patrick SullivanandSubhradeep Chakraborty 89783b7df4 fix: Fix compiler error in enterprise unit test (#5896)
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
2025-02-07 11:23:56 +05:30
Bhargav 63f88512ee fix: 5870 UI improvements (#5884) 2025-02-06 13:33:51 +05:30