Commit Graph
347 Commits
Author SHA1 Message Date
Shyam Panjiyar c84792d3eb test: streaming search pytest (#6803)
Streaming Search PyTest
2025-05-20 16:22:24 +05:30
Shyam Panjiyar 53459dcecc test: enabled streaming search playwright (#6795)
log search after streaming search enabled playwright
2025-05-20 12:04:30 +05:30
Neha P e40ecdcdcd test: e2e-cmd+enter-test (#6802)
- Verify search results are triggered when pressing Cmd+Enter or
Ctrl+Enter
- Verify query executes with keyboard shortcut after clicking elsewhere
- Verify different query executes with keyboard shortcut
- Verify SQL query executes with keyboard shortcut
2025-05-14 15:01:41 +05:30
ktx-akshay 8eed07fbb6 test: fix dashboard filter testcase (#6790)
Fixed the skipped test cases by using an appropriate selector instead of
force: true.
2025-05-14 12:13:35 +05:30
Abhay Padamaniandktx-akshay 7d6b828646 fix: dashboard panel error msg automation issue (#6787)
Co-authored-by: ktx-akshay <akshay.patil@kiara.tech>
2025-05-12 17:47:28 +05:30
Neha P 3ff97016cf test: e2e-stream-name (#6764)
Added a test to add stream name and assert if stream with uppercase
exists
add a stream with lowercase and assert if stream name with lowercase
exists
2025-05-08 20:16:03 +05:30
Hengfei Yang cb18d89e44 fix: is_querier error for super cluster (#6761) 2025-05-08 21:40:26 +08:00
ktx-akshay dbd6c1017b test(dashboard): add geomap and map test cases (#6722) 2025-05-06 19:14:13 +05:30
40f932198a feat: separate out ddl client and commands (#6675)
fixes https://github.com/openobserve/openobserve/issues/6621

This separates the functions which execute ddl statements, and only runs
them if certain entry in meta is mis-matching with what we have in our
code. Introduces a new env var `ZO_META_DDL_DSN` , which is optional and
falls back to the original DSN if not specified. Added a new client to
Postgres and Mysql both which uses this DDL DSN, and that is used only
for the ddl statements.

Also added another command to cli `upgrade-db` which runs the db
initialization functions.

For sqlite, there is no change, as it does not have concept of users.

Primarily tested on Postgres to validate that the behavior with
read-write and admin users is as expected.

`DB_SCHEMA_VERSION` is hardcoded in code, and devs are expected to
manually uptick it everytime we do any changes to db schema, similar to
how we maintain ofga.

---------

Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
2025-05-01 16:56:37 +05:30
ktx-akshay 7cc36e986b test: add dashboard methods for pom (#6671) 2025-04-30 12:31:50 +05:30
bff9364ac7 feat: add EventProgress server event (#6668)
- [x] add `percent` calc, based on the query duration

---------

Co-authored-by: ktx-vaidehi <vaidehi.akhani@kiara.tech>
Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: Kirtan Thakkar <kirtan@kiara.tech>
2025-04-29 17:38:58 +05:30
Ashish Kolhe 1f47ac2614 fix: base64 encode cookies (#6596)
fixes : [6564](https://github.com/openobserve/openobserve/issues/6564)
2025-04-28 18:08:48 +05:30
ktx-akshay 034d1d4c53 test: multi y-axis test case fix (#6641) 2025-04-28 12:19:14 +05:30
Shyam PanjiyarandTaiming Liu c3364cbab0 test: Import destination alert playwright (#6631)
1. Import webhook and email destination
2. Import scheduled and realtime alert

---------

Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
2025-04-28 10:45:46 +05:30
ktx-akshay 76582d67d5 test: visualize test cases (#6619) 2025-04-23 20:25:17 +05:30
Shyam PanjiyarandHengfei Yang e8f348819e test: Import Template Destination Alerts Playwright (#6205)
1. Import webhook and email Template

---------

Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
2025-04-23 11:55:46 +05:30
Neha P c64d04d099 test: alert-folder-testcases (#6592)
These tests include 
- Added a new alert
- Add a new folder
- Added a new alert in this folder
- Moved 1 alert in 1 folder
- Bulk move alerts from default folder to new created folder
- Deleted the alerts that were moved to new folder
- Deleted the folder
- Deleted the template and destination
2025-04-21 17:56:10 +05:30
ktx-akshay 29699ed4cb test: multi y-axis stacked charts (#6549) 2025-04-17 13:44:32 +05:30
Sai NikhilandBhargav df1f5eba31 fix: Sql mode skip query validation (#6434)
1. this PR fixes the sql mode query validation when they are not parsed
by the parser we use
2. in this case we are using node-sql parser so we will simply skip the
validation and pass it to BE

---------

Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
2025-04-17 11:27:50 +05:30
Sai NikhilandSubhradeep Chakraborty a5716d88e0 feat: bulk move and export dashboard (#6507)
1. this PR implements bulk move and export dashboards

---------

Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
2025-04-16 22:21:05 +05:30
Hengfei Yang 0f864aab47 feat: add keepalive for scheduler_jobs (#6554) 2025-04-16 19:29:25 +08:00
Shyam Panjiyar 9eec05a6ee test: websocket version 2 connection pytest (#6476) 2025-04-14 15:51:04 +05:30
Shyam Panjiyar 04b83127e7 test: sql vrl satisfied and no satisfied alert pytest (#6529) 2025-04-14 10:59:45 +05:30
Sai Nikhil 064d029939 fix: schema page tab improvements (#6524)
1. This PR adds 
**previous approach** 
1. we were showing both All Fields and UDS tabs even user doesnot have
UDS enabled
2. UDS is shown as second tab


**current approach**
1. only show UDS if enabled otherwise show only All fields 
3. if UDS enabled show it as first tab and instead of All Fields(when
UDS enabled) renamed it to Other Fields and also added tooltip with some
explanatory text after Other Fields with a info icon as
**_

> Other fields show only the schema fields that existed before the
stream was configured to use a user-defined schema.

_**
2025-04-12 18:05:36 +05:30
18d42d93e4 test(dashboard): add test cases for dashboard folder (#6500)
Co-authored-by: ktx-kirtan <124270523+ktx-kirtan@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2025-04-10 15:58:24 +05:30
Shyam Panjiyar 6bed6730ed test: histogram reappears when enabled and data exists playwright (#6455) 2025-04-03 16:08:14 +05:30
Ashish KolheandSubhra264 5f9f304379 fix: improve error handling (#6432)
Co-authored-by: Subhra264 <chakrabortysubhradeep556@gmail.com>
2025-03-31 18:13:31 +05:30
Shyam Panjiyar f21dfd871d test: root and admin search automation using pytest (#6433) 2025-03-31 16:16:13 +05:30
ktx-akshay f9a4b65641 test(dashboards): add import test cases (#6399) 2025-03-27 18:05:56 +05:30
+2 8cd0683a0b feat: ws v2 (#6404)
# WebSocket v2 Architecture

## NOTE: Open Issues:
- [ ] `session_drain` from server is experimental and not ready for
prod. Instead, the client maintains ongoing `trace_id`'s and when cookie
expires it will retry search for all the ongoing searches.

## Overview
Implements websocket v2 with the main goal to establish `Client-Router`
connections `per tab` and `Router-Querier` `persistent` connections.

## New Env (Default):
```sh
ZO_WEBSOCKET_CHECK_COOKIE_EXPIRY=true
ZO_WEBSOCKET_HEALTH_CHECK_INTERVAL_SECS=60
ZO_WEBSOCKET_IS_SESSION_DRAIN_ENABLED=false # This is experimental, not meant for prod
```

## High-Level Flow

1. Client establishes WebSocket connection with the router.  
2. Router creates a session.  
3. Router maintains persistent connections to all queriers.  
4. Client requests are routed to the appropriate querier using
consistent hashing.
5. Responses are routed back using `trace_id` mappings.  
6. Health checks maintain connection validity.  
7. Failed connections trigger automatic cleanup and reconnection.  
8. **Internal Communication Validation Using:**  
   - **8a.** Internal gRPC tokens  
   - **8b.** IP validation  
   - **8c.** WebSocket upgrade headers  

## Connection Lifecycle
```mermaid
sequenceDiagram
    participant Client
    participant Router
    participant ConnectionPool
    participant QuerierConn
    participant Querier
    participant Cluster

    Client->>Router: WS Connection Request
    Router->>SessionManager: register_client(client_id)
    
    Note over Router,SessionManager: Creates channels:<br/>response_tx/rx<br/>disconnect_tx/rx

    par Handle Incoming Messages
        Router->>Router: spawn handle_incoming task
    and Handle Outgoing Messages
        Router->>Router: spawn handle_outgoing task
    end

    Client->>Router: WS Client Event
    Router->>SessionManager: get_querier_for_trace(trace_id)
    
    alt No Existing Mapping
        Router->>Cluster: get_node_from_consistent_hash(trace_id)
        Router->>ConnectionPool: get_or_create_connection(querier)
        
        alt No Existing Connection
            ConnectionPool->>QuerierConn: create_connection
            QuerierConn->>Querier: WS Connection Request
            
            par Connection Tasks
                QuerierConn->>QuerierConn: spawn listen_to_querier_response
                QuerierConn->>QuerierConn: spawn health_check
            end
            
            Querier-->>QuerierConn: Connection Established
        end
        
        Router->>SessionManager: set_querier_for_trace(trace_id, querier)
    end

    Router->>QuerierConn: register_request(trace_id)
    Router->>QuerierConn: send_message(event)
    Querier-->>QuerierConn: Response
    QuerierConn->>Router: Route Response via response_tx
    Router->>Client: Forward Response

    Note over Router,Client: On Disconnect/Error:<br/>1. Send disconnect signal<br/>2. Clean up sessions<br/>3. Drain if needed
```

## Connections
```mermaid
graph TB
    subgraph "Client Layer"
        UI1[UI Client 1]
        UI2[UI Client 2]
        UI3[UI Client 3]
    end

    subgraph "Router Layer"
        R1[Router 1]
        R2[Router 2]
        
        subgraph "Router 1 Connection Pool"
            CP1[Connection Pool]
            QC11[Querier1 Conn]
            QC12[Querier2 Conn]
            QC13[Querier3 Conn]
            CP1 --- QC11
            CP1 --- QC12
            CP1 --- QC13
        end
        
        subgraph "Router 2 Connection Pool"
            CP2[Connection Pool]
            QC21[Querier1 Conn]
            QC22[Querier2 Conn]
            QC23[Querier3 Conn]
            CP2 --- QC21
            CP2 --- QC22
            CP2 --- QC23
        end
        
        R1 --- CP1
        R2 --- CP2
    end

    subgraph "Querier Layer"
        Q1[Querier 1]
        Q2[Querier 2]
        Q3[Querier 3]
    end

    UI1 -.->|WS| R1
    UI2 -.->|WS| R1
    UI3 -.->|WS| R2

    QC11 ===|Persistent WS| Q1
    QC12 ===|Persistent WS| Q2
    QC13 ===|Persistent WS| Q3
    QC21 ===|Persistent WS| Q1
    QC22 ===|Persistent WS| Q2
    QC23 ===|Persistent WS| Q3

    classDef client fill:#f9f,stroke:#333,stroke-width:2px
    classDef router fill:#bbf,stroke:#333,stroke-width:2px
    classDef querier fill:#bfb,stroke:#333,stroke-width:2px
    classDef pool fill:#ffd,stroke:#333,stroke-width:2px
    
    class UI1,UI2,UI3 client
    class R1,R2 router
    class Q1,Q2,Q3 querier
    class CP1,CP2 pool
```

## Components
```mermaid
classDiagram
    class WsHandler {
        +session_manager: Arc<SessionManager>
        +connection_pool: Arc<QuerierConnectionPool>
        +handle_connection()
        +remove_querier_connection()
        +_shutdown()
    }

    class SessionManager {
        +sessions: RwAHashMap<ClientId, SessionInfo>
        +mapped_queriers: RwAHashMap<QuerierName, HashSet<TraceId>>
        +register_client()
        +update_session_activity()
        +remove_trace_id()
        +unregister_client()
        +set_querier_for_trace()
        +get_querier_for_trace()
    }

    class QuerierConnectionPool {
        +connections: RwAHashMap<QuerierName, Arc<QuerierConnection>>
        +get_or_create_connection()
        +remove_querier_connection()
        +maintain_connections()
        +get_active_connection()
    }

    class QuerierConnection {
        +querier_name: QuerierName
        +write: Arc<Mutex<Option<SplitSink>>>
        +response_router: Arc<ResponseRouter>
        +is_connected: Arc<AtomicBool>
        +connect()
        +disconnect()
        +send_message()
        +register_request()
        +unregister_request()
    }

    class ResponseRouter {
        +routes: RwAHashMap<TraceId, Sender<WsServerEvents>>
        +route_response()
        +flush()
        +spawn_cleanup_task()
    }

    WsHandler --> SessionManager
    WsHandler --> QuerierConnectionPool
    QuerierConnectionPool --> QuerierConnection
    QuerierConnection --> ResponseRouter
```


Related Issues: #6396

---------

Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
Co-authored-by: Abhay Padamani <141122205+ktx-abhay@users.noreply.github.com>
Co-authored-by: Ashish Kolhe <35160958+oasisk@users.noreply.github.com>
Co-authored-by: udev <uddhav@openobserve.ai>
Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech>
Co-authored-by: Omkar Kesarkhane <127078473+omkarK06@users.noreply.github.com>
Co-authored-by: udev <uddhavdave1@gmail.com>
Co-authored-by: Subhra264 <chakrabortysubhradeep556@gmail.com>
Co-authored-by: ktx-vaidehi <134508096+ktx-vaidehi@users.noreply.github.com>
2025-03-26 11:43:40 -07:00
Shyam Panjiyar b0ac97c523 test: no histogram displays if no data available playwright (#6391) 2025-03-26 13:33:37 +05:30
Shyam Panjiyar cac2ded2cf test: Create Objects and Refactoring POM PyTest (#6357) 2025-03-24 16:50:30 +05:30
c3759a4231 fix: Migration for scheduled_jobs table to use alert id as module_key (#6365)
Co-authored-by: udev <uddhavdave1@gmail.com>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2025-03-24 15:10:44 +05:30
06920b203b fix: previous histogram getting display even after there is no data (#6346)
Co-authored-by: ktx-vaidehi <vaidehi.akhani@kiara.tech>
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
2025-03-24 11:27:20 +05:30
Sai Nikhil c5be09fbf4 fix: changes for icon bar in logs page (#6362)
This PR includes changes 
1. introduced icons for histogram , sql mode, quickmode and moved search
history , download logs , search scheduled jobs options into menu that
will be there at right end
2. this will make the UI to accommodate upto 1200px with datetime as
relative , 1350 when it is absolute
3. introducing icons saved space of ~200px in total for both sides
2025-03-23 10:16:34 +05:30
71b32480ab feat: alerts folder implementation (UI) (#6228)
implemented alerts folders in the UI

1. search across folders
2. alert listing page moved condition and description column to the
expand so whenever user clicks on any row it will show respective
condition and description if there
3. provided fixed width for columns so that all columns are in view port
visibility expect name column as it can be varied depending upon name
given by user
4. search across all folders implemented in this PR only
Import alert add ons
1 additional things done to import alert as well like introducing the
folders in import page

---------

Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
2025-03-21 13:31:54 +05:30
Omkar Kesarkhane 205965f854 fix: fixed histogram toggle playwright test (#6305) 2025-03-18 12:05:54 -07:00
Taiming Liu f931ac00fc fix: playwrite test selecting quick mode (#6304) 2025-03-18 11:29:22 -07:00
Omkar Kesarkhaneandudev bb4f30f21d feat: Actions in Logs and Alert Destinations (#6129)
Co-authored-by: udev <uddhavdave1@gmail.com>
2025-03-18 14:42:30 +05:30
ktx-kirtanandSubhra264 022506462d test: dashboard filters remove failed test case (#6286)
Co-authored-by: Subhra264 <chakrabortysubhradeep556@gmail.com>
2025-03-17 19:42:39 +05:30
73b49ea686 test: dashboard filter testcases (#6284)
Co-authored-by: ktx-kirtan <124270523+ktx-kirtan@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Kirtan Thakkar <kirtan@kiara.tech>
2025-03-17 15:13:49 +05:30
Bhargav 0fc1e65fc7 fix: added debounce in stream list filter (#6253) 2025-03-13 22:57:32 +05:30
77a70ef4fa fix: search type to lowercase (#6197)
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
2025-03-13 21:43:05 +05:30
ktx-akshayandHengfei Yang d7b90a48e0 test: dashboard timeout testcases (#6184)
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
2025-03-07 10:46:54 +05:30
Hengfei Yang 595490ea33 ci: update rust to nightly-2025-03-02 (#6145) 2025-03-02 17:27:10 +08:00
Subhra264 4a7784b409 feat: Scheduler redesign with mpsc channels (#6121)
Uses mpsc channels to process pulled scheduled jobs so that it does not
block the entire alert manager to pull other jobs when there is
capacity. Makes use of semaphores so that we pull the same number of
jobs as the number of available workers.

### The new design is -
- We spawn the same number of worker threads as the configured max concurreny env.
- After pulling jobs, it sends to each worker through the mpsc channel
- Each worker processes the received trigger
- After the scheduler interval, the puller job checks how many workers are still available i.e. how many workers are done with processing triggers.
- It only pulls the same number of jobs from db as the number of available workers at that particular time.
The cycle continues.

So, the scheduler pulls at max the "max concurrency" number of jobs at a time depending on the availability of worker threads.
2025-03-01 23:25:46 +05:30
Hengfei Yang f8c5f012bd feat: remove configure ZO_USAGE_ORG (#6142) 2025-03-01 21:41:39 +08:00
Patrick Sullivan 3f4074b8cf fix: Misc V2 alerts routes fixes (#6137)
Makes the following fixes to these routes:
- `ListAlerts`
  - adds `enabled` to the response body
  - adds `name_substring` to the request body
- `MoveAlerts`
- changes the method type to `PATCH` since this is a non-idempotent
update
- changing to `PATCH` coincidentally happens to fix a routing issue
where requests to `MoveAlerts` were being routed to the `UpdateAlerts`
handler
- `EnableAlert`
- changes the method type to `PATCH` since this is a non-idempotent
update
- `TriggerAlert`
- changes the method type to `PATCH` since this is a non-idempotent
update
2025-02-28 17:19:22 -06:00
Sai Nikhil 2bea89307c fix: search jobs UI (#5706)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

## Release Notes

- **New Features**
  - Introduced search job scheduling functionality.
- Added ability to create, list, cancel, retry, and delete scheduled
searches.
- Enhanced user interface for managing scheduled search jobs, including
a new component for displaying scheduled jobs.
- Implemented conditional rendering for search history and search
scheduler components.
- Added new methods for handling scheduled searches within the service
layer.

- **Improvements**
  - Refined search bar interface with new scheduling options.
  - Updated pagination handling for job-related data.
- Enhanced overall functionality related to job management and
pagination.
- Improved loading indicators and user experience during data retrieval.

- **Bug Fixes**
  - Corrected conditional rendering of search history components.
  - Optimized job data retrieval and pagination logic.
  - Simplified sorting logic within the search history component.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-02-27 16:07:45 +05:30
Sai NikhilandTaiming Liu 85e8af977d fix: Pipeline improvements (#6078)
1. this PR introduces new design for query node 
2. dynamic stream name for destination node instead of create new stream
it allows user to input any non existing stream name as destination and
at the time of ingestion it will be created dynamically
3. Fixes issues of #5886

---------

Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
2025-02-26 14:05:11 -08:00