mirror of
https://github.com/openobserve/openobserve.git
synced 2026-11-04 16:09:05 +00:00
1a15c45e0e
# Objectives:
1. Stream results to the client using web-socket.
2. Cache results for a query as a single result cache.
## New Endpoint:
```sh
/api/{org_id}/ws/{req_id}
```
## WS Design:
1. The request_id is used to handle the sessions `pub static
WS_SESSIONS: Lazy<RwHashMap<String, WsSession>> =
Lazy::new(DashMap::default);`.
2. The request_id is removed from the session cache and the web-socket
is disconnected once single search is complete. (e.g. for 5 dashboards
in a single page, 5 individual requests are made). The design choice is
discussed [here](https://github.com/openobserve/walmart1/issues/207).
3. The supported events and responses the web-socket are described
below.
4. Auditing for web-socket linked
[here](https://github.com/openobserve/o2-enterprise/pull/248)
5. Org level setting to enable and disable web-socket communication.
(Default: disabled)
## New Env:
```sh
ZO_WEBSOCKET_ENABLED=true
```
## Env to be enabled:
```sh
ZO_USE_MULTIPLE_RESULT_CACHE=true
ZO_RESULT_CACHE_ENABLED=true
```
- If `ZO_RESULT_CACHE_ENABLED` is set to `false` which is default.
Results cache will not be written.
## Nginx config:
```yaml
nginx.ingress.kubernetes.io/proxy-http-version: "1.1" # Enable HTTP/1.1 for WebSockets
nginx.ingress.kubernetes.io/enable-websocket: "true"
nginx.ingress.kubernetes.io/proxy-set-headers: |
Upgrade $http_upgrade;
Connection "Upgrade";
```
## Note: Disable `keep-alive` settings if present
```yaml
# nginx.ingress.kubernetes.io/connection-proxy-header: keep-alive
```
## Client Behavior before this pr:
1. For a single query, client calls the `_search_partition` api which
returns the list of partitions `Vec<[start_time, end_time]>` -> then
makes a `_search` api call for each of the partition.
2. The client also handles the logic for `max_query_range` for the
stream.
3. Handle the logic for dashboards loading from right to left
irrespective of `order_by` but honors the result by `order_by`
## New Behavior:
1. Client makes a web socket request with a request_id (uuid). This
req_id is stored in-memory cache along with its session object.
2. Server Search
- STEP 1: Search result cache
- STEP 1(a): Handle the cache responses & query the deltas
- STEP 2: Search without cache if no caches found
- STEP 3: Write to result cache
6. Server will now handle querying the partitions and then calling
search for each.
7. The results are streamed as in when the responses are received.
8. Server will handle the logic for dashboards loading from right to
left irrespective of `order_by` but honors the result by `order_by`
## Modified Code:
1. The `search` service will now use `write_results_v2` when writing to
cache.
2. The main difference between `write_results` and `write_results_v2` is
the strategy for deduplication. In `write_results_v2` we remove all the
hits within the same Date, Hour, Minute and Second; before writing to
cache.
```rust
/// # Caching Strategy
/// 1. **Select Hit for Deduplication**:
/// - Selects either the first or last record based on the `is_descending` flag.
/// - `is_descending = true`: Selects the last record.
/// - `is_descending = false`: Selects the first record.
///
/// 2. **Remove All Hits Within the Same Date, Hour, Minute, and Second for Deduplication**:
/// - Identifies the timestamp of the hit to remove (based on the `is_descending` flag, either
/// the first or the last record is chosen).
/// - Removes all hits that match the exact same date, hour, minute, and second
/// (`YYYY-MM-DDTHH:MM:SS`) as the identified hit.
/// - Example: If the timestamp to remove is `2024-12-06T04:15:30`, only hits with the exact
/// timestamp `2024-12-06T04:15:30` are removed, while others with different seconds (e.g.,
/// `2024-12-06T04:15:29` or `2024-12-06T04:15:31`) are retained.
```
## Web-socket Events:
1. Search
```sh
{
"type": "search",
"content": {
"trace_id": "d56755eb02cf47ecb3064ee096735c4a",
"payload": {
"query": {
"sql": "select histogram(_timestamp, '1 hour') AS zo_sql_key, count(*) AS zo_sql_num from \"olympics\" GROUP BY zo_sql_key ORDER BY zo_sql_key",
"start_time": 1733499294217000,
"end_time": 1734708894217000,
"size": -1,
"query_fn": "LmFiYywgZXJyID0gZ2V0X2VucmljaG1lbnRfdGFibGVfcmVjb3JkKCJvbHltcGljcyIsCnsKICAicmFuayI6IHRvX3N0cmluZyEoLnJhbmspCn0pCi5yYW5rX2tleXdvcmQgPSAuYWJjLmtleXdvcmQKLg..",
"sql_mode": "full"
}
},
"stream_type": "enrichment_tables",
"search_type": "ui",
"use_cache": true
}
}
```
2. Cancel
```sh
{
"type": "cancel",
"content": {
"trace_id": "b221bb019f3742e8bd93f817d530ef70"
}
}
```
Related Issues: #5173, #5188, #5266, #5320
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Introduced WebSocket search functionality with a toggle in settings.
- Added support for real-time data retrieval via WebSocket in various
components.
- Enhanced search operations with new methods for handling WebSocket
queries.
- Added new properties and methods to improve handling of WebSocket
sessions and requests.
- New interface for search request payloads and improvements in query
handling.
- Implemented WebSocket proxy handling for improved connection
management.
- Enhanced caching functionality with new methods for managing cached
responses.
- Introduced optional field for enabling WebSocket search in
organization settings.
- Added a new constant for handling partial error responses in search
operations.
- Added a new configuration option for enabling WebSocket support.
- Updated package version and various dependency versions to enhance
stability.
- **Bug Fixes**
- Improved error handling and logging for search requests and caching
mechanisms.
- Enhanced validation and URL construction for alerts.
- **Documentation**
- Updated comments and documentation for clarity on new features and
changes.
- **Style**
- Minor formatting adjustments for consistency across components and
services.
- **Chores**
- Updated dependency versions to enhance stability and performance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech>
Co-authored-by: hengfei yang <hengfei.yang@gmail.com>
web
This template should help get you started developing with Vue 3 in Vite.
Recommended IDE Setup
VSCode + Volar (and disable Vetur) + TypeScript Vue Plugin (Volar).
Type Support for .vue Imports in TS
TypeScript cannot handle type information for .vue imports by default, so we replace the tsc CLI with vue-tsc for type checking. In editors, we need TypeScript Vue Plugin (Volar) to make the TypeScript language service aware of .vue types.
If the standalone TypeScript plugin doesn't feel fast enough to you, Volar has also implemented a Take Over Mode that is more performant. You can enable it by the following steps:
- Disable the built-in TypeScript Extension
- Run
Extensions: Show Built-in Extensionsfrom VSCode's command palette - Find
TypeScript and JavaScript Language Features, right click and selectDisable (Workspace)
- Run
- Reload the VSCode window by running
Developer: Reload Windowfrom the command palette.
Customize configuration
See Vite Configuration Reference.
Project Setup
npm install
Compile and Hot-Reload for Development
For stand-alone UI development environment, create .env file and which contains below environment variable
VITE_OPENOBSERVE_ENDPOINT=http://localhost:5080/
npm run dev
Type-Check, Compile and Minify for Production
npm run build
Run Unit Tests with Vitest
npm run test:unit
Run End-to-End Tests with Cypress
npm run test:e2e:dev
This runs the end-to-end tests against the Vite development server. It is much faster than the production build.
But it's still recommended to test the production build with test:e2e before deploying (e.g. in CI environments):
npm run build
npm run test:e2e
Lint with ESLint
npm run lint