This pr aligns the next_run_at of the alerts/pipelines according to the
timezone of the given alert/pipeline. For example, if the current time
is 11:03pm, and the frequency is 5 mins, the next run at should be
11:05pm, then 11:10pm, and so on. In future, we may want to make
aligntime an optional feature, by giving a toggle option in the
alert/pipeline itself.
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
## What is changed
- [x] Try to improve task starving issue
- [x] Update mmdb frequency
- [x] Improve slow log detection
## MMDB
- Removed ENV `ZO_MMDB_UPDATE_DURATION` because is in seconds
- Add new ENV `ZO_MMDB_UPDATE_DURATION_DAYS` is in days
## Tokio task
We try to use `task::coop` to improve task starving issue in Tokio.
<img width="1496" alt="image"
src="https://github.com/user-attachments/assets/1666488a-146b-49d7-b460-c629528e06f9"
/>
The second chart is for this PR, it is a big slow but more stable.
This PR has some improves:
- Improve tantivy generate, from 7.7s -> 5.9s
- Improve STREAM_SETTINGS cache with atomic
- Add process metrics
- Remove zo_node_open_fd, node_tcp_conn_resets( use
`node_tcp_connections{state="resets"}`)
- Add debug logs for bulk api
- Add debug logs for tantivy panic
This adds support of cipher keys, along with corresponding udfs for
encrypting and decrypting fields from the log.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes: Enterprise Encryption and Key Management Features
- **New Features**
- Added enterprise-level cipher key management functionality.
- Introduced encryption and decryption capabilities for data protection.
- Implemented key storage and retrieval mechanisms.
- New Vue components for adding and managing cipher keys and encryption
mechanisms.
- Added a new settings page for managing cipher keys.
- Integrated advanced key management with multiple provider types.
- Added user-defined functions for encryption and decryption within
DataFusion.
- Introduced a new migration for creating the `cipher_keys` database
table.
- Enhanced HTTP request handling for cipher key operations.
- Added validation for encryption configurations and improved error
handling.
- New asynchronous tasks for managing cipher keys during initialization.
- Enhanced error handling and notifications for cipher key changes.
- **Security Enhancements**
- Added support for managing encryption keys across the platform.
- Implemented secure key storage and rotation mechanisms.
- Enhanced migration logic for cipher keys.
- **User Interface Updates**
- Enhanced user navigation with new tabs for cipher key management.
- Introduced stepper interface for key creation and management.
- Improved localization support for cipher key management.
- **Performance Improvements**
- Optimized query handling for encrypted data.
- Enhanced DataFusion query optimizer with cipher key support.
- **Compatibility**
- Enterprise feature flag controls advanced encryption capabilities.
- Supports multiple key storage and authentication types.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
Co-authored-by: Subhra264 <chakrabortysubhradeep556@gmail.com>
Co-authored-by: ShyamOOAI <shyam@openobserve.ai>
### Summary
This PR updates the logic that is used to generate KSUID primary keys
for the `folders`, `dashboards`, and `alerts` tables in existing
migrations which transform data from the `meta` table into the
`folders`, `dashboards`, and `alerts` tables.
The new logic generates 160-bit "KSUIDs" which are actually 160-bit
SHA-1 hashes of the records being transformed rather than being truly
random KSUIDs.
### Motivation
In the existing implementation, the migrations generate random KSUIDs
for the records that are transformed from the `meta` table and into the
different `folders`, `dashboards`, and `alerts` tables. When running the
application in a single cluster this is acceptable behavior.
However when running the application in a multi-cluster environment this
causes problems. In a multi-cluster environment, each cluster will have
it's own copy of the application database, and each database should
contain the exact same data in the `meta` table. When we run the
migration scripts in each cluster, the data from the `meta` table will
be transformed into the `folders`, `dashboards`, and `alerts` tables.
However if the migration generates a random KSUID for each new record
inserted into those new tables, then different clusters will each
generate different KSUID `id`s for rows in the `folders`, `dashboards`,
and `alerts` tables which should actually have the same `id`s.
Our solution to this problem is that during the migration from the
`meta` table to the `folders`, `dashboards`, and `alerts` tables, rather
than generating random 160-bit KSUID `id`s for the new records, we
generate 160-bit SHA-1 hashes for each record and we use that SHA-1 hash
as the `id` of the each new record. This ensure that each record that is
transformed from the `meta` table will have an `id` in the `folders`,
`alerts`, or `dashboards` tables that is consistent across all clusters.
Newly generated records (ie, records generated AFTER the migration) in
the `folders`, `alerts`, and `dashboards` tables will continue to be
generated with random KSUID `id`s since these `id`s can be shared with
other clusters via NATS super-cluster events and then records with those
`id`s can be inserted into databases in other clusters, ensuring that
records have consistent `id`s across clusters.
### Notes about KSUID generation
To generate a KSUID this function generates the 160-bit SHA-1 hash of
the alert's `org`, `stream_type`, `stream_name`, and `name` and
interprets that 160-bit hash as a 160-bit KSUID. Therefore two KSUIDs
generated in this manner will always be equal if the alerts have the
same `org`, `stream_type`, `stream_name`, and `name`.
⚠️ It is important to note that although KSUIDs generated in this manner
can be parsed as KSUIDs since they are 160-bit values, the KSUIDs
generated in this manner will have timestamp bits which are effectively
random, meaning that the timestamp in any KSUID generated with this
function will be random. This will render the timestamp-sortability
property of these KSUIDs useless. This probably isn't a big deal since
we can always add and sort by a `created_at` column if we want to, but
we should be aware of this limitation.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added SHA-1 hashing dependency to generate unique identifiers (KSUIDs)
for alerts, folders, and dashboards.
- Implemented KSUID generation based on specific entity attributes.
- **Refactor**
- Transitioned database tables from integer-based to KSUID-based primary
keys.
- Updated `StreamType` enum to support more flexible usage.
- **Chores**
- Updated project dependencies to include SHA-1 hashing library.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
# Objectives:
1. Stream results to the client using web-socket.
2. Cache results for a query as a single result cache.
## New Endpoint:
```sh
/api/{org_id}/ws/{req_id}
```
## WS Design:
1. The request_id is used to handle the sessions `pub static
WS_SESSIONS: Lazy<RwHashMap<String, WsSession>> =
Lazy::new(DashMap::default);`.
2. The request_id is removed from the session cache and the web-socket
is disconnected once single search is complete. (e.g. for 5 dashboards
in a single page, 5 individual requests are made). The design choice is
discussed [here](https://github.com/openobserve/walmart1/issues/207).
3. The supported events and responses the web-socket are described
below.
4. Auditing for web-socket linked
[here](https://github.com/openobserve/o2-enterprise/pull/248)
5. Org level setting to enable and disable web-socket communication.
(Default: disabled)
## New Env:
```sh
ZO_WEBSOCKET_ENABLED=true
```
## Env to be enabled:
```sh
ZO_USE_MULTIPLE_RESULT_CACHE=true
ZO_RESULT_CACHE_ENABLED=true
```
- If `ZO_RESULT_CACHE_ENABLED` is set to `false` which is default.
Results cache will not be written.
## Nginx config:
```yaml
nginx.ingress.kubernetes.io/proxy-http-version: "1.1" # Enable HTTP/1.1 for WebSockets
nginx.ingress.kubernetes.io/enable-websocket: "true"
nginx.ingress.kubernetes.io/proxy-set-headers: |
Upgrade $http_upgrade;
Connection "Upgrade";
```
## Note: Disable `keep-alive` settings if present
```yaml
# nginx.ingress.kubernetes.io/connection-proxy-header: keep-alive
```
## Client Behavior before this pr:
1. For a single query, client calls the `_search_partition` api which
returns the list of partitions `Vec<[start_time, end_time]>` -> then
makes a `_search` api call for each of the partition.
2. The client also handles the logic for `max_query_range` for the
stream.
3. Handle the logic for dashboards loading from right to left
irrespective of `order_by` but honors the result by `order_by`
## New Behavior:
1. Client makes a web socket request with a request_id (uuid). This
req_id is stored in-memory cache along with its session object.
2. Server Search
- STEP 1: Search result cache
- STEP 1(a): Handle the cache responses & query the deltas
- STEP 2: Search without cache if no caches found
- STEP 3: Write to result cache
6. Server will now handle querying the partitions and then calling
search for each.
7. The results are streamed as in when the responses are received.
8. Server will handle the logic for dashboards loading from right to
left irrespective of `order_by` but honors the result by `order_by`
## Modified Code:
1. The `search` service will now use `write_results_v2` when writing to
cache.
2. The main difference between `write_results` and `write_results_v2` is
the strategy for deduplication. In `write_results_v2` we remove all the
hits within the same Date, Hour, Minute and Second; before writing to
cache.
```rust
/// # Caching Strategy
/// 1. **Select Hit for Deduplication**:
/// - Selects either the first or last record based on the `is_descending` flag.
/// - `is_descending = true`: Selects the last record.
/// - `is_descending = false`: Selects the first record.
///
/// 2. **Remove All Hits Within the Same Date, Hour, Minute, and Second for Deduplication**:
/// - Identifies the timestamp of the hit to remove (based on the `is_descending` flag, either
/// the first or the last record is chosen).
/// - Removes all hits that match the exact same date, hour, minute, and second
/// (`YYYY-MM-DDTHH:MM:SS`) as the identified hit.
/// - Example: If the timestamp to remove is `2024-12-06T04:15:30`, only hits with the exact
/// timestamp `2024-12-06T04:15:30` are removed, while others with different seconds (e.g.,
/// `2024-12-06T04:15:29` or `2024-12-06T04:15:31`) are retained.
```
## Web-socket Events:
1. Search
```sh
{
"type": "search",
"content": {
"trace_id": "d56755eb02cf47ecb3064ee096735c4a",
"payload": {
"query": {
"sql": "select histogram(_timestamp, '1 hour') AS zo_sql_key, count(*) AS zo_sql_num from \"olympics\" GROUP BY zo_sql_key ORDER BY zo_sql_key",
"start_time": 1733499294217000,
"end_time": 1734708894217000,
"size": -1,
"query_fn": "LmFiYywgZXJyID0gZ2V0X2VucmljaG1lbnRfdGFibGVfcmVjb3JkKCJvbHltcGljcyIsCnsKICAicmFuayI6IHRvX3N0cmluZyEoLnJhbmspCn0pCi5yYW5rX2tleXdvcmQgPSAuYWJjLmtleXdvcmQKLg..",
"sql_mode": "full"
}
},
"stream_type": "enrichment_tables",
"search_type": "ui",
"use_cache": true
}
}
```
2. Cancel
```sh
{
"type": "cancel",
"content": {
"trace_id": "b221bb019f3742e8bd93f817d530ef70"
}
}
```
Related Issues: #5173, #5188, #5266, #5320
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Introduced WebSocket search functionality with a toggle in settings.
- Added support for real-time data retrieval via WebSocket in various
components.
- Enhanced search operations with new methods for handling WebSocket
queries.
- Added new properties and methods to improve handling of WebSocket
sessions and requests.
- New interface for search request payloads and improvements in query
handling.
- Implemented WebSocket proxy handling for improved connection
management.
- Enhanced caching functionality with new methods for managing cached
responses.
- Introduced optional field for enabling WebSocket search in
organization settings.
- Added a new constant for handling partial error responses in search
operations.
- Added a new configuration option for enabling WebSocket support.
- Updated package version and various dependency versions to enhance
stability.
- **Bug Fixes**
- Improved error handling and logging for search requests and caching
mechanisms.
- Enhanced validation and URL construction for alerts.
- **Documentation**
- Updated comments and documentation for clarity on new features and
changes.
- **Style**
- Minor formatting adjustments for consistency across components and
services.
- **Chores**
- Updated dependency versions to enhance stability and performance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech>
Co-authored-by: hengfei yang <hengfei.yang@gmail.com>
#5253
Part 1 in a series of PRs to add new Alerts endpoints that support
parent folders and KSUIDs
This PR introduces HTTP layer models for the new Alerts endpoints
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced new dependencies to enhance database interactions and
unique identifier generation.
- Added a unique identifier field to the Alert structure.
- Established a comprehensive framework for managing alerts, including
new request and response structures for HTTP handling.
- Expanded OpenAPI documentation to include new alert-related
components.
- **Bug Fixes**
- Enhanced error handling for alert operations, providing more specific
error reporting.
- **Refactor**
- Updated function signatures to improve flexibility and modularity in
database interactions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This is the first PR for #5068 .
We need to move `dashboards` into the `config` crate so that DB access
functions in the `infra` crate can return types defined in the
`dashboards` module.
When the `dashboards` module is in the `openobserve` crate we cannot use
them in the `infra` crate since that would introduce a circular
dependency.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Introduced a new function to move dashboards, enhancing dashboard
management capabilities.
- Added a new `SendReport` trait for improved report handling and
modularity.
- Enhanced error handling across various dashboard and report management
functions.
- **Improvements**
- Updated import paths for better clarity and organization, particularly
for dashboard-related modules.
- Improved timestamp handling for alerts to ensure accurate tracking of
updates.
- Enhanced OpenAPI documentation to reflect updated schema references
for dashboards.
- Expanded development dependencies to support improved functionality.
- Enhanced the logic for handling report triggers based on frequency
types.
- **Bug Fixes**
- Refined error handling in dashboard creation and deletion functions to
provide clearer responses.
- Standardized error handling across report management functions for
consistency.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
#5069
**Description**
- Introduces migrations to
1. Create a new `folder` table
2. Transform folder records from `meta` table into `folder` table
3. Delete folder records from `meta` table
- Updates folder and dashboard service logic to use the new `folder`
table rather than the `meta` table
**Prior steps**
Before this PR is merged, two others must be merged first
1. https://github.com/openobserve/openobserve/pull/5095
2. https://github.com/openobserve/openobserve/pull/5096
After these other PRs are merged this PR will be updated to merge into
`main` rather than into `ps/5069/move-folder-2`.
**Next steps**
Feature #5069 will be complete with this PR.
This will open us up to starting on feature #5068.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced new folder management functionalities, including creating,
updating, listing, and deleting folders.
- Added support for error handling improvements in dashboard and folder
operations.
- Implemented a new README for SeaORM entities detailing usage and
generation instructions.
- **Bug Fixes**
- Enhanced error handling in various functions to ensure appropriate
responses based on success or failure.
- **Documentation**
- Added documentation for SeaORM entities and their generation process.
- **Refactor**
- Streamlined folder and dashboard management functions for better
clarity and maintainability.
- Updated API structure for folder-related endpoints to improve clarity
and response handling.
- Improved error handling strategy across multiple service functions.
- Enhanced database interaction methods for folders and dashboards,
promoting modularity and consistency.
- Modified initialization workflow to ensure database schema updates on
application start.
- Simplified error handling in folder management service functions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a garbage collection mechanism for the local cache,
enhancing resource management.
- **Improvements**
- Enhanced error handling during configuration initialization for better
debugging.
- Updated time calculations to improve accuracy and consistency.
- **Dependency Updates**
- Updated versions of key dependencies to leverage new features and
fixes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Subhra264andgreptile-apps[bot]
Ashish Kolheandloaki07
Hengfei YangandTaiming Liu
HuaijinandHengfei Yang
udev
Zhao.wei
Patrick Sullivan
