This pr aligns the next_run_at of the alerts/pipelines according to the
timezone of the given alert/pipeline. For example, if the current time
is 11:03pm, and the frequency is 5 mins, the next run at should be
11:05pm, then 11:10pm, and so on. In future, we may want to make
aligntime an optional feature, by giving a toggle option in the
alert/pipeline itself.
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
## What is changed
- [x] Try to improve task starving issue
- [x] Update mmdb frequency
- [x] Improve slow log detection
## MMDB
- Removed ENV `ZO_MMDB_UPDATE_DURATION` because is in seconds
- Add new ENV `ZO_MMDB_UPDATE_DURATION_DAYS` is in days
## Tokio task
We try to use `task::coop` to improve task starving issue in Tokio.
<img width="1496" alt="image"
src="https://github.com/user-attachments/assets/1666488a-146b-49d7-b460-c629528e06f9"
/>
The second chart is for this PR, it is a big slow but more stable.
This PR has some improves:
- Improve tantivy generate, from 7.7s -> 5.9s
- Improve STREAM_SETTINGS cache with atomic
- Add process metrics
- Remove zo_node_open_fd, node_tcp_conn_resets( use
`node_tcp_connections{state="resets"}`)
- Add debug logs for bulk api
- Add debug logs for tantivy panic
At times same function is used different pipelines , the function would
like to add stream name to current row, added `org_id` and `stream_name`
It can be used in vrl with below syntax:
`.stream_name_value = %stream_name`
Insert timestamp into json record before pipeline processing, to avoid
inconsistent timestamps among different pipeline destinations.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved timestamp handling across multiple service modules.
- Enhanced error handling for timestamp parsing and processing.
- Refined timestamp extraction and validation in log and metrics
ingestion.
- **Refactor**
- Streamlined timestamp management in pipeline and trace processing.
- Simplified error reporting mechanisms for timestamp-related issues.
- Removed timestamp tracking functionality in the pipeline execution
struct.
- **Chores**
- Updated code to improve robustness of timestamp handling.
- Consolidated timestamp handling logic for clarity and efficiency.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This adds support of cipher keys, along with corresponding udfs for
encrypting and decrypting fields from the log.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes: Enterprise Encryption and Key Management Features
- **New Features**
- Added enterprise-level cipher key management functionality.
- Introduced encryption and decryption capabilities for data protection.
- Implemented key storage and retrieval mechanisms.
- New Vue components for adding and managing cipher keys and encryption
mechanisms.
- Added a new settings page for managing cipher keys.
- Integrated advanced key management with multiple provider types.
- Added user-defined functions for encryption and decryption within
DataFusion.
- Introduced a new migration for creating the `cipher_keys` database
table.
- Enhanced HTTP request handling for cipher key operations.
- Added validation for encryption configurations and improved error
handling.
- New asynchronous tasks for managing cipher keys during initialization.
- Enhanced error handling and notifications for cipher key changes.
- **Security Enhancements**
- Added support for managing encryption keys across the platform.
- Implemented secure key storage and rotation mechanisms.
- Enhanced migration logic for cipher keys.
- **User Interface Updates**
- Enhanced user navigation with new tabs for cipher key management.
- Introduced stepper interface for key creation and management.
- Improved localization support for cipher key management.
- **Performance Improvements**
- Optimized query handling for encrypted data.
- Enhanced DataFusion query optimizer with cipher key support.
- **Compatibility**
- Enterprise feature flag controls advanced encryption capabilities.
- Supports multiple key storage and authentication types.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
Co-authored-by: Subhra264 <chakrabortysubhradeep556@gmail.com>
Co-authored-by: ShyamOOAI <shyam@openobserve.ai>
### Summary
This PR updates the logic that is used to generate KSUID primary keys
for the `folders`, `dashboards`, and `alerts` tables in existing
migrations which transform data from the `meta` table into the
`folders`, `dashboards`, and `alerts` tables.
The new logic generates 160-bit "KSUIDs" which are actually 160-bit
SHA-1 hashes of the records being transformed rather than being truly
random KSUIDs.
### Motivation
In the existing implementation, the migrations generate random KSUIDs
for the records that are transformed from the `meta` table and into the
different `folders`, `dashboards`, and `alerts` tables. When running the
application in a single cluster this is acceptable behavior.
However when running the application in a multi-cluster environment this
causes problems. In a multi-cluster environment, each cluster will have
it's own copy of the application database, and each database should
contain the exact same data in the `meta` table. When we run the
migration scripts in each cluster, the data from the `meta` table will
be transformed into the `folders`, `dashboards`, and `alerts` tables.
However if the migration generates a random KSUID for each new record
inserted into those new tables, then different clusters will each
generate different KSUID `id`s for rows in the `folders`, `dashboards`,
and `alerts` tables which should actually have the same `id`s.
Our solution to this problem is that during the migration from the
`meta` table to the `folders`, `dashboards`, and `alerts` tables, rather
than generating random 160-bit KSUID `id`s for the new records, we
generate 160-bit SHA-1 hashes for each record and we use that SHA-1 hash
as the `id` of the each new record. This ensure that each record that is
transformed from the `meta` table will have an `id` in the `folders`,
`alerts`, or `dashboards` tables that is consistent across all clusters.
Newly generated records (ie, records generated AFTER the migration) in
the `folders`, `alerts`, and `dashboards` tables will continue to be
generated with random KSUID `id`s since these `id`s can be shared with
other clusters via NATS super-cluster events and then records with those
`id`s can be inserted into databases in other clusters, ensuring that
records have consistent `id`s across clusters.
### Notes about KSUID generation
To generate a KSUID this function generates the 160-bit SHA-1 hash of
the alert's `org`, `stream_type`, `stream_name`, and `name` and
interprets that 160-bit hash as a 160-bit KSUID. Therefore two KSUIDs
generated in this manner will always be equal if the alerts have the
same `org`, `stream_type`, `stream_name`, and `name`.
⚠️ It is important to note that although KSUIDs generated in this manner
can be parsed as KSUIDs since they are 160-bit values, the KSUIDs
generated in this manner will have timestamp bits which are effectively
random, meaning that the timestamp in any KSUID generated with this
function will be random. This will render the timestamp-sortability
property of these KSUIDs useless. This probably isn't a big deal since
we can always add and sort by a `created_at` column if we want to, but
we should be aware of this limitation.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added SHA-1 hashing dependency to generate unique identifiers (KSUIDs)
for alerts, folders, and dashboards.
- Implemented KSUID generation based on specific entity attributes.
- **Refactor**
- Transitioned database tables from integer-based to KSUID-based primary
keys.
- Updated `StreamType` enum to support more flexible usage.
- **Chores**
- Updated project dependencies to include SHA-1 hashing library.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Enhanced folder management with improved unique identifier generation.
- Added enterprise-specific folder processing for create, update, and
delete operations.
- Introduced new asynchronous functions for processing folder-related
messages.
- Added optional ID parameter for alert creation, enhancing flexibility.
- **Changes**
- Updated folder-related function signatures to return more
comprehensive information.
- Integrated enterprise-level folder message processing in super cluster
queue.
- Relocated and centralized message definitions for alerts and
dashboards.
- **Improvements**
- Refined error handling for folder operations.
- Centralized message handling for folder-related events.
- Improved clarity and maintainability in dashboard moving logic.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
# Objectives:
1. Stream results to the client using web-socket.
2. Cache results for a query as a single result cache.
## New Endpoint:
```sh
/api/{org_id}/ws/{req_id}
```
## WS Design:
1. The request_id is used to handle the sessions `pub static
WS_SESSIONS: Lazy<RwHashMap<String, WsSession>> =
Lazy::new(DashMap::default);`.
2. The request_id is removed from the session cache and the web-socket
is disconnected once single search is complete. (e.g. for 5 dashboards
in a single page, 5 individual requests are made). The design choice is
discussed [here](https://github.com/openobserve/walmart1/issues/207).
3. The supported events and responses the web-socket are described
below.
4. Auditing for web-socket linked
[here](https://github.com/openobserve/o2-enterprise/pull/248)
5. Org level setting to enable and disable web-socket communication.
(Default: disabled)
## New Env:
```sh
ZO_WEBSOCKET_ENABLED=true
```
## Env to be enabled:
```sh
ZO_USE_MULTIPLE_RESULT_CACHE=true
ZO_RESULT_CACHE_ENABLED=true
```
- If `ZO_RESULT_CACHE_ENABLED` is set to `false` which is default.
Results cache will not be written.
## Nginx config:
```yaml
nginx.ingress.kubernetes.io/proxy-http-version: "1.1" # Enable HTTP/1.1 for WebSockets
nginx.ingress.kubernetes.io/enable-websocket: "true"
nginx.ingress.kubernetes.io/proxy-set-headers: |
Upgrade $http_upgrade;
Connection "Upgrade";
```
## Note: Disable `keep-alive` settings if present
```yaml
# nginx.ingress.kubernetes.io/connection-proxy-header: keep-alive
```
## Client Behavior before this pr:
1. For a single query, client calls the `_search_partition` api which
returns the list of partitions `Vec<[start_time, end_time]>` -> then
makes a `_search` api call for each of the partition.
2. The client also handles the logic for `max_query_range` for the
stream.
3. Handle the logic for dashboards loading from right to left
irrespective of `order_by` but honors the result by `order_by`
## New Behavior:
1. Client makes a web socket request with a request_id (uuid). This
req_id is stored in-memory cache along with its session object.
2. Server Search
- STEP 1: Search result cache
- STEP 1(a): Handle the cache responses & query the deltas
- STEP 2: Search without cache if no caches found
- STEP 3: Write to result cache
6. Server will now handle querying the partitions and then calling
search for each.
7. The results are streamed as in when the responses are received.
8. Server will handle the logic for dashboards loading from right to
left irrespective of `order_by` but honors the result by `order_by`
## Modified Code:
1. The `search` service will now use `write_results_v2` when writing to
cache.
2. The main difference between `write_results` and `write_results_v2` is
the strategy for deduplication. In `write_results_v2` we remove all the
hits within the same Date, Hour, Minute and Second; before writing to
cache.
```rust
/// # Caching Strategy
/// 1. **Select Hit for Deduplication**:
/// - Selects either the first or last record based on the `is_descending` flag.
/// - `is_descending = true`: Selects the last record.
/// - `is_descending = false`: Selects the first record.
///
/// 2. **Remove All Hits Within the Same Date, Hour, Minute, and Second for Deduplication**:
/// - Identifies the timestamp of the hit to remove (based on the `is_descending` flag, either
/// the first or the last record is chosen).
/// - Removes all hits that match the exact same date, hour, minute, and second
/// (`YYYY-MM-DDTHH:MM:SS`) as the identified hit.
/// - Example: If the timestamp to remove is `2024-12-06T04:15:30`, only hits with the exact
/// timestamp `2024-12-06T04:15:30` are removed, while others with different seconds (e.g.,
/// `2024-12-06T04:15:29` or `2024-12-06T04:15:31`) are retained.
```
## Web-socket Events:
1. Search
```sh
{
"type": "search",
"content": {
"trace_id": "d56755eb02cf47ecb3064ee096735c4a",
"payload": {
"query": {
"sql": "select histogram(_timestamp, '1 hour') AS zo_sql_key, count(*) AS zo_sql_num from \"olympics\" GROUP BY zo_sql_key ORDER BY zo_sql_key",
"start_time": 1733499294217000,
"end_time": 1734708894217000,
"size": -1,
"query_fn": "LmFiYywgZXJyID0gZ2V0X2VucmljaG1lbnRfdGFibGVfcmVjb3JkKCJvbHltcGljcyIsCnsKICAicmFuayI6IHRvX3N0cmluZyEoLnJhbmspCn0pCi5yYW5rX2tleXdvcmQgPSAuYWJjLmtleXdvcmQKLg..",
"sql_mode": "full"
}
},
"stream_type": "enrichment_tables",
"search_type": "ui",
"use_cache": true
}
}
```
2. Cancel
```sh
{
"type": "cancel",
"content": {
"trace_id": "b221bb019f3742e8bd93f817d530ef70"
}
}
```
Related Issues: #5173, #5188, #5266, #5320
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Introduced WebSocket search functionality with a toggle in settings.
- Added support for real-time data retrieval via WebSocket in various
components.
- Enhanced search operations with new methods for handling WebSocket
queries.
- Added new properties and methods to improve handling of WebSocket
sessions and requests.
- New interface for search request payloads and improvements in query
handling.
- Implemented WebSocket proxy handling for improved connection
management.
- Enhanced caching functionality with new methods for managing cached
responses.
- Introduced optional field for enabling WebSocket search in
organization settings.
- Added a new constant for handling partial error responses in search
operations.
- Added a new configuration option for enabling WebSocket support.
- Updated package version and various dependency versions to enhance
stability.
- **Bug Fixes**
- Improved error handling and logging for search requests and caching
mechanisms.
- Enhanced validation and URL construction for alerts.
- **Documentation**
- Updated comments and documentation for clarity on new features and
changes.
- **Style**
- Minor formatting adjustments for consistency across components and
services.
- **Chores**
- Updated dependency versions to enhance stability and performance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech>
Co-authored-by: hengfei yang <hengfei.yang@gmail.com>
#5253
Part 1 in a series of PRs to add new Alerts endpoints that support
parent folders and KSUIDs
This PR introduces HTTP layer models for the new Alerts endpoints
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced new dependencies to enhance database interactions and
unique identifier generation.
- Added a unique identifier field to the Alert structure.
- Established a comprehensive framework for managing alerts, including
new request and response structures for HTTP handling.
- Expanded OpenAPI documentation to include new alert-related
components.
- **Bug Fixes**
- Enhanced error handling for alert operations, providing more specific
error reporting.
- **Refactor**
- Updated function signatures to improve flexibility and modularity in
database interactions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced support for an "Enterprise" database type in the Geoip
functionality.
- Added new utilities for file downloads and checksum verification.
- Enhanced downloader logic to support enterprise features.
- **Bug Fixes**
- Improved error handling for file downloads and checksum checks.
- **Chores**
- Added a new dependency for cryptographic hashing.
- **Documentation**
- Updated module structure to include new download utilities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#5253
The overall goal of this PR is to migrate alerts data out of `meta`
table into `alerts` table. To accomplish this three main changes are
made. Unfortunately, all three of these changes need to be synchronized
and submitted in the same PR.
1. A new migration script is added to
- Parse alerts JSON currently stored in the `meta` table into Rust
models for alerts. These Rust models are isolated to the migration
script since they represent the schema of alerts as they are stored in
the `meta` table before this migration executes.
- Convert those Rust models into Sea ORM entities for alerts.
- Insert all the Sea ORM entities for alerts into the new `alerts`
table.
2. Create get/put/delete/list methods for accessing the data in the
`alerts` table
- These functions use SeaORM to query and update the database.
- These functions must also convert between SeaORM alert entities and
the application's current data model for alerts (as defined in
`config::meta::alerts`.
- It's entirely expected that the models inside `config::meta::alerts`
will change over time. When this happens in the future, developers will
need to update the transformation functions between SeaORM alert
entities and the models in `config::meta::alerts`. Luckily, the compiler
should help to direct developers to where these transformation functions
need to be updated.
3. Update the old logic in `/src/service/db/alerts.rs` that previously
read and wrote alerts to the `meta` table so that it now reads and
writes alerts to the `alerts` table.
- The methods for accessing the `meta` table included logic for
triggering "cache update events" to the cluster coordinator and for
sending events to the super cluster queue. Since the new logic accesses
the `alerts` table rather than the `meta` table, this logic for emitting
events had to be reproduced.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new `ListAlertsParams` struct for flexible querying of
alerts.
- Enhanced `TriggerCondition` struct with new timing parameters.
- Added new error variants for improved error handling in alert
management.
- Created a comprehensive alert management system with CRUD operations.
- Added integration tests for various functionalities related to alerts
and dashboards.
- Introduced a new `Alerts` variant in the `FolderType` enum for better
folder management.
- **Bug Fixes**
- Improved error handling for folder deletion when alerts are present.
- **Documentation**
- Updated documentation for several functions to clarify their scope and
functionality.
- **Chores**
- Added new dependencies to support additional functionality.
- Introduced migration scripts for database schema changes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The previous mechanism for generating hashes for dashboard models
depended on hashing the JSON representation of the dashboard that was
stored in the `meta` table.
As part of the migration of dashboards out of the `meta` table and into
a dedicated `dashboards` table, this ☝️ presented some problems:
Specifically when we want to translate from the Dashboard ORM model to a
`core::meta::dashboards::Dashboard` model, we need to generate the hash
at the time of translation since `hash` is a field on
`core::meta::dashboards::Dashboard`. We can't simply generate a hash
from the JSON stored in the `dashboards.data` column because this JSON
no longer contains fields for `title`, `owner`, `description`, and
`role`. Those fields are now stored in DB columns, and changes to those
columns in the DB columns would *not* result in changes to the hash on
the JSON blob.
Therefore, rather than hashing the JSON blob, this PR implements hash
functions on the Rust types for the dashboards. Therefore changes to
fields in those Rust structs will result in changes to the hash values,
but changes to the way in which the data is stored in the DB will not
affect the hash values.
⚠️ Note that this PR is changing the hash function on the dashboard
models, so hashes generated for the same model before and after this
deployment will not be the same. My understanding is that this shouldn't
be an issue since hashes are just used as a way of checking on the UI if
the dashboard model has been updated to avoid overwriting the changes of
another user. I believe that this change to the dashboard function might
just have the effect (around the time of deployment) where it would
appear to the UI that the dashboard has been updated when, in fact, it
has not.
https://github.com/openobserve/o2-enterprise/actions/runs/11979723563
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new type alias `OrdF64` for enhanced precision in
floating-point operations.
- Added the `Hash` trait to multiple dashboard-related structs and
enums, enabling their use in hash-based collections.
- **Bug Fixes**
- Improved error handling and assertion clarity in API testing for
folder and dashboard operations.
- **Chores**
- Standardized hashing mechanism for dashboard objects in database
operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This is the first PR for #5068 .
We need to move `dashboards` into the `config` crate so that DB access
functions in the `infra` crate can return types defined in the
`dashboards` module.
When the `dashboards` module is in the `openobserve` crate we cannot use
them in the `infra` crate since that would introduce a circular
dependency.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Introduced a new function to move dashboards, enhancing dashboard
management capabilities.
- Added a new `SendReport` trait for improved report handling and
modularity.
- Enhanced error handling across various dashboard and report management
functions.
- **Improvements**
- Updated import paths for better clarity and organization, particularly
for dashboard-related modules.
- Improved timestamp handling for alerts to ensure accurate tracking of
updates.
- Enhanced OpenAPI documentation to reflect updated schema references
for dashboards.
- Expanded development dependencies to support improved functionality.
- Enhanced the logic for handling report triggers based on frequency
types.
- **Bug Fixes**
- Refined error handling in dashboard creation and deletion functions to
provide clearer responses.
- Standardized error handling across report management functions for
consistency.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Subhra264andgreptile-apps[bot]
Hengfei Yang
Bhargav
dependabot[bot]
Ashish Kolhe
HuaijinandHengfei Yang
Zhao.wei
Patrick Sullivan
