- [x] Enrichment tables are not listed correctly with RBAC.
- [x] Create ofga ownership for metadata streams
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
- **New Features**
- Enhanced handling of stream types for improved flexibility and
functionality.
- Introduced enterprise-level ownership management features for metadata
handling in the context of the enterprise feature.
- **Bug Fixes**
- Streamlined logic for retrieving model keys associated with stream
types, ensuring more accurate results.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Added `owner` and `enabled` URI query parameter for the
`{org_id}/alerts` and `{org}/{stream_name}/alerts` `GET` APIs. E.g. -
`{org}/alerts?owner=user@user.com&enabled=true`.
Fixes#4273
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new filtering capability for alerts, allowing users to
filter by owner and enabled status.
- Added a new structure, `AlertListFilter`, to manage alert filtering
criteria.
- **Improvements**
- Enhanced alert retrieval functions to support user-defined query
parameters for more granular control over alert listings, improving the
overall usability of the alert system.
- Expanded filtering logic in the AlertList component for a more
comprehensive search capability within the alert list.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
Fixes#4455
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced authorization checks for `query`, `query_range`, and `series`
functions to dynamically reflect stream types.
- Improved permission validation logic in the `search` and
`search_multi` functions, allowing for more accurate access control
based on user roles and stream types.
- Updated logic in the `get_latest_traces` function to refine the
construction of the `o2_type` field for better permission checks.
- **Bug Fixes**
- Corrected the mapping of stream types to model keys, enhancing
security and accuracy in permission checks.
- **Chores**
- Added necessary imports for `OFGA_MODELS` to support new permission
logic.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- Fixes#4394
- When cloning alert, don't copy `last_satisfied_at`,
`last_triggered_at`
When the alert query is invalid, the scheduler stops processing the
alert and returns, but the error message is not reported in the triggers
usage stream. This PR fixes that.
When the alert query is invalid, it will be retried again and again
(default maximum number of retries is `3`). After that, the associated
`scheduled_jobs` record will be automatically removed from the table and
hence the alert will not be triggered anymore.
In order to schedule the alert again, the user needs to update the query
and again save it, it will start scheduling again.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
- **New Features**
- Enhanced alert handling logic for improved robustness and clarity.
- Introduced structured handling of alert data with better error
management.
- Added initialization of `last_triggered_at` and `last_satisfied_at`
fields to ensure clear state management for alerts.
- **Bug Fixes**
- Improved error handling for alert evaluation, ensuring accurate status
updates for alert triggers.
- Enhanced state preservation for alerts by updating the
`last_satisfied_at` timestamp during saves.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
issue #4326
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced new structures for updating stream settings, allowing for
more granular management of stream configurations.
- Added an asynchronous function for updating stream settings via HTTP
PUT requests.
- Implemented a new service route for handling updates to stream
settings.
- Enhanced the `AddStream` component to retrieve updated settings.
- **Improvements**
- Improved schema management by tracking previous settings during
updates.
- Enhanced utility functions for comparing and managing stream settings.
- **Documentation**
- Updated documentation to reflect changes in API functionality for
creating and updating stream settings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
Fixes#4291
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new method for handling 409 Conflict responses in the
HTTP responses, improving error messaging clarity.
- **Bug Fixes**
- Updated error handling to return conflict responses instead of
forbidden responses when certain delete operations fail due to resource
conflicts.
- Enhanced clarity in user membership responses by changing the error
message when a user is already part of an organization.
These changes improve user experience by providing clearer and more
accurate feedback on error conditions related to resource management.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Addresses #4273
- Adds `last_triggered_at`, `owner`, `updated_at` and `last_edited_by`
fields (similar to Reports) to Alerts.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced alert and report management with additional metadata fields
for tracking last triggered times, owners, and edit history.
- Introduced user identification in report creation and updates for
better ownership tracking.
- **Bug Fixes**
- Improved logic for saving alerts and reports to accurately reflect
their statuses and metadata during creation and updates.
- **Documentation**
- Restructured import organization for improved clarity and
maintainability in the codebase.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new structure for enhanced management of cached query
responses, allowing for detailed handling of multiple cached results.
- Added functionality to retrieve multiple cached results in a single
request, optimizing performance and reducing service calls.
- Implemented a mechanism for efficient caching and retrieval of query
results in a distributed cluster environment.
- Enhanced configuration options for managing result caching behavior.
- **Bug Fixes**
- Improved handling of cache misses and response merging to ensure
robustness against empty responses.
- **Refactor**
- Updated internal logic to support new cached query response structure,
streamlining caching operations and enhancing clarity.
- Introduced new modules to improve code organization related to
multi-cluster operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Improved Period of derived stream to be same as frequency
Complete the `Frequency` task in the tracking issue #4170
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
- **New Features**
- Introduced a new component for managing scheduled alerts with a tabbed
interface for custom alerts, SQL queries, and PromQL queries.
- Enhanced user experience with dynamic input fields, real-time
validation, and aggregation settings for alert conditions.
- Added cron job configuration options for scheduling alerts.
- **Changes**
- Renamed the scheduled alert component to reflect the new
functionality, impacting how the application handles alert triggering
and scheduling.
- Adjusted timing and frequency parameters for triggering conditions in
the alert management system.
- **Improvements**
- Enhanced logging functionality for monitoring enrichment data saving,
providing insights into operations and parameters involved.
- Improved error handling and response validation processes for
ingestion tasks, ensuring more informative logging and error reporting.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
`DerivedStream`, as part of `Pipeline`, enables the use case of querying
an existing stream, `source`, with optional conditions and aggregations.
The resulted data will then be ingested into a new stream,
`destination`.
This PR provides the feature's initial offering. Please follow below
issue to track & suggest future improvements
#4170
Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: hengfei yang <hengfei.yang@gmail.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved caching behavior and response processing in search requests.
- Enhanced error handling and response generation for better user
experience.
- **Tests**
- Disabled specific UI interaction in Playwright tests to prevent test
failures.
- **Chores**
- Updated configuration defaults to improve clarity and efficiency.
- Made internal constants publicly accessible for broader usage.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- [x] Add two consistent hashing table for `Interactive` and
`Background` querier group
- [x] Only cache latest parquet files for `Interactive` queriers.
Categorizes nodes into role groups selected to serve different query
requests based on query search event type.
---
Each node can be configured into different role groups by using the new
env: `ZO_NODE_ROLE_GROUP`
- default: `` (empty: none)
- node for high-priority tasks: `interactive`
- node for low-priority tasks: `background`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced `node_role_group` configuration to specify role groups for
nodes.
- Added `SearchEventType` to `SearchRequest` to enhance search
capabilities.
- Implemented `RoleGroup` enum to categorize nodes based on priority.
- **Improvements**
- Updated node registration and status setting to include `role_group`.
- Enhanced logging to include node's role group in cluster operations.
- Streamlined search logic by assigning `search_type` earlier in the
request handling process.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added sorting functionality for search results based on an
`is_descending` flag.
- Enhanced logging to provide information about cache and result
lengths.
- **Improvements**
- Optimized internal calculations for search results cache.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Refactor**
- Improved tracing instrumentation across various search-related
functions by removing the `trace_id` field and refining how `trace_id`
is handled in logs.
- **Bug Fixes**
- Enhanced logging consistency and accuracy by standardizing trace ID
processing in `get_file_list`, `search`, `cluster_search`,
`search_parquet`, and `search_memtable` functions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- User roles are now dynamically assigned based on the `role` parameter
during user addition to an organization.
- **Enhancements**
- Improved role management by incorporating a feature flag for
enterprise-level role determination.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added new fields `is_descending` and `limit` to query responses for
enhanced sorting and pagination.
- **Improvements**
- Enhanced logging and tracing for better query time tracking and
debugging.
- Streamlined span handling for more efficient tracing and
instrumentation.
- **Bug Fixes**
- Fixed validation for `result_cache_enabled` configuration to prevent
cache-related errors.
- Improved accuracy in cache limit handling in query responses.
- **Chores**
- Updated Rust toolchain version for improved compatibility and
performance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved the request handling for job cancellation, ensuring a
smoother and more reliable user experience when canceling multiple jobs.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added support for canceling queries within a specific organization.
- Introduced a new RPC method, `ClusterCancelQuery`, for enhanced query
management.
- **Enhancements**
- Improved the `cancel_query` functionality to include organization ID
parameters for more targeted query management.
- Updated error handling and logging for better debugging and user
clarity.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced authentication and authorization handling for user requests
with more efficient token validation.
- **Bug Fixes**
- Improved error handling and response based on user permissions and
authorization status.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added `result_cache_ratio` to the search feature to improve response
management.
- Introduced `discard_interval` parameter to enhance cache handling.
- **Improvements**
- Enhanced cache retention and discarding logic with `discard_interval`
for better performance.
- Improved timestamp parsing and rounding functions for more accurate
data handling.
- **Refactor**
- Reorganized logging and utility functions within the search and cache
modules for better code maintainability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced trace ID management with functions for adding, removing,
and generating trace contexts.
- Added the ability to cancel running queries.
- **Enhancements**
- Integrated tracing functionality into search operations for improved
observability.
- Updated CORS configuration to allow `traceparent` headers.
- **Debugging**
- Added console logs for traceparent parameters in search functions.
- **Localization**
- Added new localization key for "cancel" message in English.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Subhra264
Yashodhan Joshi
Taiming Liu
Hengfei Yang
Huaijin
Ashish Kolhe
胡同里有猫
baerwang
Ankur Srivastava