- [x] for search job, in `total` field, will return the total records
numbers.
- [x] if not set the from and size when get result, we set default from
= 0 and size = 100
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Security Enhancement**
- Added validation to prevent service accounts from logging in
- Implemented additional authentication checks during user login process
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Enhanced error handling for dashboard movement operations
- Improved error reporting when deleting dashboards from old folders
during move operations
- **New Features**
- Added more detailed error tracking for dashboard-related actions
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added a new configuration option for minimum auto-refresh interval.
- Expanded configuration response to include minimum auto-refresh
interval setting.
- **Improvements**
- Enhanced configuration capabilities for controlling automatic refresh
frequency.
- Corrected type for result cache discard duration to ensure it is
treated as a numeric value.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
for debug the #5398
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **Configuration**
- Added new configuration option to disable keep-alive connections
- Adjusted HTTP server connection timeout settings
- **Authentication**
- Enhanced token validation with improved traceability
- Updated authentication methods to support more detailed request
tracking
- **Performance**
- Optimized client lock management in MaxMind database downloader
- Refined request handling in RUM ingestion APIs
- **Logging**
- Improved logging for authentication and request processing
- Added trace ID support for better request tracking
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: domyway <weizhao029@gmail.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Added a new function testing capability for VRL (Variable Runtime
Language) functions.
- Introduced interactive UI components for testing and managing
functions, including `FunctionsToolbar`, `TestFunction`, and
`FullViewContainer`.
- New API endpoint for testing VRL functions against a set of events.
- Added new localization strings for user interface elements.
- **Improvements**
- Refined UI components for better user experience, including sidebar
management and editor functionalities.
- Enhanced function management with improved error handling and
validation.
- Improved code structure and maintainability across multiple
components.
- **Bug Fixes**
- Addressed various minor formatting and syntax issues.
- Improved error handling in function testing and management.
- **Enhancements**
- Updated the `QueryEditor` component for improved formatting and error
diagnostics.
- Enhanced Playwright tests for better targeting and reliability in
function management scenarios.
- Improved layout and responsiveness of several components, including
`AssociateFunction` and `ScheduledAlert`.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: udev <uddhavdave1@gmail.com>
Co-authored-by: udev <uddhav@openobserve.ai>
Try to match on the v2 alert endpoints in the router **before** the v1
endpoints.
If we try to match on the v1 endpoints first then a call to a v2
endpoint such as
`GET: /api/v2/my-org/alerts`
will actually match with the v1 endpoint
`GET: /api/{org_id}/alerts`
where `v2` will be bound to `{org_id}`.
If we swap the matching order then requests to v2 endpoints will be
correctly routed to v2 endpoints and requests to v1 endpoints will be
correctly routed to v1 endpoints.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Refactor**
- Reorganized alert-related service routes in the application
- Removed and re-added existing alert service routes without changing
their core functionality
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced alert listing query parameters with optional stream type and
name
- Made page size and index optional in alert queries
- **Refactor**
- Simplified alert query structure by consolidating stream and
pagination parameters
- Marked certain alert fields as read-only in schema generation
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced enterprise-level access control for user authentication.
- Added stricter login requirements for non-root users when
`root_only_login` is enabled.
- **Bug Fixes**
- Improved authentication validation process to prevent unauthorized
access.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
# Objectives:
1. Stream results to the client using web-socket.
2. Cache results for a query as a single result cache.
## New Endpoint:
```sh
/api/{org_id}/ws/{req_id}
```
## WS Design:
1. The request_id is used to handle the sessions `pub static
WS_SESSIONS: Lazy<RwHashMap<String, WsSession>> =
Lazy::new(DashMap::default);`.
2. The request_id is removed from the session cache and the web-socket
is disconnected once single search is complete. (e.g. for 5 dashboards
in a single page, 5 individual requests are made). The design choice is
discussed [here](https://github.com/openobserve/walmart1/issues/207).
3. The supported events and responses the web-socket are described
below.
4. Auditing for web-socket linked
[here](https://github.com/openobserve/o2-enterprise/pull/248)
5. Org level setting to enable and disable web-socket communication.
(Default: disabled)
## New Env:
```sh
ZO_WEBSOCKET_ENABLED=true
```
## Env to be enabled:
```sh
ZO_USE_MULTIPLE_RESULT_CACHE=true
ZO_RESULT_CACHE_ENABLED=true
```
- If `ZO_RESULT_CACHE_ENABLED` is set to `false` which is default.
Results cache will not be written.
## Nginx config:
```yaml
nginx.ingress.kubernetes.io/proxy-http-version: "1.1" # Enable HTTP/1.1 for WebSockets
nginx.ingress.kubernetes.io/enable-websocket: "true"
nginx.ingress.kubernetes.io/proxy-set-headers: |
Upgrade $http_upgrade;
Connection "Upgrade";
```
## Note: Disable `keep-alive` settings if present
```yaml
# nginx.ingress.kubernetes.io/connection-proxy-header: keep-alive
```
## Client Behavior before this pr:
1. For a single query, client calls the `_search_partition` api which
returns the list of partitions `Vec<[start_time, end_time]>` -> then
makes a `_search` api call for each of the partition.
2. The client also handles the logic for `max_query_range` for the
stream.
3. Handle the logic for dashboards loading from right to left
irrespective of `order_by` but honors the result by `order_by`
## New Behavior:
1. Client makes a web socket request with a request_id (uuid). This
req_id is stored in-memory cache along with its session object.
2. Server Search
- STEP 1: Search result cache
- STEP 1(a): Handle the cache responses & query the deltas
- STEP 2: Search without cache if no caches found
- STEP 3: Write to result cache
6. Server will now handle querying the partitions and then calling
search for each.
7. The results are streamed as in when the responses are received.
8. Server will handle the logic for dashboards loading from right to
left irrespective of `order_by` but honors the result by `order_by`
## Modified Code:
1. The `search` service will now use `write_results_v2` when writing to
cache.
2. The main difference between `write_results` and `write_results_v2` is
the strategy for deduplication. In `write_results_v2` we remove all the
hits within the same Date, Hour, Minute and Second; before writing to
cache.
```rust
/// # Caching Strategy
/// 1. **Select Hit for Deduplication**:
/// - Selects either the first or last record based on the `is_descending` flag.
/// - `is_descending = true`: Selects the last record.
/// - `is_descending = false`: Selects the first record.
///
/// 2. **Remove All Hits Within the Same Date, Hour, Minute, and Second for Deduplication**:
/// - Identifies the timestamp of the hit to remove (based on the `is_descending` flag, either
/// the first or the last record is chosen).
/// - Removes all hits that match the exact same date, hour, minute, and second
/// (`YYYY-MM-DDTHH:MM:SS`) as the identified hit.
/// - Example: If the timestamp to remove is `2024-12-06T04:15:30`, only hits with the exact
/// timestamp `2024-12-06T04:15:30` are removed, while others with different seconds (e.g.,
/// `2024-12-06T04:15:29` or `2024-12-06T04:15:31`) are retained.
```
## Web-socket Events:
1. Search
```sh
{
"type": "search",
"content": {
"trace_id": "d56755eb02cf47ecb3064ee096735c4a",
"payload": {
"query": {
"sql": "select histogram(_timestamp, '1 hour') AS zo_sql_key, count(*) AS zo_sql_num from \"olympics\" GROUP BY zo_sql_key ORDER BY zo_sql_key",
"start_time": 1733499294217000,
"end_time": 1734708894217000,
"size": -1,
"query_fn": "LmFiYywgZXJyID0gZ2V0X2VucmljaG1lbnRfdGFibGVfcmVjb3JkKCJvbHltcGljcyIsCnsKICAicmFuayI6IHRvX3N0cmluZyEoLnJhbmspCn0pCi5yYW5rX2tleXdvcmQgPSAuYWJjLmtleXdvcmQKLg..",
"sql_mode": "full"
}
},
"stream_type": "enrichment_tables",
"search_type": "ui",
"use_cache": true
}
}
```
2. Cancel
```sh
{
"type": "cancel",
"content": {
"trace_id": "b221bb019f3742e8bd93f817d530ef70"
}
}
```
Related Issues: #5173, #5188, #5266, #5320
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Introduced WebSocket search functionality with a toggle in settings.
- Added support for real-time data retrieval via WebSocket in various
components.
- Enhanced search operations with new methods for handling WebSocket
queries.
- Added new properties and methods to improve handling of WebSocket
sessions and requests.
- New interface for search request payloads and improvements in query
handling.
- Implemented WebSocket proxy handling for improved connection
management.
- Enhanced caching functionality with new methods for managing cached
responses.
- Introduced optional field for enabling WebSocket search in
organization settings.
- Added a new constant for handling partial error responses in search
operations.
- Added a new configuration option for enabling WebSocket support.
- Updated package version and various dependency versions to enhance
stability.
- **Bug Fixes**
- Improved error handling and logging for search requests and caching
mechanisms.
- Enhanced validation and URL construction for alerts.
- **Documentation**
- Updated comments and documentation for clarity on new features and
changes.
- **Style**
- Minor formatting adjustments for consistency across components and
services.
- **Chores**
- Updated dependency versions to enhance stability and performance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech>
Co-authored-by: hengfei yang <hengfei.yang@gmail.com>
#5253
Miscellaneous cleanup tasks for the new alerts v2 endpoints:
- fixes some of the new route paths
- removes utoipa schema attribute from a Unix timestamp field on the
HTTP model
- removes unnecessary call to DB in "delete" operation
#5253
Part 8 in a series of PRs to add new Alerts endpoints that support
parent folders and KSUIDs
This PR introduces the new `List` endpoint which returns the parent
folder information for each folder in the response
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new endpoint for listing alerts, enhancing alert
management capabilities.
- Added pagination support for alert listing through
`ListAlertsQueryPageParams`.
- New functionality to retrieve alerts along with associated folders.
- **Bug Fixes**
- Improved error handling for alert management operations.
- **Refactor**
- Enhanced data structures for alert parameters and queries.
- Updated existing functions for better clarity and modularity in alert
management.
- **Documentation**
- Updated OpenAPI documentation to include the new alert listing
endpoint.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#5253
Part 7 in a series of PRs to add new Alerts endpoints that support
parent folders and KSUIDs
This PR introduces the new `DeleteAlert`, `TriggerAlert`, and
`EnableAlert` endpoints which identify the alert by its KSUID
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced new API endpoints for managing alerts: delete, enable, and
trigger alerts.
- Enhanced alert management capabilities with comprehensive CRUD
operations for alerts.
- **Bug Fixes**
- Improved error handling for alert operations, ensuring better
management of alert states.
- **Documentation**
- Updated API documentation to reflect new functionalities for alert
management.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#5253
Part 6 in a series of PRs to add new Alerts endpoints that support
parent folders and KSUIDs
This PR introduces the new `UpdateAlert` endpoint which identifies the
alert to update by its KSUID
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new HTTP endpoint for updating alerts.
- Added functionality to update existing alerts in the database.
- Enhanced OpenAPI documentation with the new alert update path.
- **Bug Fixes**
- Improved error handling for folder-related operations during alert
updates.
- **Refactor**
- Renamed several alert management functions for clarity and
consistency.
- Updated existing functions to streamline alert creation and updating
processes.
- **Documentation**
- Updated API documentation to reflect new alert management
capabilities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#5253
Part 5 in a series of PRs to add new Alerts endpoints that support
parent folders and KSUIDs
This PR introduces the new `CreateAlert` endpoint which takes a
`folder_id` parameter to specify the parent folder of the new alert
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new HTTP POST endpoint for creating alerts.
- Added functionality for creating alerts in the database.
- Enhanced API documentation to include the new alert creation endpoint.
- **Bug Fixes**
- Improved error handling for alert creation, including a specific error
for non-existent folders.
- **Chores**
- Updated import statements to include necessary types for new
functionalities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#5253
Part 4 in a series of PRs to add new Alerts endpoints that support
parent folders and KSUIDs
This PR introduces the new Get endpoint for getting an alert by its
KSUID ID
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new endpoint for retrieving alerts by ID.
- Added functionality to fetch alerts by ID in the alert management
system.
- **Bug Fixes**
- Improved error handling for alert retrieval and management.
- **Documentation**
- Updated OpenAPI documentation to include the new alert retrieval
endpoint.
- **Refactor**
- Renamed existing functions for clarity and consistency in alert
management.
- **Chores**
- Added necessary imports to support new functionalities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#5253
Part 3 in a series of PRs to add new Alerts endpoints that support
parent folders and KSUIDs
This PR deprecates old Alerts endpoints
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new namespace for alert services, marking existing routes
as deprecated.
- **Bug Fixes**
- Updated alert-related functions to reflect changes in their underlying
implementations while maintaining functionality.
- **Documentation**
- Revised OpenAPI documentation to reflect the transition to deprecated
alert handlers.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#5253
Part 2 in a series of PRs to add new Alerts endpoints that support
parent folders and KSUIDs
This PR just renames various methods
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced new alert management structures and request/response bodies
for creating, updating, and listing alerts.
- Added unique identifier support for alerts.
- **Bug Fixes**
- Enhanced error handling for alert creation and updates.
- **Documentation**
- Updated OpenAPI documentation to include new alert-related request and
response types.
- **Refactor**
- Renamed functions for clarity in alert management, including retrieval
and deletion by name.
- **Tests**
- Updated integration tests to reflect changes in alert deletion
methods.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#5253
Part 1 in a series of PRs to add new Alerts endpoints that support
parent folders and KSUIDs
This PR introduces HTTP layer models for the new Alerts endpoints
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced new dependencies to enhance database interactions and
unique identifier generation.
- Added a unique identifier field to the Alert structure.
- Established a comprehensive framework for managing alerts, including
new request and response structures for HTTP handling.
- Expanded OpenAPI documentation to include new alert-related
components.
- **Bug Fixes**
- Enhanced error handling for alert operations, providing more specific
error reporting.
- **Refactor**
- Updated function signatures to improve flexibility and modularity in
database interactions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Introduces domain-specific errors for alerts in the service layer. These
errors are then mapped to HTTP responses in the HTTP layer.
This PR will be followed up by another PR to introduce new alerts
endpoints for accessing alerts by their new KSUIDs and for accessing
information about each alert's parent folder.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new `AlertError` enum for improved error handling in
alert operations.
- Enhanced error handling for alert management functions, allowing for
more specific error responses.
- **Bug Fixes**
- Streamlined return types for alert management functions to enhance
clarity and reduce complexity in error handling.
- **Documentation**
- Updated method signatures to reflect changes in error handling and
return types across various alert management functions.
- **Chores**
- Refined error reporting mechanisms in multiple modules for consistency
in error handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
impl #4281
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Enhanced organizational metrics with total counts for functions and
dashboards.
- Added new utility functions for formatting numerical data.
- **Bug Fixes**
- Improved display logic for stream and document counts in the user
interface.
- **Documentation**
- Updated localization keys for clarity and added new summary labels.
- **Refactor**
- Restructured summary data generation for better clarity and
organization.
- **Style**
- Increased width of certain layout elements for improved visibility.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: hengfei yang <hengfei.yang@gmail.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added labels for "Index Size", "Docs Count", "Function", "Pipeline",
"Pipelines & Functions", "Alerts", and "Dashboards" in the home section.
- Enhanced the home view to display additional metrics: Index Size,
Document Count, Dashboard Count, and Pipelines alongside existing data.
- Introduced new summaries for pipelines, alerts, and dashboards in
organizational data.
- **Bug Fixes**
- Improved layout and styling for better data presentation in the home
view.
- Updated validation logic for the home page to reflect new label
changes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
related to #5388
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Introduced a gRPC search method for enhanced search capabilities.
- Added support for multi-query requests with the new
`MultiStreamRequest` struct.
- **Improvements**
- Enhanced error handling and tracing across various search functions.
- Simplified request handling by removing unnecessary conversion
methods.
- **Breaking Changes**
- Removed the previous `SearchRequest` and `SearchResponse` messages
from the protocol, impacting existing integrations.
- **Refactor**
- Restructured search request processing to streamline operations and
improve maintainability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Overview
This adds new CRUDL folder endpoints that include `folder_type` as URL
parameter. These new endpoints can be used to
create/read/update/delete/list folders of the type specified in the
`folder_type` URL parameter.
The older CRUDL folder endpoints only worked on dashboard folders. They
are being deprecated in this PR. Once the UI stops calling these
endpoints they can be removed entirely.
## URL path design
he URLs for the new endpoints generally follow the pattern
`{org_id}/folders/{folder_type}/{folder_id}`. The position of the
`{folder_type}` parameter in these URLs is specifically intended to
represent the place of "folder types" within the hierarchy of
resources/concepts in the URL. Specifically the hierarchy is as
follows...
- `{org_id}` - represents an org which owns many folders
- `folders` - represents all folders
- `{folder_type}` - represents a class or subset of folders
- `{folder_id}` - identifies a single folder; this ID is only uniquely
identifying in the context of all concepts higher in the hierarchy,
including `{folder_type}`
## Alternatives
An alternative approach to this PR would be to update the existing
endpoints so that either
- `{folder_type}` is an optional final URL parameter (after
`{folder_id}`) that defaults to `dashboards`
- `{folder_type}` is an optional query parameter that defaults to
`dashboards`
Although pursuing one of these approaches would have reduced the need to
add new routes and deprecate old ones, I think the design of the URL
path would suffer and be less clear with either of these approaches
since in either of these approaches the path would not actually be
formatted to represent the resource hierarchy described above.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new `FolderType` enum for folder data categorization.
- Enhanced folder management API with new functions that include a
`folder_type` parameter.
- **Bug Fixes**
- Deprecated old folder-related functions while maintaining backward
compatibility.
- **Documentation**
- Updated API documentation to reflect changes in folder management
functions, including new deprecated endpoints and the `FolderType`
schema.
- **Chores**
- Added deprecation warnings for the `folders` module to inform users
about outdated methods.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#5253
The overall goal of this PR is to migrate alerts data out of `meta`
table into `alerts` table. To accomplish this three main changes are
made. Unfortunately, all three of these changes need to be synchronized
and submitted in the same PR.
1. A new migration script is added to
- Parse alerts JSON currently stored in the `meta` table into Rust
models for alerts. These Rust models are isolated to the migration
script since they represent the schema of alerts as they are stored in
the `meta` table before this migration executes.
- Convert those Rust models into Sea ORM entities for alerts.
- Insert all the Sea ORM entities for alerts into the new `alerts`
table.
2. Create get/put/delete/list methods for accessing the data in the
`alerts` table
- These functions use SeaORM to query and update the database.
- These functions must also convert between SeaORM alert entities and
the application's current data model for alerts (as defined in
`config::meta::alerts`.
- It's entirely expected that the models inside `config::meta::alerts`
will change over time. When this happens in the future, developers will
need to update the transformation functions between SeaORM alert
entities and the models in `config::meta::alerts`. Luckily, the compiler
should help to direct developers to where these transformation functions
need to be updated.
3. Update the old logic in `/src/service/db/alerts.rs` that previously
read and wrote alerts to the `meta` table so that it now reads and
writes alerts to the `alerts` table.
- The methods for accessing the `meta` table included logic for
triggering "cache update events" to the cluster coordinator and for
sending events to the super cluster queue. Since the new logic accesses
the `alerts` table rather than the `meta` table, this logic for emitting
events had to be reproduced.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new `ListAlertsParams` struct for flexible querying of
alerts.
- Enhanced `TriggerCondition` struct with new timing parameters.
- Added new error variants for improved error handling in alert
management.
- Created a comprehensive alert management system with CRUD operations.
- Added integration tests for various functionalities related to alerts
and dashboards.
- Introduced a new `Alerts` variant in the `FolderType` enum for better
folder management.
- **Bug Fixes**
- Improved error handling for folder deletion when alerts are present.
- **Documentation**
- Updated documentation for several functions to clarify their scope and
functionality.
- **Chores**
- Added new dependencies to support additional functionality.
- Introduced migration scripts for database schema changes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This PR introduces a feature to exempt data form the normal data
retention cycle and allows it to be stored for a longer period of time.
This time period is configurable via a global environment variable.
- [x] Add new feature: extend data retention
- [x] Fix issue #5418
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
- **New Features**
- Introduced a new field for extended data retention days in the
compacting process.
- Added a flexible update settings structure for managing stream
settings.
- Implemented a new time range management feature for deletion tasks.
- Enhanced configuration response to include retention days.
- **Bug Fixes**
- Improved error handling for stream settings related to retention days.
- **Tests**
- Expanded test suite with new tests for time range functionality.
- **Documentation**
- Updated OpenAPI documentation to reflect changes in stream settings
structures.
- Removed outdated components from OpenAPI definition.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
There are 3 types of partial responses:
1. VRL error
2. Super cluster error
3. Range error (max_query_limit)
Cache partial results only if there is a range error.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced search functionality with improved error handling for query
ranges.
- Introduced caching logic for partial results based on specific error
conditions.
- **Bug Fixes**
- Streamlined response generation and error reporting in search-related
functions, ensuring clarity and efficiency.
- **Documentation**
- Updated method signatures to reflect the latest changes in
functionality.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
In this [other PR
](https://github.com/openobserve/openobserve/pull/5390) I need to be
able to add logic to create a default "alerts" folder.
In order to do that, we first need to enable callers to pass a
`folder_type` parameter to different internal methods for
getting/creating/updating/deleting/listing folders. Previously those
methods assumed that the folder type was always `Dashboards` since thus
far that was the only folder type that existed.
This fixes two bugs -
1. The _values api used `Count(*)` function in query to get count.
However in case of distinct streams we already aggregate the count in
`count` field. Thus we need to use `sum(count)` instead.
2. In https://github.com/openobserve/openobserve/pull/5220 , the logic
which adds/removes the distinct values when dashboard is added/deleted
was removed accidentally. This Also re-adds that logic back.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced bulk deletion of dashboards with associated distinct
values.
- Added new methods for managing distinct fields in dashboards.
- Enhanced search functionality with improved handling of distinct
streams.
- **Bug Fixes**
- Improved error handling for dashboard operations and search requests,
including new error messages.
- **Documentation**
- Updated documentation for methods to clarify functionality and usage.
- **Refactor**
- Streamlined query variable extraction logic for dashboards.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new endpoint for listing user roles, excluding certain
roles.
- Added a new endpoint for token exchange requests for service accounts.
- **Enhancements**
- Improved role management logic and error handling in user management
functions.
- Enhanced error handling for authentication and token validation
processes.
- Updated the handling of claims and user roles based on login flow.
- Enhanced the logic for retrieving login URLs and managing
authentication states.
- **Bug Fixes**
- Prevented unauthorized updates and retrievals for external service
accounts.
- **Documentation**
- Updated method signatures to reflect new parameters and functionality.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced precision for storage size metrics across various components
by updating data types to floating-point numbers.
- **Bug Fixes**
- Adjusted calculations for size metrics to ensure accurate
floating-point operations.
- **Chores**
- Updated test cases to reflect new floating-point type handling for
storage metrics.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This PR introduces the service accounts with tokens
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Introduced a new module for managing service accounts, including
functionalities for listing, creating, updating, and deleting service
accounts.
- Added a new tab for "Service Accounts" in the Identity Access
Management interface.
- Enhanced user management interfaces to support service account
operations, including the ability to add and edit service accounts.
- Implemented a new component for managing service accounts within
groups and roles.
- Added new API endpoints for service account management, including
token operations.
- Introduced a comprehensive Vue component for managing service accounts
with enhanced features, including filtering, searching, and token
management.
- Enhanced permission checks and error handling for service accounts
across various functions.
- Added support for external user permissions in service account
operations.
- Enhanced error handling for various components to prevent unnecessary
notifications on permission-related issues.
- Added localization support for service account functionalities,
providing necessary labels for user interface elements.
- **Bug Fixes**
- Improved error handling and validation across user management and
service account functionalities.
- **Documentation**
- Updated localization files to support new service account features and
labels.
- **Style**
- Adjusted styles in various components to enhance usability and
consistency.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
Previously the `GetDashboard` and `DeleteDashboard` endpoints required
the client to pass in the `folder_id` query parameter. The query
parameter was actually superfluous because the `org_id` and
`dashboard_id` URL parameters provided to each endpoint already uniquely
identified a dashboard. Therefore, this PR removes `folder_id` as a
query parameter from those endpoints.
This PR introduces the optional `page_size` query parameter to the
`ListDashboards` endpoint. When set this will limit the size of the
result set to the specified value. The client/frontend will be able to
set this query parameter in calls to `ListDashboards` to limit the
number of results returned - specifically in requests to search
dashboards across folders.
This PR only implements limiting the result set size and does not
actually expose pagination as functionality of the `ListDashboards`
endpoint. My hope however is that this implementation will put us in a
good place to easily expose pagination on this endpoint if that's
something we want to do in the future - though it would by no means be
necessary.
There are a couple of idiosyncrasies here.
1. The `ListDashboards` endpoint does not read or use any `page_idx` or
`page_number` query parameter, as one might expect from a paginated
endpoint. Therefore if the client sets the `page_size` parameter to some
number N then the `ListDashboards` endpoint will always return the top N
results. (Considered in a different way, the endpoint always acts as if
the requested page index is `0`.)
- In the future we might want to add an additional `page_idx` query
parameter as well to enable "true" pagination (rather than simply
limiting the result set to the first N results). However there are no
immediate plans for this.
2. The `page_size` query parameter is only read/used by the endpoint
when the `title` parameter is **also** set. This means that when the
`title` parameter is **not** set the endpoint will ignore the
`page_size` parameter and will always return all dashboards that match
the other query parameter, `folder`.
- In the future we could choose to use this parameter any time it is
provided, not just when `title` is also provided. To do that there would
need to first be an update to the UI to address the issue described
below.
- For the time being this is necessary in order to not break the UI.
This is because currently the UI always sends the query parameter
`page_size=1000` to the endpoint, even though before now that query
parameter was never recognized by the endpoint. Additionally, the UI
assumes that the calls it makes to `ListDashboards` in order to show the
dashboards in a selected folder always return **all** dashboards, and in
such requests the frontend sets the `folder` query parameter but not the
`title` query parameter. Therefore, if we actually to use the
`page_size` parameter in **those** requests, then in its current
implementation the UI would fail to show all dashboards when a folder
with over 1000 dashboards is selected.
Hengfei Yang
Huaijin
Patrick Sullivan
Ashish Kolhe
Subhra264
Taiming Liu
Bhargavandhengfei yang
Yashodhan JoshiandYashodhan Joshi
