Currently, the backend returns a raw database internal server error
(500) when updating or creating a folder with a name that already exists
in the same organization. This pr avoids exposing the raw database error
to the api and instead returns a 400 bad request error.
Addresses #5723
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced folder management with prevention of duplicate folder names
within an organization
- **Bug Fixes**
- Added validation to prevent creating folders with names that already
exist
- **Improvements**
- Improved error handling for folder creation and update scenarios
- More descriptive error messages for folder-related operations
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Now that the old alerts code, which stored alerts in the `meta` table,
has been removed, `alert/mod.rs` mostly just serves as a wrapper around
`alert/new.rs`.
This PR removes `alert/new.rs` and moves its logic into the parent
`alert` module. Also `alert/mod.rs` is replaced with `alert.rs` since
the `alert` module no longer contains child modules.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Enhanced alert management functionality with improved listing and
caching capabilities
- Added new methods for retrieving and managing alerts more flexibly
- **Changes**
- Updated alert listing method to support more specific filtering
parameters
- Simplified alert retrieval and management processes
- **Refactor**
- Restructured alert-related functions to improve code organization and
maintainability
- Streamlined database interaction methods for alerts
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Follow-up:
Consolidate the `alert/new.rs` and `alert/mod.rs` files into one file,
since now the `alert/mod.rs` file mostly just acts as a wrapper around
the `alert/new.rs` file
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Refactor**
- Removed legacy alert handling module
- Simplified alert management logic
- Eliminated dual-table approach for alerts
- Streamlined alert operations to use new alerts table exclusively
- **Chores**
- Removed deprecated code and conditional checks related to old alert
management system
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
impl #5656
## Todo
- [x] Router consistent hashing for PromQL
- [x] Remove the key from index if the key get nothing from cache
- [x] Some cache item is empty, it should not
- [x] Add cache control ENV: `ZO_METRICS_CACHE_ENABLED=true`
- [ ] Optimize cache gc logic
- [ ] New encoding of RangeValue for cache
- [x] New cache key generate
- [x] Load cache from disk
- [ ] Histogram query OOM without cache for 2 days
- [x] Fix response series order
- [x] Need to check stream is exists
- [x] Add trace_id to the response
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Enhanced folder management with improved unique identifier generation.
- Added enterprise-specific folder processing for create, update, and
delete operations.
- Introduced new asynchronous functions for processing folder-related
messages.
- Added optional ID parameter for alert creation, enhancing flexibility.
- **Changes**
- Updated folder-related function signatures to return more
comprehensive information.
- Integrated enterprise-level folder message processing in super cluster
queue.
- Relocated and centralized message definitions for alerts and
dashboards.
- **Improvements**
- Refined error handling for folder operations.
- Centralized message handling for folder-related events.
- Improved clarity and maintainability in dashboard moving logic.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
We need to be able to reference the `FolderType` enum from
`o2-enterprise` without introducing a dependency on the `table` module.
Therefore, this PR moves the `FolderType` enum out of the `table` module
and into the shared `config::meta` module.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new `FolderType` enum to categorize folders as Dashboards
or Alerts.
- Enhanced folder management with more precise type handling across
different services.
- **Refactor**
- Updated import statements and type references for `FolderType` across
multiple modules.
- Standardized folder type conversion and handling mechanisms.
- **Improvements**
- Added more robust folder creation and validation logic.
- Improved error handling for folder-related operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
### Summary
Folder names within an individual organization and within a specific
folder type should be unique.
This migration adds a migration to...
1. Rename any folders that have the same name, organization, and folder
type by appending a 27-character random KSUID to the name to ensure that
each name is unique.
2. Create a unique constraint on the `folders` table on the composite
key of `org`, `type`, and `name`. This will prevent new folders from
having duplicate names in the future.
### Follow-up work
If a user tries to create or update a folder so that the folder would
have the same name as another folder with the same `org` and `type`,
then the database will reject the query and the application will return
an error response. However the error response will likely just contain
the error text returned by SeaORM.
We should also update the logic in `/src/service/folders.rs` to check
for duplicate folder names when folders are created or updated and to
gracefully return a useful error.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Implemented a database migration to ensure unique folder names within
an organization.
- Added a mechanism to automatically rename duplicate folders by
appending a unique identifier.
- Created a unique constraint for folder names across organization and
type.
- Introduced a new API endpoint to retrieve folders by name.
- Added a deprecated API endpoint for retrieving folders by name with a
different structure.
- **Improvements**
- Enhanced database integrity by preventing naming conflicts in folder
management.
- Improved authorization logic for GET requests targeting folders by
name.
- Added functionality to retrieve folders by their name in the service
layer.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
Adds a new endpoint for moving an alert between different folders.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added ability to move alerts between folders.
- Introduced new endpoint for moving alerts.
- Enhanced alert management with improved validation and error handling.
- **Bug Fixes**
- Improved alert retrieval to include folder context.
- **Documentation**
- Updated OpenAPI documentation to include new alert movement endpoint.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
When we move a dashboard from one folder to another, currently the rbac
relations are not updated, so the folder still points to the old folder
as `parent` in the ofga `tuple` database.
This PR fixes this issue by updating the `parent` relationship between
the dashboard and the old and new folders.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced dashboard management with improved access control integration
- Added support for setting and removing parent relations for dashboards
using OpenFGA system
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
close#5645
## New Env
```
ZO_SEARCH_JOB_RETENTION
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Added configurable search job retention setting
- Implemented job retention management for search jobs
- **Enterprise Features**
- Enhanced job deletion mechanism based on retention period
- Added ability to automatically clean up older search jobs
- **Configuration**
- New option to specify how long search jobs are kept (default: 30 days)
These updates improve system management and provide more control over
job lifecycle, particularly for enterprise users.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved alert update and enablement process
- Enhanced transactional handling for alert state changes
- **Refactor**
- Updated alert management logic to provide more robust state
transitions
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Enhanced error handling for dashboard movement operations
- Improved error reporting when deleting dashboards from old folders
during move operations
- **New Features**
- Added more detailed error tracking for dashboard-related actions
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
for debug the #5398
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **Configuration**
- Added new configuration option to disable keep-alive connections
- Adjusted HTTP server connection timeout settings
- **Authentication**
- Enhanced token validation with improved traceability
- Updated authentication methods to support more detailed request
tracking
- **Performance**
- Optimized client lock management in MaxMind database downloader
- Refined request handling in RUM ingestion APIs
- **Logging**
- Improved logging for authentication and request processing
- Added trace ID support for better request tracking
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: domyway <weizhao029@gmail.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Added a new function testing capability for VRL (Variable Runtime
Language) functions.
- Introduced interactive UI components for testing and managing
functions, including `FunctionsToolbar`, `TestFunction`, and
`FullViewContainer`.
- New API endpoint for testing VRL functions against a set of events.
- Added new localization strings for user interface elements.
- **Improvements**
- Refined UI components for better user experience, including sidebar
management and editor functionalities.
- Enhanced function management with improved error handling and
validation.
- Improved code structure and maintainability across multiple
components.
- **Bug Fixes**
- Addressed various minor formatting and syntax issues.
- Improved error handling in function testing and management.
- **Enhancements**
- Updated the `QueryEditor` component for improved formatting and error
diagnostics.
- Enhanced Playwright tests for better targeting and reliability in
function management scenarios.
- Improved layout and responsiveness of several components, including
`AssociateFunction` and `ScheduledAlert`.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: udev <uddhavdave1@gmail.com>
Co-authored-by: udev <uddhav@openobserve.ai>
This moves the methods for emitting and listening for alert-related
cluster coordinator events to the infra crate.
We are doing this so that logic in the enterprise crate can call the
methods to emit cluster coordinator events when responding to a super
cluster queue event.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced alert management functionality in the cluster coordinator
- Added new methods for emitting and watching alert events
- Introduced improved event handling mechanisms for alerts
- **Refactor**
- Restructured alert event processing logic
- Modularized cluster coordinator event management
- Streamlined cache update mechanisms for alerts
- **Chores**
- Updated infrastructure and service layer modules
- Added new module for cluster coordinator functionality
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This adds support for super_cluster queue notification for
addition/removal of distinct value fields.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added serialization capabilities for `OriginType` and
`DistinctFieldRecord`.
- Introduced new asynchronous functions for managing distinct field
records in the database.
- Added enterprise-specific event emission for record operations.
- **Refactor**
- Updated import statements across multiple service modules.
- Reorganized distinct values handling in the database service.
- **Chores**
- Added a new module for distinct values management.
- Improved error handling for dashboard updates and stream operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
reuse the function `filter_file_list_by_tantivy_index` in log tantivy
index to impl the inverted index for promql.
the inverted index for promql only support Equal and Regex, don't support NotEqual and NotRegex.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved alert permission filtering to support a more flexible
permission structure.
- Enhanced alert access control to ensure accurate alert visibility
based on updated permissions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Enhanced serialization support for short URL records
- Improved event handling for short URL operations
- **Refactor**
- Modularized event emission logic for short URL management
- Introduced separate modules for standard and enterprise cluster event
handling
- **Chores**
- Added support for conditional enterprise features
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Hengfei Yang
Taiming Liu
Subhra264
Patrick Sullivan
Huaijin
iQQBot
Loakesh Indiran