This is a hack to fix#5740 . As mentioned in
https://github.com/openobserve/openobserve/pull/5752, with postgresql
there is an issue when upgrading from v0.14.0 to v0.14.1 or v0.14.2 -
The migration `m20250109_092400_recreate_tables_with_ksuids` fails with
an error message. So after testing, it looks like the earlier mentioned
migration requires the `m20241209_120000_create_alerts_table` migration
to be committed to db first (so upto
`m20241217_155000_populate_alerts_table` migration).
Earlier https://github.com/openobserve/openobserve/pull/5722 was used to
address the issue, but unfortunately it works when upgrading from below
`v0.14.0` to `v0.14.2`. Currently the way to upgrade to v0.14.2 from
v0.14.0 with postgresql is to first upgrade from `v0.14.0` to
`v0.14.1-rc1` and then to `v0.14.2`.
This PR should make the direct upgrade from v0.14.0 to v0.14.3 possible
with postgresql.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved migration process for alerts table to ensure correct index
selection and prevent potential migration failures.
- **Documentation**
- Enhanced comments to provide clearer context about migration logic and
order of execution.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved WebSocket URL conversion logic to ensure more accurate URL
handling in proxy scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Currently, the backend returns a raw database internal server error
(500) when updating or creating a folder with a name that already exists
in the same organization. This pr avoids exposing the raw database error
to the api and instead returns a 400 bad request error.
Addresses #5723
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced folder management with prevention of duplicate folder names
within an organization
- **Bug Fixes**
- Added validation to prevent creating folders with names that already
exist
- **Improvements**
- Improved error handling for folder creation and update scenarios
- More descriptive error messages for folder-related operations
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
### Summary
This PR updates the logic that is used to generate KSUID primary keys
for the `folders`, `dashboards`, and `alerts` tables in existing
migrations which transform data from the `meta` table into the
`folders`, `dashboards`, and `alerts` tables.
The new logic generates 160-bit "KSUIDs" which are actually 160-bit
SHA-1 hashes of the records being transformed rather than being truly
random KSUIDs.
### Motivation
In the existing implementation, the migrations generate random KSUIDs
for the records that are transformed from the `meta` table and into the
different `folders`, `dashboards`, and `alerts` tables. When running the
application in a single cluster this is acceptable behavior.
However when running the application in a multi-cluster environment this
causes problems. In a multi-cluster environment, each cluster will have
it's own copy of the application database, and each database should
contain the exact same data in the `meta` table. When we run the
migration scripts in each cluster, the data from the `meta` table will
be transformed into the `folders`, `dashboards`, and `alerts` tables.
However if the migration generates a random KSUID for each new record
inserted into those new tables, then different clusters will each
generate different KSUID `id`s for rows in the `folders`, `dashboards`,
and `alerts` tables which should actually have the same `id`s.
Our solution to this problem is that during the migration from the
`meta` table to the `folders`, `dashboards`, and `alerts` tables, rather
than generating random 160-bit KSUID `id`s for the new records, we
generate 160-bit SHA-1 hashes for each record and we use that SHA-1 hash
as the `id` of the each new record. This ensure that each record that is
transformed from the `meta` table will have an `id` in the `folders`,
`alerts`, or `dashboards` tables that is consistent across all clusters.
Newly generated records (ie, records generated AFTER the migration) in
the `folders`, `alerts`, and `dashboards` tables will continue to be
generated with random KSUID `id`s since these `id`s can be shared with
other clusters via NATS super-cluster events and then records with those
`id`s can be inserted into databases in other clusters, ensuring that
records have consistent `id`s across clusters.
### Notes about KSUID generation
To generate a KSUID this function generates the 160-bit SHA-1 hash of
the alert's `org`, `stream_type`, `stream_name`, and `name` and
interprets that 160-bit hash as a 160-bit KSUID. Therefore two KSUIDs
generated in this manner will always be equal if the alerts have the
same `org`, `stream_type`, `stream_name`, and `name`.
⚠️ It is important to note that although KSUIDs generated in this manner
can be parsed as KSUIDs since they are 160-bit values, the KSUIDs
generated in this manner will have timestamp bits which are effectively
random, meaning that the timestamp in any KSUID generated with this
function will be random. This will render the timestamp-sortability
property of these KSUIDs useless. This probably isn't a big deal since
we can always add and sort by a `created_at` column if we want to, but
we should be aware of this limitation.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added SHA-1 hashing dependency to generate unique identifiers (KSUIDs)
for alerts, folders, and dashboards.
- Implemented KSUID generation based on specific entity attributes.
- **Refactor**
- Transitioned database tables from integer-based to KSUID-based primary
keys.
- Updated `StreamType` enum to support more flexible usage.
- **Chores**
- Updated project dependencies to include SHA-1 hashing library.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Added a targeted migration step to resolve an issue with alerts
migration
- Implemented a temporary workaround to ensure successful database
migration process
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Now that the old alerts code, which stored alerts in the `meta` table,
has been removed, `alert/mod.rs` mostly just serves as a wrapper around
`alert/new.rs`.
This PR removes `alert/new.rs` and moves its logic into the parent
`alert` module. Also `alert/mod.rs` is replaced with `alert.rs` since
the `alert` module no longer contains child modules.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Enhanced alert management functionality with improved listing and
caching capabilities
- Added new methods for retrieving and managing alerts more flexibly
- **Changes**
- Updated alert listing method to support more specific filtering
parameters
- Simplified alert retrieval and management processes
- **Refactor**
- Restructured alert-related functions to improve code organization and
maintainability
- Streamlined database interaction methods for alerts
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Follow-up:
Consolidate the `alert/new.rs` and `alert/mod.rs` files into one file,
since now the `alert/mod.rs` file mostly just acts as a wrapper around
the `alert/new.rs` file
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Refactor**
- Removed legacy alert handling module
- Simplified alert management logic
- Eliminated dual-table approach for alerts
- Streamlined alert operations to use new alerts table exclusively
- **Chores**
- Removed deprecated code and conditional checks related to old alert
management system
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
impl #5656
## Todo
- [x] Router consistent hashing for PromQL
- [x] Remove the key from index if the key get nothing from cache
- [x] Some cache item is empty, it should not
- [x] Add cache control ENV: `ZO_METRICS_CACHE_ENABLED=true`
- [ ] Optimize cache gc logic
- [ ] New encoding of RangeValue for cache
- [x] New cache key generate
- [x] Load cache from disk
- [ ] Histogram query OOM without cache for 2 days
- [x] Fix response series order
- [x] Need to check stream is exists
- [x] Add trace_id to the response
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Enhanced folder management with improved unique identifier generation.
- Added enterprise-specific folder processing for create, update, and
delete operations.
- Introduced new asynchronous functions for processing folder-related
messages.
- Added optional ID parameter for alert creation, enhancing flexibility.
- **Changes**
- Updated folder-related function signatures to return more
comprehensive information.
- Integrated enterprise-level folder message processing in super cluster
queue.
- Relocated and centralized message definitions for alerts and
dashboards.
- **Improvements**
- Refined error handling for folder operations.
- Centralized message handling for folder-related events.
- Improved clarity and maintainability in dashboard moving logic.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
We need to be able to reference the `FolderType` enum from
`o2-enterprise` without introducing a dependency on the `table` module.
Therefore, this PR moves the `FolderType` enum out of the `table` module
and into the shared `config::meta` module.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new `FolderType` enum to categorize folders as Dashboards
or Alerts.
- Enhanced folder management with more precise type handling across
different services.
- **Refactor**
- Updated import statements and type references for `FolderType` across
multiple modules.
- Standardized folder type conversion and handling mechanisms.
- **Improvements**
- Added more robust folder creation and validation logic.
- Improved error handling for folder-related operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This is the follow-up PR to [this other
PR](https://github.com/openobserve/o2-enterprise/pull/281#issue-2773724675).
Please see the other PR description for a detailed explanation of the
changes.
This PR copies the supercluster queue message handling code from
`o2-enterprise` which updates the database when different messages are
received. In this PR these message-handling functions are injected into
`o2-enterprise`'s various `SuperClusterQueue` structures so that the
`SuperClusterQueue`s can execute the logic to update the `openobserve`
database without the `o2-enterprise` crate needing a dependency on the
`table` module.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added comprehensive message processing capabilities for various
enterprise components including alerts, dashboards, distinct values,
meta, pipelines, schemas, search jobs, and short URLs
- Introduced a new super cluster queue system with modular message
handling and initialization
- **Chores**
- Updated library and main application to support enterprise feature
initialization
- Added conditional compilation for enterprise-specific modules
- **Infrastructure**
- Implemented robust error handling and logging for message processing
across different system components
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
### Summary
Folder names within an individual organization and within a specific
folder type should be unique.
This migration adds a migration to...
1. Rename any folders that have the same name, organization, and folder
type by appending a 27-character random KSUID to the name to ensure that
each name is unique.
2. Create a unique constraint on the `folders` table on the composite
key of `org`, `type`, and `name`. This will prevent new folders from
having duplicate names in the future.
### Follow-up work
If a user tries to create or update a folder so that the folder would
have the same name as another folder with the same `org` and `type`,
then the database will reject the query and the application will return
an error response. However the error response will likely just contain
the error text returned by SeaORM.
We should also update the logic in `/src/service/folders.rs` to check
for duplicate folder names when folders are created or updated and to
gracefully return a useful error.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Implemented a database migration to ensure unique folder names within
an organization.
- Added a mechanism to automatically rename duplicate folders by
appending a unique identifier.
- Created a unique constraint for folder names across organization and
type.
- Introduced a new API endpoint to retrieve folders by name.
- Added a deprecated API endpoint for retrieving folders by name with a
different structure.
- **Improvements**
- Enhanced database integrity by preventing naming conflicts in folder
management.
- Improved authorization logic for GET requests targeting folders by
name.
- Added functionality to retrieve folders by their name in the service
layer.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
Adds a new endpoint for moving an alert between different folders.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added ability to move alerts between folders.
- Introduced new endpoint for moving alerts.
- Enhanced alert management with improved validation and error handling.
- **Bug Fixes**
- Improved alert retrieval to include folder context.
- **Documentation**
- Updated OpenAPI documentation to include new alert movement endpoint.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
When we move a dashboard from one folder to another, currently the rbac
relations are not updated, so the folder still points to the old folder
as `parent` in the ofga `tuple` database.
This PR fixes this issue by updating the `parent` relationship between
the dashboard and the old and new folders.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced dashboard management with improved access control integration
- Added support for setting and removing parent relations for dashboards
using OpenFGA system
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Enhanced database migration script to improve record processing order
- Added logging for KSUID generation during migration process
- Updated import statements to support query ordering functionality
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- [x] for search job, in `total` field, will return the total records
numbers.
- [x] if not set the from and size when get result, we set default from
= 0 and size = 100
close#5645
## New Env
```
ZO_SEARCH_JOB_RETENTION
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Added configurable search job retention setting
- Implemented job retention management for search jobs
- **Enterprise Features**
- Enhanced job deletion mechanism based on retention period
- Added ability to automatically clean up older search jobs
- **Configuration**
- New option to specify how long search jobs are kept (default: 30 days)
These updates improve system management and provide more control over
job lifecycle, particularly for enterprise users.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Database Migration**
- Transitioned database tables to use KSUIDs (K-Sortable Unique
Identifiers) as primary keys.
- Recreated `folders`, `dashboards`, and `alerts` tables with a new
identifier system.
- Updated `id` and `folder_id` fields in `alerts`, `dashboards`, and
`folders` tables from numeric to string types.
- Ensured unique identifiers are generated upon creation of new folders
and dashboards.
- Preserved existing data during the migration process.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved alert update and enablement process
- Enhanced transactional handling for alert state changes
- **Refactor**
- Updated alert management logic to provide more robust state
transitions
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced configuration handling for enterprise geographic IP
enrichment when the enterprise feature is enabled.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Security Enhancement**
- Added validation to prevent service accounts from logging in
- Implemented additional authentication checks during user login process
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Refactor**
- Simplified the Maxmind database download and client initialization
process.
- Streamlined logic for handling file downloads and client updates.
- Removed unnecessary conditional checks, enhancing the overall flow of
operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Hengfei Yang
Subhra264
Huaijin
Loakesh Indiran
Taiming Liu
Patrick Sullivan
Ashish Kolhe
Zhao.wei