fixed#4382
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced conditional compilation for the `ofga` module, enabling
more flexible builds based on the "enterprise" feature.
- Enhanced the `init` function to validate store IDs when the super
cluster feature is active.
- Added a new function to handle database updates, improving modularity.
- **Bug Fixes**
- Improved error handling in the initialization process to enhance
system observability.
- **Style**
- Restructured import statements for better organization and clarity in
the `ofga` module.
- Simplified conditional logic formatting in the `Sql` implementation
for improved readability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a `histogram_interval` field to enhance query response
capabilities, allowing for more detailed data representation.
- Enhanced caching logic to set the `histogram_interval` based on the
`discard_interval`, improving the handling of cached query responses.
- **Improvements**
- Added conditional logic in response handling to include
`histogram_interval` when specific criteria are met, improving response
accuracy and robustness.
- Refined checks for empty responses, enhancing the robustness of the
merging process for query responses.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced response handling by including the histogram interval in
search results.
- **Bug Fixes**
- Adjusted hit filtering logic to improve accuracy by changing the
timestamp comparison criteria.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- Fixes#4394
- When cloning alert, don't copy `last_satisfied_at`,
`last_triggered_at`
When the alert query is invalid, the scheduler stops processing the
alert and returns, but the error message is not reported in the triggers
usage stream. This PR fixes that.
When the alert query is invalid, it will be retried again and again
(default maximum number of retries is `3`). After that, the associated
`scheduled_jobs` record will be automatically removed from the table and
hence the alert will not be triggered anymore.
In order to schedule the alert again, the user needs to update the query
and again save it, it will start scheduling again.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
- **New Features**
- Enhanced alert handling logic for improved robustness and clarity.
- Introduced structured handling of alert data with better error
management.
- Added initialization of `last_triggered_at` and `last_satisfied_at`
fields to ensure clear state management for alerts.
- **Bug Fixes**
- Improved error handling for alert evaluation, ensuring accurate status
updates for alert triggers.
- Enhanced state preservation for alerts by updating the
`last_satisfied_at` timestamp during saves.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The `last_satisfied_at` field in the alert tracks when the alert
condition last met. This is in UTC microseconds.
This PR fixes the issue in #4379
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
- **New Features**
- Introduced a new timestamp field to track when alerts were last
satisfied.
- Enhanced alert handling logic to manage and update satisfaction
timestamps effectively.
- Added a new column in the Alert List to display the last satisfied
timestamp for alerts.
- Improved localization support by adding a label for the last satisfied
timestamp.
- **Bug Fixes**
- Improved accuracy in alert state management by ensuring timestamps are
recorded correctly.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Sai Nikhil <nikhil@openobserve.ai>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced query hash calculation for improved uniqueness and
flexibility in search functionality.
- **Bug Fixes**
- Simplified control flow in the hash generation process, reducing
potential errors in query handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fixes#4388
- If the VRL function in alert contains only '.', no need to store the
VRL function in the alert
- Remove alert evaluation in backend while saving alert
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved logic for saving alerts by preventing the assignment of
non-meaningful values (specifically a single period) to alert
conditions, enhancing data integrity.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fixes#4360
In case some `scheduled_job` records (required for alerts to get
scheduled properly) are missing for some alerts, they can be recreated
by simply disabling and enabling it.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Improvements**
- Enhanced error logging for trigger updates and creations, providing
clearer context with the inclusion of the `schedule_key`.
- Improved robustness of trigger management logic, facilitating easier
diagnosis of issues related to alert triggers.
- Increased flexibility in alert handling by dynamically setting the
`is_realtime` property based on the alert's characteristics.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
issue #4326
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced new structures for updating stream settings, allowing for
more granular management of stream configurations.
- Added an asynchronous function for updating stream settings via HTTP
PUT requests.
- Implemented a new service route for handling updates to stream
settings.
- Enhanced the `AddStream` component to retrieve updated settings.
- **Improvements**
- Improved schema management by tracking previous settings during
updates.
- Enhanced utility functions for comparing and managing stream settings.
- **Documentation**
- Updated documentation to reflect changes in API functionality for
creating and updating stream settings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced cache checking logic to improve performance and handling of
aggregate queries.
- **Bug Fixes**
- Improved error logging for cached results to streamline
troubleshooting.
- **Refactor**
- Refined control flow and code readability in cache-related functions
without changing core functionality.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced search functionality with integrated performance metrics,
allowing for better monitoring and optimization of search operations.
- **Bug Fixes**
- Improved logging behavior in the caching mechanism to avoid
unnecessary log entries, providing clearer and more relevant log
outputs.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fixes#4291
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new method for handling 409 Conflict responses in the
HTTP responses, improving error messaging clarity.
- **Bug Fixes**
- Updated error handling to return conflict responses instead of
forbidden responses when certain delete operations fail due to resource
conflicts.
- Enhanced clarity in user membership responses by changing the error
message when a user is already part of an organization.
These changes improve user experience by providing clearer and more
accurate feedback on error conditions related to resource management.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new configurable caching strategy, allowing users to
choose between "Overlap," "Duration," or "Both" for result caching.
- Added a new public field for `result_cache_selection_strategy` in the
application configuration, providing improved flexibility in managing
caching behavior.
- **Improvements**
- Enhanced the function for processing cache results to accommodate the
new caching strategies, allowing for more nuanced result management
based on user selection.
- Updated the `check_cache` functionality to conditionally sort cached
results and optimize performance based on the state of the cache.
- Improved tracking of the size of hits in cache responses.
- Added a helper function to streamline the calculation of the discard
timestamp for cached results.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- #4302
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
- **New Features**
- Introduced new configuration options for dashboard panels, including
settings for "Top Results" and an "Others" series to enhance data
visualization.
- Added an input field in the dashboard configuration panel for users to
specify the number of top values displayed.
- **Improvements**
- Enhanced data processing capabilities for SQL data, allowing better
aggregation and sorting of results.
- Streamlined the dashboard configuration flow for an improved user
experience.
- Improved error handling and logging in data processing to enhance
debugging and visibility.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced validation for alert creation to prevent the use of "SELECT
*" in SQL queries, improving data integrity and security during the
alert setup process.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fixes : https://github.com/openobserve/openobserve/issues/4274
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced search response quality by implementing a deduplication step,
ensuring unique entries based on timestamps in the search results.
- Improved delta calculations in the caching mechanism with new
parameters for granular control over time intervals, enhancing the
accuracy of cached data responses.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fixes#4241
- Adds a new `data` column to `scheduled_jobs` table for the purpose of
storing different data as required by different modules.
- Uses the last end time of stream as the start time of derived stream
calculation.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added a new `data` field to the `Trigger` structure, allowing for
additional information storage in scheduled jobs.
- Introduced a new `DerivedTriggerData` struct for improved data
handling within the scheduler module.
- Enhanced job evaluation and query conditions with new optional
`start_time` parameters for greater flexibility.
- **Bug Fixes**
- Improved error handling for database schema changes to maintain
backward compatibility with older versions.
- **Refactor**
- Simplified logic within various functions to improve code clarity and
performance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced the `ResultCacheMeta` struct to support equality comparisons,
improving filtering and duplicate checks.
- Updated response merging to account for histogram intervals in cache
responses.
- Improved filtering logic to optimize processing of metadata, including
early returns for empty collections.
- **Bug Fixes**
- Refined logic to ensure correct filtering based on response times for
metadata processing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Addresses #4273
- Adds `last_triggered_at`, `owner`, `updated_at` and `last_edited_by`
fields (similar to Reports) to Alerts.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced alert and report management with additional metadata fields
for tracking last triggered times, owners, and edit history.
- Introduced user identification in report creation and updates for
better ownership tracking.
- **Bug Fixes**
- Improved logic for saving alerts and reports to accurately reflect
their statuses and metadata during creation and updates.
- **Documentation**
- Restructured import organization for improved clarity and
maintainability in the codebase.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fixes : https://github.com/openobserve/openobserve/issues/4260
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved error handling in cache response calculations to prevent
division by zero when there are no search results. This change enhances
the stability and reliability of the application.
- Enhanced the conditional checks in the search function to ensure
accurate results are returned and prevent false positives when results
are absent.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new structure for enhanced management of cached query
responses, allowing for detailed handling of multiple cached results.
- Added functionality to retrieve multiple cached results in a single
request, optimizing performance and reducing service calls.
- Implemented a mechanism for efficient caching and retrieval of query
results in a distributed cluster environment.
- Enhanced configuration options for managing result caching behavior.
- **Bug Fixes**
- Improved handling of cache misses and response merging to ensure
robustness against empty responses.
- **Refactor**
- Updated internal logic to support new cached query response structure,
streamlining caching operations and enhancing clarity.
- Introduced new modules to improve code organization related to
multi-cluster operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fixes#4243
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Improvements**
- Enhanced logging for alert and report triggers to provide clearer and
more informative messages.
- Updated status reporting for trigger conditions, improving operational
insights.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fixes : grpc: search err: Status { code: OutOfRange, message: "Error,
message length too large: found 7163638 bytes, the limit is: 4194304
bytes", metadata: MetadataMap { headers: {"content-type":
"application/grpc","content-length": "0"} }, source: None }
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced gRPC server initialization with configurable maximum message
sizes for improved message handling.
- **Bug Fixes**
- Improved robustness in handling larger messages during gRPC
communication to prevent potential issues.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fixes#4129
- Current workflow for the pipelining -
ingestion to the main stream -> apply functions of main stream with
`apply_before_flattening` flag -> flatten data -> routing -> apply rest
of the functions of the routed stream. `apply_before_flattening`
functions will be ignored for the routed stream because data is already
flattened before routing.
- Fix of bug: For non-bulk apis for log ingestion, if the first record
is routed (i.e. the routing condition matched), the rest of the records
also are routed even if the condition does not match.
- Fix of bug: For json log ingestion, routing was happening before
flattening of stream data, this PR fixes that.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced stream processing functionality with improved control flow
and routing logic.
- Introduced structured management of stream transformations using
HashMaps for better clarity and maintainability.
- **Bug Fixes**
- Improved robustness of stream processing by ensuring all relevant
streams are processed without premature termination.
- **Chores**
- Renamed variables for clarity regarding their roles in stream
processing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced handling of the `#ResultArray#` marker in query inputs,
allowing for more robust pattern detection and transformation.
- **Bug Fixes**
- Improved the accuracy of the query function's transformation logic by
switching from a simple prefix check to a regex-based matching approach.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced cached search results filtering with a new `discard_interval`
parameter for improved time-based adjustments.
- **Bug Fixes**
- Adjusted logic to ensure accurate filtering of cached hits based on
the newly implemented time boundaries.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Improved Period of derived stream to be same as frequency
Complete the `Frequency` task in the tracking issue #4170
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
- **New Features**
- Introduced a new component for managing scheduled alerts with a tabbed
interface for custom alerts, SQL queries, and PromQL queries.
- Enhanced user experience with dynamic input fields, real-time
validation, and aggregation settings for alert conditions.
- Added cron job configuration options for scheduling alerts.
- **Changes**
- Renamed the scheduled alert component to reflect the new
functionality, impacting how the application handles alert triggering
and scheduling.
- Adjusted timing and frequency parameters for triggering conditions in
the alert management system.
- **Improvements**
- Enhanced logging functionality for monitoring enrichment data saving,
providing insights into operations and parameters involved.
- Improved error handling and response validation processes for
ingestion tasks, ensuring more informative logging and error reporting.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced the SQL query analysis to better identify aggregate queries
with additional checks for `GROUP BY` and `HAVING` clauses.
- Improved the evaluation of various expression types to determine
aggregation, making the SQL parser more robust.
- **Bug Fixes**
- Adjusted performance by modifying function signatures for improved
efficiency and reduced memory allocation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- Includes the `functionContent` query parameter in the alert url
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Improved handling of input for alert conditions, ensuring only valid
`vrl` values are used.
- Enhanced URL generation for alerts by incorporating refined logic
based on the presence of `vrl_function`.
- **Bug Fixes**
- Fixed potential issues in alert processing due to better validation of
input values.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- Convert `[:?#\s'"&%]` characters into `_`
- Migration for updating existing alerts/destinations/templates/reports
with name containing the above characters.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
- **New Features**
- Enhanced initialization process with better tuple management for data
migration.
- Added conditional execution for migration processes based on node type
and version.
- Introduced new functions to validate and format names according to
OFGA requirements.
- Improved handling of destination, alert, and report names in API
interactions for better compliance.
- **Bug Fixes**
- URL parameters are now properly encoded to prevent malformed API
requests.
- **Documentation**
- Improved error handling and logging for migration processes for better
visibility.
- **Style**
- Minor formatting improvements for better code readability and
consistency.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
Hengfei Yang
Ashish Kolhe
TessaIO
Subhra264
Huaijin
BhargavandSubhradeep Chakraborty
Yashodhan Joshi
ktx-vaidehi
