- fix PromQL functions: irate, topk, bottomk, and histogram_quantile
- perf: remove rayon `par_iter()` and some unnecessary clones
This PR improved compliance coverage from 95.07% to
```bash
================================================================================
General query tweaks:
* Openobserve is sometimes off by 1ms when parsing floating point start/end timestamps.
* Openobserve fractional tolerance amount for <agg>_over_time queries
================================================================================
Total: 538 / 548 (98.18%) passed, 0 unsupported
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved label handling and processing in the data evaluation
functions.
- Optimized value processing by removing unnecessary cloning operations.
- Enhanced accuracy in time difference calculations in rate functions.
- **Performance**
- Streamlined iteration and filtering operations for better performance.
- Simplified `Engine` instance creation to reduce overhead.
- **Refactor**
- Refined logic in histogram functions to handle empty buckets more
effectively.
- Adjusted control flow in extrapolated rate calculations for better
efficiency.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Refactor**
- Reformatted the `usage_publish_interval` configuration attribute for
better readability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fixes#3373
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Improved the handling and publishing of usage data, ensuring more
efficient and timely updates.
- **Refactor**
- Streamlined the internal control flow for managing usage data,
enhancing overall system performance.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced `searchType` attribute across various components to enhance
search functionality.
- Added support for different search event types like `UI`,
`Dashboards`, `Reports`, `Alerts`, `Values`, and `RUM`.
- **Bug Fixes**
- Improved search parameter handling in multiple components and API
calls.
- **Enhancements**
- Updated search-related functions to include `searchType` for more
precise search results.
- Enhanced dashboard and RUM views with new `searchType` attributes for
better data filtering.
- **Refactor**
- Standardized search functionality across different modules and
components.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved the logic for formatting record batches to handle cases where
schema fields are empty before comparison, ensuring more robust data
processing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The query `size` works like this, it should more clear now:
1. size > 0, it will return hits equal to `size`
2. size = 0, it will return empty hits, this used to other purposes,
like: get total, get aggs.
3. size = -1, it will use a backend default limit to return hits,
setting by `ZO_QUERY_DEFAULT_LIMIT=1000`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Corrected handling of query parameters by changing types from `usize`
to `i64` to support larger values and prevent potential overflow issues.
- **Improvements**
- Enhanced the logic for setting query limits to handle negative values
more gracefully, ensuring better default behavior.
- **Refactor**
- Updated various internal structures and function signatures to use
`i64` instead of `usize` for consistency and improved range handling.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fixes#3640
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Improved the efficiency of updating and fetching scheduled jobs by
using dynamic SQL query construction based on job IDs.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced search functionality with indexed matching options.
- **Bug Fixes**
- Updated SQL queries in tests to align with new matching functions.
- **Improvements**
- Standardized naming conventions for matching functions to improve
clarity and consistency.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- [x] If the search request size is 0 and in full sql mod and track
total hits, will not apply default limit `1000`.
- [x] change consistent hashing vnode default value to `16`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Improvements**
- Enhanced performance and accuracy of data aggregation by updating
internal logic and default values.
- Improved configuration flexibility by allowing custom `history`
values.
These changes aim to provide a more robust and configurable experience
for end-users.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- #3460
- #2015
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added multi-select functionality for variables in dashboards.
- Introduced a custom value selector component for dashboard settings.
- Updated the app to use Quasar's fullscreen module.
- **Enhancements**
- Improved query verification process in DashboardQueryEditor.
- Enhanced filtering logic for variable placeholders and values in
queries.
- Added "IN" operator support for dashboard queries.
- **Bug Fixes**
- Fixed issues with handling and displaying selected values in various
components.
- **Performance Improvements**
- Increased size parameter for dashboard panel data loading from 10 to
100.
- **Localization**
- Added new localization key: "multiSelect" for multiple selection
support.
- **UI/UX**
- Updated styles and key bindings in DrilldownPopUp.
- Enhanced display logic for selected values in
VariableQueryValueSelector.
- **Dependencies**
- Updated Quasar package from version ^2.7.5 to ^2.15.4.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech>
- Multiple filters for the same field.
- Added p99, p95, p90 and p50 as an option on y-axis
- In Drilldown popup will now get variables value of current dashboard
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Added new percentile aggregation options (`p50`, `p90`, `p95`, `p99`)
across various dashboard components.
- Introduced `variablesData` prop to enhance data handling in several
components.
- **Improvements**
- Enhanced error handling and asynchronous operations in dashboard panel
functionalities.
- Improved readability and maintainability of aggregation functions in
query components.
- **Localization**
- Added new keys for percentiles (`p50`, `p90`, `p95`, `p99`) in the
English language file.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The new endpoint `/config/reload` will refresh the
environment variables.
There are a bunch of env-vars which will not be reloaded
for e.g. HTTP_PORT etc. these will need a restart.
---------
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
match_all renamed to match_all_raw
match_all_indexed renamed to match_all
same pattern applied to *_ignore_case functions
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved clarity in function names for better understanding and
usability.
- **Tests**
- Updated SQL queries in test cases to reflect changes in function
names, ensuring accurate test results.
- **Chores**
- Adjusted logging statements to provide better insights during
debugging.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Improvements**
- Enhanced initialization process to prevent redundant cluster lock
creation when the existing OFGA model version matches the fetched
version. This improvement reduces unnecessary operations and improves
system efficiency.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
There are three cases to consider:
if ZO_ENABLE_EMBEDDED_REPORT_SERVER=true
If running a binary, alertmanager node will spawn an embedded server
running at localhost:5082.
All the requests from alert-manager will go to localhost:5082 for pdf
generation related tasks.
For single node k8s deployment, as this is running in a container there
is no chromium installed, so even if you set the variable to true, the
report server will panic due to missing dependencies. For this situation
helm chart will spawn a report-server instance which can be used.
Same for cluster deployment.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new report server for generating and sending reports.
- Added health check and report sending endpoints.
- **Improvements**
- Enhanced logging for report server operations.
- Configurable option to skip TLS verification for report server
connections.
- **Bug Fixes**
- Fixed formatting issue in HTTP status request handler.
- **Chores**
- Updated dependencies and configurations for the new report server
module.
- **Documentation**
- Added new structures and functions for email configuration, report
generation, and server operations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
add a new test environment:
```
ZO_DATAFUSION_PARQUET_STAT_DISABLE_FOR_AGGS=false
```
Set to `true` will skip collect stat for parquet files, then it will
faster.
And, i am thinking the final solution should be cache parquet
statistics, or we can optimize datafusion let it only collect stat for
registered schema fields.
---------
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
Fixes#3355
User must use a separate chrome server if `ZO_CHROME_PATH` is not
specified. `ZO_REPORT_SERVER_URL` is introduced to specify the report
server url.
- Add cluster selection in logs search queries for additional filtering
based on region/clusters
- add org/stream info in ingestion http handler error logging
---------
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
Fixes#3558
- [x] When SSO is enabled reports don't work , because of UI having two
login options
- [x] Scheduled job has end_time which is not current & retries are not
working
- [x] If a report is paused during execution , the reports status get
changed after execution to enabled = true again
- [x] After editing a report that was set to once to per min, the report
is not sent every minute
- [x] RBAC error while trying to enable/disable a report
Load small parquet files into memory to create a merged RecordBatch to
be sorted directly instead of using querying DataFusion for the same
purpose.
Merging 6 parquet files, each with 20 row by 30k columns
**Before**
openobserve::service::compact::merge: merge_parquet_files took: 35.473
openobserve::service::compact::merge: merge_parquet_files took: 37.249
openobserve::service::compact::merge: merge_parquet_files took: 38.337
openobserve::service::compact::merge: merge_parquet_files took: 38.759
openobserve::service::compact::merge: merge_parquet_files took: 38.920
**After**
openobserve::service::compact::merge: merge_parquet_files took 3.399
openobserve::service::compact::merge: merge_parquet_files took 3.781
openobserve::service::compact::merge: merge_parquet_files took 4.017
openobserve::service::compact::merge: merge_parquet_files took 4.162
openobserve::service::compact::merge: merge_parquet_files took 4.441
Taiming Liu
Hengfei Yang
Subhra264
Ashish Kolheandktx-abhay
ktx-vaidehiandktx-abhay
Ankur SrivastavaandHengfei Yang
