while importing the pipeline if the node is function or condition node
then we dont have org_id in the data object so we need to skip the
validation the node is fn or condition
These tests include
- Added a new alert
- Add a new folder
- Added a new alert in this folder
- Moved 1 alert in 1 folder
- Bulk move alerts from default folder to new created folder
- Deleted the alerts that were moved to new folder
- Deleted the folder
- Deleted the template and destination
Adds aligntime feature for alerts periods. If the frequency and period
is 5 mins, current time is 11:53pm, it will use the period of 11:45pm to
11:50pm to evaluate the alert. The next time it will be triggered at at
11:55pm, and data evaluation timerange will be 11:50pm to 11:55pm
1. some times due to same trace id for multiple rows when we expand it
will lead to expanding the same trace id rows (multiple) for that we had
a fix like combination of trace_id + sql which would be unique but due
to some times we might get same trace_id and sql as well
2. to fix this when we are receiving response we are dynamically
generating UUID for each hit and assigning it to each row so that we
will expand based on UUID
This PR add two new options for stream setting:
```
index_original_data: false
index_all_values: false
```
## index_original_data
Enable it will create Full text index on the original data.
> This will force enable `store_original_data: true`
## index_all_values
Enable it will create Full text index on the field `_all_values` which
collect all the fields values.
## Todo
- [x] Add options for index all
- [x] Handle ingestion for the new options
- [x] Handle search for the new options
- [x] exclude the `_all_values` for usage report
- [x] exclude `_all_values` and `_original` from search response for
`select *`
- [x] Performance testing for `index_original_data`
- [x] Performance testing for `index_all_values`
## index_original_data
`4.6`GB real data from production. ingest 10 times.
cpu: 800% mem: 17GB, took: 19.62s, 21.37s, 17.89s, avg: 19.63s
size:
```
9.1G data/openobserve/stream/files/default/index
18G data/openobserve/stream/files/default/logs
```
search:
```
SELECT count(*) FROM default where match_all('1724740197727131372')
```
total: 296790
took: 279ms, 274ms, 266ms, avg: 273ms
## index_all_values
`4.6`GB real data from production. ingest 10 times.
cpu: 900% mem: 15GB, took: 18.99s, 21.21s, 24.74s, avg: 21.65s
size:
```
8.9G data/openobserve/stream/files/default/index
17G data/openobserve/stream/files/default/logs
```
search:
```
SELECT count(*) FROM default where match_all('1724740197727131372')
```
total: 296790
took: 269ms, 275ms, 262ms, avg: 268.67ms
## Disable index all
`4.6`GB real data from production. ingest 10 times.
cpu: 600% mem: 12GB, took: 10.72s, 10.82s, 11.14s, avg: 10.89s
size:
```
0 --- no index
7.8G data/openobserve/stream/files/default/logs
```
search:
```
SELECT count(*) FROM default where match_all('1724740197727131372')
```
total: 0
no index, can't run this query.
## Test result
`index_all_values` compare to `index_original_data`, will use `90%` MEM,
but `110%` CPU, reduce `5%` of storage. search speed almost same.
But `index_all` compare to `none-index` will increased `50%` CPU and
`25%` MEM, also `triple` the storage size: ( `9GB` index, log data
`doubled`, because of store original_data or a new field with all
values).
---------
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
1. this PR fixes the sql mode query validation when they are not parsed
by the parser we use
2. in this case we are using node-sql parser so we will simply skip the
validation and pass it to BE
---------
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
API layer enforcement: scheduled pipeline queries must generate a
`_timestamp` filed in the result, use alias for functions.
recommended:
```sql
SELECT MAX(_timestamp) AS _timestamp, ... FROM ...
```
***This rule is not applied to any existing scheduled pipelines**
```
.country_data =get_enrichment_table_record!("countries",
{
"region_code":"142",
"country_code":"4"
},
case_sensitive: true)
```
Earlier it would consider only first condition , now it considers all
This pr aligns the next_run_at of the alerts/pipelines according to the
timezone of the given alert/pipeline. For example, if the current time
is 11:03pm, and the frequency is 5 mins, the next run at should be
11:05pm, then 11:10pm, and so on. In future, we may want to make
aligntime an optional feature, by giving a toggle option in the
alert/pipeline itself.
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
# Mini Partitions for Faster Initial Search Results
## Summary of Changes
This PR introduces "mini partitions" to improve user experience for
search queries by providing faster initial results, especially for large
time range queries.
## Old Partition Logic
- Created partitions of roughly equal size based on time range and data
volume
- Optimized for balanced workload distribution across processing nodes
- Users had to wait until a full partition was processed before seeing
any results
- For large time ranges (e.g., 7 days), the first result could take long
time to appear
## New Partition Logic
- Creates small "mini partitions" at the beginning of the time range
- Followed by regular-sized partitions for the remaining time range
- Preserves all the original constraints and optimizations for regular
partitions
- Users see initial results within seconds, even for large time range
queries
## New Env
```sh
ZO_MINI_SEARCH_PARTITION_DURATION_SECS: Duration of each mini partition in seconds (default: 60)
```
Now we have two ENV for control ingestion time:
- ZO_INGEST_ALLOWED_UPTO: old than x hours will discard
- ZO_INGEST_ALLOWED_IN_FUTURE: future than x hours will discard
```
ZO_INGEST_ALLOWED_UPTO=5
ZO_INGEST_ALLOWED_IN_FUTURE=24
```
Default is `5` hours and `24` hours.
- When the report cron expression is invalid, return a proper error
message saying the cron expression is invalid.
- When uploading Enrichment table check the payload size irrespective of
append=false/true.
Sai Nikhil
Loakesh IndiranandSubhradeep Chakraborty
ktx-vaidehiandKirtan Thakkar
Omkar Kesarkhane
Hengfei Yang
udev
Zhao.wei
Neha P
haryoiro
dependabot[bot]
ktx-akshay
Ashish Kolhe
Abhay Padamani
Shyam Panjiyar