This pr uses the meta table to store the start_time and end_time of
enrichment table, instead of relying on the stream_stats table which
takes some time to update. When we cache enrichment table for vrls, we
use the above start_time and end_time instead of `BASE_TIMESTAMP`
(because when enrichment table is overridden, deletion of old data
happens in the background and is not realtime, so using `BASE_TIMESTAMP`
included old data)
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
1. This PR fixes the issue that sometimes BE may return id's in non
sorting manner which is used in FE so now we are making the indexing in
the FE so that it will always be in sorted manner
2.it was handled in both normal cluster and super cluster as well
This implements the file list dump feature, where we dump the file list
by hour after compaction into parquet files and use those files while
searching through files lists. This also adds following variables -
- ZO_FILE_LIST_DUMP_ENABLED : if this dumping is enabled or not, default
is false
- ZO_FILE_LIST_DUMP_DUAL_WRITE : should the file list entries be kept in
the file list even after dumping. default is true, recommended is false.
Default can be changed after more verification.
- ZO_FILE_LIST_DUMP_MIN_HOUR : number of hours in past, only files older
than this will be considered for dumping, to avoid hot files issues.
The dump files entries themselves are stored in the file_list table as
was decided, so they also become a part of the normal compactor flow.
However, there is check to ensure these files are not dumped.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
- [x] better lock handling at router
- [x] send `CancelResponse` if search task errors out due to `search
query cancelled`
- [x] optimize `remove_querier_connection` at router `SessionManager`
while importing the pipeline if the node is function or condition node
then we dont have org_id in the data object so we need to skip the
validation the node is fn or condition
These tests include
- Added a new alert
- Add a new folder
- Added a new alert in this folder
- Moved 1 alert in 1 folder
- Bulk move alerts from default folder to new created folder
- Deleted the alerts that were moved to new folder
- Deleted the folder
- Deleted the template and destination
Adds aligntime feature for alerts periods. If the frequency and period
is 5 mins, current time is 11:53pm, it will use the period of 11:45pm to
11:50pm to evaluate the alert. The next time it will be triggered at at
11:55pm, and data evaluation timerange will be 11:50pm to 11:55pm
1. some times due to same trace id for multiple rows when we expand it
will lead to expanding the same trace id rows (multiple) for that we had
a fix like combination of trace_id + sql which would be unique but due
to some times we might get same trace_id and sql as well
2. to fix this when we are receiving response we are dynamically
generating UUID for each hit and assigning it to each row so that we
will expand based on UUID
This PR add two new options for stream setting:
```
index_original_data: false
index_all_values: false
```
## index_original_data
Enable it will create Full text index on the original data.
> This will force enable `store_original_data: true`
## index_all_values
Enable it will create Full text index on the field `_all_values` which
collect all the fields values.
## Todo
- [x] Add options for index all
- [x] Handle ingestion for the new options
- [x] Handle search for the new options
- [x] exclude the `_all_values` for usage report
- [x] exclude `_all_values` and `_original` from search response for
`select *`
- [x] Performance testing for `index_original_data`
- [x] Performance testing for `index_all_values`
## index_original_data
`4.6`GB real data from production. ingest 10 times.
cpu: 800% mem: 17GB, took: 19.62s, 21.37s, 17.89s, avg: 19.63s
size:
```
9.1G data/openobserve/stream/files/default/index
18G data/openobserve/stream/files/default/logs
```
search:
```
SELECT count(*) FROM default where match_all('1724740197727131372')
```
total: 296790
took: 279ms, 274ms, 266ms, avg: 273ms
## index_all_values
`4.6`GB real data from production. ingest 10 times.
cpu: 900% mem: 15GB, took: 18.99s, 21.21s, 24.74s, avg: 21.65s
size:
```
8.9G data/openobserve/stream/files/default/index
17G data/openobserve/stream/files/default/logs
```
search:
```
SELECT count(*) FROM default where match_all('1724740197727131372')
```
total: 296790
took: 269ms, 275ms, 262ms, avg: 268.67ms
## Disable index all
`4.6`GB real data from production. ingest 10 times.
cpu: 600% mem: 12GB, took: 10.72s, 10.82s, 11.14s, avg: 10.89s
size:
```
0 --- no index
7.8G data/openobserve/stream/files/default/logs
```
search:
```
SELECT count(*) FROM default where match_all('1724740197727131372')
```
total: 0
no index, can't run this query.
## Test result
`index_all_values` compare to `index_original_data`, will use `90%` MEM,
but `110%` CPU, reduce `5%` of storage. search speed almost same.
But `index_all` compare to `none-index` will increased `50%` CPU and
`25%` MEM, also `triple` the storage size: ( `9GB` index, log data
`doubled`, because of store original_data or a new field with all
values).
---------
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
1. this PR fixes the sql mode query validation when they are not parsed
by the parser we use
2. in this case we are using node-sql parser so we will simply skip the
validation and pass it to BE
---------
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
API layer enforcement: scheduled pipeline queries must generate a
`_timestamp` filed in the result, use alias for functions.
recommended:
```sql
SELECT MAX(_timestamp) AS _timestamp, ... FROM ...
```
***This rule is not applied to any existing scheduled pipelines**
```
.country_data =get_enrichment_table_record!("countries",
{
"region_code":"142",
"country_code":"4"
},
case_sensitive: true)
```
Earlier it would consider only first condition , now it considers all
Taiming Liuandsai nikhil kethe
udev
Ashish KolheandBhargav
Loakesh Indiran
ktx-akshay
Shyam PanjiyarandHengfei Yang
Omkar Kesarkhane
ktx-vaidehiandKirtan Thakkar
Zhao.wei
Neha P
haryoiro
dependabot[bot]
