while importing the pipeline if the node is function or condition node
then we dont have org_id in the data object so we need to skip the
validation the node is fn or condition
1. some times due to same trace id for multiple rows when we expand it
will lead to expanding the same trace id rows (multiple) for that we had
a fix like combination of trace_id + sql which would be unique but due
to some times we might get same trace_id and sql as well
2. to fix this when we are receiving response we are dynamically
generating UUID for each hit and assigning it to each row so that we
will expand based on UUID
This PR add two new options for stream setting:
```
index_original_data: false
index_all_values: false
```
## index_original_data
Enable it will create Full text index on the original data.
> This will force enable `store_original_data: true`
## index_all_values
Enable it will create Full text index on the field `_all_values` which
collect all the fields values.
## Todo
- [x] Add options for index all
- [x] Handle ingestion for the new options
- [x] Handle search for the new options
- [x] exclude the `_all_values` for usage report
- [x] exclude `_all_values` and `_original` from search response for
`select *`
- [x] Performance testing for `index_original_data`
- [x] Performance testing for `index_all_values`
## index_original_data
`4.6`GB real data from production. ingest 10 times.
cpu: 800% mem: 17GB, took: 19.62s, 21.37s, 17.89s, avg: 19.63s
size:
```
9.1G data/openobserve/stream/files/default/index
18G data/openobserve/stream/files/default/logs
```
search:
```
SELECT count(*) FROM default where match_all('1724740197727131372')
```
total: 296790
took: 279ms, 274ms, 266ms, avg: 273ms
## index_all_values
`4.6`GB real data from production. ingest 10 times.
cpu: 900% mem: 15GB, took: 18.99s, 21.21s, 24.74s, avg: 21.65s
size:
```
8.9G data/openobserve/stream/files/default/index
17G data/openobserve/stream/files/default/logs
```
search:
```
SELECT count(*) FROM default where match_all('1724740197727131372')
```
total: 296790
took: 269ms, 275ms, 262ms, avg: 268.67ms
## Disable index all
`4.6`GB real data from production. ingest 10 times.
cpu: 600% mem: 12GB, took: 10.72s, 10.82s, 11.14s, avg: 10.89s
size:
```
0 --- no index
7.8G data/openobserve/stream/files/default/logs
```
search:
```
SELECT count(*) FROM default where match_all('1724740197727131372')
```
total: 0
no index, can't run this query.
## Test result
`index_all_values` compare to `index_original_data`, will use `90%` MEM,
but `110%` CPU, reduce `5%` of storage. search speed almost same.
But `index_all` compare to `none-index` will increased `50%` CPU and
`25%` MEM, also `triple` the storage size: ( `9GB` index, log data
`doubled`, because of store original_data or a new field with all
values).
---------
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
1. this PR fixes the sql mode query validation when they are not parsed
by the parser we use
2. in this case we are using node-sql parser so we will simply skip the
validation and pass it to BE
---------
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
1. This PR adds
**previous approach**
1. we were showing both All Fields and UDS tabs even user doesnot have
UDS enabled
2. UDS is shown as second tab
**current approach**
1. only show UDS if enabled otherwise show only All fields
3. if UDS enabled show it as first tab and instead of All Fields(when
UDS enabled) renamed it to Other Fields and also added tooltip with some
explanatory text after Other Fields with a info icon as
**_
> Other fields show only the schema fields that existed before the
stream was configured to use a user-defined schema.
_**
1. This PR handles the issue of alerts of being overridden when user
tries to shift the folders rapidly
example:
folder1 -> 100alerts
folder2 -> 2alerts
folder3 -> 0 alerts
so first users goes to folder2 and fetches the alerts meanwhile user
goes to folder1 and immediately switches to folder3 , lets assume the
folder3 alerts are already fetched and they are in store then it will
show folder1 alerts because api takes time to resolve, overrides the
current alerts after resolving so tries to show the wrong folder alerts
this PR fixes
1. histogram loader should while both search and histogram request as
well as no data found should only show when there is no actual data not
at the time of loading
2. in the detail table when we add a field to the table it again shows
add a field to the table instead should show remove from the table
3. when user try to add fields using quick mode with sql mode off there
is one console error
1. this PR fixes the issue that is related to when user tries to run
distinct query when histogram is on
eg:
`SELECT DISTINCT body FROM "default"`
currently it will show total events as if 50 records per page it will
show
1 to 50 out of 50 events
- [x] `Router-Client` if sending `pong` from `router` fails force close
the ws connection
- [ ] `Router-Querier` conn if `pong` is not received from querier for
certain time (ping_interval + 10 secs) then force close the connection
- [x] ws behavior should be such that if one querier disconnects from
the connection pool the entire `Router-Client` connection should not be
disconnected, searches/events should continue with other queriers in the
pool. Also, if one of the querier disconnects send `should_client_retry`
as `true`
---------
Co-authored-by: omkark06 <omkar@zinclabs.io>
- [x] fix: `session_drain` i.e. when a cookie expires no new `search`
events will be accepted by the router/server while the ongoing searches
will be drained to completion or allowed up to the
`ZO_WEBSOCKET_SESSION_IDLE_TIMEOUT_SECS` (default is 5 mins) after which
the session will be closed by the server.
- [x] track search duration for every `trace_id` and log it
---------
Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech>
Sai Nikhil
ktx-vaidehiandKirtan Thakkar
Omkar Kesarkhane
haryoiro
Hengfei YangandTaiming Liu
udevandomkark06
Abhay Padamani
Loakesh Indiranandomkark06
Ashish KolheandHengfei Yang
Zhao.wei
