1. this PR fixes not showing what is the search field that was not found
in both logs page and dashbaords panel
2. alert list pagination fix
---------
Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech>
1. This PR fixes the minor issues that are there in node list UI
2. --> when the node is offline or prepare state then the border to the
particular is not shown
3. --> change header of TCP to TCP Connections in table
- [x] fix: `tokio_tungstenite` max msg limit config
- [x] honor user cancel in ui (logs, dashboards) if
`should_client_retry` is `true`, and is in loop.
- [x] fix: `values` event to send `cancel` response if search query
cancelled
- [x] handle `values` event error in ui
---------
Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: udev <uddhav@openobserve.ai>
1. This PR fixes the issue that sometimes BE may return id's in non
sorting manner which is used in FE so now we are making the indexing in
the FE so that it will always be in sorted manner
2.it was handled in both normal cluster and super cluster as well
while importing the pipeline if the node is function or condition node
then we dont have org_id in the data object so we need to skip the
validation the node is fn or condition
1. some times due to same trace id for multiple rows when we expand it
will lead to expanding the same trace id rows (multiple) for that we had
a fix like combination of trace_id + sql which would be unique but due
to some times we might get same trace_id and sql as well
2. to fix this when we are receiving response we are dynamically
generating UUID for each hit and assigning it to each row so that we
will expand based on UUID
This PR add two new options for stream setting:
```
index_original_data: false
index_all_values: false
```
## index_original_data
Enable it will create Full text index on the original data.
> This will force enable `store_original_data: true`
## index_all_values
Enable it will create Full text index on the field `_all_values` which
collect all the fields values.
## Todo
- [x] Add options for index all
- [x] Handle ingestion for the new options
- [x] Handle search for the new options
- [x] exclude the `_all_values` for usage report
- [x] exclude `_all_values` and `_original` from search response for
`select *`
- [x] Performance testing for `index_original_data`
- [x] Performance testing for `index_all_values`
## index_original_data
`4.6`GB real data from production. ingest 10 times.
cpu: 800% mem: 17GB, took: 19.62s, 21.37s, 17.89s, avg: 19.63s
size:
```
9.1G data/openobserve/stream/files/default/index
18G data/openobserve/stream/files/default/logs
```
search:
```
SELECT count(*) FROM default where match_all('1724740197727131372')
```
total: 296790
took: 279ms, 274ms, 266ms, avg: 273ms
## index_all_values
`4.6`GB real data from production. ingest 10 times.
cpu: 900% mem: 15GB, took: 18.99s, 21.21s, 24.74s, avg: 21.65s
size:
```
8.9G data/openobserve/stream/files/default/index
17G data/openobserve/stream/files/default/logs
```
search:
```
SELECT count(*) FROM default where match_all('1724740197727131372')
```
total: 296790
took: 269ms, 275ms, 262ms, avg: 268.67ms
## Disable index all
`4.6`GB real data from production. ingest 10 times.
cpu: 600% mem: 12GB, took: 10.72s, 10.82s, 11.14s, avg: 10.89s
size:
```
0 --- no index
7.8G data/openobserve/stream/files/default/logs
```
search:
```
SELECT count(*) FROM default where match_all('1724740197727131372')
```
total: 0
no index, can't run this query.
## Test result
`index_all_values` compare to `index_original_data`, will use `90%` MEM,
but `110%` CPU, reduce `5%` of storage. search speed almost same.
But `index_all` compare to `none-index` will increased `50%` CPU and
`25%` MEM, also `triple` the storage size: ( `9GB` index, log data
`doubled`, because of store original_data or a new field with all
values).
---------
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
1. this PR fixes the sql mode query validation when they are not parsed
by the parser we use
2. in this case we are using node-sql parser so we will simply skip the
validation and pass it to BE
---------
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
1. This PR adds
**previous approach**
1. we were showing both All Fields and UDS tabs even user doesnot have
UDS enabled
2. UDS is shown as second tab
**current approach**
1. only show UDS if enabled otherwise show only All fields
3. if UDS enabled show it as first tab and instead of All Fields(when
UDS enabled) renamed it to Other Fields and also added tooltip with some
explanatory text after Other Fields with a info icon as
**_
> Other fields show only the schema fields that existed before the
stream was configured to use a user-defined schema.
_**
1. This PR handles the issue of alerts of being overridden when user
tries to shift the folders rapidly
example:
folder1 -> 100alerts
folder2 -> 2alerts
folder3 -> 0 alerts
so first users goes to folder2 and fetches the alerts meanwhile user
goes to folder1 and immediately switches to folder3 , lets assume the
folder3 alerts are already fetched and they are in store then it will
show folder1 alerts because api takes time to resolve, overrides the
current alerts after resolving so tries to show the wrong folder alerts
udev
Sai Nikhil
Abhay Padamaniandktx-akshay
ktx-vaidehi
Subhra264andsai nikhil kethe
Omkar Kesarkhane
Bhargav
dependabot[bot]
Taiming Liuandsai nikhil kethe
Ashish KolheandBhargav
haryoiro
Hengfei YangandTaiming Liu