Refreshing in-memory cache in such NATs event trigger by NATs client's
callback closure.
---------
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
There is a bug in the handling of super cluster scheduler event messages
for derived streams. The `trigger_modify_module_key` function which was
supposed to be executed for alert messages only, also gets applied to
scheduled pipeline events containing a different module key format than
desired. Hence, it keeps throwing error in the compactor. This pr fixes
this issue.
In PR for separating ddl client from normal db client, a new function
was created which would do all db initialization. Consequently, fn call
to create the local db dir was also moved into that function. However,
this function is called only if version key in db is mismatch, so if the
version in db matches, this function would not be called, and thus db
dir creation will aalso not happen.
This works on stateful set, because most of the time db dir will already
be present. However, for non-stateful sets such as router or compactor,
this errors because of missing db dir.
This PR fixes this by always calling create_dir_all in infra::init. They
way things are setup, this will get called twice if f db_init also runs,
but that is ok. Tested the commit on dev3/dev4 for ~2 hours, container
init and restart work fine with this change.
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
fixes https://github.com/openobserve/openobserve/issues/6621
This separates the functions which execute ddl statements, and only runs
them if certain entry in meta is mis-matching with what we have in our
code. Introduces a new env var `ZO_META_DDL_DSN` , which is optional and
falls back to the original DSN if not specified. Added a new client to
Postgres and Mysql both which uses this DDL DSN, and that is used only
for the ddl statements.
Also added another command to cli `upgrade-db` which runs the db
initialization functions.
For sqlite, there is no change, as it does not have concept of users.
Primarily tested on Postgres to validate that the behavior with
read-write and admin users is as expected.
`DB_SCHEMA_VERSION` is hardcoded in code, and devs are expected to
manually uptick it everytime we do any changes to db schema, similar to
how we maintain ofga.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
Currently, the scheduler trace_ids are not searchable in the traces
page, because they are only used in the logs and not in the
instrumentation. This pr fixes that - so, the scheduler query trace id
(which can be obtained from the `triggers` stream, field name -
`scheduler_trace_id`, the trace_id after the `/` is the scheduler query
trace_id), can be used as `trace_id` to search the trace in the traces
page.
Also, the `grpc::search` and `grpc::search_multi` does not use the
trace_id from the grpc "request" for instrumentation. This pr fixes that
as well.
---------
Co-authored-by: hengfei yang <hengfei.yang@gmail.com>
We recently started using grpc search for alerts inside alert manager.
Now, the grpc search uses `cache::search` (and `service::search` when no
data in cache). `service::search` already reports usage for alert search
event, and also the same is done by `cache::search` as well. Hence,
there are duplicate search usage reporting for alerts. In this pr, only
the `cache::search` reports usage in case of alerts.
Recently a label was added to one of our metrics in main. In file dump
PR I had added couple of new DB functions which uses this metric,
however when testing the label changes were not merged in main yet.
Thus, I did not catch that the additional label needed to be added, and
now it panics because of incorrect labels. Fix is simple, add an empty
string for the missing label. No other change. Tested it on usertest for
~12 hours, didn't see any issues.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
- When in the alert, frequency is changed from cron to minutes, we don't
delete the old cron expression. This pr fixes that.
- Fixes false reporting of skipped alert events
- [x] fix: add retry if download file get size mismatch
- [x] fix: delete from file_cache index if one file delete from disk
- [x] fix: remove trace_id for file_cache and download, it is noise
- [x] fix: download file other node, add file size mismatch check, and
skip the not found error
We should extent slice not replace slice for download with streaming.
Testing this feature need to enable ENV:
```
ZO_CACHE_LATEST_FILES_ENABLED = true
ZO_CACHE_LATEST_FILES_DOWNLOAD_FROM_NODE=true
```
- [x] fix: `tokio_tungstenite` max msg limit config
- [x] honor user cancel in ui (logs, dashboards) if
`should_client_retry` is `true`, and is in loop.
- [x] fix: `values` event to send `cancel` response if search query
cancelled
- [x] handle `values` event error in ui
---------
Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: udev <uddhav@openobserve.ai>
This pr uses the meta table to store the start_time and end_time of
enrichment table, instead of relying on the stream_stats table which
takes some time to update. When we cache enrichment table for vrls, we
use the above start_time and end_time instead of `BASE_TIMESTAMP`
(because when enrichment table is overridden, deletion of old data
happens in the background and is not realtime, so using `BASE_TIMESTAMP`
included old data)
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
This implements the file list dump feature, where we dump the file list
by hour after compaction into parquet files and use those files while
searching through files lists. This also adds following variables -
- ZO_FILE_LIST_DUMP_ENABLED : if this dumping is enabled or not, default
is false
- ZO_FILE_LIST_DUMP_DUAL_WRITE : should the file list entries be kept in
the file list even after dumping. default is true, recommended is false.
Default can be changed after more verification.
- ZO_FILE_LIST_DUMP_MIN_HOUR : number of hours in past, only files older
than this will be considered for dumping, to avoid hot files issues.
The dump files entries themselves are stored in the file_list table as
was decided, so they also become a part of the normal compactor flow.
However, there is check to ensure these files are not dumped.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
- [x] better lock handling at router
- [x] send `CancelResponse` if search task errors out due to `search
query cancelled`
- [x] optimize `remove_querier_connection` at router `SessionManager`
Hengfei Yang
Ashish Kolhe
Taiming Liu
Subhra264
Abhay Padamaniandloaki07
Abhishek Tripathi
Yashodhan JoshiandYashodhan Joshi
ByteBaker
Zhao.wei
