Previously we did a footer check even when after all retries the blob
download was partial. Now we skip and return error, as no sense in
checking footer when we know the blob size is incomplete.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
- [x] fix: alert can't query latest data for separate alert query
- [x] feat: add size limit for download from other nodes
- [x] fix: use real time id generator
- get file size from db and download file from blob
- if mismatch in size between db and file downloaded then
- get content length of file from blob
- if content length != file downloaded -> we had a partial download from
blob -> retry download with exponential backoff
- if content length == file downloaded -> check footer of downloaded
file to see if parquet in blob is actually corrupt
- if file is corrupt -> log as info and mark deleted in db
- if file is good -> update size in db
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
This adds the error header to all cases of internal server error so they
show up properly in audit. This is done by one of two approaches -
1. preferably using the `map_error_to_http_response` function , so that
everything goes via the same function
2. directly adding the header where the above function cannot be used
because is is not an error per say like trying to ingest at
non-ingester.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Refreshing in-memory cache in such NATs event trigger by NATs client's
callback closure.
---------
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
There is a bug in the handling of super cluster scheduler event messages
for derived streams. The `trigger_modify_module_key` function which was
supposed to be executed for alert messages only, also gets applied to
scheduled pipeline events containing a different module key format than
desired. Hence, it keeps throwing error in the compactor. This pr fixes
this issue.
In PR for separating ddl client from normal db client, a new function
was created which would do all db initialization. Consequently, fn call
to create the local db dir was also moved into that function. However,
this function is called only if version key in db is mismatch, so if the
version in db matches, this function would not be called, and thus db
dir creation will aalso not happen.
This works on stateful set, because most of the time db dir will already
be present. However, for non-stateful sets such as router or compactor,
this errors because of missing db dir.
This PR fixes this by always calling create_dir_all in infra::init. They
way things are setup, this will get called twice if f db_init also runs,
but that is ok. Tested the commit on dev3/dev4 for ~2 hours, container
init and restart work fine with this change.
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
fixes https://github.com/openobserve/openobserve/issues/6621
This separates the functions which execute ddl statements, and only runs
them if certain entry in meta is mis-matching with what we have in our
code. Introduces a new env var `ZO_META_DDL_DSN` , which is optional and
falls back to the original DSN if not specified. Added a new client to
Postgres and Mysql both which uses this DDL DSN, and that is used only
for the ddl statements.
Also added another command to cli `upgrade-db` which runs the db
initialization functions.
For sqlite, there is no change, as it does not have concept of users.
Primarily tested on Postgres to validate that the behavior with
read-write and admin users is as expected.
`DB_SCHEMA_VERSION` is hardcoded in code, and devs are expected to
manually uptick it everytime we do any changes to db schema, similar to
how we maintain ofga.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
Currently, the scheduler trace_ids are not searchable in the traces
page, because they are only used in the logs and not in the
instrumentation. This pr fixes that - so, the scheduler query trace id
(which can be obtained from the `triggers` stream, field name -
`scheduler_trace_id`, the trace_id after the `/` is the scheduler query
trace_id), can be used as `trace_id` to search the trace in the traces
page.
Also, the `grpc::search` and `grpc::search_multi` does not use the
trace_id from the grpc "request" for instrumentation. This pr fixes that
as well.
---------
Co-authored-by: hengfei yang <hengfei.yang@gmail.com>
We recently started using grpc search for alerts inside alert manager.
Now, the grpc search uses `cache::search` (and `service::search` when no
data in cache). `service::search` already reports usage for alert search
event, and also the same is done by `cache::search` as well. Hence,
there are duplicate search usage reporting for alerts. In this pr, only
the `cache::search` reports usage in case of alerts.
Recently a label was added to one of our metrics in main. In file dump
PR I had added couple of new DB functions which uses this metric,
however when testing the label changes were not merged in main yet.
Thus, I did not catch that the additional label needed to be added, and
now it panics because of incorrect labels. Fix is simple, add an empty
string for the missing label. No other change. Tested it on usertest for
~12 hours, didn't see any issues.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
- When in the alert, frequency is changed from cron to minutes, we don't
delete the old cron expression. This pr fixes that.
- Fixes false reporting of skipped alert events
- [x] fix: add retry if download file get size mismatch
- [x] fix: delete from file_cache index if one file delete from disk
- [x] fix: remove trace_id for file_cache and download, it is noise
- [x] fix: download file other node, add file size mismatch check, and
skip the not found error
We should extent slice not replace slice for download with streaming.
Testing this feature need to enable ENV:
```
ZO_CACHE_LATEST_FILES_ENABLED = true
ZO_CACHE_LATEST_FILES_DOWNLOAD_FROM_NODE=true
```
Huaijinandgreptile-apps[bot]
Taiming Liu
udev
Ashish Kolheandhengfei yang
Loakesh Indiran
Abhay Padamaniandloaki07
Abhishek Tripathi
ByteBaker
Zhao.wei