- [x] fix: alert can't query latest data for separate alert query
- [x] feat: add size limit for download from other nodes
- [x] fix: use real time id generator
Refreshing in-memory cache in such NATs event trigger by NATs client's
callback closure.
---------
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
fixes https://github.com/openobserve/openobserve/issues/6621
This separates the functions which execute ddl statements, and only runs
them if certain entry in meta is mis-matching with what we have in our
code. Introduces a new env var `ZO_META_DDL_DSN` , which is optional and
falls back to the original DSN if not specified. Added a new client to
Postgres and Mysql both which uses this DDL DSN, and that is used only
for the ddl statements.
Also added another command to cli `upgrade-db` which runs the db
initialization functions.
For sqlite, there is no change, as it does not have concept of users.
Primarily tested on Postgres to validate that the behavior with
read-write and admin users is as expected.
`DB_SCHEMA_VERSION` is hardcoded in code, and devs are expected to
manually uptick it everytime we do any changes to db schema, similar to
how we maintain ofga.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
- [x] better lock handling at router
- [x] send `CancelResponse` if search task errors out due to `search
query cancelled`
- [x] optimize `remove_querier_connection` at router `SessionManager`
# Mini Partitions for Faster Initial Search Results
## Summary of Changes
This PR introduces "mini partitions" to improve user experience for
search queries by providing faster initial results, especially for large
time range queries.
## Old Partition Logic
- Created partitions of roughly equal size based on time range and data
volume
- Optimized for balanced workload distribution across processing nodes
- Users had to wait until a full partition was processed before seeing
any results
- For large time ranges (e.g., 7 days), the first result could take long
time to appear
## New Partition Logic
- Creates small "mini partitions" at the beginning of the time range
- Followed by regular-sized partitions for the remaining time range
- Preserves all the original constraints and optimizations for regular
partitions
- Users see initial results within seconds, even for large time range
queries
## New Env
```sh
ZO_MINI_SEARCH_PARTITION_DURATION_SECS: Duration of each mini partition in seconds (default: 60)
```
Now we have two ENV for control ingestion time:
- ZO_INGEST_ALLOWED_UPTO: old than x hours will discard
- ZO_INGEST_ALLOWED_IN_FUTURE: future than x hours will discard
```
ZO_INGEST_ALLOWED_UPTO=5
ZO_INGEST_ALLOWED_IN_FUTURE=24
```
Default is `5` hours and `24` hours.
- [x] ws v2 to use tokio `RwHashMap` instead of `Dashmap` to avoid dead
locks
- [x] `trace_id` when retry to be different after the first retry to get
a different querier from consistent hash
- [x] `user_id` for enterprise search event when run as single node
---------
Co-authored-by: loaki07 <rustyloaki@gmail.com>
- [x] `Router-Client` if sending `pong` from `router` fails force close
the ws connection
- [ ] `Router-Querier` conn if `pong` is not received from querier for
certain time (ping_interval + 10 secs) then force close the connection
- [x] ws behavior should be such that if one querier disconnects from
the connection pool the entire `Router-Client` connection should not be
disconnected, searches/events should continue with other queriers in the
pool. Also, if one of the querier disconnects send `should_client_retry`
as `true`
---------
Co-authored-by: omkark06 <omkar@zinclabs.io>
- [x] add `jitter` to `ping_interval` for heartbeats
- [x] deprecate `is_connected` in `Connection` trait
- [x] remove `maintain_connection`, instead the `health_check` tokio
tasks for each querier will do the similar functionality by manual heart
beats
- [x] add `clean_up` for `QuerrierConnectionPool` & `QuerierConnection`
with graceful shutdown
- [x] use `ZO_INSTANCE_NAME` as the `req_id` for the connection between
the `Rotuer-Querier`
- [x] remove ws v1
When introducing service accounts rbac, we didn't do a rbac openfga
migration to save openfga tuples (`_all_{org_id}`) for existing
organizations. Hence as a result, viewer and editor are not able to
view/edit service accounts by default. Currently, they can do it only
after given service accounts explicitly through the roles edit UI page.
This pr fixes this by adding a migration which saves `_all_{org_id}`
tuples for every organizations.
## What is changed
- [x] Try to improve task starving issue
- [x] Update mmdb frequency
- [x] Improve slow log detection
## MMDB
- Removed ENV `ZO_MMDB_UPDATE_DURATION` because is in seconds
- Add new ENV `ZO_MMDB_UPDATE_DURATION_DAYS` is in days
## Tokio task
We try to use `task::coop` to improve task starving issue in Tokio.
<img width="1496" alt="image"
src="https://github.com/user-attachments/assets/1666488a-146b-49d7-b460-c629528e06f9"
/>
The second chart is for this PR, it is a big slow but more stable.
This PR has some improves:
- Improve tantivy generate, from 7.7s -> 5.9s
- Improve STREAM_SETTINGS cache with atomic
- Add process metrics
- Remove zo_node_open_fd, node_tcp_conn_resets( use
`node_tcp_connections{state="resets"}`)
- Add debug logs for bulk api
- Add debug logs for tantivy panic
This API will return cluster specific information. This iteration will
report the pending compaction jobs in this format.
```
{
"regions": {
"openobserve": {
"zo1": {
"pending_jobs": 0
}
}
}
}
```
API supports filtering by region.
- Fixes 403 error when alert move api is called
- 403 error when trying to create alert in default folder even after
having permission
- Fixes # Regular user without `GET` user permission should not be able
to get list of all users
- Fixes#6289
- Also, Removes use of semaphore inside the scheduler. Currently it only
uses the mpsc channel and waits if the mpsc channel is full. Apart from
this, it uses db locks in postgres and mysql when pulling
scheduled_jobs.
udev
Hengfei Yang
Subhra264andsai nikhil kethe
Taiming Liu
Ashish Kolhe
Loakesh Indiran
Zhao.weiandhengfei yang
