- [x] feat: use local node for load enrichmentable
- [x] feat: improve search around with multiple streams
- [x] perf: skip the querier with have no file to query
Currently, individual level RBAC permission does not work for
dashboards. This pr fixes that. So a role permission like below
<img width="1399" alt="Screenshot 2025-03-04 at 5 27 11 PM"
src="https://github.com/user-attachments/assets/06c2030d-4227-4109-b73f-dc2337151362"
/>
will allow the user to access only the dashboards with `Get` and `All`
permission even if the `Get` permission for the folder is not specified.
Ref: https://github.com/openobserve/openobserve/issues/5974
Issue: Close frame description field causes frame malformation in
Chrome/Brave
- Description field causes inconsistent frame encoding
- Firefox handles it fine, but Chrome/Brave show malformed data
- Workaround: Skip description field in close frames
Actions:
- Set `CloseFrame` description at both `router` & `querier` to `None`
Makes the following fixes to these routes:
- `ListAlerts`
- adds `enabled` to the response body
- adds `name_substring` to the request body
- `MoveAlerts`
- changes the method type to `PATCH` since this is a non-idempotent
update
- changing to `PATCH` coincidentally happens to fix a routing issue
where requests to `MoveAlerts` were being routed to the `UpdateAlerts`
handler
- `EnableAlert`
- changes the method type to `PATCH` since this is a non-idempotent
update
- `TriggerAlert`
- changes the method type to `PATCH` since this is a non-idempotent
update
## Goal:
- The goal is to not sleep/timeout but also wait a tiny bit before
sending close and at the same time not keeping the threads hanging which
could lead to resource exhaustion on the querier
## Changes:
1. **Message In-Flight Control**:
```rust
message_in_flight: AtomicBool
```
- Uses atomic boolean for lightweight synchronization
- Ensures only one message is being sent at a time
- Prevents message overlap and potential race conditions
2. **Limited Retry Mechanism**:
```rust
let mut attempts = 0;
while attempts < 3 {
match session.message_in_flight.compare_exchange(...) {
// ...
Err(_) => {
attempts += 1;
if attempts < 3 {
tokio::task::yield_now().await;
}
}
}
}
```
- Maximum 3 retries
- Uses `yield_now()` instead of sleep
- Quick failure path if message can't be sent
3. **Non-Blocking Cleanup Before Sending Close**:
```rust
while session.message_in_flight.load(Ordering::SeqCst) && retries < 3 {
tokio::task::yield_now().await;
retries += 1;
}
```
- Limited retries during cleanup
- Non-blocking approach to message completion
- Proceeds with close even if message is still in flight
**Why This Prevents Querier Resource Exhaustion**:
1. No thread blocking or sleeping
2. Limited retry attempts (max 3)
3. Quick failure paths
4. `yield_now()` only gives up CPU time slice, doesn't block
5. Resources are cleaned up even on failure
1. this PR introduces new design for query node
2. dynamic stream name for destination node instead of create new stream
it allows user to input any non existing stream name as destination and
at the time of ingestion it will be created dynamically
3. Fixes issues of #5886
---------
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
- [x] Add support for alerts
- [x] Add support for search
- [x] Add Test API
- [x] Improve Update API
---------
Co-authored-by: Yashodhan Joshi <yashodhan@openobserve.ai>
New Env:
```sh
# default 0ms
# preferred to be set at 10ms to 100ms
# milliseconds
ZO_WEBSOCKET_CLOSE_FRAME_DELAY=10
```
Changes:
- [x] add support to dashboard for streaming aggregate queries from in
memory cache
- [x] new sort logic for web socket search results
> - [x] introduced a new `optional` parameter `fallback_order_by_col`
for ws search req `SearchEventReq`
> - For queries where there is no `_timestamp` field and no `order_by`
specified:
```sql
SELECT level, COUNT(*) as count
FROM logs
GROUP BY level
```
> - The default sql behavior is to return the results in a non-ordered
manner.
> - However, with web sockets this behavior impacts the UX in both
`logs` and `dashboards` when `streaming aggregates` is `enabled` as the
results in page/dashboards will keep fluctuating.
> - To fix this there a new sort behavior introduced in web sockets
```rust
/// Represents different sorting strategies
enum SortStrategy {
SqlOrderBy,
FallbackColumn(String, OrderBy),
AutoDetermine(String, bool), // (column, is_string)
NoSort,
}
```
> 1. If `order_by` is already specified in the sql query and the search
service has populated the `order_by` field in its response for the
search, then there is no sort action required on the search results.
> 2. If `fallback_order_by_col` is specified that col is selected for
sort. This will mainly be used in `Dashboards`.
> 3. If none of the above is specified, there is `auto-determine`
strategy which would check for the `first non-numeric col` else the
`first numeric col`
> 4. `non-numeric` col i.e. `text/string` is always sorted in `asc`.
While, `numeric` col is sorted in `desc`. This applies to both
`FallbackColumn` & `AutoDetermine` strategies.
- [x] redesign web socket search event handling
> - Previously the `cleanup_and_close_session` used to happen inside the
search task thread. Which is suspected :suspect: to be one of the causes
for the `CloseFrame` description and the `DataFrame` data to get merged
and malform the message in the socket.
> - New behavior `cleanup_and_close_session` will happen outside of the
search task spawned once search is complete.
> - Since its suspected :suspect: that there is some race condition 🏃🏾
when the web socket frames are being sent the env variable
`ZO_WEBSOCKET_CLOSE_FRAME_DELAY` default `0` is created to introduce a
delay preferred `10 ms` to ` 100ms` before sending the `CloseFrame`
> - `SEARCH_REGISTRY` now will hold the different states of search which
are `Running`, `Cancelled` & `Completed`
> - The search task is modified to now listen for a `cancel` from the
`mpsc` channel and does `cleanup_search_resources`
Related Issues:
- #5967
- #5974
---------
Co-authored-by: loaki07 <rustyloaki@gmail.com>
Co-authored-by: Loakesh Indiran <66478092+Loaki07@users.noreply.github.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
Fixes an issue reported by user where if the field name is a reserved
keyword in sql, the values api would error out. By adding quotes around
the field, we specifically indicate that it is a field name and not a
kw, so it works. Also checked for non-kw names, also works.
fixes#5917
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
The`Sql::new()` has optimization where it removes all the filters in
sql, when the tantivy index is enable.
By using the sql from the original req, the above scenario can be
avoided.
- [x] Add a gRPC API for get stats from other cluster
- [x] Update stream stats cache logic to get stats from all of the
clusters.
---------
Co-authored-by: hengfei yang <hengfei.yang@gmail.com>
Note: This PR has an [accompanying
PR](https://github.com/openobserve/o2-enterprise/pull/331) in the
enterprise repo
This splits the `openfga` and `dex` modules into their own crates,
`o2_openfga` and `o2_dex`.
**Why?**
We want to be able to reference the `o2_enterprise::openfga` module from
within the `infra::table` module so that we can perform OpenFGA data
migrations as part of data transformation migrations in the
`openobserve` app. However, the `o2_enterprise` module references the
`infra::table` module in various places (eg: inside
`o2_enterprise::enterprise::actions::action_manager`). Therefore we
cannot reference `o2_enterprise::enterprise::openfga` from
`infra::table` without introducing a circular dependency.
The solution presented by this PR is to move the `openfga` module into
its own crate, so that `infra::table` can reference the new create
without creating a circular dependency. As part of this PR we also move
the `dex` module into its own crate since the `openfga` module depends
on the `dex` module, and we do not want the new `o2_openfga` crate to
depend on the `o2_enterprise` crate.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Refactor**
- Simplified the construction of search queries to ensure consistent
behavior without altering error notifications.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- #5844
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
Based on the comprehensive summary of changes, here are the updated
release notes:
- **New Features**
- Added timed annotations functionality for dashboards.
- Introduced ability to create, edit, and delete annotations on panels.
- Implemented annotation management across frontend and backend systems.
- **Improvements**
- Enhanced chart rendering to support annotation visualization.
- Added new API endpoints for annotation operations.
- Improved dashboard interaction with annotation mode.
- **Backend Changes**
- Created new database migrations for timed annotations and timed
annotation panels.
- Added database entities and relationships for annotations.
- Implemented service-level logic for annotation management.
- **Frontend Changes**
- Added new components for annotation management, including
`AddAnnotation`.
- Integrated annotation services and composables.
- Updated chart rendering to display annotations and improved panel
interactivity.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: ktx-vaidehi <vaidehi.akhani@kiara.tech>
Co-authored-by: Kirtan Thakkar <kirtan@kiara.tech>
Co-authored-by: Shyam Panjiyar <shyam@openobserve.ai>
Co-authored-by: ktx-vaidehi <134508096+ktx-vaidehi@users.noreply.github.com>
Co-authored-by: Huaijin <haohuaijin@gmail.com>
Co-authored-by: Neha P <144308844+neha00290@users.noreply.github.com>
Co-authored-by: Subhra264 <chakrabortysubhradeep556@gmail.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Refactor**
- Improved code readability by adjusting indentation and line
organization in the permissions checking utility
- No functional changes to the existing code logic
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Closes: #5761
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
- **New Features**
- Enhanced search functionality with improved streaming output handling.
- Added a new optional property for streaming aggregations in search
responses.
- Introduced a new `TimeOffset` struct for better representation of
time-related data in responses.
- New method `setCancelSearchError` to manage error states for cancelled
search queries.
- **Bug Fixes**
- Refined search response management to prevent unintended result
duplication.
- Improved handling of total count in search responses based on
streaming aggregations.
- Added pagination checks to prevent unnecessary requests when limits
are reached.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: omkark06 <omkar@zinclabs.io>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **License Management**
- Added new allowed license: BSL-1.0
- Removed license exceptions for `clipboard-win` and `error-code`
- **Code Improvements**
- Updated import statements to include additional enterprise action
functions
- Removed conditional compilation for `APP_DEPLOYER` variable, making it
universally accessible
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Hengfei Yang
udev
Yashodhan Joshi
Patrick Sullivan
Subhra264
Loakesh Indiran
Sai NikhilandTaiming Liu
Abhay PadamaniandYashodhan Joshi
Ashish Kolhe
Zhao.weiandhengfei yang
