At times same function is used different pipelines , the function would
like to add stream name to current row, added `org_id` and `stream_name`
It can be used in vrl with below syntax:
`.stream_name_value = %stream_name`
Currently, individual level RBAC permission does not work for
dashboards. This pr fixes that. So a role permission like below
<img width="1399" alt="Screenshot 2025-03-04 at 5 27 11 PM"
src="https://github.com/user-attachments/assets/06c2030d-4227-4109-b73f-dc2337151362"
/>
will allow the user to access only the dashboards with `Get` and `All`
permission even if the `Get` permission for the folder is not specified.
Fixes three problems -
- [x] Pipeline pauses automatically after reaching max retries.
- [x] There could be some data loss if `(now - start_time)` is not a
multiple of `period_num_microseconds + 1`.
- [x] If derived stream fails, it should retry immediately in the next
batch until it reaches max retries. After reaching max retries, it
should retry at the next scheduled run.
---------
Co-authored-by: sai nikhil kethe <nikhil@openobserve.ai>
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
This uses background jobs in order to download files asynchronously, and
a mpsc queue in order to send file names to the jobs threads. This
limits the number of threads we spawn to a fixed number configurable via
config var.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Ref: https://github.com/openobserve/openobserve/issues/5974
Issue: Close frame description field causes frame malformation in
Chrome/Brave
- Description field causes inconsistent frame encoding
- Firefox handles it fine, but Chrome/Brave show malformed data
- Workaround: Skip description field in close frames
Actions:
- Set `CloseFrame` description at both `router` & `querier` to `None`
1. reduced query time for sql validation from 15mins to 5 seconds
2. silence column is removed from the pipeline list as we are not
supporting in the UI
Uses mpsc channels to process pulled scheduled jobs so that it does not
block the entire alert manager to pull other jobs when there is
capacity. Makes use of semaphores so that we pull the same number of
jobs as the number of available workers.
### The new design is -
- We spawn the same number of worker threads as the configured max concurreny env.
- After pulling jobs, it sends to each worker through the mpsc channel
- Each worker processes the received trigger
- After the scheduler interval, the puller job checks how many workers are still available i.e. how many workers are done with processing triggers.
- It only pulls the same number of jobs from db as the number of available workers at that particular time.
The cycle continues.
So, the scheduler pulls at max the "max concurrency" number of jobs at a time depending on the availability of worker threads.
Makes the following fixes to these routes:
- `ListAlerts`
- adds `enabled` to the response body
- adds `name_substring` to the request body
- `MoveAlerts`
- changes the method type to `PATCH` since this is a non-idempotent
update
- changing to `PATCH` coincidentally happens to fix a routing issue
where requests to `MoveAlerts` were being routed to the `UpdateAlerts`
handler
- `EnableAlert`
- changes the method type to `PATCH` since this is a non-idempotent
update
- `TriggerAlert`
- changes the method type to `PATCH` since this is a non-idempotent
update
## Goal:
- The goal is to not sleep/timeout but also wait a tiny bit before
sending close and at the same time not keeping the threads hanging which
could lead to resource exhaustion on the querier
## Changes:
1. **Message In-Flight Control**:
```rust
message_in_flight: AtomicBool
```
- Uses atomic boolean for lightweight synchronization
- Ensures only one message is being sent at a time
- Prevents message overlap and potential race conditions
2. **Limited Retry Mechanism**:
```rust
let mut attempts = 0;
while attempts < 3 {
match session.message_in_flight.compare_exchange(...) {
// ...
Err(_) => {
attempts += 1;
if attempts < 3 {
tokio::task::yield_now().await;
}
}
}
}
```
- Maximum 3 retries
- Uses `yield_now()` instead of sleep
- Quick failure path if message can't be sent
3. **Non-Blocking Cleanup Before Sending Close**:
```rust
while session.message_in_flight.load(Ordering::SeqCst) && retries < 3 {
tokio::task::yield_now().await;
retries += 1;
}
```
- Limited retries during cleanup
- Non-blocking approach to message completion
- Proceeds with close even if message is still in flight
**Why This Prevents Querier Resource Exhaustion**:
1. No thread blocking or sleeping
2. Limited retry attempts (max 3)
3. Quick failure paths
4. `yield_now()` only gives up CPU time slice, doesn't block
5. Resources are cleaned up even on failure
Apart from filtering dashboard list based on
`dashboard:{folder_id}/dashboard_id` and `dashboard:_all_{org_id}`,
filter with `dashboard:{dashboard_id}` as well.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Introduced search job scheduling functionality.
- Added ability to create, list, cancel, retry, and delete scheduled
searches.
- Enhanced user interface for managing scheduled search jobs, including
a new component for displaying scheduled jobs.
- Implemented conditional rendering for search history and search
scheduler components.
- Added new methods for handling scheduled searches within the service
layer.
- **Improvements**
- Refined search bar interface with new scheduling options.
- Updated pagination handling for job-related data.
- Enhanced overall functionality related to job management and
pagination.
- Improved loading indicators and user experience during data retrieval.
- **Bug Fixes**
- Corrected conditional rendering of search history components.
- Optimized job data retrieval and pagination logic.
- Simplified sorting logic within the search history component.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
1. no data error before rendering
2. chart rerenders even after loading
3. all charts rererenders if we click on refresh of one chart
---------
Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech>
1. this PR introduces new design for query node
2. dynamic stream name for destination node instead of create new stream
it allows user to input any non existing stream name as destination and
at the time of ingestion it will be created dynamically
3. Fixes issues of #5886
---------
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
This PR adds a `source_node` field to the `triggers` stream to indicate
which alert manager processed the particular trigger. This is the name
of the node.
---------
Co-authored-by: hengfei yang <hengfei.yang@gmail.com>
Abhay Padamani
Subhra264andHengfei Yang
ktx-akshayandHengfei Yang
udev
Ashish Kolhe
ktx-vaidehiandHengfei Yang
Zhao.weiandHengfei Yang
Patrick Sullivan
Yashodhan JoshiandYashodhan Joshi
Loakesh Indiran
HuaijinandHengfei Yang
Bhargav
Shyam Panjiyar