Uses mpsc channels to process pulled scheduled jobs so that it does not
block the entire alert manager to pull other jobs when there is
capacity. Makes use of semaphores so that we pull the same number of
jobs as the number of available workers.
### The new design is -
- We spawn the same number of worker threads as the configured max concurreny env.
- After pulling jobs, it sends to each worker through the mpsc channel
- Each worker processes the received trigger
- After the scheduler interval, the puller job checks how many workers are still available i.e. how many workers are done with processing triggers.
- It only pulls the same number of jobs from db as the number of available workers at that particular time.
The cycle continues.
So, the scheduler pulls at max the "max concurrency" number of jobs at a time depending on the availability of worker threads.
This PR adds a `source_node` field to the `triggers` stream to indicate
which alert manager processed the particular trigger. This is the name
of the node.
---------
Co-authored-by: hengfei yang <hengfei.yang@gmail.com>
#5253
Part 2 in a series of PRs to add new Alerts endpoints that support
parent folders and KSUIDs
This PR just renames various methods
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced new alert management structures and request/response bodies
for creating, updating, and listing alerts.
- Added unique identifier support for alerts.
- **Bug Fixes**
- Enhanced error handling for alert creation and updates.
- **Documentation**
- Updated OpenAPI documentation to include new alert-related request and
response types.
- **Refactor**
- Renamed functions for clarity in alert management, including retrieval
and deletion by name.
- **Tests**
- Updated integration tests to reflect changes in alert deletion
methods.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#5253
The overall goal of this PR is to migrate alerts data out of `meta`
table into `alerts` table. To accomplish this three main changes are
made. Unfortunately, all three of these changes need to be synchronized
and submitted in the same PR.
1. A new migration script is added to
- Parse alerts JSON currently stored in the `meta` table into Rust
models for alerts. These Rust models are isolated to the migration
script since they represent the schema of alerts as they are stored in
the `meta` table before this migration executes.
- Convert those Rust models into Sea ORM entities for alerts.
- Insert all the Sea ORM entities for alerts into the new `alerts`
table.
2. Create get/put/delete/list methods for accessing the data in the
`alerts` table
- These functions use SeaORM to query and update the database.
- These functions must also convert between SeaORM alert entities and
the application's current data model for alerts (as defined in
`config::meta::alerts`.
- It's entirely expected that the models inside `config::meta::alerts`
will change over time. When this happens in the future, developers will
need to update the transformation functions between SeaORM alert
entities and the models in `config::meta::alerts`. Luckily, the compiler
should help to direct developers to where these transformation functions
need to be updated.
3. Update the old logic in `/src/service/db/alerts.rs` that previously
read and wrote alerts to the `meta` table so that it now reads and
writes alerts to the `alerts` table.
- The methods for accessing the `meta` table included logic for
triggering "cache update events" to the cluster coordinator and for
sending events to the super cluster queue. Since the new logic accesses
the `alerts` table rather than the `meta` table, this logic for emitting
events had to be reproduced.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced a new `ListAlertsParams` struct for flexible querying of
alerts.
- Enhanced `TriggerCondition` struct with new timing parameters.
- Added new error variants for improved error handling in alert
management.
- Created a comprehensive alert management system with CRUD operations.
- Added integration tests for various functionalities related to alerts
and dashboards.
- Introduced a new `Alerts` variant in the `FolderType` enum for better
folder management.
- **Bug Fixes**
- Improved error handling for folder deletion when alerts are present.
- **Documentation**
- Updated documentation for several functions to clarify their scope and
functionality.
- **Chores**
- Added new dependencies to support additional functionality.
- Introduced migration scripts for database schema changes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
This PR only refactors code and should not result in any behavioral
changes.
This PR follows #5220 and creates separate "HTTP Layer" models for each
our dashboard HTTP API endpoints.
By separating the "HTTP Layer" models from the "Service Layer" dashboard
models we will now (in future PRs) be able to...
- add folder name and folder ID fields to each item returned by the
`ListDashboards` endpoint - Step 3 of #5198
- remove dashboard details from the each item returned by the
`ListDashboards` endpoint - Step 1 of #5196
Unfortunately, this PR does not yet accomplish fully decoupling the
"HTTP Layer" dashboard model from the "DB Layer" dashboard model since
both layers still use the same same JSON de/serialization logic defined
on the structs inside `config/meta/dashboards`. So changes to that model
and the corresponding JSON de/serialization logic could still introduce
silent breaking changes into both the interface with the DB and the HTTP
API. This PR moves us closer to decoupling those models, though future
work will be necessary to fully decouple them.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced new data structures for handling dashboard-related HTTP
requests and responses, enhancing clarity and type safety.
- Added support for creating, updating, and listing dashboards with
structured request and response bodies.
- **Documentation**
- Updated OpenAPI documentation to reflect new schemas for dashboard
operations, improving API clarity and specificity.
- **Refactor**
- Removed outdated dashboard structures and streamlined request handling
logic for improved maintainability.
- Updated integration tests to align with the new dashboard data
structures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This is the first PR for #5068 .
We need to move `dashboards` into the `config` crate so that DB access
functions in the `infra` crate can return types defined in the
`dashboards` module.
When the `dashboards` module is in the `openobserve` crate we cannot use
them in the `infra` crate since that would introduce a circular
dependency.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Introduced a new function to move dashboards, enhancing dashboard
management capabilities.
- Added a new `SendReport` trait for improved report handling and
modularity.
- Enhanced error handling across various dashboard and report management
functions.
- **Improvements**
- Updated import paths for better clarity and organization, particularly
for dashboard-related modules.
- Improved timestamp handling for alerts to ensure accurate tracking of
updates.
- Enhanced OpenAPI documentation to reflect updated schema references
for dashboards.
- Expanded development dependencies to support improved functionality.
- Enhanced the logic for handling report triggers based on frequency
types.
- **Bug Fixes**
- Refined error handling in dashboard creation and deletion functions to
provide clearer responses.
- Standardized error handling across report management functions for
consistency.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
This PR implements the new design of **Pipeline** introduced by #4432
---
**2 types of pipelines differentiates by the source**
Type I: **Realtime**
- source is a stream (Logs, Metrics, & Traces)
- takes effect when the source stream is being ingested
Type II: **Scheduled** (aka DerivedSteram)
- source is a SQL query
- takes effect based on the given schedule
The new pipeline is represented as a graph consists of different types
of Nodes that are connected by edges.
**Pipeline Data Model**
```rs
pub struct Pipeline {
pub id: String,
pub version: i32,
pub enabled: bool,
pub org: String,
pub name: String,
pub description: String,
pub source: PipelineSource,
pub nodes: Vec<Node>,
pub edges: Vec<Edge>,
}
```
**4 types of node**
I. StreamNode: either used for source node of realtime pipeline or for
destination nodes
II. QueryNode: used for source node of scheduled pipeline
III. FunctionNode: used for executing vrl functions
IV. ConditionNode: used for checking routing conditions
Rules applied to validating a pipeline when it's created:
1. non-empty nodes list
2. non-empty edges list
3. 1st node in nodes list is either StreamNode or QueryNode
4. non-empty `conditions` in all ConditionNode nodes in nodes list
5. all leaf nodes are of type StreamNode
6. In the same branch, unchecked `after_flattened` FunctionNode can't
follow checked
`after_flattened` checked FunctionNode
Pipeline execution is implemented with the struct `ExecutablePipeline`,
a ready-execute pipeline object cached in memory based on the Pipeline
objects modified on the UI side.
`ExecutablePipeline` object
- builds the relationships among all the nodes based on the edges
- topologically sorts the nodes based on the level to determine which
node to process first
- registers all the vrl function node once
`ExecutablePipeline` object processes ingested records in batch.
Starting from the source node, each connecting edge has a channel
connecting the parent and child node, used for passing the records. This
approach enables batch processing in parallel.
---
**Deprecated Features**
1. previous implementation of pipeline
2. Function x Stream associations
- Functions can still be added/edited/removed as before
- Needs to create a pipeline in order to apply chosen functions to the
desired stream
The new release when this pr is merged will automatically migrate **old
pipelines** and **Function x Stream associations** to new pipeline
format and can be found in `Stream Pipelines` tab with `Migrated-`
prefix names.
**_Note: Auto generated pipelines are paused by default. Please verify
the pipelines before enabling them_**
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
New integration tests for cargo to test alert scheduling
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Tests**
- Introduced new asynchronous integration tests for multi-range alert
handling.
- Enhanced existing alert tests with improved assertions for creation
and deletion processes.
- Updated alert trigger conditions to ensure comprehensive validation of
alert functionalities.
- Added tests for alert behavior during retries after destination
failures and evaluation errors.
- Updated tests for the `SearchHistoryQueryBuilder` to reflect new SQL
query structure tailored to search events.
- Modified tests for `RedirectResponse` to accommodate multiple query
parameter orders.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
issue #4326
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Introduced new structures for updating stream settings, allowing for
more granular management of stream configurations.
- Added an asynchronous function for updating stream settings via HTTP
PUT requests.
- Implemented a new service route for handling updates to stream
settings.
- Enhanced the `AddStream` component to retrieve updated settings.
- **Improvements**
- Improved schema management by tracking previous settings during
updates.
- Enhanced utility functions for comparing and managing stream settings.
- **Documentation**
- Updated documentation to reflect changes in API functionality for
creating and updating stream settings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Subhradeep Chakraborty <chakrabortysubhradeep556@gmail.com>
The new endpoint `/config/reload` will refresh the
environment variables.
There are a bunch of env-vars which will not be reloaded
for e.g. HTTP_PORT etc. these will need a restart.
---------
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
## Design Spec:
- POST `/api/:org/ingest/metrics/_json`
```json
[
{
"__name__": "stream name",
"__type__": "counter / gauge / histogram / summary",
"label_name1": "label_value1",
"label_name2": "label_value2",
"_timestamp": 0, // unix timestamp in millisecond
"value": 0.0 // float64 value
}
]
```
## How to use this API:
- must use `__name__` set stream name.
- must use `__type__` set metrics type. and the type can't be updated
after first record
- `_timestamp` needs UNIX timestamp or empty will fill by current
timestamp
- `value` should be float64 or int64
## Example:
POST `/api/default/ingest/metrics/_json`
```json
[
{
"__name__": "cpu",
"__type__": "gauge",
"namespace": "ziox",
"container": "zo1",
"_timestamp": 1687326643584062,
"value": 10.1
},
{
"__name__": "cpu",
"__type__": "gauge",
"namespace": "ziox",
"container": "zo2",
"value": 0.1
}
]
```
Hengfei Yang
Taiming Liu
Patrick Sullivan
Huaijin
Prabhat Sharma
John Vandenberg
Ankur SrivastavaandHengfei Yang
Nrishinghananda Roy
Ashish Kolhe