## Summary
This PR only refactors code and should not result in any behavioral
changes.
This PR follows #5206 and attempts to refactor dashboards in a similar
way to how that PR refactors folders. This PR aims to divide our
dashboard code into distinct "HTTP Layer" and "Service Layer"s - where
the "HTTP Layer" contains only logic and models related to
serializing/deserializing for the REST HTTP API, and the "Service Layer"
contains only domain models/logic and is agnostic to the REST HTTP API.
This PR only partially achieves the above goals. Specifically, it makes
following changes...
1. Move all HTTP responses out of the `service` module and and into the
`http/request` module.
2. Introduce a new `DashboardError` in the `service` module to model all
types of "domain logic" errors that occur when interacting with folders.
3. Remove the `service/db/folders` module since, after the migration of
folders from the `meta` table to the `folders` table, this module mostly
just acted as a wrapper around the `infra/table/folders` module.
This PR is still missing the following changes. These changes will be
saved for a subsequent PR in order to keep the scope of this PR limited:
- Introduce new `...Response` and `...Request` types in the
`http/request/models` module whose role is specifically to define
serialization/deserialization definitions for the REST HTTP API.
⚠️ Notably, at this point the "HTTP Layer", "Service Layer", and "DB
Layer" models for dashboards are still tightly coupled together since
all still use the shared JSON serialization/deserialization logic
defined on the shared `Dashboard` struct in `config::meta`. Decoupling
the models and de/serialization logic at each layer will take more
effort than folders since the `Dashboard` struct is large and
complicated. I plan to start on this decoupling in the follow-up PR,
though that PR likely won't complete the process.
## Overall design goals
The design goals here are the same as in #5206 . Please see that PR's
description for a detailed explanation.
## Immediate motivation and next steps
Motivations and next steps are generally the same as in #5206 . Please
see that PR's description for a detailed explanation.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Enhanced dashboard creation and update processes with improved error
handling and response management.
- Added support for parsing dashboard data across multiple versions.
- **Bug Fixes**
- Streamlined error handling for dashboard operations, ensuring more
consistent and informative responses.
- **Chores**
- Refactored dashboard management logic to remove unnecessary
complexity, improving maintainability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This PR implements the new design of **Pipeline** introduced by #4432
---
**2 types of pipelines differentiates by the source**
Type I: **Realtime**
- source is a stream (Logs, Metrics, & Traces)
- takes effect when the source stream is being ingested
Type II: **Scheduled** (aka DerivedSteram)
- source is a SQL query
- takes effect based on the given schedule
The new pipeline is represented as a graph consists of different types
of Nodes that are connected by edges.
**Pipeline Data Model**
```rs
pub struct Pipeline {
pub id: String,
pub version: i32,
pub enabled: bool,
pub org: String,
pub name: String,
pub description: String,
pub source: PipelineSource,
pub nodes: Vec<Node>,
pub edges: Vec<Edge>,
}
```
**4 types of node**
I. StreamNode: either used for source node of realtime pipeline or for
destination nodes
II. QueryNode: used for source node of scheduled pipeline
III. FunctionNode: used for executing vrl functions
IV. ConditionNode: used for checking routing conditions
Rules applied to validating a pipeline when it's created:
1. non-empty nodes list
2. non-empty edges list
3. 1st node in nodes list is either StreamNode or QueryNode
4. non-empty `conditions` in all ConditionNode nodes in nodes list
5. all leaf nodes are of type StreamNode
6. In the same branch, unchecked `after_flattened` FunctionNode can't
follow checked
`after_flattened` checked FunctionNode
Pipeline execution is implemented with the struct `ExecutablePipeline`,
a ready-execute pipeline object cached in memory based on the Pipeline
objects modified on the UI side.
`ExecutablePipeline` object
- builds the relationships among all the nodes based on the edges
- topologically sorts the nodes based on the level to determine which
node to process first
- registers all the vrl function node once
`ExecutablePipeline` object processes ingested records in batch.
Starting from the source node, each connecting edge has a channel
connecting the parent and child node, used for passing the records. This
approach enables batch processing in parallel.
---
**Deprecated Features**
1. previous implementation of pipeline
2. Function x Stream associations
- Functions can still be added/edited/removed as before
- Needs to create a pipeline in order to apply chosen functions to the
desired stream
The new release when this pr is merged will automatically migrate **old
pipelines** and **Function x Stream associations** to new pipeline
format and can be found in `Stream Pipelines` tab with `Migrated-`
prefix names.
**_Note: Auto generated pipelines are paused by default. Please verify
the pipelines before enabling them_**
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
`DerivedStream`, as part of `Pipeline`, enables the use case of querying
an existing stream, `source`, with optional conditions and aggregations.
The resulted data will then be ingested into a new stream,
`destination`.
This PR provides the feature's initial offering. Please follow below
issue to track & suggest future improvements
#4170
Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: hengfei yang <hengfei.yang@gmail.com>
The new endpoint `/config/reload` will refresh the
environment variables.
There are a bunch of env-vars which will not be reloaded
for e.g. HTTP_PORT etc. these will need a restart.
---------
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
## Reports
Both `ZO_CHROME_ENABLED` and `ZO_SMTP_ENABLED` ENV variables need to be
`true` for reports to work.
**Report APIs -**
```
POST /api/{org_id}/reports
PUT /api/{org_id}/reports/{report_name}
GET /api/{org_id}/reports
GET /api/{org_id}/reports/{report_name}
DELETE /api/{org_id}/reports/{report_name}
PUT /api/{org_id}/reports/{report_name}/enable?value=<true|false>
```
**Report body**
```
{
"createdAt": "2024-03-23T09:46:32.913574782Z",
"dashboards": [
{
"dashboard": "7177215109754781699",
"folder": "default",
"tabs": [
"default"
],
"timerange": {
"from": 0,
"period": "1h",
"to": 0,
"type": "relative"
},
"variables": []
}
],
"description": "Report description",
"destinations": [
{
"email": "mail@gmail.com"
}
],
"enabled": false,
"frequency": {
"cron": "",
"interval": 1,
"type": "weeks"
},
"lastEditedBy": "root@example.com",
"mediaType": "pdf",
"message": "Email message",
"name": "report_name",
"orgId": "default",
"owner": "root@example.com",
"password": "ComplexPass#123",
"start": 1711187192913571,
"timezone": "Asia/Calcutta",
"timezoneOffset": 330,
"title": "Email subject",
"user": "root@example.com"
}
```
**ENVs**
| ENV | Description |
|------|---------------|
| `ZO_CHROME_ENABLED` | Must be `true` for reports, Default - `false` |
| `ZO_CHROME_PATH` | If chrome is enabled, custom `chrome` executable
path can be specified. If not specified, it looks for chrome executable
in default locations. If still not found, it automatically downloads a
good known version of `chromium`. |
| `ZO_CHROME_CHECK_DEFAULT_PATH` | If `false`, it does not look for
chromium in default locations, Default is `true`. |
| `ZO_CHROME_DOWNLOAD_PATH` | If chromium can not be found in default
locations and also `ZO_CHROME_PATH` is not specified, it downloads the
system specific chromium in the given path. Default is `./download`
(gitignored). `chromium` is downloaded for the first time only,
afterwords, `chromium` is fetched from the given path. If there is any
error regarding download of `chromium`, delete the download folder as it
might be in a bad state. |
| `ZO_CHROME_NO_SANDBOX` | If `true`, it launches chromium in
`no-sandbox` environment. Default is `false` |
| `ZO_CHROME_SLEEP_SECS` | Specify the number of seconds the
`service::dashboards::reports::generate_report()` will wait to load all
the data in the headless chromium dashboard. Default is `20` seconds. |
| `ZO_CHROME_WINDOW_WIDTH` | Specifies the width of the headless
chromium browser. Default is `1370` |
| `ZO_CHROME_WINDOW_HEIGHT` | Specifies the height of the headless
chromium browser. Default is `730` |
## Scheduler
Currently `Scheduler` is implemented for only `Sqlite`, `Postgres` and
`MySql`.
**API**
```rust
pub trait Scheduler: Sync + Send + 'static {
async fn create_table(&self) -> Result<()>;
async fn create_table_index(&self) -> Result<()>;
async fn push(&self, trigger: Trigger) -> Result<()>;
async fn delete(&self, org: &str, module: TriggerModule, key: &str) -> Result<()>;
async fn update_status(
&self,
org: &str,
module: TriggerModule,
key: &str,
status: TriggerStatus,
retries: i32,
) -> Result<()>;
async fn update_trigger(&self, trigger: Trigger) -> Result<()>;
async fn pull(
&self,
concurrency: i64,
alert_timeout: i64,
report_timeout: i64,
) -> Result<Vec<Trigger>>;
async fn clean_complete(&self, interval: u64);
async fn watch_timeout(&self, interval: u64);
async fn len_module(&self, module: TriggerModule) -> usize;
async fn len(&self) -> usize;
async fn is_empty(&self) -> bool;
async fn clear(&self) -> Result<()>;
}
```
**ENVs**
| ENV | Description |
|------|---------------|
| `ZO_ALERT_SCHEDULE_CONCURRENCY` | The number of `scheduled_jobs` the
`alert_manager` can process at a time. Default is `5` |
| `ZO_ALERT_SCHEDULE_TIMEOUT` | Visibility timeout for alerts. Default
is `90` seconds |
| `ZO_REPORT_SCHEDULE_TIMEOUT` | Visibility timeout for reports. Default
is `300` seconds |
| `ZO_SCHEDULER_MAX_RETRIES` | Maximum number of retries that will be
made for a `scheduled_job`. Default is `3` |
| `ZO_SCHEDULER_CLEAN_INTERVAL` | Time interval after which the
`Scheduler` will clean up the queue. Default is `30` seconds |
| `ZO_SCHEDULER_WATCH_INTERVAL` | Time interval after which the
`Scheduler` will watch for timeout for `scheduled_jobs`. Default is `30`
seconds. |
**Workflow**
Patrick Sullivan
udev
Hengfei Yang
Prabhat Sharma
John Vandenberg
Ankur SrivastavaandHengfei Yang
Subhra264
baerwangandHengfei Yang