fixes https://github.com/openobserve/openobserve/issues/6621
This separates the functions which execute ddl statements, and only runs
them if certain entry in meta is mis-matching with what we have in our
code. Introduces a new env var `ZO_META_DDL_DSN` , which is optional and
falls back to the original DSN if not specified. Added a new client to
Postgres and Mysql both which uses this DDL DSN, and that is used only
for the ddl statements.
Also added another command to cli `upgrade-db` which runs the db
initialization functions.
For sqlite, there is no change, as it does not have concept of users.
Primarily tested on Postgres to validate that the behavior with
read-write and admin users is as expected.
`DB_SCHEMA_VERSION` is hardcoded in code, and devs are expected to
manually uptick it everytime we do any changes to db schema, similar to
how we maintain ofga.
---------
Signed-off-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Yashodhan Joshi <yjdoc2@gmail.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
Now we have two ENV for control ingestion time:
- ZO_INGEST_ALLOWED_UPTO: old than x hours will discard
- ZO_INGEST_ALLOWED_IN_FUTURE: future than x hours will discard
```
ZO_INGEST_ALLOWED_UPTO=5
ZO_INGEST_ALLOWED_IN_FUTURE=24
```
Default is `5` hours and `24` hours.
The migration to drop an index on the `folders` table fails for MySQL.
The `DROP INDEX` statement in MySQL requires specifying the table name
but including the table name is not allowed for PostgreSQL and Sqlite.
⚠️ **Migration change:** This PR modifies an existing migration, so we
should be very careful not to alter the behavior of a migration that has
already been applied to users' databases. In this case, this PR only
changes the behavior of the `aler_folder_table_idx` migration when the
database backend is MySQL. Prior to this PR that migration would panic
when ran on a MySQL database, so no user with a MySQL database will have
been able to successfully run this migration yet. Therefore there should
be not risk of creating diverging migration behavior in this PR.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Chores**
- Updated migration logic for the `folders` table to enhance indexing
clarity.
- Improved index drop operations for the dashboards, alerts, and search
queue tables by specifying the associated tables.
- Introduced a new CLI command `seaorm-rollback` for rolling back
migration steps, with options to rollback all or the last N migrations.
- Added an asynchronous function to facilitate migration rollbacks with
optional step specification.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#5307
### Summary
From the issue description:
> We would like to deprecate various old migrations. If these migrations
are marked as deprecated in v0.14.0, then they will be officially
removed from the app initialization logic in v0.17.0.
>
> The ramification of this is that users will not be able to upgrade
directly from versions < 0.14.0 to versions >= v0.17.0. Instead, for
example, a user on v0.13.0 would first need to upgrade to v0.14.0, and
then from v0.14.0 they could upgrade to v0.17.0.
>
> Removing these migrations will reduce the risk that the behavior of
some of these migrations, particularly those that reference data
structures or logic in other parts of the application code base,
accidentally changes in the future resulting in inconsistent migration
behavior.
### Technical details
This PR just marks certain sections of the code as deprecated. This
deprecated logic will be removed some time after v0.17.0.
### 📣 Note to reviewers
Please comment on any additional migrations that we would like to
deprecate for v0.14.0 and I can include it in this PR. Or feel free to
commit to this branch to mark additional migrations as deprecated. If
you commit to this branch please use the `#[deprecated]` attribute if
possible or include the word `deprecated` in a comment on the code; I
plan on searching the source code for the word `"deprecated"` after
v0.17.0 to remove the deprecated code, so this would help with that.
### Todo in v0.14.0 release
We will want to include information in our official v0.14.0 release
notes about...
- The deprecation of migration/upgrade logic
- The incompatibility of direct upgrades from < v0.14.0 to >= v0.17.0
- Instructions for upgrading from < v0.14.0 to >= v0.17.0
- Users will first need to upgrade to some version >= v0.14.0 and <
v0.17.0. And then they will be able to upgrade to v0.17.0
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- Enhanced error handling for the `delete-parquet` command, providing
clearer feedback on deletion success or failure.
- Added deprecation warnings for several commands and functions,
indicating future removals.
- **Bug Fixes**
- Improved feedback for unsupported components in the `view` command.
- **Documentation**
- Added comments clarifying the deprecation status of certain migration
logic and functions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
This PR only refactors code and should not result in any behavioral
changes.
This PR follows #5206 and attempts to refactor dashboards in a similar
way to how that PR refactors folders. This PR aims to divide our
dashboard code into distinct "HTTP Layer" and "Service Layer"s - where
the "HTTP Layer" contains only logic and models related to
serializing/deserializing for the REST HTTP API, and the "Service Layer"
contains only domain models/logic and is agnostic to the REST HTTP API.
This PR only partially achieves the above goals. Specifically, it makes
following changes...
1. Move all HTTP responses out of the `service` module and and into the
`http/request` module.
2. Introduce a new `DashboardError` in the `service` module to model all
types of "domain logic" errors that occur when interacting with folders.
3. Remove the `service/db/folders` module since, after the migration of
folders from the `meta` table to the `folders` table, this module mostly
just acted as a wrapper around the `infra/table/folders` module.
This PR is still missing the following changes. These changes will be
saved for a subsequent PR in order to keep the scope of this PR limited:
- Introduce new `...Response` and `...Request` types in the
`http/request/models` module whose role is specifically to define
serialization/deserialization definitions for the REST HTTP API.
⚠️ Notably, at this point the "HTTP Layer", "Service Layer", and "DB
Layer" models for dashboards are still tightly coupled together since
all still use the shared JSON serialization/deserialization logic
defined on the shared `Dashboard` struct in `config::meta`. Decoupling
the models and de/serialization logic at each layer will take more
effort than folders since the `Dashboard` struct is large and
complicated. I plan to start on this decoupling in the follow-up PR,
though that PR likely won't complete the process.
## Overall design goals
The design goals here are the same as in #5206 . Please see that PR's
description for a detailed explanation.
## Immediate motivation and next steps
Motivations and next steps are generally the same as in #5206 . Please
see that PR's description for a detailed explanation.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
- **New Features**
- Enhanced dashboard creation and update processes with improved error
handling and response management.
- Added support for parsing dashboard data across multiple versions.
- **Bug Fixes**
- Streamlined error handling for dashboard operations, ensuring more
consistent and informative responses.
- **Chores**
- Refactored dashboard management logic to remove unnecessary
complexity, improving maintainability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This PR implements the new design of **Pipeline** introduced by #4432
---
**2 types of pipelines differentiates by the source**
Type I: **Realtime**
- source is a stream (Logs, Metrics, & Traces)
- takes effect when the source stream is being ingested
Type II: **Scheduled** (aka DerivedSteram)
- source is a SQL query
- takes effect based on the given schedule
The new pipeline is represented as a graph consists of different types
of Nodes that are connected by edges.
**Pipeline Data Model**
```rs
pub struct Pipeline {
pub id: String,
pub version: i32,
pub enabled: bool,
pub org: String,
pub name: String,
pub description: String,
pub source: PipelineSource,
pub nodes: Vec<Node>,
pub edges: Vec<Edge>,
}
```
**4 types of node**
I. StreamNode: either used for source node of realtime pipeline or for
destination nodes
II. QueryNode: used for source node of scheduled pipeline
III. FunctionNode: used for executing vrl functions
IV. ConditionNode: used for checking routing conditions
Rules applied to validating a pipeline when it's created:
1. non-empty nodes list
2. non-empty edges list
3. 1st node in nodes list is either StreamNode or QueryNode
4. non-empty `conditions` in all ConditionNode nodes in nodes list
5. all leaf nodes are of type StreamNode
6. In the same branch, unchecked `after_flattened` FunctionNode can't
follow checked
`after_flattened` checked FunctionNode
Pipeline execution is implemented with the struct `ExecutablePipeline`,
a ready-execute pipeline object cached in memory based on the Pipeline
objects modified on the UI side.
`ExecutablePipeline` object
- builds the relationships among all the nodes based on the edges
- topologically sorts the nodes based on the level to determine which
node to process first
- registers all the vrl function node once
`ExecutablePipeline` object processes ingested records in batch.
Starting from the source node, each connecting edge has a channel
connecting the parent and child node, used for passing the records. This
approach enables batch processing in parallel.
---
**Deprecated Features**
1. previous implementation of pipeline
2. Function x Stream associations
- Functions can still be added/edited/removed as before
- Needs to create a pipeline in order to apply chosen functions to the
desired stream
The new release when this pr is merged will automatically migrate **old
pipelines** and **Function x Stream associations** to new pipeline
format and can be found in `Stream Pipelines` tab with `Migrated-`
prefix names.
**_Note: Auto generated pipelines are paused by default. Please verify
the pipelines before enabling them_**
Co-authored-by: Taiming Liu <liutaiming3@gmail.com>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
`DerivedStream`, as part of `Pipeline`, enables the use case of querying
an existing stream, `source`, with optional conditions and aggregations.
The resulted data will then be ingested into a new stream,
`destination`.
This PR provides the feature's initial offering. Please follow below
issue to track & suggest future improvements
#4170
Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: hengfei yang <hengfei.yang@gmail.com>
The new endpoint `/config/reload` will refresh the
environment variables.
There are a bunch of env-vars which will not be reloaded
for e.g. HTTP_PORT etc. these will need a restart.
---------
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
## Reports
Both `ZO_CHROME_ENABLED` and `ZO_SMTP_ENABLED` ENV variables need to be
`true` for reports to work.
**Report APIs -**
```
POST /api/{org_id}/reports
PUT /api/{org_id}/reports/{report_name}
GET /api/{org_id}/reports
GET /api/{org_id}/reports/{report_name}
DELETE /api/{org_id}/reports/{report_name}
PUT /api/{org_id}/reports/{report_name}/enable?value=<true|false>
```
**Report body**
```
{
"createdAt": "2024-03-23T09:46:32.913574782Z",
"dashboards": [
{
"dashboard": "7177215109754781699",
"folder": "default",
"tabs": [
"default"
],
"timerange": {
"from": 0,
"period": "1h",
"to": 0,
"type": "relative"
},
"variables": []
}
],
"description": "Report description",
"destinations": [
{
"email": "mail@gmail.com"
}
],
"enabled": false,
"frequency": {
"cron": "",
"interval": 1,
"type": "weeks"
},
"lastEditedBy": "root@example.com",
"mediaType": "pdf",
"message": "Email message",
"name": "report_name",
"orgId": "default",
"owner": "root@example.com",
"password": "ComplexPass#123",
"start": 1711187192913571,
"timezone": "Asia/Calcutta",
"timezoneOffset": 330,
"title": "Email subject",
"user": "root@example.com"
}
```
**ENVs**
| ENV | Description |
|------|---------------|
| `ZO_CHROME_ENABLED` | Must be `true` for reports, Default - `false` |
| `ZO_CHROME_PATH` | If chrome is enabled, custom `chrome` executable
path can be specified. If not specified, it looks for chrome executable
in default locations. If still not found, it automatically downloads a
good known version of `chromium`. |
| `ZO_CHROME_CHECK_DEFAULT_PATH` | If `false`, it does not look for
chromium in default locations, Default is `true`. |
| `ZO_CHROME_DOWNLOAD_PATH` | If chromium can not be found in default
locations and also `ZO_CHROME_PATH` is not specified, it downloads the
system specific chromium in the given path. Default is `./download`
(gitignored). `chromium` is downloaded for the first time only,
afterwords, `chromium` is fetched from the given path. If there is any
error regarding download of `chromium`, delete the download folder as it
might be in a bad state. |
| `ZO_CHROME_NO_SANDBOX` | If `true`, it launches chromium in
`no-sandbox` environment. Default is `false` |
| `ZO_CHROME_SLEEP_SECS` | Specify the number of seconds the
`service::dashboards::reports::generate_report()` will wait to load all
the data in the headless chromium dashboard. Default is `20` seconds. |
| `ZO_CHROME_WINDOW_WIDTH` | Specifies the width of the headless
chromium browser. Default is `1370` |
| `ZO_CHROME_WINDOW_HEIGHT` | Specifies the height of the headless
chromium browser. Default is `730` |
## Scheduler
Currently `Scheduler` is implemented for only `Sqlite`, `Postgres` and
`MySql`.
**API**
```rust
pub trait Scheduler: Sync + Send + 'static {
async fn create_table(&self) -> Result<()>;
async fn create_table_index(&self) -> Result<()>;
async fn push(&self, trigger: Trigger) -> Result<()>;
async fn delete(&self, org: &str, module: TriggerModule, key: &str) -> Result<()>;
async fn update_status(
&self,
org: &str,
module: TriggerModule,
key: &str,
status: TriggerStatus,
retries: i32,
) -> Result<()>;
async fn update_trigger(&self, trigger: Trigger) -> Result<()>;
async fn pull(
&self,
concurrency: i64,
alert_timeout: i64,
report_timeout: i64,
) -> Result<Vec<Trigger>>;
async fn clean_complete(&self, interval: u64);
async fn watch_timeout(&self, interval: u64);
async fn len_module(&self, module: TriggerModule) -> usize;
async fn len(&self) -> usize;
async fn is_empty(&self) -> bool;
async fn clear(&self) -> Result<()>;
}
```
**ENVs**
| ENV | Description |
|------|---------------|
| `ZO_ALERT_SCHEDULE_CONCURRENCY` | The number of `scheduled_jobs` the
`alert_manager` can process at a time. Default is `5` |
| `ZO_ALERT_SCHEDULE_TIMEOUT` | Visibility timeout for alerts. Default
is `90` seconds |
| `ZO_REPORT_SCHEDULE_TIMEOUT` | Visibility timeout for reports. Default
is `300` seconds |
| `ZO_SCHEDULER_MAX_RETRIES` | Maximum number of retries that will be
made for a `scheduled_job`. Default is `3` |
| `ZO_SCHEDULER_CLEAN_INTERVAL` | Time interval after which the
`Scheduler` will clean up the queue. Default is `30` seconds |
| `ZO_SCHEDULER_WATCH_INTERVAL` | Time interval after which the
`Scheduler` will watch for timeout for `scheduled_jobs`. Default is `30`
seconds. |
**Workflow**
Taiming Liu
Hengfei Yang
Patrick Sullivan
udev
Prabhat Sharma
John Vandenberg
Ankur SrivastavaandHengfei Yang
Subhra264
baerwangandHengfei Yang