Commit Graph
21 Commits
Author SHA1 Message Date
Hengfei Yang 8d1ead5105 fix: add more debug logs for search (#3280) 2024-04-19 20:01:39 +08:00
Hengfei Yang 7e457c3538 ci: add tokio-console build (#3264) 2024-04-17 15:20:38 +08:00
Hengfei Yang a48326861f feat: complete super cluster (#3226) 2024-04-12 13:45:02 +08:00
Huaijin 951c55bd92 feat: make user can cancel query task by session_id (#3130) 2024-04-08 23:43:05 +08:00
Hengfei Yang 611b955b10 refactor: separate search structure to config crate (#3186)
- [x] Separate search structure into config crate
- [x] fixed #3084
2024-04-07 12:01:47 +08:00
Subhra264 08d12f8571 feat: introduce reports in backend and replace TRIGGERS with infra::scheduler (#3086)
## Reports
Both `ZO_CHROME_ENABLED` and `ZO_SMTP_ENABLED` ENV variables need to be
`true` for reports to work.

**Report APIs -**
```
POST /api/{org_id}/reports
PUT /api/{org_id}/reports/{report_name}
GET /api/{org_id}/reports
GET /api/{org_id}/reports/{report_name}
DELETE /api/{org_id}/reports/{report_name}
PUT /api/{org_id}/reports/{report_name}/enable?value=<true|false>
```

**Report body**
```
{
  "createdAt": "2024-03-23T09:46:32.913574782Z",
  "dashboards": [
    {
      "dashboard": "7177215109754781699",
      "folder": "default",
      "tabs": [
        "default"
      ],
      "timerange": {
        "from": 0,
        "period": "1h",
        "to": 0,
        "type": "relative"
      },
      "variables": []
    }
  ],
  "description": "Report description",
  "destinations": [
    {
      "email": "mail@gmail.com"
    }
  ],
  "enabled": false,
  "frequency": {
    "cron": "",
    "interval": 1,
    "type": "weeks"
  },
  "lastEditedBy": "root@example.com",
  "mediaType": "pdf",
  "message": "Email message",
  "name": "report_name",
  "orgId": "default",
  "owner": "root@example.com",
  "password": "ComplexPass#123",
  "start": 1711187192913571,
  "timezone": "Asia/Calcutta",
  "timezoneOffset": 330,
  "title": "Email subject",
  "user": "root@example.com"
}
```

**ENVs**
| ENV |  Description | 
|------|---------------|
| `ZO_CHROME_ENABLED` | Must be `true` for reports, Default - `false` |
| `ZO_CHROME_PATH` | If chrome is enabled, custom `chrome` executable
path can be specified. If not specified, it looks for chrome executable
in default locations. If still not found, it automatically downloads a
good known version of `chromium`. |
| `ZO_CHROME_CHECK_DEFAULT_PATH` | If `false`, it does not look for
chromium in default locations, Default is `true`. |
| `ZO_CHROME_DOWNLOAD_PATH` | If chromium can not be found in default
locations and also `ZO_CHROME_PATH` is not specified, it downloads the
system specific chromium in the given path. Default is `./download`
(gitignored). `chromium` is downloaded for the first time only,
afterwords, `chromium` is fetched from the given path. If there is any
error regarding download of `chromium`, delete the download folder as it
might be in a bad state. |
| `ZO_CHROME_NO_SANDBOX` | If `true`, it launches chromium in
`no-sandbox` environment. Default is `false` |
| `ZO_CHROME_SLEEP_SECS` | Specify the number of seconds the
`service::dashboards::reports::generate_report()` will wait to load all
the data in the headless chromium dashboard. Default is `20` seconds. |
| `ZO_CHROME_WINDOW_WIDTH` | Specifies the width of the headless
chromium browser. Default is `1370` |
| `ZO_CHROME_WINDOW_HEIGHT` | Specifies the height of the headless
chromium browser. Default is `730` |

## Scheduler

Currently `Scheduler` is implemented for only `Sqlite`, `Postgres` and
`MySql`.
**API**
```rust
pub trait Scheduler: Sync + Send + 'static {
    async fn create_table(&self) -> Result<()>;
    async fn create_table_index(&self) -> Result<()>;
    async fn push(&self, trigger: Trigger) -> Result<()>;
    async fn delete(&self, org: &str, module: TriggerModule, key: &str) -> Result<()>;
    async fn update_status(
        &self,
        org: &str,
        module: TriggerModule,
        key: &str,
        status: TriggerStatus,
        retries: i32,
    ) -> Result<()>;
    async fn update_trigger(&self, trigger: Trigger) -> Result<()>;
    async fn pull(
        &self,
        concurrency: i64,
        alert_timeout: i64,
        report_timeout: i64,
    ) -> Result<Vec<Trigger>>;
    async fn clean_complete(&self, interval: u64);
    async fn watch_timeout(&self, interval: u64);
    async fn len_module(&self, module: TriggerModule) -> usize;
    async fn len(&self) -> usize;
    async fn is_empty(&self) -> bool;
    async fn clear(&self) -> Result<()>;
}
```

**ENVs**
| ENV |  Description | 
|------|---------------|
| `ZO_ALERT_SCHEDULE_CONCURRENCY` | The number of `scheduled_jobs` the
`alert_manager` can process at a time. Default is `5` |
| `ZO_ALERT_SCHEDULE_TIMEOUT` | Visibility timeout for alerts. Default
is `90` seconds |
| `ZO_REPORT_SCHEDULE_TIMEOUT` | Visibility timeout for reports. Default
is `300` seconds |
| `ZO_SCHEDULER_MAX_RETRIES` | Maximum number of retries that will be
made for a `scheduled_job`. Default is `3` |
| `ZO_SCHEDULER_CLEAN_INTERVAL` | Time interval after which the
`Scheduler` will clean up the queue. Default is `30` seconds |
| `ZO_SCHEDULER_WATCH_INTERVAL` | Time interval after which the
`Scheduler` will watch for timeout for `scheduled_jobs`. Default is `30`
seconds. |

**Workflow**

![openobserve_scheduler_workflow](https://github.com/openobserve/openobserve/assets/59690052/68beb43b-d04f-49dc-9006-a30942acd99b)
2024-03-25 18:24:14 +05:30
7c52e0ea7a fix: schema versions as separate key (#3021)
To enable row per schema version use:
ZO_ENTRY_PER_SCHEMA_VERSION_ENABLED = true.

One can migrate schemas from single row to row per schema version using
: migrate-schemas command

Before migration :

- DROP INDEX meta_module_key2_idx ON meta
- ALTER TABLE meta ADD COLUMN start_dt BIGINT NOT NULL DEFAULT 0;
- CREATE UNIQUE INDEX meta_module_start_dt_idx ON meta (start_dt, key2,
key1, module);

---------

Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
Co-authored-by: Ankur Srivastava <ansrivas@users.noreply.github.com>
Co-authored-by: Huaijin <haohuaijin@gmail.com>
Co-authored-by: ktx-abhay <141122205+ktx-abhay@users.noreply.github.com>
Co-authored-by: Subhra264 <chakrabortysubhradeep556@gmail.com>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
Co-authored-by: ktx-vaidehi <vaidehi.akhani@kiara.tech>
Co-authored-by: Omkar Kesarkhane <127078473+omkarK06@users.noreply.github.com>
Co-authored-by: Kirtan Thakkar <kirtan@kiara.tech>
2024-03-24 21:25:06 +05:30
Hengfei Yang fd49360f49 feat: add quick mode fields (#3068) 2024-03-22 15:54:39 +08:00
Hengfei Yang f1a7ad0961 feat: update datafusion to 36 (#3035) 2024-03-18 22:50:26 +08:00
Hengfei Yangandoasisk e1097f86e2 feat: add fast mode for query (#2876)
Use **Fast Mode** when dealing with thousands of fields. It retrieves
the first N fields for a quicker response.

Enable it through this configuration: 

```
ZO_FAST_MODE_NUM_FIELDS=50
ZO_FAST_MODE_STRATEGY=first/last/both
```

And use the search API:

```json
{
	"query": {
		"sql": "select * from default ",
		"start_time": 1709711116396000,
		"end_time": 1709712016396000,
		"from": 0,
		"size": 250,
		"sql_mode": "full",
		"fast_mode": true
	}
}
```

When the backend identifies a request:
- Using `fast_mode` 
- With an `SQL` prefix of `select *` 
- Where schema fields exceed `ZO_FAST_MODE_NUM_FIELDS` 

It will then rewrite the `SQL` to `select f1,f2,f3`, where 'fn'
represents the first 'n' fields from schema. The value of 'n' is
determined by `ZO_FAST_MODE_NUM_FIELDS`.

In the end you can set `ZO_FAST_MODE_STRATEGY=first/last/both` to
control the order for pickup fields from schema.

---------

Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
2024-03-06 19:50:30 +05:30
Hengfei Yang 79682120f0 feat: search queue management (#2621) 2024-02-01 11:54:01 +08:00
Ashish Kolhe 98c9af0356 fix: report user email as part of usage reporting (#2610) 2024-01-31 15:31:08 +05:30
Hengfei Yang 325aa4b3f2 fix: unit tests (#2542)
- [x] change sqlite backend to a Mutex
- [x] fix unit tests
2024-01-24 22:42:20 +08:00
Hengfei Yang ab28f13afb feat: migrate from any to any (#2472) 2024-01-14 22:54:38 +08:00
Hengfei Yang 150d251983 perf: keeping optimize search (#2362) 2024-01-09 19:49:17 +08:00
Hengfei Yang bb927db797 chore: delete migrate-file-list-from-dynamo (#2361) 2024-01-03 11:32:46 +08:00
Hengfei Yang a4bae1a584 feat: Implement memtable + wal (#2261) 2023-12-28 07:40:22 +08:00
Ashish Kolhe 3cc141d778 fix: user api checks (#2294)
- [x] Check role during User Creation
- [x] Added checks during User Deletion/removal from org
2023-12-23 22:48:59 +05:30
Ashish Kolhe 122d5d52c4 feat: dex integration (#2277) 2023-12-22 09:05:46 +05:30
Hengfei Yang 27c8d2420f chore: update datafusion to 34 (#2222) 2023-12-18 10:49:08 +08:00
baerwangandHengfei Yang 7b32681617 feat: cli import or export data (#2071)
feat #2056

Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
2023-12-16 12:45:09 +08:00