Commit Graph
362 Commits
Author SHA1 Message Date
4b92c092a9 feat: Query Cancellation (#3207)
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
2024-04-30 22:41:56 +05:30
Subhra264andoasisk f9e200f82b fix: same role and group name in rbac with diff org conflicts (#3359)
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
2024-04-29 11:31:43 +05:30
Ashish Kolhe 15f9788a6e fix: format stream name for _bulk api (#3371) 2024-04-28 23:12:35 +05:30
Hengfei Yang 8a73f2cf05 fix: value API is always waiting (#3364)
fixed #3334
2024-04-28 22:30:07 +08:00
Subhra264 09441cf4fb fix: Add logo settings to rbac (#3366) 2024-04-28 18:35:27 +05:30
Ashish Kolhe a7222f7d65 fix: id token length issue (#3360) 2024-04-28 15:16:05 +05:30
Ashish Kolhe 4f29f564e6 fix: user login flow fixes (#3358) 2024-04-27 23:41:09 +05:30
Ashish KolheandHengfei Yang a88c5c4646 fix: user session id (#3343)
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
2024-04-25 20:51:25 +08:00
Subhra264 7bf9ee1fab feat: introduce cached_ratio to usage reporting for search usage type (#3341)
Fixes #3336
2024-04-25 14:08:23 +08:00
Hengfei Yang 15b22c1876 fix: schema cache error with prefix (#3338) 2024-04-25 10:54:40 +08:00
Subhra264 8482a913b3 fix: show savedview and report resources in role permissions and fix list_objects_for_user for _all_{org} (#3284)
For enterprise in rbac, `GET` methods for `reports` and `savedviews`
consists of `{org_id}/<reports|savedviews>/{id}` format, i.e. `url_len`
is 3. This PR corrects how the fga object string is extracted (which
should be `report:{report_name}` or `savedview:{view_id}`) from the url
path with the above format with `GET` method.
2024-04-23 15:09:09 +05:30
Hengfei Yang 1738a51653 fix: API rule optimize let search UI works without ingester (#3314) 2024-04-23 12:26:26 +08:00
Subhra264 1e8d1e3cc1 fix: don't use stream_type from query params for alert create/update (#3311)
Fixes #3283
2024-04-23 07:25:21 +08:00
Hengfei Yang 84db0f3bf3 feat: sync scheduler in super cluster (#3309) 2024-04-22 19:34:31 +08:00
Hengfei Yang 8d1ead5105 fix: add more debug logs for search (#3280) 2024-04-19 20:01:39 +08:00
Ashish Kolheandomkark06 ebdc19d02e feat: routing & user defined schema restrictions (#3254)
Co-authored-by: omkark06 <omkar@zinclabs.io>
2024-04-19 06:00:56 +05:30
momantech 24bd5f4fe8 docs: fix some typos in comments (#3267) 2024-04-17 19:31:33 +08:00
Huaijin 0ec3285e1a fix: add order by field to select when need (#3259) 2024-04-16 18:04:55 +08:00
Subhra264 e9e8160391 feat: add audit trails for enterprise version (#3211) 2024-04-16 11:40:32 +05:30
Ashish KolheandBhargav 1c1b5c5e7f fix: cookies issue for SSO (#3244)
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
2024-04-16 07:27:33 +05:30
HuaijinandHengfei Yang cacf46e480 feat: only enable tracing in search (#3230)
close #3215 
use follow to only enable tracing in search
```
ZO_TRACING_ENABLED=false
ZO_TRACING_SEARCH_ENABLED=true 
```

---------

Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
2024-04-15 18:43:34 +08:00
Hengfei Yang 3da8c96e92 fix: openAPI spec (#3239) 2024-04-13 10:57:50 +08:00
Hengfei Yang a48326861f feat: complete super cluster (#3226) 2024-04-12 13:45:02 +08:00
Ankur Srivastavaandomkark06 3826f87640 fix: ensure that rum proxy endpoint uses cookies (#3210)
Due to changes in the current architecture the access-token is being
sent as a cookie - `access_token` so this change facilitates accessing
the proxy url via the same mechanism

---------

Co-authored-by: omkark06 <omkar@zinclabs.io>
2024-04-11 18:13:21 +02:00
Hengfei Yang d4ff1888e4 feat: super cluster (#3039) 2024-04-11 10:01:56 +08:00
HuaijinandHengfei Yang e7f17e0541 chore: rename search_job to query_manager (#3214)
rename cancel job related api
from 
```
#[delete("/search_job/{trace_id}")]
#[get("/search_job/status")]
```
to 
```
#[delete("/query_manager/{trace_id}")]
#[get("/query_manager/status")]
```

---------

Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
2024-04-10 22:50:14 +08:00
Ashish Kolhe 076dda1488 fix: add logout endpoint (#3212) 2024-04-10 20:02:01 +08:00
Hengfei Yangandoasisk a40540c64c fix: parquet file job should be concurrent (#3208)
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
2024-04-10 14:38:06 +08:00
Huaijin 35d42fe228 feat: replace sessiond_id to trace_id when enable tracing (#3204) 2024-04-10 13:38:22 +08:00
Huaijin b8e1790030 fix: modify cancel query status code to 429 (#3206) 2024-04-10 13:17:12 +08:00
Hengfei Yang d4be2560bd perf: use limit for scan_files (#3198)
- [x] Improve scan_files with limit
- [x] Fix router error: `Internal error: connector has been
disconnected`
2024-04-09 13:24:56 +08:00
Huaijin 951c55bd92 feat: make user can cancel query task by session_id (#3130) 2024-04-08 23:43:05 +08:00
Hengfei Yang a6068264d5 fix: search for wal with prefix (#3189) 2024-04-08 20:30:44 +08:00
Ashish Kolhe 7472a0fcf0 fix: use async walkdir (#3193) 2024-04-08 17:06:54 +05:30
Hengfei Yang b017352b11 fix: traces metadata write error (#3187) 2024-04-07 20:09:10 +08:00
Hengfei Yang 611b955b10 refactor: separate search structure to config crate (#3186)
- [x] Separate search structure into config crate
- [x] fixed #3084
2024-04-07 12:01:47 +08:00
Hengfei Yang 42cffffe31 fix: node cache status (#3170) 2024-04-05 19:04:45 +08:00
Hengfei Yang 0c33ee62d7 perf: node cache status (#3169) 2024-04-05 16:29:49 +08:00
Hengfei Yang aba980714f feat: add auth cookie max age to 30 days (#3165) 2024-04-05 15:23:34 +08:00
Ashish Kolhe 82eb6e6e15 fix: cookie settings as env variable (#3152) 2024-04-03 15:24:21 +05:30
Ashish Kolhe 00fdce8c83 chore: cookies for authentication (#3144)
This will break openobserve cloud , DON'T deploy UI changes until we
accommodate cookies for cloud
2024-04-03 05:35:54 +05:30
Hengfei Yang 6d0fbdd91d refactor: separate proto to a new crate (#3146) 2024-04-02 23:15:14 +08:00
Hengfei Yang 7d502419f1 feat: use database lock replace etcd lock (#3107) 2024-04-02 21:09:16 +08:00
Ashish Kolhe 91f02d1e6b chore: update native login & openapi specs (#3132) 2024-04-01 16:05:25 +05:30
Subhra264andomkark06 62b79416e3 feat: add custom hide menu field in the /config (#3125)
config settings for hiding menu items

---------

Co-authored-by: omkark06 <omkar@zinclabs.io>
2024-03-28 23:34:20 +05:30
Subhra264 f4a5150895 feat: add query start_time and end_time for reporting a search query usage (#3097)
Reports `query.start_time` and `query.end_time` as `min_ts` and `max_ts`
respectively for the "Search" event in `usage` stream.
2024-03-26 13:05:46 +05:30
Subhra264 08d12f8571 feat: introduce reports in backend and replace TRIGGERS with infra::scheduler (#3086)
## Reports
Both `ZO_CHROME_ENABLED` and `ZO_SMTP_ENABLED` ENV variables need to be
`true` for reports to work.

**Report APIs -**
```
POST /api/{org_id}/reports
PUT /api/{org_id}/reports/{report_name}
GET /api/{org_id}/reports
GET /api/{org_id}/reports/{report_name}
DELETE /api/{org_id}/reports/{report_name}
PUT /api/{org_id}/reports/{report_name}/enable?value=<true|false>
```

**Report body**
```
{
  "createdAt": "2024-03-23T09:46:32.913574782Z",
  "dashboards": [
    {
      "dashboard": "7177215109754781699",
      "folder": "default",
      "tabs": [
        "default"
      ],
      "timerange": {
        "from": 0,
        "period": "1h",
        "to": 0,
        "type": "relative"
      },
      "variables": []
    }
  ],
  "description": "Report description",
  "destinations": [
    {
      "email": "mail@gmail.com"
    }
  ],
  "enabled": false,
  "frequency": {
    "cron": "",
    "interval": 1,
    "type": "weeks"
  },
  "lastEditedBy": "root@example.com",
  "mediaType": "pdf",
  "message": "Email message",
  "name": "report_name",
  "orgId": "default",
  "owner": "root@example.com",
  "password": "ComplexPass#123",
  "start": 1711187192913571,
  "timezone": "Asia/Calcutta",
  "timezoneOffset": 330,
  "title": "Email subject",
  "user": "root@example.com"
}
```

**ENVs**
| ENV |  Description | 
|------|---------------|
| `ZO_CHROME_ENABLED` | Must be `true` for reports, Default - `false` |
| `ZO_CHROME_PATH` | If chrome is enabled, custom `chrome` executable
path can be specified. If not specified, it looks for chrome executable
in default locations. If still not found, it automatically downloads a
good known version of `chromium`. |
| `ZO_CHROME_CHECK_DEFAULT_PATH` | If `false`, it does not look for
chromium in default locations, Default is `true`. |
| `ZO_CHROME_DOWNLOAD_PATH` | If chromium can not be found in default
locations and also `ZO_CHROME_PATH` is not specified, it downloads the
system specific chromium in the given path. Default is `./download`
(gitignored). `chromium` is downloaded for the first time only,
afterwords, `chromium` is fetched from the given path. If there is any
error regarding download of `chromium`, delete the download folder as it
might be in a bad state. |
| `ZO_CHROME_NO_SANDBOX` | If `true`, it launches chromium in
`no-sandbox` environment. Default is `false` |
| `ZO_CHROME_SLEEP_SECS` | Specify the number of seconds the
`service::dashboards::reports::generate_report()` will wait to load all
the data in the headless chromium dashboard. Default is `20` seconds. |
| `ZO_CHROME_WINDOW_WIDTH` | Specifies the width of the headless
chromium browser. Default is `1370` |
| `ZO_CHROME_WINDOW_HEIGHT` | Specifies the height of the headless
chromium browser. Default is `730` |

## Scheduler

Currently `Scheduler` is implemented for only `Sqlite`, `Postgres` and
`MySql`.
**API**
```rust
pub trait Scheduler: Sync + Send + 'static {
    async fn create_table(&self) -> Result<()>;
    async fn create_table_index(&self) -> Result<()>;
    async fn push(&self, trigger: Trigger) -> Result<()>;
    async fn delete(&self, org: &str, module: TriggerModule, key: &str) -> Result<()>;
    async fn update_status(
        &self,
        org: &str,
        module: TriggerModule,
        key: &str,
        status: TriggerStatus,
        retries: i32,
    ) -> Result<()>;
    async fn update_trigger(&self, trigger: Trigger) -> Result<()>;
    async fn pull(
        &self,
        concurrency: i64,
        alert_timeout: i64,
        report_timeout: i64,
    ) -> Result<Vec<Trigger>>;
    async fn clean_complete(&self, interval: u64);
    async fn watch_timeout(&self, interval: u64);
    async fn len_module(&self, module: TriggerModule) -> usize;
    async fn len(&self) -> usize;
    async fn is_empty(&self) -> bool;
    async fn clear(&self) -> Result<()>;
}
```

**ENVs**
| ENV |  Description | 
|------|---------------|
| `ZO_ALERT_SCHEDULE_CONCURRENCY` | The number of `scheduled_jobs` the
`alert_manager` can process at a time. Default is `5` |
| `ZO_ALERT_SCHEDULE_TIMEOUT` | Visibility timeout for alerts. Default
is `90` seconds |
| `ZO_REPORT_SCHEDULE_TIMEOUT` | Visibility timeout for reports. Default
is `300` seconds |
| `ZO_SCHEDULER_MAX_RETRIES` | Maximum number of retries that will be
made for a `scheduled_job`. Default is `3` |
| `ZO_SCHEDULER_CLEAN_INTERVAL` | Time interval after which the
`Scheduler` will clean up the queue. Default is `30` seconds |
| `ZO_SCHEDULER_WATCH_INTERVAL` | Time interval after which the
`Scheduler` will watch for timeout for `scheduled_jobs`. Default is `30`
seconds. |

**Workflow**

![openobserve_scheduler_workflow](https://github.com/openobserve/openobserve/assets/59690052/68beb43b-d04f-49dc-9006-a30942acd99b)
2024-03-25 18:24:14 +05:30
Hengfei Yang d4b0580205 feat: add new endpoint for health check (#3090) 2024-03-25 10:22:36 +08:00
7c52e0ea7a fix: schema versions as separate key (#3021)
To enable row per schema version use:
ZO_ENTRY_PER_SCHEMA_VERSION_ENABLED = true.

One can migrate schemas from single row to row per schema version using
: migrate-schemas command

Before migration :

- DROP INDEX meta_module_key2_idx ON meta
- ALTER TABLE meta ADD COLUMN start_dt BIGINT NOT NULL DEFAULT 0;
- CREATE UNIQUE INDEX meta_module_start_dt_idx ON meta (start_dt, key2,
key1, module);

---------

Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
Co-authored-by: Ankur Srivastava <ansrivas@users.noreply.github.com>
Co-authored-by: Huaijin <haohuaijin@gmail.com>
Co-authored-by: ktx-abhay <141122205+ktx-abhay@users.noreply.github.com>
Co-authored-by: Subhra264 <chakrabortysubhradeep556@gmail.com>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
Co-authored-by: ktx-vaidehi <vaidehi.akhani@kiara.tech>
Co-authored-by: Omkar Kesarkhane <127078473+omkarK06@users.noreply.github.com>
Co-authored-by: Kirtan Thakkar <kirtan@kiara.tech>
2024-03-24 21:25:06 +05:30
Hengfei Yang fd49360f49 feat: add quick mode fields (#3068) 2024-03-22 15:54:39 +08:00