Commit Graph
1709 Commits
Author SHA1 Message Date
dependabot[bot] 0a55c32c8f chore(deps): bump EmbarkStudios/cargo-deny-action from 1.6.1 to 1.6.2 (#3100) 2024-03-30 21:40:55 +08:00
dependabot[bot] 600006a429 chore(deps-dev): bump express from 4.18.2 to 4.19.2 in /web (#3113) 2024-03-30 21:40:10 +08:00
dongjinlong 202f4eeded chore: fix some typos (#3104) 2024-03-30 21:39:47 +08:00
Neha P 898cb1cc47 test: testcases for quick mode (#3124)
- Verify if quick mode is displayed
- Verify if user is able to click on interesting fields
- Verify on clicking interesting fields user is able to view the field
in sql mode
- Verify if on clicking interesting field when histogram is enabled,
expected results are displayed
- Verify if error displayed on entering random text in histogram mode
when quick mode is on
- Verify should display order by as default when switched to sql mode
- Verify if on page refresh the orderby is still present by default in
sql mode
- Verify if on page adding removing the interesting field removes in
from sql mode and the results too
2024-03-30 11:07:21 +05:30
Ashish Kolhe 8a65ade1a2 fix: schema inference issue (#3128) 2024-03-29 17:29:53 +05:30
Ashish Kolhe 943873b071 fix: schema inference issue (#3127) 2024-03-29 13:38:11 +05:30
Omkar Kesarkhane 5be80450de fix: stream names with hyphen (-) were not getting parsed correctly (#3126)
Stream names with hyphen (-) were not getting parsed correctly
2024-03-29 12:40:05 +05:30
Subhra264andomkark06 62b79416e3 feat: add custom hide menu field in the /config (#3125)
config settings for hiding menu items

---------

Co-authored-by: omkark06 <omkar@zinclabs.io>
2024-03-28 23:34:20 +05:30
Subhra264 0bf8c8aa5c feat: frequently check for span element render for dashboard report (#3102)
Instead of waiting for `CONFIG.chrome.chrome_sleep_secs` to let the data
load in the dashboard for report, frequently, check for the `span`
element with a loop.
2024-03-28 17:21:15 +05:30
Ashish Kolhe d307f2a2ea fix: blocked streams (#3123) 2024-03-28 16:06:31 +05:30
Ashish Kolhe ba1ff75900 fix: prod panic debug (#3117) 2024-03-28 15:49:07 +05:30
Ankur Srivastava 22fd621f40 test: run the forks on default github-runners (#3116)
For the incoming PRs from external contributors run the action on github
managed runners and everything else on ubicloud-runners
2024-03-27 18:13:39 +01:00
Ankur Srivastava 0c9ad5f357 test: use full name before checking out the repo (#3115) 2024-03-27 16:23:34 +01:00
Omkar Kesarkhane 6e47372cfe fix: saved view issues (#3112)
fix #3109
2024-03-27 19:37:07 +05:30
Ashish Kolhe 124a5f59dd fix: schema settings to be retained (#3114) 2024-03-27 18:41:23 +05:30
Omkar Kesarkhane 6af0e663d7 feat: Dashboard Reporting UI (#3098) 2024-03-27 18:11:13 +05:30
Ankur Srivastava 77c34185b5 test: Improve secret handling in forks (#3111)
ensure that recording functionality is skipped during the forked PRs for
the cypress tests.
2024-03-27 13:12:15 +01:00
Ashish Kolhe 318bbe306e feat: schema lock per request per stream (#3110) 2024-03-27 16:20:54 +05:30
Bhargav 96a08ef31c fix: append condition (#3108) 2024-03-26 22:55:58 +05:30
Bhargav 001ed9b735 fix: Saved views (#3105) 2024-03-26 22:03:14 +05:30
Omkar Kesarkhane aabeb04254 fix: added order by _timestamp desc in logs query (#3103)
In sql mode, if query does not contains  " order by " it throws error.

Solution:
On switching to sql mode, by default added "order by _timestamp desc" in
logs query
2024-03-26 20:34:49 +05:30
Bhargav 09b3b571d3 fix: function icon (#3106) 2024-03-26 18:54:15 +05:30
ktx-vaidehi a008b75de3 fix: dashboard variable dropdown on typing getting error (#3101)
While implementing a filter search feature with dropdown menus for
variables, an error occurred due to undefined values. This error
hindered the functionality of the search feature, causing unexpected
behavior when attempting to filter results.
2024-03-26 14:09:24 +05:30
Subhra264 f4a5150895 feat: add query start_time and end_time for reporting a search query usage (#3097)
Reports `query.start_time` and `query.end_time` as `min_ts` and `max_ts`
respectively for the "Search" event in `usage` stream.
2024-03-26 13:05:46 +05:30
Subhra264 08d12f8571 feat: introduce reports in backend and replace TRIGGERS with infra::scheduler (#3086)
## Reports
Both `ZO_CHROME_ENABLED` and `ZO_SMTP_ENABLED` ENV variables need to be
`true` for reports to work.

**Report APIs -**
```
POST /api/{org_id}/reports
PUT /api/{org_id}/reports/{report_name}
GET /api/{org_id}/reports
GET /api/{org_id}/reports/{report_name}
DELETE /api/{org_id}/reports/{report_name}
PUT /api/{org_id}/reports/{report_name}/enable?value=<true|false>
```

**Report body**
```
{
  "createdAt": "2024-03-23T09:46:32.913574782Z",
  "dashboards": [
    {
      "dashboard": "7177215109754781699",
      "folder": "default",
      "tabs": [
        "default"
      ],
      "timerange": {
        "from": 0,
        "period": "1h",
        "to": 0,
        "type": "relative"
      },
      "variables": []
    }
  ],
  "description": "Report description",
  "destinations": [
    {
      "email": "mail@gmail.com"
    }
  ],
  "enabled": false,
  "frequency": {
    "cron": "",
    "interval": 1,
    "type": "weeks"
  },
  "lastEditedBy": "root@example.com",
  "mediaType": "pdf",
  "message": "Email message",
  "name": "report_name",
  "orgId": "default",
  "owner": "root@example.com",
  "password": "ComplexPass#123",
  "start": 1711187192913571,
  "timezone": "Asia/Calcutta",
  "timezoneOffset": 330,
  "title": "Email subject",
  "user": "root@example.com"
}
```

**ENVs**
| ENV |  Description | 
|------|---------------|
| `ZO_CHROME_ENABLED` | Must be `true` for reports, Default - `false` |
| `ZO_CHROME_PATH` | If chrome is enabled, custom `chrome` executable
path can be specified. If not specified, it looks for chrome executable
in default locations. If still not found, it automatically downloads a
good known version of `chromium`. |
| `ZO_CHROME_CHECK_DEFAULT_PATH` | If `false`, it does not look for
chromium in default locations, Default is `true`. |
| `ZO_CHROME_DOWNLOAD_PATH` | If chromium can not be found in default
locations and also `ZO_CHROME_PATH` is not specified, it downloads the
system specific chromium in the given path. Default is `./download`
(gitignored). `chromium` is downloaded for the first time only,
afterwords, `chromium` is fetched from the given path. If there is any
error regarding download of `chromium`, delete the download folder as it
might be in a bad state. |
| `ZO_CHROME_NO_SANDBOX` | If `true`, it launches chromium in
`no-sandbox` environment. Default is `false` |
| `ZO_CHROME_SLEEP_SECS` | Specify the number of seconds the
`service::dashboards::reports::generate_report()` will wait to load all
the data in the headless chromium dashboard. Default is `20` seconds. |
| `ZO_CHROME_WINDOW_WIDTH` | Specifies the width of the headless
chromium browser. Default is `1370` |
| `ZO_CHROME_WINDOW_HEIGHT` | Specifies the height of the headless
chromium browser. Default is `730` |

## Scheduler

Currently `Scheduler` is implemented for only `Sqlite`, `Postgres` and
`MySql`.
**API**
```rust
pub trait Scheduler: Sync + Send + 'static {
    async fn create_table(&self) -> Result<()>;
    async fn create_table_index(&self) -> Result<()>;
    async fn push(&self, trigger: Trigger) -> Result<()>;
    async fn delete(&self, org: &str, module: TriggerModule, key: &str) -> Result<()>;
    async fn update_status(
        &self,
        org: &str,
        module: TriggerModule,
        key: &str,
        status: TriggerStatus,
        retries: i32,
    ) -> Result<()>;
    async fn update_trigger(&self, trigger: Trigger) -> Result<()>;
    async fn pull(
        &self,
        concurrency: i64,
        alert_timeout: i64,
        report_timeout: i64,
    ) -> Result<Vec<Trigger>>;
    async fn clean_complete(&self, interval: u64);
    async fn watch_timeout(&self, interval: u64);
    async fn len_module(&self, module: TriggerModule) -> usize;
    async fn len(&self) -> usize;
    async fn is_empty(&self) -> bool;
    async fn clear(&self) -> Result<()>;
}
```

**ENVs**
| ENV |  Description | 
|------|---------------|
| `ZO_ALERT_SCHEDULE_CONCURRENCY` | The number of `scheduled_jobs` the
`alert_manager` can process at a time. Default is `5` |
| `ZO_ALERT_SCHEDULE_TIMEOUT` | Visibility timeout for alerts. Default
is `90` seconds |
| `ZO_REPORT_SCHEDULE_TIMEOUT` | Visibility timeout for reports. Default
is `300` seconds |
| `ZO_SCHEDULER_MAX_RETRIES` | Maximum number of retries that will be
made for a `scheduled_job`. Default is `3` |
| `ZO_SCHEDULER_CLEAN_INTERVAL` | Time interval after which the
`Scheduler` will clean up the queue. Default is `30` seconds |
| `ZO_SCHEDULER_WATCH_INTERVAL` | Time interval after which the
`Scheduler` will watch for timeout for `scheduled_jobs`. Default is `30`
seconds. |

**Workflow**

![openobserve_scheduler_workflow](https://github.com/openobserve/openobserve/assets/59690052/68beb43b-d04f-49dc-9006-a30942acd99b)
2024-03-25 18:24:14 +05:30
Huaijin cd264bee5a perf: improve field search by hashmap (#3096) 2024-03-25 19:35:29 +08:00
Bhargav 8dca74b03e fix: added pagination in logs and stream page for schema field list (#3093) 2024-03-25 15:12:21 +05:30
Hengfei Yang d4b0580205 feat: add new endpoint for health check (#3090) 2024-03-25 10:22:36 +08:00
7c52e0ea7a fix: schema versions as separate key (#3021)
To enable row per schema version use:
ZO_ENTRY_PER_SCHEMA_VERSION_ENABLED = true.

One can migrate schemas from single row to row per schema version using
: migrate-schemas command

Before migration :

- DROP INDEX meta_module_key2_idx ON meta
- ALTER TABLE meta ADD COLUMN start_dt BIGINT NOT NULL DEFAULT 0;
- CREATE UNIQUE INDEX meta_module_start_dt_idx ON meta (start_dt, key2,
key1, module);

---------

Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
Co-authored-by: Ankur Srivastava <ansrivas@users.noreply.github.com>
Co-authored-by: Huaijin <haohuaijin@gmail.com>
Co-authored-by: ktx-abhay <141122205+ktx-abhay@users.noreply.github.com>
Co-authored-by: Subhra264 <chakrabortysubhradeep556@gmail.com>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
Co-authored-by: ktx-vaidehi <vaidehi.akhani@kiara.tech>
Co-authored-by: Omkar Kesarkhane <127078473+omkarK06@users.noreply.github.com>
Co-authored-by: Kirtan Thakkar <kirtan@kiara.tech>
2024-03-24 21:25:06 +05:30
Bhargav ea9adbb114 fix: deselect all field should replace * (#3085) 2024-03-23 15:20:18 +05:30
Bhargav 41603561c7 fix: Update IndexList.vue (#3083) 2024-03-23 15:00:28 +05:30
Bhargav b42f2708d2 fix: fixed custom logo spacing issue & add remove interesting field (#3082) 2024-03-23 13:23:14 +05:30
Bhargav 0d2b35906f feat: Quick mode fixes (#3079) 2024-03-23 10:18:33 +05:30
BhargavandHengfei Yang a9285e731c fix: url safe encoding. (#3061)
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
2024-03-23 10:16:34 +08:00
Hengfei Yang f12c697be2 fix: disk cache with s3 bucket prefix (#3075)
fixed #3074
2024-03-23 09:34:54 +08:00
Bhargav 9b7b360b90 fix: minor changes (#3072) 2024-03-22 18:54:58 +05:30
Neha P d0b9d5043f test: added drilldown tests (#3070)
- should not allow adding drilldown without name
- should add variable under drilldown and verify if drilldown is added 
- delete the created dashboard once verification is done
2024-03-22 17:42:07 +05:30
BhargavandHengfei Yang 0ad6888e5c feat: Interesting fields UI changes (#3058)
- The notification has been eliminated during the configuration of
absolute dates.
- The SQL mode should not trigger the query automatically upon
deactivation if run query flag is true.
- The Histogram feature should remain inactive while SQL mode is
enabled.
- Modifications related to the quick mode.
- Within SQL mode, the "_timestamp" field is required, and an "order by"
clause is necessary.

---------

Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
2024-03-22 16:55:10 +05:30
dependabot[bot] 8164ae155a chore(deps-dev): bump webpack-dev-middleware from 5.3.3 to 5.3.4 in /web (#3069) 2024-03-22 16:35:00 +08:00
Hengfei Yang fd49360f49 feat: add quick mode fields (#3068) 2024-03-22 15:54:39 +08:00
Hengfei Yang 32bd2b8814 fix: column duplicated in quick mode (#3066) 2024-03-22 00:03:42 +08:00
Neha Pandktx-vaidehi 48b5247126 test: added aggregation queries and folders testcase fix (#3064)
- added search API tests for aggregation queries
- fixed dashboard-folders tests

---------

Co-authored-by: ktx-vaidehi <vaidehi.akhani@kiara.tech>
2024-03-21 18:19:20 +05:30
Omkar Kesarkhane 7a4b29af53 fix: enabled metadata streams in streams page (#3059)
fix #2849
2024-03-21 17:34:22 +05:30
Omkar Kesarkhane a831b4bf29 fix: updated traces duration filter functionality (#3052)
fix #3005 
1. By default added 0 min  100000 max duration
2. Removed duration slider
2024-03-21 17:10:47 +05:30
Omkar Kesarkhane 0ef345bfa9 fix: fixed event filtering condition in Session Viewer events query (#3049)
Problem:
Observed that the User Session Viewer was yielding incorrect results
when filtering session events data. The root cause was identified as the
improper handling of a compound predicate within the SQL query used for
data retrieval.

Solution:
When parentheses were added around the conditions `session_id='xyz' &&
(type='error' or type='action' or type='view')` to logically group them,
this fixed the issue.
2024-03-21 16:39:52 +05:30
胡同里有猫 3865a8d126 feat: add storage_type label for storage_time (#3063) 2024-03-21 17:28:02 +08:00
ktx-vaidehi c9ef8917a9 fix: error on blank input in dashboard panel config (#3043)
- #3042
2024-03-21 11:33:38 +05:30
ktx-vaidehi 6d30788861 fix: tabs default edit delete (#3033) 2024-03-21 10:55:28 +05:30
ktx-vaidehi 23d391477f feat: dashboard table text wrap (#3026) 2024-03-21 10:31:44 +05:30
Subhra264 423c1a44ef feat: introduce enterprise api for uploading/deleting custom logo (#3034) 2024-03-21 11:36:29 +08:00