Commit Graph
201 Commits
Author SHA1 Message Date
Ankur Srivastava 76d7678acd fix: fixes the create-folder docs in #1649 (#1654) 2023-10-15 21:20:35 -05:00
Hengfei Yang 13ad3f3163 feat: use the node which has the latest file_list (#1591) 2023-10-02 08:52:04 -05:00
Ashish Kolhe 8cd8e0b550 fix: logs ingestion refactor (#1527) 2023-10-02 11:20:30 +05:30
Hengfei Yang 031d5dc856 fix: optimize file list broadcast (#1589) 2023-09-30 22:25:35 -05:00
Hengfei Yangandoasisk bb1bd0049b feat: optimize file list (#1577)
## Implement

- [x] Implement file_list API for gRPC
- [x] Debuging deadlock issue

## Todo

- [ ] Querier should use the file_list API for search

---------

Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
2023-09-29 15:01:10 +05:30
Ashish Kolhe 84fc09ef0b fix: default folder rename issue (#1585)
We create default folder , when we first receive call to create a
dashboard in default folder. If an API call tries to create a default
folder , it should not be allowed
2023-09-29 12:32:31 +05:30
Ashish Kolhe 9063a8c7b3 fix: Chunking of Data Prevents Uploads of Large CSV Files to Enrichment Tables (#1560) 2023-09-25 11:31:32 +05:30
Ashish Kolhe 47bea0abb9 feat: folders for dashboards (#1559)
Adds support for folder for dashboards, which helps user organise
dashboards in folders.

We should support folders for dashboards like grafana
:https://grafana.com/docs/grafana-cloud/infrastructure-as-code/grizzly/dashboards-folders-datasources/#add-a-folder

Provides API's for :
https://github.com/openobserve/openobserve/issues/1053
2023-09-22 15:54:10 +05:30
Ashish Kolhe af2d805be2 fix: OTLP ingestion fixes (#1547)
Fixes https://github.com/openobserve/openobserve/issues/1544

- [x] Accept ingestion token for v1/logs v1/metrics 
- [x] Introduce optional header Stream_Name to accept stream name
- [x] change zinc_org_id to Organisation
2023-09-20 13:05:26 +05:30
Hengfei Yang c6a46ac315 feat: upgrade datafusion to 31 (#1506) 2023-09-19 17:43:17 +08:00
Ashish Kolhe 9900b795e9 fix: Syslog ingestion message values are 'cut off' #1514 (#1529)
Before the fix supported length in bytes by both TCP & UDP was limited
to 1024 bytes, which resulted in values being cut off.

We had two options as fix : 

- Support max size in line with standard [IEEE
802.3](https://en.wikipedia.org/wiki/IEEE_802.3) , which limits 1500
bytes of payload , with this we can support 1472 bytes (1500 - 20 (IP
header ) -8 (UDP header) ) for UDP & 1460 bytes(1500 - 20 (IP header )
-20 (TCP header) ) for TCP
- JUMBO frames - Ruled out because : for jumbo frames , frame size used
by an end-to-end connection is typically limited by the lowest frame
size in intermediate links ,not only sender & receiver has to enable
jumbo frames but all intermediate links will have to support it.

Fixes #1514

**Summary : we will support max 1472 bytes for UDP & 1460 bytes for
TCP**
2023-09-18 22:47:27 +05:30
Hengfei Yang e885950b70 feat: add more debug logs (#1515) 2023-09-15 12:30:57 +08:00
Hengfei Yang 98c00665a4 fix: dynamo migration (#1504) 2023-09-13 19:53:07 +08:00
Ashish Kolhe e2c0c8c91b fix: vrl , values api filtering (#1490)
- [x] Add . if not already added to vrl functions
- [x] Server side fix for :
https://github.com/openobserve/openobserve/issues/699
- [x] Add error logs
- [x] Fixes : https://github.com/openobserve/openobserve/issues/1381
2023-09-12 14:44:33 +05:30
Hengfei Yang 9d1f82d0fd fix: cancel datafusion execution when it timeout (#1481) 2023-09-11 18:46:59 +08:00
Ashish Kolhe 56655cdbef feat: otlp http for logs & metrics (#1482) 2023-09-11 15:25:31 +05:30
Hengfei Yang 019c4ccc53 fix: swagger under sub directory (#1476) 2023-09-09 12:36:23 +08:00
Hengfei Yang c690bca900 fix: top values api error when the filter is empty (#1475) 2023-09-09 11:01:31 +08:00
Hengfei Yang 7d9f0d7e92 feat: add disk cache (#1472) 2023-09-08 18:40:31 +08:00
Ashish Kolhe c0114e36f6 feat: dashboard versioning (#1464) 2023-09-06 15:45:44 +05:30
Hengfei Yang 7b688e7236 feat: disable dictionary for log fields (#1460) 2023-09-06 09:18:29 +08:00
Hengfei Yang 5a3173c689 feat: implement sqlite for metadata (#1449) 2023-09-04 11:00:28 +08:00
Hengfei Yang c9bfc5bbb0 feat: implement stream stats with file list (#1433) 2023-08-29 19:00:05 +08:00
Hengfei Yang 8b74b952f7 feat: new file list (#1376) 2023-08-25 09:49:30 +08:00
Ashish Kolheandhengfeiyang 6312be63f9 feat: grpc logs & metrics support (#1399)
Co-authored-by: hengfeiyang <hengfei.yang@gmail.com>
2023-08-23 14:00:17 +05:30
Hengfei Yang bf7f6f5d13 feat: improve wal file with partition key (#1407) 2023-08-23 15:47:53 +08:00
Ashish Kolhe c013e90532 fix: traces issues 1316 1090 (#1410) 2023-08-22 16:06:20 +08:00
Hengfei Yang af521fde54 fix: update license (#1404) 2023-08-21 15:37:50 +08:00
Hengfei Yang 5f3739c45c fix: debug deadlock (#1389) 2023-08-16 20:06:13 +08:00
Hengfei Yang f4b8e7c820 perf: improve search around (#1334) 2023-08-04 21:51:08 +08:00
Hengfei Yang 610c1ec5a6 fix: file_list with daily partition (#1330) 2023-08-04 20:35:18 +08:00
Hengfei Yang b3189f246c perf: optimize parquet write params (#1323)
- [x] optimize parquet write params
- [x] add schedule fetch latest file_list to local cache
- [x] optimize CI for release mode
- [x] deadlock by usage report
2023-08-04 11:28:33 +05:30
Ashish Kolhe 785516f26b fix: calculate stats (#1322) 2023-08-03 15:59:17 +05:30
xiaoziv 80d7c9c662 refactor: remove expensive clone operation (#1291) 2023-07-31 16:19:02 +08:00
Hengfei Yang 339787b015 feat: add prometheus format query API (#1240) 2023-07-23 22:43:51 +08:00
Ashish Kolheandhengfeiyang ffacf619c6 feat: use dynamo db (#1220)
Co-authored-by: hengfeiyang <hengfei.yang@gmail.com>
2023-07-21 22:22:35 +05:30
Hengfei Yang a7152f1e3f fix: broadcast event failed (#1188) 2023-07-15 19:25:52 +08:00
Ashish Kolhe 45b018f02b fix: data casting & report usage (#1185) 2023-07-14 11:32:56 +05:30
Hengfei Yang 97d65e86cf fix: optimize some query cases (#1172) 2023-07-13 20:46:27 +08:00
Hengfei Yangandoasisk f1fa874faa fix: upgrade to datafusion 27 (#1140)
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
2023-07-11 20:41:16 +08:00
Ankur Srivastava e2dd047bf5 refactor: move infra and meta inside common (#1105)
This is the first step towards splitting the repo into smaller sub
projects.
2023-07-06 12:32:07 +05:30
Hengfei Yang cbbcdc135f fix: organization orders (#1082) 2023-07-04 12:26:39 +05:30
Hengfei Yang 4829ed7cf0 fix: add records for scan stats report (#1060) 2023-06-29 00:18:35 +08:00
Ankur Srivastava c505d78c46 feat: add redirects for swagger docs (#1049)
- /swagger -> /swagger/
- /docs    -> /swagger/
2023-06-27 18:01:09 +05:30
Ashish Kolhe f58ad652d1 fix: functions disabled due to feature flag (#1047) 2023-06-27 18:43:40 +08:00
Hengfei Yang 820ded2086 fix: root user should list all organizations include created empty orgs (#1038) 2023-06-27 00:19:52 +08:00
Ashish Kolhe b23751e4ce feat: usage & billing changes (#1033) 2023-06-26 17:32:35 +05:30
Hengfei Yangandoasisk 5bfe676244 perf: some improve (#1008)
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
2023-06-22 14:25:18 +05:30
Hengfei Yang 7e6ec82674 fix: readme link (#1005) 2023-06-22 09:53:31 +08:00
Hengfei Yang 8c74eaac84 feat: impl metrics http json API (#999)
## Design Spec:

- POST `/api/:org/ingest/metrics/_json`

```json
[
    {
        "__name__": "stream name",
        "__type__": "counter / gauge / histogram / summary",
        "label_name1": "label_value1",
        "label_name2": "label_value2",
        "_timestamp": 0, // unix timestamp in millisecond
        "value": 0.0 // float64 value
    }
]
```

## How to use this API:
- must use `__name__` set stream name.
- must use `__type__` set metrics type. and the type can't be updated
after first record
- `_timestamp` needs UNIX timestamp or empty will fill by current
timestamp
- `value` should be float64 or int64

## Example:

POST `/api/default/ingest/metrics/_json`
```json
[
    {
        "__name__": "cpu",
        "__type__": "gauge",
        "namespace": "ziox",
        "container": "zo1",
        "_timestamp": 1687326643584062,
        "value": 10.1
    },
    {
        "__name__": "cpu",
        "__type__": "gauge",
        "namespace": "ziox",
        "container": "zo2",
        "value": 0.1
    }
]
```
2023-06-21 23:02:41 +08:00