Commit Graph
303 Commits
Author SHA1 Message Date
Hengfei Yang 611f2e1c29 perf: improve ingestion (#3678) 2024-06-12 22:01:03 +08:00
Subhra264 9e178fd0de fix: include trace_id in search usage reporting (#3703)
Fixes #3695 

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added `trace_id` support across various functions to enhance tracking
and debugging capabilities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-06-10 12:34:21 +05:30
04b7df63be feat: support presigned GET URLS (#3667)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Enhanced user authentication by adding extended password fields and
support for generating presigned login URLs.
- Improved user management with new functions for handling user data and
authentication tokens.

- **Bug Fixes**
- Fixed issues with user data handling by passing references instead of
values in database operations.

- **Refactor**
- Streamlined authentication and user management code for better
maintainability and performance.

- **Tests**
- Added new test cases to validate password hashing and presigned URL
generation functionalities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
2024-06-07 14:20:21 +02:00
Ashish Kolheandktx-abhay 79fe002f60 feat: include search type in search calls (#3648)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Introduced `searchType` attribute across various components to enhance
search functionality.
- Added support for different search event types like `UI`,
`Dashboards`, `Reports`, `Alerts`, `Values`, and `RUM`.

- **Bug Fixes**
- Improved search parameter handling in multiple components and API
calls.

- **Enhancements**
- Updated search-related functions to include `searchType` for more
precise search results.
- Enhanced dashboard and RUM views with new `searchType` attributes for
better data filtering.

- **Refactor**
- Standardized search functionality across different modules and
components.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: ktx-abhay <abhay.padamani@kiara.tech>
2024-06-03 14:14:54 +05:30
Hengfei Yang 3ccf0be933 feat: add limit -1 for query (#3658)
The query `size` works like this, it should more clear now:
1. size > 0, it will return hits equal to `size`
2. size = 0, it will return empty hits, this used to other purposes,
like: get total, get aggs.
3. size = -1, it will use a backend default limit to return hits,
setting by `ZO_QUERY_DEFAULT_LIMIT=1000`

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Bug Fixes**
- Corrected handling of query parameters by changing types from `usize`
to `i64` to support larger values and prevent potential overflow issues.

- **Improvements**
- Enhanced the logic for setting query limits to handle negative values
more gracefully, ensuring better default behavior.

- **Refactor**
- Updated various internal structures and function signatures to use
`i64` instead of `usize` for consistency and improved range handling.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-06-01 11:25:12 +05:30
Hengfei Yang 569bb121ce feat: add new node role flatten_compactor (#3589) 2024-05-30 18:34:41 +08:00
Ankur SrivastavaandHengfei Yang 8f3df9e7a1 feat: support reloading config vars using an API (#3623)
The new endpoint `/config/reload` will refresh the
environment variables.

There are a bunch of env-vars which will not be reloaded
for e.g. HTTP_PORT etc. these will need a restart.

---------

Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
2024-05-30 08:23:27 +08:00
Hengfei Yang c830fa6532 perf: use parquet file static cache (#3604) 2024-05-28 10:43:24 +08:00
Ankur Srivastava 28cf5b0776 feat: integrate embedded reportserver (#3583)
There are three cases to consider:

if ZO_ENABLE_EMBEDDED_REPORT_SERVER=true

If running a binary, alertmanager node will spawn an embedded server
running at localhost:5082.
All the requests from alert-manager will go to localhost:5082 for pdf
generation related tasks.

For single node k8s deployment, as this is running in a container there
is no chromium installed, so even if you set the variable to true, the
report server will panic due to missing dependencies. For this situation
helm chart will spawn a report-server instance which can be used.

Same for cluster deployment.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
  - Introduced a new report server for generating and sending reports.
  - Added health check and report sending endpoints.

- **Improvements**
  - Enhanced logging for report server operations.
- Configurable option to skip TLS verification for report server
connections.

- **Bug Fixes**
  - Fixed formatting issue in HTTP status request handler.

- **Chores**
- Updated dependencies and configurations for the new report server
module.

- **Documentation**
- Added new structures and functions for email configuration, report
generation, and server operations.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-05-27 11:19:50 +02:00
Hengfei Yang 470714616b fix: search partition with base64 (#3597)
fix #3557
2024-05-26 17:00:44 +08:00
Ashish Kolhe a000dad31b fix: get auth headers (#3578) 2024-05-25 03:17:57 -07:00
Taiming LiuandNeha P 37ebd33637 fix: _around api for query with limit (#3590)
fix #3587 

<img width="1286" alt="image"
src="https://github.com/openobserve/openobserve/assets/90490946/38ed9e95-6569-4015-a156-6040b0d3fc7c">

---------

Co-authored-by: Neha P <neha@openobserve.ai>
2024-05-24 16:50:35 -07:00
Ankur Srivastava e4f4389974 feat: support build-type in configuration (#3581)
![image](https://github.com/openobserve/openobserve/assets/1695056/35b2adcf-b461-4471-8962-0bd0410dca31)


![image](https://github.com/openobserve/openobserve/assets/1695056/118726c1-b276-4cae-8689-cb21ef3ae53c)
2024-05-23 18:20:18 +02:00
Taiming LiuandHengfei Yang a83c46c656 feat: cluster selection in logs (#3559)
- Add cluster selection in logs search queries for additional filtering
based on region/clusters
- add org/stream info in ingestion http handler error logging

---------

Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
2024-05-22 07:33:34 -07:00
Ashish Kolhe a03b4d14e0 feat: login using basic auth headers (#3550) 2024-05-21 12:13:09 +05:30
Ashish Kolhe 798c3469ae fix: invalid peer certificate (#3505)
- [x] invalid peer certificate: UnknownIssuer
- [x] SSO user - infinite loop when accessing query cancelation UI
- [x] Clusters API should be accessible with SSO user
- [x] Handle _timestamp for _all fields
2024-05-14 17:22:05 +05:30
Hengfei Yang 6f975a02f6 fix: around API with function (#3497) 2024-05-13 16:08:15 +08:00
Hengfei Yang 1878b7eb50 fix: region filter for _around API (#3492)
#3488
2024-05-12 19:37:22 +08:00
Ashish KolheandHengfei Yang ed73fcd5ff feat: only write defined schema fields to parquet (#3454)
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
2024-05-12 01:12:43 +08:00
Omkar Kesarkhaneandoasisk ee3a9c16e5 feat: User defined schema for existing streams (#3477)
1. Users can define schema fields for existing streams.
2. During querying/exploring only those defined schema fields will be
available in schema, rest all fields will be available in "_all" field
in the schema.

---------

Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
2024-05-11 11:34:45 +05:30
Ashish Kolhe 70741119fb fix: pipeline RBAC issues (#3470) 2024-05-10 17:54:37 +05:30
Hengfei Yang 517de8f05d fix: add region filter to _value and _around API (#3449) 2024-05-09 20:00:32 +08:00
Hengfei Yang dba09d7297 feat: add config for super cluster (#3440) 2024-05-09 10:41:36 +08:00
Hengfei Yang 0e986c4728 fix: super cluster region filter (#3437) 2024-05-09 09:06:22 +08:00
Hengfei Yang 654ed8f612 perf: value api (#3435) 2024-05-09 00:04:33 +08:00
Hengfei Yang c3ec7ebbd9 fix: value API (#3429) 2024-05-08 19:08:26 +08:00
Hengfei Yang 32b7ce4c26 feat: improve super cluster (#3428)
- [x] add timeout and partical success for super cluster
- [x] add region filter for super cluster
2024-05-08 16:32:54 +08:00
c9a4e00d69 feat: data ingestion pipeline (#3282)
Co-authored-by: omkark06 <omkar@zinclabs.io>
Co-authored-by: Neha P <neha@openobserve.ai>
2024-05-06 19:31:46 +05:30
Ashish KolheandNeha P a413179068 fix: only get required versions while watching (#3399)
Co-authored-by: Neha P <neha@openobserve.ai>
2024-05-06 16:24:46 +05:30
Ashish Kolhe 8a1cfddcc7 fix: RBAC for native users (#3404) 2024-05-03 12:44:32 +05:30
Hengfei Yang 13e1442acc perf: get schema versions for the time range (#3397) 2024-05-02 13:36:23 +08:00
Hengfei Yang 3ce71a1be3 fix: add compress for schema cache (#3396) 2024-05-01 22:54:50 +08:00
4b92c092a9 feat: Query Cancellation (#3207)
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
2024-04-30 22:41:56 +05:30
Subhra264andoasisk f9e200f82b fix: same role and group name in rbac with diff org conflicts (#3359)
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
2024-04-29 11:31:43 +05:30
Ashish Kolhe 15f9788a6e fix: format stream name for _bulk api (#3371) 2024-04-28 23:12:35 +05:30
Hengfei Yang 8a73f2cf05 fix: value API is always waiting (#3364)
fixed #3334
2024-04-28 22:30:07 +08:00
Ashish Kolhe a7222f7d65 fix: id token length issue (#3360) 2024-04-28 15:16:05 +05:30
Ashish KolheandHengfei Yang a88c5c4646 fix: user session id (#3343)
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
2024-04-25 20:51:25 +08:00
Subhra264 7bf9ee1fab feat: introduce cached_ratio to usage reporting for search usage type (#3341)
Fixes #3336
2024-04-25 14:08:23 +08:00
Hengfei Yang 15b22c1876 fix: schema cache error with prefix (#3338) 2024-04-25 10:54:40 +08:00
Subhra264 8482a913b3 fix: show savedview and report resources in role permissions and fix list_objects_for_user for _all_{org} (#3284)
For enterprise in rbac, `GET` methods for `reports` and `savedviews`
consists of `{org_id}/<reports|savedviews>/{id}` format, i.e. `url_len`
is 3. This PR corrects how the fga object string is extracted (which
should be `report:{report_name}` or `savedview:{view_id}`) from the url
path with the above format with `GET` method.
2024-04-23 15:09:09 +05:30
Subhra264 1e8d1e3cc1 fix: don't use stream_type from query params for alert create/update (#3311)
Fixes #3283
2024-04-23 07:25:21 +08:00
Hengfei Yang 8d1ead5105 fix: add more debug logs for search (#3280) 2024-04-19 20:01:39 +08:00
Ashish Kolheandomkark06 ebdc19d02e feat: routing & user defined schema restrictions (#3254)
Co-authored-by: omkark06 <omkar@zinclabs.io>
2024-04-19 06:00:56 +05:30
momantech 24bd5f4fe8 docs: fix some typos in comments (#3267) 2024-04-17 19:31:33 +08:00
Huaijin 0ec3285e1a fix: add order by field to select when need (#3259) 2024-04-16 18:04:55 +08:00
Subhra264 e9e8160391 feat: add audit trails for enterprise version (#3211) 2024-04-16 11:40:32 +05:30
Ashish KolheandBhargav 1c1b5c5e7f fix: cookies issue for SSO (#3244)
Co-authored-by: Bhargav <BJP232004@GMAIL.COM>
2024-04-16 07:27:33 +05:30
HuaijinandHengfei Yang cacf46e480 feat: only enable tracing in search (#3230)
close #3215 
use follow to only enable tracing in search
```
ZO_TRACING_ENABLED=false
ZO_TRACING_SEARCH_ENABLED=true 
```

---------

Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
2024-04-15 18:43:34 +08:00
Hengfei Yang a48326861f feat: complete super cluster (#3226) 2024-04-12 13:45:02 +08:00