Commit Graph
197 Commits
Author SHA1 Message Date
Hengfei Yang 3ce0db827e fix: #4927 (#4928) 2024-10-28 20:00:32 +08:00
Hengfei Yang ff5a2cdeb2 fix: change some print to logs (#4926) 2024-10-28 17:02:39 +08:00
Hengfei Yang 87d2bb862a fix: bugs of v0.13.0-rc1 (#4916) 2024-10-26 09:08:09 +08:00
Huaijin 56dcacb513 feat: use seaorm to refactor short_urls (#4862) 2024-10-24 19:59:06 +08:00
Hengfei Yang 71876ab5b2 fix: can't query data in the case with multiple memtable buckets (#4867) 2024-10-22 19:39:04 +08:00
Hengfei Yang b72ac5b8d7 feat: Alerts support SNS destination (#4824) 2024-10-18 20:27:43 +08:00
Yashodhan Joshi a46ad6cea7 feat: add prefix based paritioning for index (#4709) 2024-10-13 19:40:23 +08:00
Huaijin 07b7df6fed fix: upgrade dependencies (#4762) 2024-10-11 18:40:27 +08:00
Loakesh Indiran f69593ba26 feat: short url service (#4697) 2024-10-07 20:03:37 +08:00
Hengfei Yang 309cddaf27 perf: query file_list in parallels by day (#4669) 2024-10-01 23:21:32 +08:00
Ashish Kolhe 35d25d7691 fix: add partial response warning (#4658)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
	- Enhanced query handling with restrictions based on stream settings.
	- Improved response messaging to indicate partial data availability.

- **Bug Fixes**
- Adjusted query start time when exceeding maximum duration limits,
ensuring compliance with stream settings.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-09-27 15:41:19 +05:30
Hengfei Yang 9a1c7eedf4 feat: add without connection pool mode for router (#4609) 2024-09-25 23:43:54 +08:00
John Vandenberg d0079bb9bf docs: Fix typos (#4492) 2024-09-22 17:57:47 +08:00
Hengfei Yang d1ed457369 feat: new search engine (#4182) 2024-09-20 15:10:54 +08:00
Yashodhan Joshi 8d5ebbb3c8 feat: optimise file list querying for better search perf (#4518) 2024-09-19 22:45:33 +08:00
Yashodhan Joshi d45ac8f9ad fix: unify the trace ingestion handling (#4435) 2024-09-05 13:33:15 +08:00
Hengfei Yang bb4881c99c feat: add datafusion json functions (#4321) 2024-09-04 11:30:24 +08:00
Taiming Liu 7f75a4d6db feat: use FST for inverted indexing (#4270) 2024-09-02 22:39:44 +08:00
Hengfei Yang 3a079ec06d feat: upgrade to datafusion 41 (#4180) 2024-08-12 13:07:55 +08:00
df2488fa44 feat: optimize sql to plan execution (#4100)
Now we use sql to plan to execute the query. 

we split the PhysicalPlan to two stage:
1. Partial plan, it can be distributed will run on all the nodes.
2. Final plan, it can't be distributed will run on the leader querier
and merge the result.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit


- **New Features**
- Updated to version 0.10.9, introducing enhancements and new
dependencies for improved data processing functionalities.
- Added new Protocol Buffers schemas to define structures for cluster
management and data processing frameworks, enhancing integration
capabilities.

- **Bug Fixes**
  - Improved error handling in configuration to prevent runtime panics.
- Enhanced logic in caching mechanisms for more reliable query
responses.

- **Documentation**
- Added module declarations to improve organization and clarity within
the codebase.

- **Refactor**
- Streamlined execution plans and SQL processing for better performance
and maintainability.

- **Tests**
- Introduced new functions for log ingestion in test suites to enhance
reliability and coverage.

- **Style**
- Made formatting adjustments across various files to improve
readability and consistency.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Neha P <neha@openobserve.ai>
Co-authored-by: oasisk <ashish.j.kolhe@gmail.com>
2024-08-05 18:38:35 +08:00
Ashish Kolhe 8ec1be9eaa chore: upgrade vrl version (#4105)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit


- **New Features**
	- Updated critical dependencies to enhance software capabilities.
- Improved data structure management in StreamTable and Geoip
implementations for better performance and clarity.

- **Bug Fixes**
	- Enhanced efficiency of entry removal in the data persistence process.

- **Chores**
- Routine updates to external package versions for improved
functionality and security.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-08-05 09:29:01 +05:30
Hengfei Yang 766d5a0244 fix: search partition with order by time asc (#4009) 2024-07-18 23:27:07 +08:00
Hengfei Yang 99970f7d0c feat: use cache for value API (#3997)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **Bug Fixes**
	- Improved caching behavior and response processing in search requests.
- Enhanced error handling and response generation for better user
experience.

- **Tests**
- Disabled specific UI interaction in Playwright tests to prevent test
failures.

- **Chores**
	- Updated configuration defaults to improve clarity and efficiency.
	- Made internal constants publicly accessible for broader usage.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-07-18 08:09:12 +05:30
Hengfei Yang effdc28b3f ci: bump datafusion to 40 (#3968) 2024-07-13 20:16:18 +08:00
Hengfei Yang 274a4e3ff1 fix: invalid peer certificate: UnknownIssuer for report traces (#3949) 2024-07-10 20:33:44 +08:00
Hengfei Yang 46a5af14fe feat: secondary index (#3870) 2024-07-10 15:08:07 +08:00
dependabot[bot] a5960b8226 chore(deps): bump zerovec-derive from 0.10.2 to 0.10.3 (#3936) 2024-07-10 12:17:27 +08:00
dependabot[bot] 3129a58e5a chore(deps): bump zerovec from 0.10.2 to 0.10.4 (#3927) 2024-07-09 12:16:43 +08:00
Ashish Kolhe 45dce09ff3 fix: query result cache v2 (#3895)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added new fields `is_descending` and `limit` to query responses for
enhanced sorting and pagination.

- **Improvements**
- Enhanced logging and tracing for better query time tracking and
debugging.
- Streamlined span handling for more efficient tracing and
instrumentation.

- **Bug Fixes**
- Fixed validation for `result_cache_enabled` configuration to prevent
cache-related errors.
  - Improved accuracy in cache limit handling in query responses.

- **Chores**
- Updated Rust toolchain version for improved compatibility and
performance.


<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-07-08 20:33:18 +05:30
Hengfei Yang 03f1b192c2 feat: ugprade rust to 2024-07-07 (#3922) 2024-07-08 20:56:56 +08:00
Huaijin 376aa00972 fix: sort record batches when persist to parquet (#3890) 2024-07-03 17:34:55 +08:00
Huaijin 20bc0f80b2 feat: add tracing for get file list (#3806) 2024-06-24 22:30:58 +08:00
Hengfei Yang e7a158eb87 fix: partition time level check for compactor job (#3782)
We need check the partition time level with stream type.
2024-06-20 11:59:14 -04:00
dependabot[bot] 01e38fc48f chore(deps): bump curve25519-dalek from 4.1.2 to 4.1.3 (#3765) 2024-06-19 08:59:15 +08:00
Hengfei Yang 02a1fe4a9d feat: set bloom filter for index stream (#3739)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Enhanced schema handling logic to ensure correct processing based on
metadata.
- Added bloom filter settings for specific fields in schema creation and
updates.

- **Bug Fixes**
- Improved error handling in schema index generation to manage missing
schemas effectively.

- **Chores**
	- Updated version from "0.10.7" to "0.10.8".

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-06-14 21:32:46 +05:30
Hengfei Yang 53e45b6b90 chore(deps): update to datafusion 39 (#3734) 2024-06-14 11:43:43 +08:00
Hengfei Yang 611f2e1c29 perf: improve ingestion (#3678) 2024-06-12 22:01:03 +08:00
Hengfei Yang ac1ced2ac6 feat: allow disable gxhash dependency (#3660) 2024-06-03 14:16:36 +08:00
Hengfei Yang 950e14c54d perf: improve file move on ingester (#3622) 2024-05-30 20:55:16 +08:00
Hengfei Yang 7d6b983955 fix: reload .env (#3625) 2024-05-30 09:48:27 +08:00
Ankur SrivastavaandHengfei Yang 8f3df9e7a1 feat: support reloading config vars using an API (#3623)
The new endpoint `/config/reload` will refresh the
environment variables.

There are a bunch of env-vars which will not be reloaded
for e.g. HTTP_PORT etc. these will need a restart.

---------

Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
2024-05-30 08:23:27 +08:00
Hengfei Yang c830fa6532 perf: use parquet file static cache (#3604) 2024-05-28 10:43:24 +08:00
Ankur Srivastava 28cf5b0776 feat: integrate embedded reportserver (#3583)
There are three cases to consider:

if ZO_ENABLE_EMBEDDED_REPORT_SERVER=true

If running a binary, alertmanager node will spawn an embedded server
running at localhost:5082.
All the requests from alert-manager will go to localhost:5082 for pdf
generation related tasks.

For single node k8s deployment, as this is running in a container there
is no chromium installed, so even if you set the variable to true, the
report server will panic due to missing dependencies. For this situation
helm chart will spawn a report-server instance which can be used.

Same for cluster deployment.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
  - Introduced a new report server for generating and sending reports.
  - Added health check and report sending endpoints.

- **Improvements**
  - Enhanced logging for report server operations.
- Configurable option to skip TLS verification for report server
connections.

- **Bug Fixes**
  - Fixed formatting issue in HTTP status request handler.

- **Chores**
- Updated dependencies and configurations for the new report server
module.

- **Documentation**
- Added new structures and functions for email configuration, report
generation, and server operations.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-05-27 11:19:50 +02:00
Hengfei Yang d84726e5db refactor: merge parquet files interface (#3547)
Prepare for vertical partition compact, change the interface first.
2024-05-20 15:31:25 +08:00
Ashish Kolhe 798c3469ae fix: invalid peer certificate (#3505)
- [x] invalid peer certificate: UnknownIssuer
- [x] SSO user - infinite loop when accessing query cancelation UI
- [x] Clusters API should be accessible with SSO user
- [x] Handle _timestamp for _all fields
2024-05-14 17:22:05 +05:30
Ashish KolheandHengfei Yang 6d90c53323 fix: move files remove get meta using datafusion (#3486)
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
2024-05-12 17:26:26 +05:30
HuaijinandHengfei Yang 1d97e05f89 chore: upgrade datafusion to v38 (#3487)
Co-authored-by: Hengfei Yang <hengfei.yang@gmail.com>
2024-05-12 14:09:45 +08:00
Hengfei Yang c3ec7ebbd9 fix: value API (#3429) 2024-05-08 19:08:26 +08:00
Hengfei Yang 32b7ce4c26 feat: improve super cluster (#3428)
- [x] add timeout and partical success for super cluster
- [x] add region filter for super cluster
2024-05-08 16:32:54 +08:00
Hengfei Yang 0fbebbe6c0 feat: add file_list history for data retention (#3416)
impl #3241 
```
ZO_COMPACT_DATA_RETENTION_HISTORY = true
```

Enable this option will move the `file_list` into `file_list_history`
and don't delete files from storage.
2024-05-06 18:16:39 +08:00